Detecting vehicle maneuvers with mobile phones

Abstract

A vehicle maneuver detection application is proposed for driving assistant systems. The application can accurately and inexpensively detect and differentiate vehicle steering maneuvers by utilizing built-in sensors on smartphones or other portable computing device residing in a vehicle. By leveraging an effective bump detection algorithm and studying the nature of steering, the application is capable of differentiating various steering patterns, such as lane change, turn, and driving on curvy roads. Practicality of the application is demonstrates by two use cases: careless steering detection and fine-grained lane guidance. Thus, the application provides new functionalities without relying on cameras to provide a broader range of driving assistance.

Claims

1. A method for detecting vehicle maneuvers using a mobile phone residing in the vehicle, comprising: receiving, by a processor of the mobile phone, signals from two or more sensors residing in the mobile phone while the vehicle is moving, where the signals are indicative of vehicle movement and reported relative to a coordinate system associated with the mobile phone; converting, by the processor, the signals from the two or more sensors from the phone coordinate system to a geographic coordinate system; deriving, by the processor, a detection signal indicative of angular speed of the vehicle about a vertical axis of the geographic coordinate system, where the detection signal is derived from the converted signals; correlating, by the processor, the detection signal to one of a plurality of predefined signatures, where each signature represents a different maneuver made by a vehicle; determining, by the processor, a direction of travel for the vehicle; determining, by the processor, a horizontal displacement of the vehicle relative to the direction of travel; and identifying, by the processor, a maneuver made by the vehicle based on the correlation of the detected signal to one of the plurality of predefined signatures and magnitude of the horizontal displacement of the vehicle.

2. The method of claim 1 further comprises aligning the phone coordinate system with the geographic coordinate system prior to the step of receiving signals.

3. The method of claim 1 wherein the geographic coordinate system includes a vertical axis defined in relation to surface the vehicle is traversing upon and the vertical axis does not align with any of the axes that define the phone coordinate system.

4. The method of claim 1 wherein correlating the detection signal to one of a plurality of predefined signatures further comprises identifying one or two bumps in the detection signal, where one bump correlates to turn by the vehicle and two bumps correlates to a lane change by the vehicle.

5. The method of claim 1 wherein identifying a maneuver made by the vehicle further comprises classifying a vehicle maneuver into one of three classes selected from a group consisting of lane change, turn and curvy road.

6. The method of claim 1 wherein identifying a maneuver made by a vehicle further comprises comparing magnitude of the horizontal displacement to a threshold; and classifying the maneuver as a curvy road in response to a determination that the magnitude of the horizontal displacement exceeds the threshold.

7. The method of claim 6 wherein the velocity of the vehicle is determined from readings of a global positioning system (GPS) residing in the mobile phone.

8. The method of claim 1 wherein determining horizontal displacement further comprises determining velocity of the vehicle along the direction of travel; determining angular speed of the vehicle about a vertical axis of the geographic coordinate system; and determining horizontal displacement as a function of velocity and angular speed of the vehicle.

9. The method of claim 1 further comprises determining a difference in vehicle heading angle between start and end of vehicle maneuver; and further classifying a turn as either a right turn, a left turn or a U-turn in accordance with the difference in vehicle heading angle.

10. The method of claim 1 further comprises measuring motion of the vehicle using a motion sensor residing in the mobile phone.

11. The method of claim 10 wherein the motion sensor is further defined as at least one of a gyroscope or an accelerometer.

12. A method for detecting vehicle maneuvers using a computing device in a vehicle, comprising: receiving, by a processor of the computing device, a signal from an inertial sensor of the computing device while the vehicle is moving, where the signal is indicative of angular speed of the vehicle about a vertical axis; detecting, by the processor, one or more pulses in the signal within a fixed time period, where one pulse with the fixed time period correlates to a possible turn by the vehicle and two pulses within the fixed time period correlates to a possible lane change by the vehicle; determining, by the processor, a direction of travel for the vehicle; determining, by the processor, a horizontal displacement of the vehicle relative to the direction of travel; comparing, by the processor, magnitude of the horizontal displacement to a first threshold, where the comparison is performed in response to detecting two pulses in the signal; classifying vehicle maneuver as a curvy road in response to a determination that the magnitude of the horizontal displacement exceeds the first threshold; and classifying vehicle maneuver as a lane change in response to a determination that the magnitude of the horizontal displacement is less than the first threshold.

13. The method of claim 12 further comprises detecting two pulses in the signal within the fixed time period; determining, by the processor, whether amplitude of first pulse of the two detected pulses is higher or lower than a baseline value; classifying the vehicle maneuver as a possible change to left lane in response to determining that the amplitude of the first pulse is higher than a baseline value; and classifying the vehicle maneuver as a possible change to right lane in response to determining that the amplitude of the first pulse is lower than a baseline value.

14. The method of claim 12 further comprises determining, by the processor, whether amplitude of the one detected pulse is higher or lower than a baseline value; classifying the vehicle maneuver as a possible left turn in response to determining that the amplitude of the one detected pulse is higher than a baseline value; and classifying the vehicle maneuver as a possible right turn in response to determining that the amplitude of the one detected pulse is lower than a baseline value.

15. The method of claim 14 further comprises comparing, by the processor, magnitude of the horizontal displacement to a right turn threshold, where the comparison is performed in response to classifying the vehicle maneuver as a possible right turn; classifying vehicle maneuver as a curvy road in response to a determination that the magnitude of the horizontal displacement exceeds the right turn threshold; and classifying vehicle maneuver as a right turn in response to a determination that the magnitude of the horizontal displacement is less than the right turn threshold.

16. The method of claim 15 further comprises comparing, by the processor, magnitude of the horizontal displacement to a left turn threshold, where the comparison is performed in response to classifying the vehicle maneuver as a possible left turn; classifying vehicle maneuver as a curvy road in response to a determination that the magnitude of the horizontal displacement exceeds the left turn threshold; and classifying vehicle maneuver as a right turn in response to a determination that the magnitude of the horizontal displacement is less than the left turn threshold, where the value of the left turn threshold differs from value of the right turn threshold.

17. The method of claim 12 further comprises converting, by the processor, the signal a device coordinate system to a geographic coordinate system, where wherein the signal is reported in the device coordinate system and the geographic coordinate system includes a vertical axis defined in relation to surface the vehicle is traversing upon and the vertical axis does not align with any of the axes that define the phone coordinate system.

18. The method of claim 12 wherein determining horizontal displacement further comprises determining velocity of the vehicle along the direction of travel; determining angular speed of the vehicle about a vertical axis of the geographic coordinate system; and determining horizontal displacement as a function of velocity and angular speed of the vehicle.

19. The method of claim 18 wherein the velocity of the vehicle is determined from readings of a global positioning system (GPS) residing in the mobile phone.

20. The method of claim 12 further comprises measuring the angular speed of the vehicle about a vertical axis using the inertial sensor in the computing device.

Description

DRAWINGS

(1) The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

(2) FIG. 1 is a flowchart depicting a technique for detecting vehicle maneuvers using a mobile device residing in a vehicle;

(3) FIG. 2 is a diagram illustrating alignment of a phone coordinate system with a geo-frame coordinate system;

(4) FIG. 3A-3D are graphs showing gyroscope readings when the vehicle makes a left turn, a right turn, a left lane change and a right lane change, respectively;

(5) FIGS. 4A and 4B are graphs showing statistical features of bumps in gyroscope readings during different steering maneuvers;

(6) FIGS. 5A-5D are diagrams showing the vehicle trajectory shape for four different scenarios: (a) lane change, (b) driving on an S-shaped curvy road, (c) turning, and (d) driving on an L-shaped curvy road;

(7) FIG. 6 is a diagram depicting how to derive horizontal displacement based on gyroscope readings and estimated velocity;

(8) FIG. 7 is a state diagram of maneuver classification in accordance with this disclosure;

(9) FIG. 8 is a graph showing the accuracy of velocity estimation by fusing sensor readings;

(10) FIG. 9 is a graph showing error of values determined by the proposed technique as compared with the ground truth;

(11) FIGS. 10A and 10B are diagrams showing real road testing routes used for evaluation in Ann Arbor, Mich.;

(12) FIG. 11 is a bar chart showing performance of recognizing different steering patterns on both test routes;

(13) FIG. 12 is a diagram depicting information flow of careless steering detection by combining the V-Sense technique with a sound detection module;

(14) FIGS. 13A and 13B are graphs showing turn signal samples from a 2006 HONDA Accord and MINI Countryman; and the matched result from a matched filter with the existence of background noises inside the car, respectively;

(15) FIG. 14 diagram depicting information flow of fine-grained lane guidance by incorporating a navigation system with the V-Sense technique;

(16) FIG. 15 is a diagram showings turns and the corresponding turning radius at a 4-lane single carriageway intersection; and

(17) FIG. 16 is a graph illustrating the performance of the InterHelper module.

(18) Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION

(19) Example embodiments will now be described more fully with reference to the accompanying drawings.

(20) FIG. 1 depicts a technique for detecting vehicle maneuvers using a mobile phone residing in the vehicle (also referred to herein as “V-Sense”). In an example embodiment, this technique is implemented as vehicle steering sensing middleware on a mobile phone, which overcomes the limitations and difficulties inherent in the existing camera-based and camera-free approaches. While reference is made throughout this disclosure to mobile phones, it is readily understood that the technique set forth herein can be implemented using other types of computing devices, such as tablets or watches.

(21) In the example embodiment, inertial measurement units (i.e., sensors) residing in a mobile phone are used to detect steering maneuvers of the vehicle. Signals from one or more inertial sensors are received at 12 by a computer processor while the vehicle is moving. Thus, the signals are indicative of vehicle movement and reported relative to a coordinate system associated with the mobile phone. The computer processor preferably resides in the mobile phone although it may part of the vehicle or reside outside of the vehicle. Other types of motion sensors are also contemplated by this disclosure.

(22) A detection signal is derived at 13 from the signals received from the inertial sensors. The detection signal is indicative of angular speed of the vehicle about a vertical axis of a geographic coordinate system, where the geographic coordinate system includes a vertical axis defined in relation to surface the vehicle is traversing upon. In most instances, the vertical axis of the geographic coordinate system will not align with any of the axes that define the phone coordinate system. FIG. 2 illustrates one technique for aligning the phone coordinates ({X.sub.p, Y.sub.p,Z.sub.p}), with the geo-frame coordinates ({X.sub.e, Y.sub.e, Z.sub.e}). This allows one to simplify the change of the readings from three degrees of freedom (DoFs) to one DoF. The key idea is that with the measurements of the direction of the applied gravity to the mobile phone (Y), the phone coordinate can be fixed within a cone. Combining the result with the angle (θ) derived from the magnetometer readings and the thus-determined rotation matrix, the phone coordinate can be aligned with the geo-frame coordinate. Further details regarding this example technique are described by P. Zhou et al in “Use it free: Instantly knowing your phone attitude” In Proc of ACM Mobicom (2014). Other alignment techniques also fall within the scope of this disclosure.

(23) When a car changes its direction via steering (e.g., changing lanes, making turns, or driving on curvy roads), the Z-axis gyroscope reading (i.e., yaw rate reading) on the phone can be utilized to represent the vehicle angular speed of that change of direction. FIGS. 3A-3D illustrates the Z-axis gyroscope measurements from the phone during a left turn, a right turn, a change to left lane and a change to right lane, respectively. During a left turn, a counter-clockwise rotation around the Z-axis occurs and thus generates positive readings (i.e., a positive bump); whereas, during a right turn, a clockwise rotation occurs and thus generates negative readings (i.e., a negative bump). Similarly, during a left lane change, a positive bump is followed by a negative bump; whereas, during a right lane change, the opposite occurs. Based on this observation, one can infer that by detecting bumps in the Z-axis gyroscope readings, a determination can be made as to whether the vehicle has made a turn or has changed a lane. That is, the angular speed signal can be correlated at 14 to one of a plurality of predefined signatures, such that each signature represents a different maneuver made by the vehicle, and a vehicle maneuver is identified at 17 based on the correlation of the angular speed signal with one of the predefined signatures.

(24) With reference to FIG. 4, an example embodiment for detecting bumps in the angular speed signal is further described. A moving average filter may be used to remove noise from the raw gyroscope readings. In the example embodiment, the delay parameter of the filter is set to 60 samples which correspond to 0.05 second in the time domain. Such a decision was made based on the experimental observation: it is short but good enough to extract the waveform of the bumps.

(25) Four systems parameters are defined: δ.sub.s, δ.sub.h, T.sub.BUMP, and T.sub.NEXT.sub._.sub.DELAY. δ.sub.s represents starting point or ending point for a bump, δ.sub.h represents height of a bump, T.sub.BUMP represents minimum duration for a bump and T.sub.NEX.sub._.sub.TDELAY represents the maximum waiting time for a second bump in case of a lane change. To reduce false positives and differentiate the bumps from jitters, a bump should satisfy the following three constraints for its validity: (1) all the readings during a bump should be larger than δ.sub.s, (2) the largest value of a bump should be no less than δ.sub.h, and (3) the duration of a bump should be no less than T.sub.BUMP. Based on these constraints of a valid bump, an example bump detection algorithm is set forth below.

(26) TABLE-US-00001 1: Inputs: State, Y (Yaw rate), System parameters 2: if State = No-Bump and |Y| > δ, then 3: (Start of 1.sup.st bump) 4: State ← One-Bump 5: Record the start point of a possible bump 6: else if State = One-Bump and |Y| > δ, then 7: Record the end point of a possible bump 8: if bump is valid then 9: State ← Waiting-for-Bump 10: else 11: State ← No-Bump 12: end if 13: else if State = Waiting-for-Bump then 14: T.sub.dwell ← State dwell duration 15: If T.sub.dwell < T.sub.NEXT_DELAY and |Y| > δ, then 16: (Start of 2.sup.nd bump) 17: If 2.sup.nd bump is valid then 18: Two valid bumps .fwdarw. “Lane change” 19: else 20: One valid bump .fwdarw. “Turn” 21: end if 22: State ← No-Bump 23: else if T.sub.dwell < T.sub.NEXT_DELAY then 24: One valid bump .fwdarw. Turn 25: State ← No-Bump 26: else 27: Continue in Waiting-for-Bump state 28: end if 29: else 30: Continue in current state 31: end if
In this bump detection algorithm there are three states: No-Bump, One-Bump and Waiting-for-Bump. Each of these states is described below.

(27) In No-Bump state, the Z-axis gyroscope readings, i.e., yaw rate are continuously monitored. When the absolute value of the measured yaw rate reaches δ.sub.s, this is interpreted as the start of a possible bump and the algorithm enters One-Bump state.

(28) The One-Bump state terminates when the yaw rate drops back to a value below δ.sub.s. If the sojourn/dwell time in One-Bump state was larger than T.sub.BUMP and the largest measured yaw rate was larger than δ.sub.h, hence satisfying the three constraints, the first detected bump is considered to be valid. In such a case, the algorithm enters Waiting-for-Bump state; otherwise, it returns to No-Bump.

(29) In Waiting-for-Bump state, the yaw rate readings are monitored for a maximum dwell time T.sub.NEXT.sub._.sub.DELAY. The yaw rate readings are monitored for the presence of a second bump. If another bump starts (i.e., the yaw rate reaching δ.sub.s with a sign opposite to the first bump's is detected), then the validation procedure for a bump is performed as described above. If the second bump is valid, this would mean that two consecutive bumps with opposite signs have been detected. In this case, the algorithm determines the maneuver to be a lane change. If the second bump turns out to be invalid, then it would mean that only a single valid bump was detected and thus the algorithm determines the maneuver to be a turn. In either case, after a determination has been made, the state is reset (at line 22) to the No-Bump state. This process is repeated (at line 27) so long as the dwell time is less than the maximum dwell time T.sub.NEXT.sub._.sub.DELAY. Once the dwell time exceeds the maximum dwell time T.sub.NEXT.sub._.sub.DELAY, the algorithm return to the No-Bump state. The bump-detection algorithm is executed iteratively for each collected sample, and goes through a different procedure depending on the current state.

(30) With large values of δ.sub.h and δ.sub.s, small bumps—which may be caused by background noise or sensing errors—can be ignored and thus reduce the false-positive rate, whereas the false-negative rate might increase. On the other hand, with small values of δ.sub.h and δ.sub.s, the false-negative rate can be reduced but will become susceptible to background noise, thus increasing the false-positive rate. From extensive road tests, it was found that parameters δ.sub.s=0.05 and δ.sub.h=0.07 represent a good tradeoff. In an example embodiment, these parameters values are used although other values may be suitable. For example, the optimal parameter setting may vary slightly with the driving habit. It is envisioned that an adaptive parameter selection mechanism may be used as well. As for the other two parameters, T.sub.BUMP represents the time duration of a valid bump; whereas, T.sub.NEXT.sub._.sub.DELAY represents the maximum waiting time for the following bump, in case of a lane change. Since the time duration of a turn or lane change is usually several seconds, T.sub.BUMP is set to 1.5 seconds and T.sub.NEXT.sub._.sub.DELAY is set to 3 seconds in the example embodiment. It is readily understood that these values may be varied within the scope of this disclosure.

(31) When the vehicle is steered, bumps in the yaw rate readings are constructed. Based on the bump detection algorithm, bumps are used to differentiate between turns and lane changes. One possible problem with this approach is when driving on a curvy road. As illustrated in FIG. 5, when driving on a curvy road, the vehicle trajectory is similar to a lane change or a turn, and hence resulting in the same number and shape of bumps. In such a case, the bump detection algorithm might misinterpret the drive on a curvy road as a lane change or turn, and thereby yield false positives/negatives. Therefore, it is imperative to differentiate between lane changes, turns, and also driving on curvy roads.

(32) To address this issue, vehicle maneuvers are also classified based on magnitude of horizontal displacement of the vehicle relative to the direction of travel. Let W.sub.LANE denote the horizontal displacement after a lane change. Since the average lane width is around 3.65 meters, W.sub.LANE is expected to be around that value after a lane change. In contrast, while driving on a curvy road, the horizontal displacement, denoted as W.sub.CURVY, is usually much larger than W.sub.LANE. Based on this observation, if two bumps are detected—which means a possible lane change—then horizontal displacement of the vehicle during that steering maneuver is also evaluated. If horizontal displacement exceeds a threshold correlated to the average lane width (e.g., 3.65 meters), then the vehicle maneuver is classified as a curvy road instead of a lane change.

(33) Similarly, horizontal displacement can be used to differentiate between turns at the intersection (i.e., a sharp turn) and driving on a curvy road. Horizontal displacement during a turn is much smaller than that during driving on a curvy road. FIGS. 5C and 5D illustrate this point, where W.sub.TURN and W.sub.CURVY represent the horizontal displacements during a turn and driving on a curvy road, respectively. In this case, if only one bump is detected, horizontal displacement is used to distinguish between turning and driving on a curvy road.

(34) FIG. 6 shows an example vehicle trajectory during a left lane change or maneuver on a curvy road as illustrated in FIGS. 5A and 5B. The dotted vertical line represents the time when the sensors are sampled with frequency of 1/T.sub.s. Here θ.sub.n denotes the angle of the vehicle's heading; whereas v.sub.n represents the average velocity during the sampling period. During each sampling period, T.sub.s, the vehicle's horizontal displacement can be expressed as:
W.sub.n=v.sub.nT.sub.s sin(θ.sub.n) (1)
Since the yaw-rate readings from the gyroscope represent the vehicle's angular velocity around the Z-axis, θ.sub.n can be expressed as:

(35) $\begin{matrix} \begin{matrix} θ_{n} = θ_{n - 1} + Y_{avg} T_{s} \\ \approx θ_{n - 1} + Y_{n} T_{s}, \end{matrix} & (2) \end{matrix}$
where Y.sub.avg represents the average yaw rate during the sampling period, and Y.sub.n the instantaneous yaw rate measured at the end of the sampling period. Note that the above approximation holds since the sampling period on smartphones can be significantly reduced. Thus, the total horizontal displacement from time 0 to NT.sub.s can be derived as:

(36) $\begin{matrix} \begin{matrix} W_{final} = {.Math.}_{n = 1}^{N} W_{n} \\ = {.Math.}_{n = 1}^{N} v_{n} T_{s} \sin (θ_{n}) \\ = {.Math.}_{n = 1}^{N} v_{n} T_{\sin} ({.Math.}_{k = 1}^{n} Y_{k} T_{s}) \end{matrix} & (3) \end{matrix}$
where T.sub.s is a predefined parameter denoting the sampling period of the application. The third equality comes from the fact that the initial angle of the vehicle's heading, θ.sub.0=0, since this is the reference point. Y.sub.k can be acquired from the gyroscope readings, while v.sub.n can be derived from the accelerometer and GPS readings. Other techniques for determining horizontal displacement also fall within the broader aspects of this disclosure.

(37) To further classify different turning maneuvers (e.g., left/right turn at the intersections, U-turn), the change in the vehicle's heading angle (i.e., the difference in the heading angle between the start and the end of a steering maneuver) may also be evaluated. As in Eq. (2), the angle of vehicle's heading at sampling time nT.sub.s can be derived by accumulating the n yaw-rate measurements. Considering the example in FIG. 6, at sampling time 3T.sub.s, the angle of the vehicle's heading would be θ.sub.3=Σ.sub.n=1.sup.3 Y.sub.nT.sub.s. In other words, the change in the vehicle's heading from time 0 to NT.sub.s can be expressed as:
θ.sub.final=Σ.sub.n=1.sup.NY.sub.nT.sub.s. (4)
For example, after making a left/right turn at the intersection, 0.sub.final≈±90°; whereas, after making a U-turn, 0.sub.final≈±180°. Thus, by exploiting the derived values, turns can be further classified as left/right turn or a U-turn.

(38) Returning to FIG. 1, the bump detection algorithm is one way that an angular speed signal can be correlated to one of a plurality of predefined signatures at 14. Other techniques for correlating also fall within the scope of this disclosure. Next, a direction of travel for the vehicle is determined at 15 and horizontal displacement of the vehicle is determined at 16. A maneuver being made by the vehicle can then be identified at 17 based on the correlation of the angular speed signal to one of the predefined signatures (i.e., number of bumps) and the magnitude of the horizontal displacement of the vehicle. It is to be understood that only the relevant steps of the proposed V-Sense technique are discussed in relation to FIG. 1, but that other software-implemented instructions may be needed to control and manage the overall operation of the system.

(39) FIG. 7 summarizes the overall maneuver classification in a state diagram. First, a determination is made as to whether the steering maneuver is a possible turn or a possible lane change, for example by calculating the number of bumps using the bump detection algorithm described above. In either case, horizontal displacement of the vehicle is also determined to help differentiate between a turn or lane change from driving on a curvy road.

(40) For a possible lane change, the magnitude of the horizontal displacement is compared to a first threshold that correlates to expected lane width (e.g., 3.65 meters). If the magnitude of the horizontal displacement exceeds the first threshold, the steering maneuver is classified as a curvy road; whereas, if the magnitude of the horizontal displacement is less than (or equal to) the first threshold, the steering maneuver is classified as a lane change.

(41) For a possible turn, the magnitude of the horizontal displacement is again compared to a threshold but the value of the threshold depends on the turn direction. For a possible right hand turn, the magnitude of the horizontal displacement is compared to right turn threshold (e.g., 13 meters). If the magnitude of the horizontal displacement exceeds the right turn threshold, the steering maneuver is classified as driving on a curvy road; whereas, if the magnitude of the horizontal displacement is less than (or equal to) the right turn threshold, the steering maneuver is classified as a right turn. On the other hand, for a possible left hand turn, the magnitude of the horizontal displacement is compared to left turn threshold (e.g., 22 meters). If the magnitude of the horizontal displacement exceeds the left turn threshold, the steering maneuver is classified as driving on a curvy road; whereas, if the magnitude of the horizontal displacement is less than (or equal to) the left turn threshold, the steering maneuver is classified as a left turn. Turns may also be further classified as U-turns based on the change in the vehicle's heading angle. Other techniques for classifying vehicle maneuvers are contemplated and fall within the broader aspects of this disclosure.

(42) In order to derive the horizontal displacement and set T.sub.BUMP and T.sub.NEXT DELAY, accurate measurements are needed for the vehicle's instantaneous velocity. Two possible ways of acquiring the velocity with a mobile phone are: reading the Speed Over Ground (SOG) output from the GPS module inside the smartphone, or exploiting the IMU. The GPS provides measurements of the velocity; whereas, acceleration can be derived from IMU readings. However, the GPS output rate is very low (e.g., 1 Hz on Samsung Galaxy S4) as shown in FIG. 8, and hence cannot properly capture velocity changes within a sampling period. On the other hand, the IMU has a much higher output rate but contains lots of noise as well as some biases as shown in FIG. 8. Thus, just simply using either the velocity measurement from GPS or taking an integral of the acceleration output from the IMU is not sufficient. Hence, in order to exploit the distinct advantages of GPS and IMU, the data is fused by using a Kalman filter to estimate the velocity.

(43) In one embodiment, a model is constructed for estimating the velocity as follows:
v(k|k−1)=v(k−1|k−1)+(a(k)−b(k−1|k−1))T.sub.s (5)
where v(k|k−1) is the estimated velocity at time k based on the optimized velocity at time k−1; v(k−1|k−1) is the optimized velocity at time k−1; a(k) is the acceleration output at time k; b(k−1|k−1) is the optimized bias of the accelerometer at time k−1; and T.sub.s is the sampling period of the accelerometer.

(44) Here, b is treated as a constant bias:
b(k|k−1)=b(k−1|k−1) (6)
Thus, we have a matrix representation of the model as:
X(k|k−1)=AX(k−1|k−1)+BU(k) (7)
where

(45) $X = [\begin{matrix} v \\ b \end{matrix}], A = [\begin{matrix} 1 & - T_{s} \\ 0 & 1 \end{matrix}], B = [\begin{matrix} T_{s} \\ 0 \end{matrix}],$
and U is the output from accelerometer. So, the covariance matrix is estimated by:

(46) $\begin{matrix} P (k | k - 1) = {APA}^{T} + Q, Q = [\begin{matrix} q_{v} & 0 \\ 0 & q_{a} \end{matrix}] & (8) \end{matrix}$
where P is the covariance matrix, and Q is the covariance of the process noise which can be regarded as the Gaussian white noise. Thus, the state can be estimated as:
X(k|k)=X(k|k−1)+g(k)(S(k)−HX(k|k−1)) (9)
where g(k) is the matrix of Kalman gain and S(k) is the speed relative to the ground measured by the GPS, and H=[1 0].

(47) FIG. 8 shows velocity estimation by using such a model based on Kalman Filter. Here the ground truth velocity is obtained by directly reading it from the OBD-II port, and comparing it with estimation results. It is shown that the velocity can be accurately estimated in real time, thus yielding accurate horizontal displacements.

(48) To evaluate the performance of V-Sense technique described above, middleware performing the technique was implemented on a Samsung Galaxy S4 with a 1.6 GHz quad-core processor running Android 4.4.1 KitKat OS and tested in variety of different conditions. First, the accuracy of the V-Sense technique was evaluated in determining the change of heading angle and the horizontal displacement in a short road test. Then, the performance of classification was evaluated with a longer road test containing various road features. The cars used for the test were a 2010 Mitsubishi Lancer and a 2006 Mazda 6. During these experiments, the mobile phone was either mounted on the windshield, or kept in the driver's pocket.

(49) By making several lane changes, turns and U-turns during a short road test, the accuracy of V-Sense technique in estimating the change of heading angle and the horizontal displacement was evaluated. During the road test, three lane changes were made (i.e., one to the left lane and the other two to the right lane) and three U-turns. The horizontal displacements and changes of heading angle were collected to check whether the estimated values are close to their ground truth. The results of the two separate tests are summarized in Table 1. For consistency, all numbers are presented as their absolute values.

(50) TABLE-US-00002 TABLE 1 Lane Change U-turn #1 #2 #3 Average #1 #2 #3 Average Displacement [m] 4.29 3.49 3.59 3.79 14.47 15.66 14.46 14.86 Angle Change [deg] 2.03 7.49 4.12 4.54 193.73 179.85 184.41 185.99

(51) During a lane change, the ground truth horizontal displacement is expected to be equal to the actual lane width, which was around 3.7 m for our experiment. However, for the change of heading angle, it is expected to be 0°, since this is a measure of the difference between the initial and the final heading angles.

(52) On the other hand, during a U-turn, the ground truth of horizontal displacement and the change of heading angle are the road width for U-turns, which was approximately 16 m in our case, and 180°, respectively.

(53) FIG. 9 shows the error ratio—which is the ratio of the absolute deviation to the ground truth value—in the two experiments. For all cases, the estimated horizontal displacement and change of heading angle have a very low error ratio, i.e., very accurate. The high accuracy of the V-Sense technique in determining the two values means that it can correctly classify various steering maneuvers, which is validated in the following subsection by conducting long road tests.

(54) To evaluate how well the V-Sense technique classifies different steering maneuvers, two long road tests were also performed. To guarantee the generality of the experiment, two different test routes were chosen as shown in FIGS. 10A and 10B. The routes run through typical urban areas and freeways, the former including residential, downtown, and school areas. The number of features in the examined routes is summarized in Table 2.

(55) TABLE-US-00003 TABLE 2 Summary of different road features in testing routes. Route Distance [miles] RT LT LC CR #1 3.4 6 5 4 11 #2 8.3 5 5 15 9
To validate the independence of the V-Sense technique from driving habits, we had five volunteers participating in the test, three male drivers and two female drivers. Each of them drove twice on both route #1 and #2. In the first test, they mounted the phone on the windshield; whereas, in the second test, the phone was kept inside the driver's pocket.

(56) With reference to FIG. 11, the on-road experimental results can be highlighted as follows. V-Sense technique achieves 100% accuracy in detecting both right and left turns, regardless of the phone's placement and road condition. This is because when turning, the heights of the bumps in the readings tend to be high enough to be accurately detected and classified. For lane changes, V-Sense technique achieves 93% accuracy when the phone is mounted on the windshield, and 85% accuracy when the phone is in the driver's pocket. The false-negative results are mostly due to the fact that the V-Sense technique occasionally misinterprets a lane change as driving on a curvy road, because a few of the lane changes in the test took longer than expected, especially on the freeway where drivers tend to take extra caution, thus making slower lane changes. The accumulated error in the gyroscope reading can also degrade the performance in such a case. However, its occurrence is expected to be rare considering the average elapsed time for lane changing, i.e., less than 6 seconds. Finally, the V-Sense technique achieves nearly 97% accuracy in detecting curvy roads with the phone mounted on the windshield, and nearly 92% accuracy with the phone kept in the driver's pocket. These results also reflect the accuracy of the coordinate alignment. Also, note that the V-Sense technique was able to detect the two long L-shaped curvy roads on the US #23 freeway using bump detection and horizontal displacement derivation.

(57) In one application, the V-Sense technique can be used to detect careless steering: changing lanes or making turns without turning on the turn signal. Detecting a driver's careless steering is important, since it would enhance the safety of not only the driver but also people/vehicles around him. Moreover, it can also be used by insurance companies in monitoring the driver's driving habit and thus determining the insurance premium accordingly, which would then motivate the drivers to avoid careless steering.

(58) FIG. 12 shows the information flow of careless steering detection using data collected from the accelerometer 131, GPS 132, gyroscope 133, and microphone 134 on a mobile phone. This application is comprised of the V-Sense module 135 as well as a sound detection module 136. The V-Sense module 135 detects possible lane changes or turns using the gyroscope readings in the manner set forth above. Upon detecting the start point of a lane change or turn (i.e., bump), the sound detection module 136 activates the microphone 134 and starts to detect the turn signal sound. If a lane change or turn is detected without the detection of signal sound, the application declares the driver is involved in careless driving, and triggers an alarm (visual, audible or otherwise) to notify the driver; otherwise, the application declares it as attentive driving.

(59) In order to detect whether the driver has used the turn signal, the sound detection module 136 uses the following three steps: (i) collect training samples of the turn signal sound; (ii) eliminate background noise with a matched filter; (iii) make a decision on whether the turn signal was used during the turn or lane change.

(60) First, the turn signal sounds are collected from two different cars, 2006 Honda Accord and MINI Countryman, which are used as sample data sets as shown in FIG. 13A. The measured steady rates of the turn signal in the 2006 Honda Accord and MINI Countryman were 161 and 163 ticks per second (shown in FIG. 16), respectively. As the turn signal sounds acquired from the 2006 Honda Accord has lower amplitude, and would thus be more difficult to detect, sound detection was studied using this data set. To test the performance of the turn signal detection module in real driving scenario, the engine was turned on and music was played which acts as background noise inside the car.

(61) To detect the sound emitted from the turn signals, the detection module has to overcome two challenges: (i) it must be resilient to the variation of SNR due to unpredictable detection conditions; and (ii) the delay in detecting a single turn signal must be low in order to be ready for detecting the subsequent signal. A matched filter 137 was used to meet these challenges.

(62) The matched filter 137 is used to detect the presence of a known signal in the unknown signals. The key idea behind the matched filter 137 is to design an impulse response that maximizes the output SNR. Due to unpredictable driving conditions, the noise inside the car cannot be easily modeled. Thus, the turn signal is modelled and the signal sound is extracted by using one convolution with the matched filter kernel. Since the turn (sound) signal can be modeled as series of discrete signals, we use the discrete version of matched filter, in which the output y[n], can be expressed as:
y[n]=Σ.sub.k=−∞.sup.∞h[n−k]h[k], (10)
where the impulse response, h[n], of the matched filter is
h[n]=g[n] custom character v[n.sub.o−n], (11)
where g[n] denotes the power spectral density of background noise. The matched signal Matched is acquired by applying
Result[n]=Signal[n]h[n], (12)
where signal[n] is the sound recorded by the smartphone's microphone inside the car and Result[n] is the output of the matched filter.

(63) If the amplitude of the matched filter output is larger than T, a pre-defined threshold set to 0.35 by default, the V-Sense technique declares the detection of a turn signal sound. If the detected turn signal is accompanied by a lane change or turn detected by the V-Sense technique, then the application declares the steering maneuver as being attentive. On the other hand, if no such turn signal sound was detected, the application declares the steering to be careless, and then alarms the driver.

(64) FIG. 13B shows the performance of the sound detection module which extracts signal sound from background noise. Experiments were conducted in a regular driving setting, where music played inside the car and the passengers were talking occasionally. The matched filter was able to extract and identify the sound of the turn signals from the background noise, even when the amplitude of the noise was very high (radio played music at the max volume). By integrating this accurate sound detection module with the V-Sense technique, the application detects careless steering and thus enhances driving safety significantly.

(65) Fine-grained lane guidance allows existing navigation systems provide higher guidance accuracy. Specifically, fine-grained lane guidance detects whether the driver is in the correct lane and alarms the driver if not. Existing navigation systems on mobile phones are constrained by the accuracy of the built-in GPS sensor, which is at best 5˜10 m. When the line-of-sight transmission between the satellite and the phone is blocked by obstacles, such as tunnels, bridges, and tall buildings, the accuracy quickly drops to 20˜100 m. Such limitations make it impossible for legacy navigation systems to recognize the exact traffic lane that the vehicle is on. The latest update of Google Maps does include a lane guidance function, but in a rather limited way: it can only provide information on which lane the vehicle should stay, not whether it is actually on that lane. Fine-grained lane guidance is important, since it can reduce abrupt lane changes, and also very helpful for drivers who have lack driving experience.

(66) Based on information from an on-line map, the correct lane for the next maneuver can be easily determined, which is a function already provided by existing navigation systems. Hence, the main challenge of realizing the fine-grained lane guidance application is the determination of the current lane that the vehicle is running on. To meet this challenge, one needs to determine the vehicle's current lane via lane change detection. The current lane can be determined based on whether and how the vehicle has changed its lane. Thus, by detecting and analyzing the lane changes made by the vehicle, one can determine the vehicle's current lane.

(67) Lane changes may take place in two situations: (i) middle of a road or; (ii) at intersections. For the first case, the V-Sense technique can reliably detect lane changes on the road using techniques described above. For the second case, an add-on module 161 (called InterHelper) was developed for the V-Sense module 135. In FIG. 14, the navigation system 162 and the V-Sense module 135 cooperate to determine the fine-grained location. The navigation system is capable for determining whether the vehicle is at the intersection. Once the vehicle reaches an intersection, the InterHelper module 161 is triggered and starts to estimate the turning radius, R.sub.i. Note that R.sub.i is equivalent to the horizontal displacement during the turn, which can be derived by using the techniques described above. This information enables the V-Sense module 135 to finally determine the fine-grained location.

(68) As shown in FIG. 15, there are four possibilities of lane change at a typical 4-lane single carriageway intersection. That is, each car has two choices of making either right or left turn. Here, it is assumed that the turning trajectory is an arc, which is a common assumption in intersection design. O.sub.1, O.sub.2, O.sub.3, O.sub.4 are centers of turning circles. InterHelper module 161 classifies each case by differentiating the turning radius, i.e., R.sub.1, R.sub.2, R.sub.3, R.sub.4.

(69) For a typical intersection, the right turn radius, R.sub.1 is 10.8 m, the left turn radius, R.sub.3 is 20.7 m, and the width of a typical intersection is 19.2 m. Moreover, the lane width is around 3.65. Based on these facts and extensive road experiments, the threshold for differentiating R.sub.1 and R.sub.2 are set as 13.1 m, and the threshold of differentiating R.sub.3 and R.sub.4 are set as 21.64 m. Using such thresholds and the horizontal displacement obtained from the V-Sense module, the application determines whether the vehicle has changed its lane during a turn at the intersection.

(70) In order to evaluate the performance of the fine-grained lane guidance application, 80 left and right turns were performed at different intersections in Ann Arbor, Mich., U.S., and the results are shown in FIG. 16. The application is shown to be able to detect 95% of right turns with R.sub.1, 90% with R.sub.2, 90% of left turns with R.sub.3 and 85% with R.sub.4. It can therefore be concluded that by integrating InterHelper into V-Sense, the application is capable of detecting lane changes in all cases, thus determining the vehicle's current lane.

(71) Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

(72) The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a tangible computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

(73) The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present disclosure is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein.

(74) The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. Individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. The same may also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Detecting vehicle maneuvers with mobile phones

Assignee

Inventors

Cpc classification

Classification Explorer

B60W2420/905

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

H04W4/48

ELECTRICITY

Classification Explorer

B60W40/10

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

H04W4/026

ELECTRICITY

Classification Explorer

B60W30/18145

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

B60W40/06

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

B60W40/09

PERFORMING OPERATIONS; TRANSPORTING

International classification

Classification Explorer

B60W40/10

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

B60W40/06

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

B60W40/09

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

H04W4/04

ELECTRICITY

Abstract

Claims

Description