Method for determining the operational state of a driver

Abstract

A method for determining an operational state of a driver of a vehicle uses an awareness detection arrangement. The awareness detection arrangement includes at least a first and a second source for generating data relating to the behavior of the driver. The method includes receiving, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle, comparing the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively, determining based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states, and weighing the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver.

Claims

1. A method for determining an operational state of a driver of a vehicle using an awareness detection arrangement, the awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, the method comprising: receiving, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle; comparing the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively; determining based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states including determining one of a probability mass function for each of the plurality of predefined driver states; and weighing the determined driver states for the first, and the second source with each other for determining an overall operational state probability for the driver.

2. Method according to claim 1, wherein the physiological data of the driver comprises information relating to at least one of eye, face, head, arm and body motion of the operator.

3. Method according to claim 1, wherein the operation of the vehicle comprises information relating to at least one of time to line crossing, distance to a further vehicle arrange in front of the vehicle, steering and/or wheel operation pattern.

4. Method according to claim 1, wherein the model of the driver operating the vehicle comprises a first component relating to at least one of the time of the day and the operational time of the vehicle and a second component relating to the drowsiness level of the driver.

5. Method according to claim 4, wherein the second component relating to the drowsiness level of the driver is based on at least one of a model of sleep latency, time of day, time on task, a circadian rhythm, and a sleep/wake homeostatic process.

6. Method according to claim 1, wherein the plurality of predefined driver states comprises at least two states defined as drowsy and alert, respectively, preferably four states defined as very alert, alert, drowsy and fighting sleep, respectively.

7. Method according to claim 1, wherein the determined state of the driver is provided to a vehicle system configured to implement a vehicle control functionality, the vehicle system adjusting the vehicle control functionality based on the driver state.

8. Method according to claim 1, wherein the deter lined state of the driver is provided to a drowsy driver detection system for generating a warning to the driver state indicate that the driver is drowsy.

9. A control system for determining an operational state of a driver of a vehicle, the control system comprising a control unit, the control unit connected to an, awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, wherein the control unit is configured to: receive, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle; compare the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively; determine based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states including determining one of a probability mass function for each of the plurality of predefined driver states; and weigh the determined driver states for the first and the second source with each other for determining air overall operational state probability for the driver.

10. Control system according to claim 9, wherein the control unit is further configure to provide the determined state of the driver to a vehicle system configured to implement a vehicle control functionality, the vehicle system adjusting the vehicle control functionality based on the driver state.

11. A vehicle system, comprising a control system for determining an operational state of a driver of a vehicle, and an awareness detection arrangement, the control system comprising a control unit, the control unit being connected to the awareness detection arrangement, the awareness detection arrangement comprising at east a first and a second source for generating data relating to the behavior of the driver, wherein the control unit is configured to: receive, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle; compare the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively; determine based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states including determining one of a probability mass function for each of the plurality of predefined driver states; and weigh the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver.

12. Vehicle system according to claim 11, wherein at least one of the first and the second source is configured to generate physiological data of the driver comprising information relating to at least one of eye, face, head, arm and body motion of the operator.

13. Vehicle system according to claim 11, wherein at least one of the first and the second source is an image capturing device.

14. Vehicle system according to claim 11, wherein at least one of the first and the second source is configured to generate operational data of the vehicle comprising information relating to at least one of time to line crossing, distance to a further vehicle arrange in front of the vehicle, steering and/or wheel operation pattern.

15. A non-transitory computer readable medium embodying a computer program product for determining an operational state of a driver of a vehicle using an awareness detection arrangement, the awareness detection arrangement comprising at least a first and a second source for generating data relating to the behavior of the driver, the computer program product comprising code configured to, when executed by a processor: receive, from the first and the second source, data relating to at least one of physiological data of the driver, the operation of the vehicle, and a model of the driver operating the vehicle; compare the data from the first and the second source with a driver state model defining a plurality of predefined driver states for each of the first and the second source, respectively; determine based on the comparison, for each of the first and the second source, a state probability for each of the plurality of predefined driver states including determining one of a probability mass function for each of the plurality of predefined driver states; and weigh the determined driver states for the first and the second source with each other for determining an overall operational state probability for the driver.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The various aspects of the invention, including its particular features and advantages, will be readily understood from the following detailed description and the accompanying drawings, in which:

(2) FIG. 1 is a perspective view of a vehicle equipped with external sensors and a coordinate system at its front end;

(3) FIG. 2 is a perspective view of the interior of the vehicle, equipped with an internal sensor;

(4) FIG. 3 illustrates a coordinate system of the face and head of a vehicle operator;

(5) FIGS. 4a and 4b illustrate the statistical likelihood calculations of two exemplary independent indicators of the driver's state used in conjunction with the inventive method;

(6) FIG. 5 conceptually illustrates a vehicle system according to a currently preferred embodiment of the invention, and

(7) FIG. 6a-c conceptually illustrates an example of the iterative adjustment of the operational state probability for the driver.

DETAILED DESCRIPTION

(8) The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which currently preferred embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided for thoroughness and completeness, and fully convey the scope of the invention to the skilled addressee. Like reference characters refer to like elements throughout.

(9) In the following, the present invention is described with reference to a system for improving a visual input quality estimation of an operator of a vehicle. The vehicle is preferably equipped with interior sensor(s) for retrieving information of the vehicle operator and external sensor(s) for retrieving information of the vehicle operation as well as the surrounding environment of the vehicle. For the sake of better understanding, the internal and external sensors will now be described in relation to FIGS. 1-3.

(10) FIG. 1 shows an exemplary vehicle, here illustrated as a car 100, in which a system according to the present invention may be incorporated. The car 100 is provided with external sensors 104 arranged to detect vehicle operation, such as overtaking, vehicle speed, vehicle yaw rate, etc, and objects, and zones, surrounding environment of the vehicle, e.g. lane markings, road marks, road curves, surrounding vehicles, etc. The external sensors 104 may be e.g. cameras or radar sensors. Preferably, a combination of camera and radar sensors may be used, since the camera provides a high precision when determining the height and width of the object, whereas a radar sensor provides a high precision when determining the distance to the object. Hereby, size, position, speed, etc. of the surrounding object can be determined. With reference to the position of the car 100, a coordinate system 102, here illustrated as a Cartesian coordinate system, is located at the front end of the car 100. The coordinate system 102 is arranged to follow the vehicle and the axis represent the longitudinal direction (x-axis), lateral direction (y-axis) and vertical direction (z-axis), respectively. The detected objects, in conjunction with the coordinate system 102 of the car 100, are provided to a system of the vehicle such that the system can determine the size and position of the object relative to the car 100. For example, the system may be continuously provided with object data from the different sensors 04. Hence it is also possible to determine speed and acceleration of surrounding traffic environment.

(11) FIG. 2 illustrates an interior of a car 100 including a vehicle operator 202, wherein the vehicle 100 is equipped with an internal sensor, here illustrated as a camera system 204. The camera system 204 is arranged to measure and detect the behavior of the vehicle operator 202 during vehicle operation, and may be configured to generate an operator motion input signal indicative of physiological data comprising information relating to at least one of eye, face, head and body motion of the operator of the vehicle.

(12) Furthermore, the camera system 204 may be arranged to focus on a predetermined number of positions of the operator's face, head, or upper body. These positions may, for example, be the eyes, eye-lids, eyebrows, nose, mouth, cheek, neck, shoulders, arms, etc. The camera system 204 may be pre-calibrated for a specific operator 202 normally operating the car or being calibrated each time an operator 202 enters the driver seat of the car 100. As the camera system 204 has detected the different positions of the operator's face or head, an estimation of facial behavior is possible for the camera system 204. The camera system 204 may hence detect, e.g. head and eye direction and movement, and derivative thereof head pose, eye saccade, combined head and eye saccade, eye closure, speed of eye closure, etc.

(13) The camera system 204 may also, by use of a coordinate system 302 in connection to the operator's face 304, e.g. a operator-centric pitch/yaw coordinate system as illustrated in FIG. 3, detect if the head, or eyes, of the operator is rotating to the right or left (yaw), 305, rotating up or down (pitch), 306, or, in the case of the head movements, leaning towards the right or left shoulder (roll), 307. The coordinate system 302 of the face 304 is preferably a polar coordinate system with its origin positioned between the eyes of the operator.

(14) Furthermore, the internal sensors may also, instead of, or additionally to the camera system 204, include other type of operator detecting means. This may, for example, include steering wheel sensors for detection of a steering behavior, sensors in the acceleration pedal and/or braking pedal for detection of inconsistent acceleration and/or braking of the car 100, sensors in various buttons of the car 100 to detect if for example, the operator 202 is adjusting any of the various functionalities of the infotainment system, etc. Further examples of internal sensors may include a breath analysis sensor or pupil size sensor for monitoring state of awareness of the operator.

(15) For providing a further understanding of the invention, an explanation is provided below in which the concept is divided into an initial preparatory phase of parameterization and calculations of the necessary reference values, and a further usage phase of continuous detection, computation and prediction of drowsiness, including the subsequent generation of warnings to the driver or control of other vehicle functions.

(16) In the initial preparatory phase, an expert based (e.g. off-line) parameterization of the possible driver states that characterizes the driver (drowsiness) state. From research it is known that this parameterization can be made strong and robust. The parameterization could in one embodiment be based on two states {alert, drowsy}. In another embodiment the parameterization could be based on four states {very alert, alert, drowsy, fighting sleep}.

(17) Further states are of course possible and within the scope of the invention. As an example, taking a completely different approach, the states can be defined to correspond to the activation triggers of other vehicle functions it is meant to feed information to. The system may then either use generic functional levels or functional states tailored for a specific vehicle system. Thus, in a one embodiment, the fusion system is designed as a pre-stage to influence specific behavior of a separate vehicle system (rather than designed to be a drowsy driver detection system), implemented e.g. as a forward collision warning (FCW) system that takes driver state into account. The FCW system may then map the output of the fusion, system directly to the internal warning decision function.

(18) The next step in the preparatory phase involves computation of a probability mass function (pmf) of each indicator (i.e. source of information) for the defined driver states. In FIGS. 4a and b, it is illustrated the pmf for two indicators, time to line crossing (TLC) and steering wheel reversal rate (SRR), using a two state parameterization {alert, drowsy}. In practice, pre collected data of actual drowsy and alert driving may be used to create the pmfs for each indicator. Since data will always be inherently ambiguous (to some extent) there is a desire to calculate the pmf of the driver state rather than a discrete classification. In other words, the shape of the pmfs describing the indicator values are used to calculate the driver state pmf. Thus there will be several pmfs describing the distribution for the data values for each indicator, and one pmf describing the distribution between valid driver states.

(19) Turning now to FIG. 5, which conceptually illustrates a vehicle system 500 according to a currently preferred embodiment of the invention, for use in relation to the usage phase. The vehicle system 500 typically comprises a detection system 502 including a plurality of sensors (e.g. image capturing device(s), radar equipment, etc. as discussed above in relation to FIG. 1), an indicator computation system 504, a prediction and data fusion system 506, a decision and feedback system 508. Additionally, the vehicle system 500 typically comprises a possibility to provide a feedback to the driver of the vehicle, possibly by means of a human machine interface.

(20) The conceptual illustration in FIG. 5 is only one of many ways of implementing the inventive concept. Additionally, the vehicle system 500 can be am asynchronously, i.e., fused estimates can be calculated at any time regardless of whether new observations have been made available. Formally, if the estimates are calculated at times T=[t1, t2, . . . , tN].sup.T, the time difference t2−t1 does not necessarily equal t3−t2.

(21) The drowsiness level at time tk is denoted xk, here assumed to be a discrete variable, and the vehicle system 500 calculates its probability mass function (or probability density function if the state is assumed to be continuous). The benefit with having a discrete state vector is twofold; the state can be designed to correspond to different interventions, and the probability mass function (pmf) can be calculated exactly rather than approximated. The time stamp of the most recent indicator value used in the calculation of the pmf is shown in the notation as p(xk|Ij); the conditional pmf. Data Ij (bold face) denotes all indicator values accumulated up to time if. Ij=

(22) I1, I2, . . . Ij].

(23) The operation of the vehicle system 500 may also be described using a “step-by-step” pseudo code for the indicator fusion:

(24) for k=1:N

(25) 1. Store output from all connected indicators (sources of information), made available in the time interval tk−tk−1, Ik=[i.sup.1, i.sup.2, . . . , i.sup.M], ordered by their timestamps.

(26) 2. Update the pmf from the previous iteration, p(xk|Ik), with the new data, Ik:

(27) for j=1:M

(28) a. Predict the drowsiness pmf to the time of the oldest indicator value, i.sup.j, in Ik

(29) Calculate p(xj|Ik−i, i.sup.1, . . . , i.sup.j-1).

(30) b. Update the predicted pfm with the new indicator value, i.sup.j:

(31) Calculate p(xj|Ik−i, i.sup.1, . . . , i.sup.j).

(32) end for

(33) 3. Predict the pmf to the desired output time:

(34) Calculate p(xk|Ik)

(35) 4. Apply a probabilistic decision making scheme to determine system output, e.g., warn the driver.

(36) end for

(37) The third step could also be used to estimate the driver state “far” in the future (e.g. 1 h) and allow for trip-planning accordingly, rather than just predict the next iteration.

(38) The fourth step enables robust decision making since not only some estimate of drowsiness are known, but rather the whole pmf. Then any optimality criterion for interventions can be incorporated in the design.

(39) Furthermore, the use of this approach allows a confidence value of each estimate to be computed easily.

(40) The decision and feedback module 508 can in one embodiment take into account both, the current estimated drowsiness level and the predicted future drowsiness level (e.g. 15 minutes from now) in determining whether to issue a warning. For instance the driver may receive a warning if he is predicted to become very drowsiness within the next 15 minutes, thus giving him a chance to act proactively.

(41) With further reference to FIGS. 6a-c where an example of the iterative adjustment of the operational state probability for a driver of a vehicle is conceptually illustrated, the driver drowsiness level x can take the values 1-3, where 1 is ‘drowsy’ and should yield a warning or an intervention, whereas 3 is ‘alert’. Assume that the proposed system calculates that
p(x)=[0.5, 0.1, 0.4]

(42) A typical fusion system which only gives an estimate of “the best guess”, would output, a ‘1’ whereas a system that calculates the mean would give a ‘2’. None of these estimates contain the necessary confidence information, in this case the state is clearly ambiguous since the states 1 and 3 are almost equally likely but are naturally contradictive—the driver cannot be drowsy and alert at the same time. According to the inventive concept, this can be considered, e.g., warn if the probability p(x=1)>0.5 and intervene if p(x=1)>0.9.

(43) In a more advanced embodiment there can be multiple actions associated with each state of the parameterization, e.g., ‘Minor warning’, ‘Major warning’, ‘Intervention’, ‘adapting the sensitivity of auto-brake systems’, ‘adapting the temperature and airflow of the climate control system’, ‘rescheduling a delivery’, etc (see table 1). To balance these actions, one cannot simply use a fusion scheme with a single output, as often proposed in existing prior art.

(44) The iteration above may accordingly be executed when a new indicator value is available, or at times when the drowsiness estimate is needed rather than when data is made available, thereby improving the functionality of the system.

(45) Even though the invention has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art. Variations to the disclosed embodiments can be understood and effected by the skilled addressee in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. For example, the invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by the skilled addressee, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims. For example, the invention is also applicable for trucks, buses, dumpers, wheel loaders and other type of vehicles than the above described car.

(46) In the claims, the word “comprises” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single computer or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage.