METHOD AND SYSTEM FOR ASSISTING A DRIVER IN DRIVING A VEHICLE AND VEHICLE ON WHICH SUCH SYSTEM IS MOUNTED

Abstract

The invention regards a method and a system for assisting a driver in driving a vehicle, as well as a vehicle with such system being mounted thereon. Information on an environment of a vehicle is obtained by sensing the environment. Furthermore, information on applicable traffic rules is obtained. A request from a driver is determined from sensing at least one of a driver's utterance, gestures, gaze and operation of vehicle control. The traffic scene as sensed is assessed on the basis of the environmental information, the applicable traffic rules and the determined driver request. Information as requested to be output is generated in response to the determined request of the driver. Finally the information in response to the request from the driver is output.

Claims

1. Method for assisting a driver in driving a vehicle, comprising the steps of: obtaining information on an environment of a vehicle, obtaining information on applicable traffic rules, determining a driver request from sensing at least one of a driver's utterance, gestures, gaze and operation of vehicle controls, assessing the traffic scene on the basis of the environmental information, applicable traffic rules and determined driver requests, generating requested information to be output as a response to the determined request, and outputting this response to the driver.

2. Method according to claim 1, wherein additional information is generated and output, wherein such additional information is generated on basis of a second level request determined on the basis of the determined driver request.

3. Method according to claim 2, wherein in the response it is announced that additional information will be output to the driver.

4. Method according to claim 1, wherein a driving intention is determined based on driver sensing, and the assessment of the traffic scene also takes account of the driving intention.

5. Method according to claim 1, wherein the driver request or the driving intention is determined based on speech analysis of a driver's utterance.

6. Method according to claim 1, wherein the requested information is output via at least one of a speaker, head up display and display.

7. Method according to claim 1, wherein the traffic scene assessment uses map data.

8. Method according to claim 1, wherein the generated requested information includes information on which of perceived traffic objects has right of way.

9. Method according to claim 1, wherein the generated requested information includes information on parking regulations with respect to an area depicted by the driver.

10. Method according to claim 1, wherein the generated requested information includes information on a currently applicable speed limit.

11. Method according to claim 1, wherein the generated requested information includes information on estimated safe driving speeds.

12. Method according to claim 1, wherein the generated requested information includes information on a recommended driving behavior for the currently encountered traffic situation.

13. Method according to claim 1, wherein for determination of the driver request or for selecting an output modality, an individual driver model is used.

14. System for assisting a driver in driving a vehicle, comprising: sensing means for sensing a vehicle environment, obtaining means for obtaining information of applicable traffic rules, determination means for determining a driver request, traffic scene assessment unit for analyzing the traffic scene on the basis of at least the sensed environment, applicable traffic rules and the determined request, and output means for outputting the information in response to the driver's request.

15. System according to claim 14, wherein the determination means is connected to sensing means including at least one of a microphone, a camera system and sensors sensing operation of vehicle controls.

16. System according to claim 14, wherein the output means includes at least one of the speaker, a head up display and a display.

17. System according to claim 14, wherein the sensing means comprises at least one of a radar sensor, a lidar sensor and a camera system and a GPS.

18. System according to claim 14, wherein the system further comprises a GPS system.

19. System according to claim 14, wherein the system further comprises a memory on which a database is stored which comprises information on traffic rules.

20. System according to claim 14, wherein the system comprises a communication means for accessing a remote database on which information and traffic rules are stored.

21. Vehicle including a system according to claim 14.

Description

BRIEF DESCRIPTION OF THE DRAWINGS:

[0026] Further features and advantages may become apparent from the following explanation which describes preferred embodiment with reference to the annexed drawings. It is shown in

[0027] FIG. 1 a block diagram of the system according to the present invention;

[0028] FIG. 2 a simplified flow chart for explaining the steps of the current invention;

[0029] FIG. 3 a first traffic situation as an example for the functioning of the present invention;

[0030] FIG. 4 a second example for illustrating the functioning of the present invention, and

[0031] FIG. 5 a third example for illustrating the functioning of the present invention.

DETAILED DESCRIPTION:

[0032] FIG. 1 shows a simplified block diagram of a system 1 that is intended for assisting a driver in driving a vehicle. The system 1 is mounted on a vehicle and comprises sensing means 2. The sensing means 2 may comprise a plurality of sensors that are mounted in a distributed fashion over the vehicle. Thus, with the sensors it is possible to achieve a 360° surveillance of the vehicle's environment. The sensors may particularly be radar sensors 3.1, lidar sensors 3.2, a camera system 3.3 and a GPS system 3.4. The sensing means 2 may in particular comprise a plurality of those sensors of different types.

[0033] The signals that are generated by the sensors 3.1 to 3.4 by physically sensing the environment of the vehicle are supplied to an environment representation generating unit 4. In the environment representation generating unit 4 a representation of the environment based on the sensor signals is generated. Generally such environment representation generation 4 per se is known. It is in particular important that by sensing the environment by means of the sensors 3.1 to 3.3 traffic objects in the surrounding of the vehicle can be perceived. In addition to recognizing the existence of such objects also velocity, heading or the like may be determined for the traffic objects. The environment representation that is generated in the environment generation representation unit 4 is then forwarded to a traffic scene interpretation unit 5. In the traffic scene interpretation unit 5 a traffic situation is determined. This means, that the information that is objectively gathered from the environment sensing is then interpreted with respect to the context and the relationship between the different identified objects. Thus, a traffic situation, like recognizing a crossing with interpreting and predicting the behavior of the individual traffic participants can be derived.

[0034] Furthermore, the system 1 needs information about the applicable traffic rules. The applicable traffic rules may either be stored in a traffic rule database 6 being stored in a memory of the system 1. Alternatively the system may also be in communication with a remote database 8. The remote database is accessed via a communication means 7 of the system 1. This has the advantage that at any time it is sufficient to update the remote database 8 and then always have up to date information on applicable traffic rules. In addition the system 1 comprises sensing means 10 that are used to determine on the one side an intended driving behavior of the driver and on the other side the requested information. The sensing means 10 may advantageously comprise a plurality of sensing units. These sensing units may include a microphone 11.1, a camera 11.2 and/or sensors that allow observing operation of vehicle controls like a steering wheel, brake, turning light switches and so on. The signals generated by the sensing means 10 from sensing a driver while driving the vehicle are supplied to a driver modeling unit 12. In the driver model firstly the requested information from the driver is determined. One central aspect when determining the requested information is an analysis of the utterance of the driver. As mentioned earlier the driver preferably uses spoken commands in order to define the information he desires to receive. It is particularly preferred that the sensor signals that are also fed to the driver modeling unit 12 are used in addition to the spoken command. As mentioned above in particular gestures, eye directions and the like may be used to limit an area of interest so that a speech command with rather general definition of requested information can be enhanced by limiting the identified request to a particular area. This will become even clearer when taking into consideration examples of traffic situations that will be explained with reference to FIGS. 3 to 5.

[0035] The sensing means 10 and the driver model 12 are not only used to determine the requested information but also may be used to determine an intended driver's behavior. This means that on the basis of the vehicle controls that are operated by the driver or even by spoken information it is possible for the system 1 to recognize an intended trajectory, an intended driving maneuver like for example a turning movement or the intention to find a parking space. The latter for example can be recognized if from the eye direction it can be gathered that the driver always looks to other vehicles that are parking alongside the road and also his driving speed is reduced compared to his usual driving. Thus, here individual aspects of currently driving person are taken also in consideration. In order to determine the driving speed being unusual low other information from the system may also be used. For example it has to be distinguished between the need of reducing the driving speed because of a preceding vehicle or if there is a free lane ahead of the vehicle. Then, it could be concluded that the slow driving is performed intentionally.

[0036] The previous description was based on the fact that in a current situation a spoken instruction or command is received from a host vehicle driver and that the response is generated on the basis of this actual spoken instruction. During use of the inventive system 1 the driver usually makes use of the system's capabilities repeatedly. Preferably, these spoken instructions can be stored. The commands or the request analyzed from the command are stored associated with the traffic scene context. Thus, since the driver assistance system 1 can permanently monitor the environment and thus remember similar traffic scenarios in a current scenario the system can autonomously generate information that is likely to be reasonable again. The corresponding settings of the system may then be retrieved from a memory. Alternatively only information on usually requested information is retrieved from the memory and the system assessed the traffic situation newly.

[0037] The system 1 may use a generic driver model 12. But advantageously, the history of interactions between the driver and the system 1 is used to generate individual driver models 12. To achieve such personalized model 12, the interactions between the driver and the system 1 is analyzed. The interactions are observed and stored associated with the respective driver. In particular, if the driver gives feedback, for example to the timing when the system 1 shall be output, the requested information is stored for each driver individually. But also the determination of intended driving behavior can be optimized so that in the end the situation analysis is improved. Other sources of information about the driver may also be exploited.

[0038] The individual driver models may in particular include preferences of the driver. One example is a level of detail of the information presented to the driver. As mentioned above the system is capable of receiving and storing feedback that is given by the driver after he received the output information (either the information output in the response or the additional information). The amount of information that is given might initially be set by a default setting. If then the driver repeatedly asks for more detailed information, the preferences regarding a level of detail with respect to this individual driver can be set to more details. Of course, this is also possible the other way round in case that the driver indicates in his feedback that he only needs less information.

[0039] But not only feedback given by the driver directly can be used to adapt the output information individually. For example, if one particular driver requests the system for several times in the same area at the same time of day if he can park here, the system might directly take into account the duration the driver parked there before. Thus, information on the driver's behavior is stored and can be exploited to adapt the driver's model. If then the system receives a new request from the driver, knowledge about the previous parking durations is used in order to directly indicate those parking spaces to the driver which allow him to park for the length of time he parked previously.

[0040] Another example for a source of information that can be used to adapt the individual driver models is a smartphone of the driver which usually is connected to an infotainment system of the vehicle, for example by Bluetooth. Thus, information that is present on the smartphone can be analyzed by the system 1. By analyzing information from the smartphone it is for example possible to determine the nationality of the driver. The driver might for example not be a national (or a resident) of the country where the trip started. This is particularly the case if he is the driver of a rental car. When then the driver requests information on traffic rules for the current situation, the system might explain them by referring to rules well known to the driver, in particular to rules applicable in his country of origin or residence. Thus, information about applicable speed limit might be for example “the speed limit on rural roads are the same as in your home country”.

[0041] All this information is then fed to an assessment unit 13. The assessment unit 13 has knowledge about a traffic situation which is interpreted from the environment representation in the traffic scene interpretation unit 5. From the traffic rule database 6 or the communication means the assessment unit 13 is provided with information about traffic rules. As mentioned above such traffic rules may either be general traffic rules or may also be regional driving practice. The assessment unit 13 is furthermore provided with the analysis result of the driver model 12 and thus information on the requested information a response to which has to be generated. In the assessment unit 13 an analysis of all these different aspects is performed which means that the information is combined. Thus, from the requested information it is determined what aspects of the traffic situations are relevant for generating a response to the request from the driver. For example if the driver requests information on priority rights at an intersection it is checked at first by the assessment unit 13 which priority rights have to be taken into consideration at that particular intersection. In order to correctly assess such priority rights the system uses information on general priority regulations of the respective country or area taken from the traffic rule database 6 but also information that is derived from the environment representation. The environment representation may for example include information about traffic signs, traffic lights or the like. In the assessment unit 13 this information is considered commonly so that the regulations that are applicable in the currently encountered traffic situation are used. Then the other traffic participants are analyzed with respect to their relation to the identified priority regulations. Thus, after having determined the applicable traffic regulations the traffic situation is further analyzed with regard to these regulations so that in response to the driver's request tailored information can be presented. This information is then output and forwarded to an information modality selector 14. In the information modality selector 14 one of a plurality of output modalities is chosen. The selection can for example be based on key words that are included in the information to be output. If for example the information includes a direction where the attention of the driver shall be drawn to it is possible to indicate the direction visually. On the other side in many cases a speech output is preferred, since such speed output can easily be grasped by the driver without the need that he turns his head or changes his eye direction. After selecting a proper output modality the signal including the information is supplied to an output means 15. The output means 15, for example comprises a speaker 16.1, a head up display 16.2 and/or a display 16.3 mounted for example on the dashboard of the vehicle.

[0042] In FIG. 2 there is shown a simplified flowchart of the inventive method. At first, in step 20 the environment of the vehicle is sensed. The signals generated by the sensing means 10 are then processed in order to generate an environment representation 21. The result of the environment representation is used to perform a traffic scene interpretation 22. Furthermore, the result of the environment representation generated in step 21 is used to identify applicable traffic rules in step 23. In step 23 also, generally applicable traffic rules that are obtained from databases as described with respect to FIG. 1 are analyzed and the identified applicable traffic rules of step 23 are used to analyze the traffic situation in step 27. Also, the requested information is used to analyze the traffic situation. For determining the requested information, a speech input is received in step 24 by means of a microphone. In addition, also the driver's gestures or operated controls by the driver are sensed in step 25. The received speech commands and the sensed control operation and/gestures of steps 24 and 25 are used to model a driver in step 26. The output of the driver modeling includes information on the requested information which is supplied and is the basis for analyzing the traffic situation in step 27 in combination with the traffic scene interpretation of step 22 and the traffic rules of step 23.

[0043] After analyzing the traffic situation on the basis of this information, output information is generated in step 28. The output information basically comprises the content of the information which is then used to select a modality for outputting the information in step 29. Thus, in step 29, the system selects if the information is output as a response by outputting speech, by displaying the information on a display 16.3 or a head up display 16.2 or if the response is output haptically. Such haptic output of information could be for example vibration of the steering wheel or generating forces on a seat, paddles or the steering wheel.

[0044] Finally, the information in response to the driver's request is output in step 30. The invention will become clearer when taking into consideration, situations and the process for generating information output in response to a driver's request as it will be described with respect to a plurality of examples in FIGS. 3 to 5.

[0045] It is to be noted that all the data and signal processing can be executed by a single processing unit or a plurality thereof.

[0046] FIG. 3 shows a top view of an intersection. The vehicle E on which the inventive system 1 is mounted is approaching the intersection. Its intended driving trajectory is indicated by the dashed arrow.

[0047] At the intersection that is observed by the sensing means 10 of the system 1, two other traffic participants are also approaching. At first, from the right relative to vehicle E there is approaching a vehicle A. And furthermore driving opposite direction of ego-vehicle E there is approaching a further vehicle B. As indicated by the traffic sign 32 and 31, vehicles E and B have to give way for vehicle A, which is driving on a priority road. When approaching the intersection, the driver of vehicle E may ask the question “Which car has the right of way?”. The system will use the results of the environment sensing which gives information on vehicle A, vehicle B as well as the traffic sign 32. Thus, the system 1 generates an environment representation in which the intersection itself with the two roads, vehicle A, vehicle B and at least traffic sign 32 is represented. It might be that traffic sign 31 cannot be identified by the system because it is sensed only from the back.

[0048] The system 1 will obtain information on the priority rights from the traffic rule database 6 and thus conclude that at first, vehicle A can cross the intersection. Furthermore, from the sensing of the driver, the system can determine that the intended trajectory of vehicle E corresponds to the dashed arrow, for example, by the turning light lever of vehicle E being operated so that the turning lights on the left side of the vehicle E are flashing. Thus, since the traffic rules that are stored preferably in a database of system 1, the system 1 can also conclude that after vehicle A has passed, also vehicle B has priority. Thus, it generates information which conveys to the driver of vehicle E that the driver has to let pass vehicle A first, then vehicle B before he can perform his turning maneuver. An output of the system 1 may be spoken information which is preferred. Thus, in the present traffic situation a response could be that “the red car approaching from the right has priority and after that you have to let the blue station wagon pass”. As indicated in the example of a response, it is possible to improve legibility of the responded information by identifying other traffic objects by mentioning their characteristic, for example the color of the car. This allows an easy understanding of the information and avoids any misinterpretation of the response.

[0049] Furthermore, the system might also announce that it will inform the driver when all other traffic participants who have right of way have passed such that the driver can then immediately start his intended maneuver. At the end of the response as depicted above (“The red car approaching from the right has priority and after that you have to let the blue station wagon pass.”) the system may first utter “I will notify you when you have the right of way”. While the last relevant traffic participant before vehicle E may drive on is passing the system 1 might utter “after the blue station wagon has fully passed you have the right of way” thereby outputting additional information. Presenting this additional information slightly before the driver can start his maneuver will also give the driver some time to prepare himself. Of course such additional information that is given at a point in time when the driver can start his intended driving maneuver or at least prepare his intended driver maneuver can be output in addition to the response even without announcing giving such additional information. For generating such additional information the system 1 determines from the driver's utterance not only a driver request but also reasons what the motivation for such request was. In the present case obviously the driver wants to know when he can perform his turning maneuver. So the additional information is given as a response to a second level request that is determined from the determined driver request. Such second level requests can be defined in a table that associate second level requests to determined driver requests.

[0050] Another situation is depicted in FIG. 4. Here, the vehicle E is standing at red traffic lights and an ambulance C is approaching from behind. The opposing lane is currently blocked and thus the ambulance may not simply overtake vehicle E. A question uttered by the driver may be “How should I yield to the ambulance I'm hearing?”. The response of the system could be spoken information again “Slowly enter the intersection in front of you despite the red traffic light”. From that example it becomes apparent that this system will not only include general traffic rules like the meaning of a red light that the driver is not allowed to enter the crossing, but also uses exceptional rules which only apply in a certain context. As the system analyzes the traffic situation and thus has knowledge about the presence of, for example an ambulance or police in the illustrated situation, it is then capable of distinguishing between the applicability of a general rule or an exceptional rule.

[0051] There may also be situations where from the sensing of the environment different types of other traffic participants can be identified. For example, vehicles can be classified into busses, trams, cars and the like. If the spoken command or question from the driver, for example is “Does the tram crossing from the right has the right of way?”, the information can be limited to this particular aspect. Thus, if there is approaching a tram, the system will limit the output information to the question of priority of the tram, even if there are other traffic participants perceived in the scene. In that case the system may answer “Yes, the tram has the right of way”.

[0052] FIG. 5 shows a further example of a traffic situation. The driver of vehicle E is driving straight on a road where a plurality of vehicles park on the right side of the road. Between the parking vehicles, two gaps 1 and 2 are present, both suitable for parking vehicle E. The driver spotted a gap 2 which is more convenient for him and is interested in parking his vehicle E there. But he is not sure if he is allowed to park there. He thus asks the system “Am I allowed to park here?”. From that question, the system 1 in first place cannot decide for sure which of gaps 1 and 2 is addressed by his questions. It is assumed that based on sensing the environment, the system is capable of recognizing gap 1 as well as gap 2. But the system 1 further comprises sensing means 10 that observe head movements and eye directions as well as gestures of the driver. Thus, since the eye direction as indicated by the dotted arrow usually is in direction of the gap of interest, the system can determine that the requested information concerns only gap 1. Thus, the system will analyze the traffic situation with respect to gap 1 only. Again, traffic signs that are sensed by a camera system are analyzed and compared to rules that have to be inferred from the respective traffic sign, for example. Alternatively, also map information can be used to determine if parking is allowed at a position of gap 1. In order to be sure which of the traffic signs is currently valid, the system 1 will while driving continuously analyze the traffic signs and label areas in the environment representation accordingly. Thus, if it is clear from a traffic sign which was already passed by vehicle E that parking is allowed in this area, but only for a limited time, the system might reply “You are only allowed to park here for one hour. After that, this space is reserved to residents”.

[0053] Since the system preferably has access to maps including information on points of interest for example parking spaces, the system might then even suggest an alternative parking space. If the request from the driver comprises additional information like for example an intended parking duration this can also be taken into consideration. In that case the system might respond for example you are not allowed to park for your intended duration “You will find an alternative parking space in a parking lot 500 meters ahead”.

[0054] According to the invention the system 1 may also assist the driver in a more general way. For example the driver crossed a border to a foreign country and thus different traffic regulations are applicable. The driver might not be so familiar with these different traffic regulations and thus after driving for sometime in a foreign country he might have forgotten particular speed limits. Because he is unsure the driver will consequently ask the system for a currently applicable speed limit or for priority regulations for example in a roundabout. One possible question is thus “How fast am I allowed to drive on this road?” or “what is the speed limit on rural roads in the UK?”, “Who has priority in a roundabout?” or the like. Again the system 1 will combine the information from the environment sensing, the traffic rule database and the determined requested information. The sensing of the environment is used in order to analyze on what type of road the driver is currently driving but also in order to identify traffic signs indicating a speed limit. The database 6 is accessed in order to further determine the general applicable speed limits so that in case that no specific speed limit traffic sign can be identified still the requested information can be output.

[0055] It is obvious that this general information being output in response to a respective request from the driver is not limited to foreign countries but can also be made in the driver's country of origin.

[0056] A final example is a driving situation in which the driver needs assistance because he is not sure how to behave because of exceptional conditions. The driver is for example driving on a rural road trough the forest. The road has sharp turns and since it is autumn there are many leaves on the ground. Thus, the driver would like to receive a recommendation for a safe driving speed for these road conditions for the next sharp turn. A question output to the system may be “At what speed should I drive in the corners not to risk to skid?”. The system assesse the traffic situation by an analysis of the turn based on for example environment sensing but furthermore based on map data. In addition the road condition is taken into consideration which is based on an identification of the leaves on the ground performed on the basis of the environment sensing. Thus, the system 1 can estimate a recommended maximum speed for safe driving. The system 1 has furthermore knowledge about the speed limit which is set either by a traffic sign or which is generally set because of traffic rules of the country and determine if the recommended speed is lower than the allowed speed. In case that the allowed speed is lower it will output an information that the for example “driving at the current speed limit of 80km/h will cause no risk to skid”. But if the applicable speed limit in that situation is for example 100km/h it might output the estimated maximum driving speed for example by saying “if you drive slower than 80km/h you do not risk to skid”.

[0057] Such output may of course be generated individually for each upcoming turn or for all turns within a certain distance, for example the next two kilometers. The driver may include such information in his spoken command by for example adding “in the corners of the next two kilometers”. On the other side, he could mention “the next corner” or “the corner ahead”. In that case, the system will determine by analyzing the spoken command that the instruction shall be limited to only a particular corner or turn and that the command thus would be repeatedly given in case the driver wanted respective information for further corners.