Driving assistance method and driving assistance system with improved response quality for driver attention delegation

Abstract

The invention relates to a system and method for assisting a driver in driving a vehicle. First information on an environment of the vehicle is obtained and an instruction from the vehicle driver is received. An evaluation task defining an aspect of a current traffic situation encountered by the vehicle and to be evaluated is defined from the received instruction. An evaluation of the obtained information according to the evaluation task is performed, and an evaluation result is generated. Other aspects of the traffic situation and their relation to the aspect defined in the task are then evaluated additionally. Finally an information on the basis of the evaluation result and the other determined aspects is generated and output.

Claims

1. Method for assisting a driver in performing a driving maneuver with a vehicle at an intersection, the vehicle comprising at least one sensor, a processing unit and an output unit, the method comprising the steps of: obtaining information on an environment of the vehicle from the at least one sensor; receiving a spoken instruction from the vehicle driver; generating, based on the received spoken instruction, an evaluation task defining an aspect of a current traffic situation encountered by the vehicle to be evaluated, wherein the evaluation task defines as the aspect to be evaluated determining an availability of a gap between other traffic participants that can be used for performing the driving maneuver at the intersection, and the aspect of a current traffic situation as defined in the instruction covers only information on a first sector in the environment for which information is obtained; performing an evaluation of the obtained information according to and limited to the evaluation task, and generating an evaluation result; evaluating the obtained information with respect to other aspects of the traffic situation and determining their relation to the aspect defined in the evaluation task, wherein the other aspects define an observation sector in the environment of the vehicle different from the first sector or a traffic participant not defined in the received spoken instruction, wherein evaluating the obtained information with respect to the observation sector or the traffic participant not defined in the received spoken instruction comprises determining whether the determined gap can be used for performing the driving maneuver; and generating and outputting, by the output unit to the driver, output information on the basis of the evaluation result and the determined other aspects, wherein the output information includes the determined availability of a gap in response to the received instruction from the driver, and at least a time of outputting the output information depends on the determined other aspects of the traffic situation.

2. Method for assisting a driver in driving a vehicle according to claim 1, wherein the time of outputting the output information is set on the basis of existence of the determined other aspects.

3. Method for assisting a driver in driving a vehicle according to claim 1, wherein the evaluation includes a judgement if a gap's size g.sub.1, g.sub.2 is larger than a minimum gap size.

4. Method for assisting a driver in driving a vehicle according to claim 3, wherein the minimum gap size is individually set for different drivers.

5. Method for assisting a driver in driving a vehicle according to claim 1, wherein a content of the output information is adapted in response to a determined driver's awareness on the aspect defined in the instruction, wherein the driver's awareness is determined by observing the driver's eye direction and gaze pattern.

6. Method for assisting a driver in driving a vehicle according to claim 5, wherein the time of outputting output information is adjusted on the basis of a driver input.

7. Method for assisting a driver in driving a vehicle according to claim 1, wherein a content of the output information depends on the determined other aspects of the traffic situation.

8. Method for assisting a driver in performing a driving maneuver with a vehicle at an intersection, the vehicle comprising at least one sensor, a processing unit and an output unit, the method comprising the steps of: obtaining information on an environment of the vehicle from the at least one sensor; receiving a spoken instruction from the vehicle driver; generating, based on the received spoken instruction, an evaluation task defining an aspect of a current traffic situation encountered by the vehicle to be evaluated, wherein the evaluation task defines as the aspect to be evaluated determining an availability of a gap between other traffic participants that can be used for performing the driving maneuver at the intersection; performing an evaluation of the obtained information according to and limited to the evaluation task, and generating an evaluation result, wherein performing an evaluation of the obtained information comprises determining whether the available gap can be used for performing the driving maneuver, and judging if the size of the gap between other traffic participants is larger than a minimum gap size; generating and outputting, by the output unit to the driver, output information on the basis of the evaluation result, wherein the output information includes the availability of the gap between the other traffic participants in response to the received instruction from the driver, and the minimum gap size is set individually for different drivers.

9. Method for assisting a driver in driving a vehicle according to claim 8, wherein the minimum gap size is adapted to at least one of a driver's feedback input, and a particular driver's behavior, and a group of drivers.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The invention will now be described in more detail which respect to the annexed drawings in which

(2) FIG. 1 shows a schematic of the layout of the entire system including the individual units that cooperate for performing the inventive method; and

(3) FIG. 2 shows an example for a traffic situation explaining the functioning of the inventive method.

DETAILED DESCRIPTION

(4) The system and method according to the invention will now be explained in its general steps and elements with respect to FIG. 1. In FIG. 1 there is shown a sensing means 2 that consists of at least one sensor that is provided in the host vehicle. The sensing means 2 can of course comprise a plurality of sensors, even of different types that all together are capable of sensing physically the environment of the vehicle. Examples for such types of sensors can be cameras, radar sensors, laser range finders, ultrasonic sensors or time-of-flight sensors. The range of the environment which is sensed by sensing means 2 is in most cases much larger that what the host vehicle driver can attend. For example, the range that is attendable by the host vehicle driver is limited to his visual field and thus, opposite directions can be viewed by the host vehicle driver only after turning his head. Contrary, a plurality of sensors in the sensing means 2 can be active at the same time and thus, provide information on the environment on the host vehicle for any direction and any area of the environment. Such information on the environment in particular comprises information on position, heading and speed of other traffic participants such as vehicles and pedestrians.

(5) Additionally the host vehicle on which the entire system 1 shown in FIG. 1 is mounted might be equipped with other information sources 3, for example, a car-to-x communication module which is a communication system that receives information on the current traffic scene from other cars or any objects that communicate with the car to transmit information about its current and future status. If information is obtained from another car which is a traffic participant, the car may for example transmit information regarding its own position and/or trajectory and/or speed. Having such an information source included in the entire system is particularly advantageous since it delivers information the sensing means 2 cannot deliver.

(6) The inventive system 1 furthermore comprises a speech recognition unit 4 that receives driver's instruction, preferably the spoken instruction from the host vehicle driver and can interpret the spoken instructions. In the following only spoken instructions will be mentioned, because this is the preferred way for the driver to input his instructions to the system 1. From the spoken instruction a system task is generated. The system task includes a definition what aspect out of a plurality of aspects of an entire traffic scene shall be assessed by the system 1. Instructions may be simple phrases like watch to the right, watch the rear, watch oncoming traffic, watch the car in front, and so on. These phrases describe the evaluation task that corresponds for example to a request of the driver to the system to observe a particular area of the vehicle's environment or evaluate some parts of a traffic situation. Of course such spoken instructions need interpretation in order to filter the core request of the host vehicle driver. The spoken instructions that are to be used in such a system 1 may be restricted to a limited number of different precisely defined phrases in order to provide for a sufficiently robust recognition. But it is not necessary to limit it to precisely defined commands as modern speech recognition systems are capable of interpreting even more complex expressions it is also possible to use instructions like tell me at the next crossing if there are vehicles approaching from the right or always inform me if there is a bicycle on my right side when I want to turn right.

(7) From these spoken instructions, a system task is generated which means that the result of the interpretation is converted into clear system interval commands that set up the advanced driver assistance system for processing the information on the environment obtained by environment sensing 2 and/or car-to-X-communication 3 to fulfill the system task and thus evaluate an aspect of a traffic scene out of a plurality of aspects constituting this traffic scene. As an easy example the spoken instructions could be checked for presence of key words watch and right. Such an identification could then be used in a mapping unit to generate the system task which defines an area (right), relative to the driving direction of the vehicle that shall be subject to the monitoring and evaluation (watch).

(8) After the system task is being defined the system task is provided from the speech recognition unit 4 to a settings controller 5. In the settings controller 5 the evaluation task is mapped on settings in the driver assistance system that are used to perform the system task. If for example only a particular area of the entire environment of the host vehicle is to be monitored this means that only information out of the entirety of obtained information regarding this particular area of the environment is used. The mapping of the system task to the settings of the system 1 can be used to limit information to only a corresponding part of the information that is provided by the sensing means 2 and/or the car-to-x communication unit or that only objects that are mentioned in the instructions which are perceived are subject to further evaluation. The result of these settings is then supplied to an attention selection section 6 which finally selects the information sources (sensor means, car2x communication system, . . . ) on the basis of the settings given by the settings controller 5.

(9) The previous description was based on the fact that in a current situation a spoken instruction is received from a host vehicle driver and that the evaluation task is generated on the basis of this actual spoken instruction. During use of the inventive system 1 the evaluation tasks that can be generated from the spoken instructions are stored. The evaluation tasks are always stored associated with the traffic scene context. Thus, since the driver assistance system 1 can permanently monitor the environment and thus remember similar traffic scenarios in a current scenario the system can autonomously generate a system task that is likely to be reasonable again. The system task or the corresponding settings may then be retrieved from memory 7 and directly supplied to the selection unit 6. Alternatively only the evaluation task is retrieved from the memory 7 and supplied to the settings controller 5 that generates then the settings and feeds the same to the selection unit 6.

(10) The system further includes an information output unit 8 for outputting a signal including information based on the evaluation results on the requested aspect by the driver. On the basis of the signal a speech output is generated by speaker 10.

(11) The idea of the invention is to optimize a response given to a driver instruction so that with minimum distraction a maximum of information can be provided to the driver. The maximization of the information is not performed by outputting additional and individual information, but by customizing the content and the timing of information in response to the driver's request. To achieve this, the environment perception unit 9 comprises two sections. Of course, the two sections can be implemented as software modules executed on a common processing unit. At first, in a partial environment perception unit 9.1 only information on the environment that is directly related to the instruction given by the driver is evaluated. The partial environment perception unit 9.1 is thus provided with information on the environment sufficient to respond to the driver's request. This is achieved by means of the selection unit 6. The selection unit 6 either literally selects portions of the available information and provides it to the partial environment perception unit 9.1. Alternatively, all the information that is obtained by the system 1 is provided to the environment perception unit 9 in general, but by means of the selection unit 6 it is indicated which of the information needs to be processed by the partial environment perception unit 9.1.

(12) In addition, the environment perception unit 9 comprises an entire environment perception unit 9.2. In the entire environment perception unit 9.2 it is possible to evaluate additionally information on the environment that was not used in the partial environment perception unit 9.1. Advantageously, the entire environment perception unit 9.2 receives information about the results of the partial environment perception unit 9.1. Thus, still referring to the example of requesting the system to give information about traffic approaching from a right side at an intersection, the entire environment perception unit 9.2 receives information from the partial environment perception unit 9.1 that a gap will be available which is suitable to be used by the driver. On the basis of this information then the entire environment perception unit 9.2 evaluates obtained information from sensing means 2 and car-to-X-communication 3 in order to determine if there are any other aspects that can be perceived in the traffic scene from the environment information which might prohibit the use of the identified gap. Such information is for example that another traffic participant is identified by evaluating also the left side of the vehicle approaching from the left. This information on an other aspect is then transferred to the information output unit 8. Of course, also the evaluation result provided by the partial environment perception unit 9.1 is provided to the speech output unit 8.

(13) In the speech output unit 8 both, the evaluation result and the evaluation of other aspects of the traffic scene are combined. This means that the content of the information which is to be output might be adapted to the additionally available information which is per se not output to the driver. In addition or instead of adapting the content to the additional evaluation result, the timing of outputting the information can be controlled by output control unit 8.2. Finally, information is output by means of a speaker 10. Of course, outputting information by a speaker 10 is only one example. Others might be use of head-up display (HUD) or a display on the dashboard or the like.

(14) In the following and with respect to the further FIG. 2, examples will be used to illustrate the inventive method, system and vehicle. When studying the example, it is to be noted that the examples given all use a right-hand-side driving traffic system for illustration reasons only. Of course the invention is not limited to such a system and might easily be adapted to a left-hand driving traffic system.

(15) An example for illustrating the operation of the inventive system and method is shown in FIG. 2. The driver of the ego-vehicle 15 is standing at an intersection where he does not have the right of way. The intended trajectory of the vehicle 15 is a left turn. The driver of the vehicle 15 instructs the advanced driver assistance system mounted in the vehicle 15 to observe a right-hand-side of the car in front of the car for example by saying Watch right. Thus, the task of observing a particular section of the vehicle's environment and evaluating relevant information obtained for this section with respect to an intended driving behavior is delegated to the system 1. From the instruction given by the driver in the example, the system 1 concludes that the evaluation task that has to be performed by the system is determining fitting gaps in traffic approaching from the right side of the road which is intended to be entered by vehicle 15. Thus, the system 1 searches for and determines fitting gaps on the right side. In the illustrated embodiment, the system 1 will detect a first gap g.sub.1 between vehicle 17 and preceding vehicle 18. The gap g.sub.1 is determined as being larger than a minimum gap size. G.sub.2 which is determined between vehicle 17 and vehicle 19 being the successor of vehicle 17 is also larger and would fulfill the requirement with respect to a minimum gap size. Thus two gaps g.sub.1 and g.sub.2 are available for the turning maneuver and their existence constitutes the evaluation result.

(16) The minimum gap size can be fixedly set when designing the system 1, but advantageously may be adapted during use of the system 1 or when initializing the system 1. The minimum gap size may be set individually for each driver of a car. The gap size might also be set using knowledge from a large pool of drivers. In order to calculate such minimum gap size, the behavior of a plurality of drivers is analyzed. From the knowledge about the size of gaps which are used by the drivers, the minimum gap size sufficient for being used can be calculated. This can of course also be done for each driver individually. In that case it is necessary that the driver performed a sufficient number of comparable maneuvers already meaning entering or not entering into a gap between two vehicles so that a conclusion can be drawn which gap size the respective driver considers as being sufficiently large. This gap size is then defined as minimum gap size.

(17) The behavior of the driver thus can be used to determine a minimum gap size, but it is also possible that a direct interaction between the driver and the system influences the minimum gap size used for evaluation of the traffic scene. For example the driver can give feedback that the suggested gap was too short or that the gap was longer than actually needed or confirm the suggestion of the system by for example saying this gap was just right. Also feedback commands like I need larger gaps or I don't need such large gaps can be used. In general, the interpretation of such spoken commands is known in the art. With respect to the present invention, it is only relevant that the content analyzed from such spoken commands leads to an adaptation of the minimum gap size used for the evaluation in the partial evaluation unit 9.1.

(18) As mentioned above, if, based on the minimum gap size actually set, the system recognizes that gaps g.sub.1 and g.sub.2 are large enough, the system will then monitor automatically also other aspects of the current traffic scene. In the illustrated embodiment, there is a vehicle 16 approaching from the left side. Thus, if the system 1 directly recommended starting the left turning maneuver after vehicle for example by outputting a spoken information Gap after the second vehicle from the right based on its evaluation of g.sub.2 a critical situation could occur.

(19) For example if the driver of the ego-vehicle 15 would rely on such immediately output suggestion made by the system 1 he possibly already starts to enter the intersecting road. According to the present invention the other aspect of the current traffic situation, namely the car approaching from the left, which is not directly involved in the estimation of gaps g.sub.1 and g.sub.2 in traffic approaching from the right, is also taken into consideration. In the example vehicle 16 blocks g.sub.1 and thus only g.sub.2 can reasonably be recommended. After such other aspect was determined by the system the output information is adapted to be useful to the driver. In the present case for example the output of the information gap after next car is delayed until vehicle 17 passed the intersection area. At that point in time it is considered by the system 1 that also vehicle 16 has already passed the intersection and thus, gap g.sub.2 is available for the ego-vehicle 15. Thus, according to one embodiment of the invention only such gaps are announced to the driver and thus form the output information that can really be taken by the ego-vehicle.

(20) In other situations the output of information might also be performed earlier. This might be advantageous for example if vehicle 16 is not too close to the intersection and thus a quick turning maneuver could make use of gap g.sub.1. Because of the earlier information output the driver will also start his turning maneuver earlier and thus not lose time.

(21) It is to be noted that during the waiting time the system could fill a silence by outputting wait . . . or UMM for example so that the vehicle driver knows that the system 1 is active and will output the desired information shortly.

(22) Furthermore and as also indicated in FIG. 2 there is a pedestrian 20 standing on the other side of the road who is ready to cross the road. When stepping on the lane which is to be used by the ego-vehicle 15 the pedestrian blocks the way and the intended trajectory of vehicle 15 might not be used. Thus, in that case the system 1 would also not output the information that there is a gap in order to avoid that the driver starts his turning maneuver.

(23) As mentioned above, the system 1 will evaluate also the reaction to an output information for adapting its future behavior. In case that for example the driver will not use a gap that was announced by the system 1 as being sufficiently large the system 1 will automatically adapt the minimum gap size in order to better fit to the driver's driving behavior.

(24) This of course works also the other way round. If a gap is considered to be not large enough by the system but nevertheless the driver performs his turning maneuver and safely enters the gap the system 1 will adjust the minimum gap size for future evaluations.

(25) The crucial aspects about the invention is that the information that is output in the end directly refers to what was the desired information of the driver by generating an evaluation result limited to an aspect of a traffic scene defined on the basis of the received instructions. Nevertheless the system 1 will automatically observe the environment not only with respect to this particular evaluation task limited to the instructions given by the driver. The response of the system 1 will give information only related to the area or more generally the aspect of a traffic scene as delegated by the driver. But the timing and possibly also the content of information output as the response to the instruction takes the situation of the entire environment into account and thus also evaluates other aspects of the traffic scene.

(26) In order to further improve the system it is possible that the system analyzes the awareness of the driver to the different aspects of the traffic scene. Thus, it can be determined if the awareness of the driver with respect to the other aspects is sufficient to make a consideration on his own. In that case it might be preferable not to adapt the content of the output information but maybe only its timing. In case that the driver is unaware and that his lack of awareness is considered critical by the system 1 even if it does not correspond to the instruction given by the driver a respective warning message might be output.

(27) Of course the driver can also set preferences or give instructions as to automatic conditions when the system 1 shall be active with respect to a particular instruction. For example the driver might instruct the system 1 to always watch the right side at intersections until we reach X, in the city, in the neighborhood Y, while it is raining and so on. Furthermore, also the feedback from the system 1 can be tailored according to the driver's needs. When the system 1 detects a suitable gap usually the information is output at a particular point in time which means relative to the passing vehicle after which the gap is available. Drivers may for example prefer to receive the information generally earlier and thus, the timing of the information output can be adjusted upon entering preference by the driver. Also the frequency of providing the driver with updates on the status of the traffic with respect to the delegated evaluation task can be adjusted. Also the degree of detail that is given in the information can be adjusted by the drivers. Spoken commands are for example tell me possible gaps earlier, give me less frequent updates on the traffic from the right, or tell me the color of the car after which there will be a gap.

(28) The more assistance a driver receives by all driver assistant systems the more there is a risk that the driver fully relies on the evaluation of the system 1. But it is still desirable that the system 1 does not take over full responsibility for the observation of a particular side of the car for example but only to give recommendations to the driver. In order to avoid that the driver might develop overconfidence in the system 1 evaluations the system can observe the driver's gaze patterns if he is still observing the environment himself. Based on such observation the system 1 might then adapt a feedback to the driver and for example express more uncertainty in its response. An example is that the information gap after next car from the right is changed to there might be a gap after the next car from the right. This triggers the attention of the driver to evaluate the situation by himself before he can enter the intersecting road.

(29) Finally it is to be noted that the system 1 can be able to support different intended maneuvers of the driver. All the aforementioned examples referred to a left turning maneuver in case of a right hand driving traffic system. Left hand side driving systems are of course also possible. Furthermore apart from turning also going straight over an intersection, leaving or entering a drive way, and other comparable driving maneuvers may benefit from the inventive method and system. Also observation of the status of traffic lights could be a delegated evaluation task. Of course the instructions that have to be given to the system differ in that case. But generally the idea of the invention is to provide the driver with information limited to respond to his original instructions or requests for observation and evaluation by the system. But to take into consideration further aspects of the entire traffic scene the interaction with the evaluation result being taken in account of adapting at least the timing of outputting the information and/or the information content.