SITUATIONAL AWARENESS IN A VEHICLE

Abstract

Enhancing situational awareness of an advanced driver assistance system in a host vehicle can be provided by acquiring, with an image sensor, an image data stream comprising a plurality of image frames. Analyzing A vision processor can analyze the image data stream to detect objects, shadows and/or lighting in the image frames. Recognizing A situation recognition engine can recognize at least one most probable traffic situation out of a set of predetermined traffic situations taking into account the detected objects, shadows and/or lighting. A processor can then control the host vehicle taking into account the at least one most probable traffic situation.

Claims

1-14. (canceled)

15. A computing device for a vehicle, including a processor and a memory configured such that the computing device is programmed to: acquire, with an image sensor, an image data stream comprising a plurality of image frames; analyze, with a vision processor, the image data stream to detect objects, shadows and/or lighting in the image frames, wherein analyzing the image data stream comprises detecting artificial lighting in the image frames, the artificial lighting being dynamic lighting that includes at least one of moving or changing size in relation to surfaces or objects being illuminated; recognize, with a situation recognition engine, at least one most probable traffic situation out of a set of predetermined traffic situations taking into account the detected objects, shadows and/or lighting; and control the vehicle taking into account the at least one most probable traffic situation.

16. The computing device of claim 15, further programmed to analyze the image data stream by detecting shadows in the image frames.

17. The computing device of claim 16, further programmed to analyze the image data stream by detecting dynamic shadows, wherein movements and/or changes in size of the dynamic shadows are detected with reference to the surfaces the respective shadows are cast on.

18. The computing device of claim 16, wherein the set of predetermined traffic situations include traffic situations for anticipating an out-of-sight traffic participant, the traffic situations including the out-of-sight traffic participant casting a shadow into the field-of-view of the image sensor, wherein the shadow is a moving shadow.

19. The computing device of claim 16, wherein the set of predetermined traffic situations include traffic situations comprising a row of parking slots, wherein a plurality, but not all, of the parking slots are occupied by respective cars, the cars in the occupied parking slots respectively casting a shadow into the field-of-view of the image sensor, an unoccupied parking slot being identifiable by lack of a corresponding shadow even if the unoccupied parking slot itself is still out-of-sight.

20. The computing device of claim 15, wherein recognizing at least one most probable traffic situation out of a set of predetermined traffic situations includes reducing a probability value relating to traffic situations comprising an object as it has been detected by the vision processor if the detected object lacks a corresponding shadow although such shadow would be expected due to existing lighting conditions, including when all other objects detected by the vision processor in vicinity of the detected object cast a respective shadow.

21. The computing device of claim 15, wherein the set of predetermined traffic situations includes traffic situations comprising a traffic participant with active vehicle lighting, in particular wherein the active vehicle lighting comprises at least one out of brake lights, reversing lights, direction indicator lights, hazard warning lights, low beams and high beams, emitting the artificial lighting to be detected in the image frames.

22. The computing device of claim 15, wherein the set of predetermined traffic situations includes traffic situations for anticipating an out-of-sight traffic participant, the traffic situations comprising the out-of-sight traffic participant with active vehicle lighting, the active vehicle lighting being detectable in the field-of-view of the image sensor and illuminating one or more of objects, surfaces, or airborne particles in the field-of-view of the image sensor.

23. The computing device of claim 15, wherein the set of predetermined traffic situations includes traffic situations for anticipating traffic participants suddenly moving into a path of the host vehicle, the traffic situations respectively comprising the traffic participant with active vehicle lighting in the field-of-view of the image sensor, the active vehicle lighting being of a defined type.

24. A method, comprising: acquiring, with an image sensor, an image data stream comprising a plurality of image frames; analyzing, with a vision processor, the image data stream to detect objects, shadows and/or lighting in the image frames, wherein analyzing the image data stream comprises detecting artificial lighting in the image frames, the artificial lighting being dynamic lighting that includes at least one of moving or changing size in relation to surfaces or objects being illuminated; recognizing, with a situation recognition engine, at least one most probable traffic situation out of a set of predetermined traffic situations taking into account the detected objects, shadows and/or lighting; and controlling the vehicle taking into account the at least one most probable traffic situation.

25. The method of claim 24, further comprising analyzing the image data stream by detecting shadows in the image frames.

26. The method of claim 25, further comprising analyzing the image data stream by detecting dynamic shadows, wherein movements and/or changes in size of the dynamic shadows are detected with reference to the surfaces the respective shadows are cast on.

27. The method of claim 25, wherein the set of predetermined traffic situations include traffic situations for anticipating an out-of-sight traffic participant, the traffic situations including the out-of-sight traffic participant casting a shadow into the field-of-view of the image sensor, wherein the shadow is a moving shadow.

28. The method of claim 25, wherein the set of predetermined traffic situations include traffic situations comprising a row of parking slots, wherein a plurality, but not all, of the parking slots are occupied by respective cars, the cars in the occupied parking slots respectively casting a shadow into the field-of-view of the image sensor, an unoccupied parking slot being identifiable by lack of a corresponding shadow even if the unoccupied parking slot itself is still out-of-sight.

29. The method of claim 24, wherein recognizing at least one most probable traffic situation out of a set of predetermined traffic situations includes reducing a probability value relating to traffic situations comprising an object as it has been detected by the vision processor if the detected object lacks a corresponding shadow although such shadow would be expected due to existing lighting conditions, including when all other objects detected by the vision processor in vicinity of the detected object cast a respective shadow.

30. The method of claim 24, wherein the set of predetermined traffic situations includes traffic situations comprising a traffic participant with active vehicle lighting, in particular wherein the active vehicle lighting comprises at least one out of brake lights, reversing lights, direction indicator lights, hazard warning lights, low beams and high beams, emitting the artificial lighting to be detected in the image frames.

31. The method of claim 24, wherein the set of predetermined traffic situations includes traffic situations for anticipating an out-of-sight traffic participant, the traffic situations comprising the out-of-sight traffic participant with active vehicle lighting, the active vehicle lighting being detectable in the field-of-view of the image sensor and illuminating one or more of objects, surfaces, or airborne particles in the field-of-view of the image sensor.

32. The method of claim 24, wherein the set of predetermined traffic situations includes traffic situations for anticipating traffic participants suddenly moving into a path of the host vehicle, the traffic situations respectively comprising the traffic participant with active vehicle lighting in the field-of-view of the image sensor, the active vehicle lighting being of a defined type.

Description

BRIEF SUMMARY OF THE DRAWINGS

[0040] The disclosure will now be described in more detail with reference to the appended figures. In the figures:

[0041] FIG. 1 shows a simplified diagram of an exemplary method.

[0042] FIG. 2 shows a schematic example of an application of the exemplary method.

[0043] FIG. 3 shows a second schematic example of an application of the exemplary method.

[0044] FIG. 4 shows a third schematic example of an application of the exemplary.

[0045] FIG. 5 shows a fourth schematic example of an application of the exemplary method.

[0046] FIG. 6 shows a fifth example of an application of the exemplary method.

DESCRIPTION

[0047] Turning to FIG. 1, which shows a simplified diagram of a method for enhancing situational awareness of an advanced driver assistance system in a host vehicle.

[0048] The method comprises the following steps:

[0049] S1: Acquiring, with an image sensor, an image data stream comprising a plurality of image frames.

[0050] S2: Analyzing, with a vision processor, the image data stream to detect objects 10, shadows 20 and/or lighting 30 in the image frames.

[0051] S3: Recognizing, with a situation recognition engine, at least one most probable traffic situation out of a set of predetermined traffic situations taking into account the detected objects 10, shadows 20 and/or lighting 30.

[0052] S4: Controlling, with a processor, the host vehicle 1 taking into account the at least one most probable traffic situation.

[0053] The method may further comprise:

[0054] S5: Reducing a probability value relating to traffic situations comprising an object 17 as it has been detected by the vision processor if the detected object 17 lacks a corresponding shadow although such shadow would be expected due to existing lighting conditions. In one example, the probability value is only reduced if all other objects 10 detected by the vision processor in vicinity of the detected object 17 cast a respective shadow 20.

[0055] The method enhances and/or facilitates the situational awareness of a highly automated driver assistance system, such as in automated valet parking, by using optical detections of shadows 20 and/or light beams 30, which are cast by other traffic participants 10.

[0056] As shadows 20 and/or light beams 30 usually reach further than their source, they can be used to already anticipate the presence of other traffic participants 11 before they are physically in the field of view of the host vehicle's 1 sensors.

[0057] Additionally, a shadow analysis can be used to evaluate the presence or absence of a possible ghost object. A ghost object 17 is a wrongly detected, fake object.

[0058] The method is further explained with regard to the respective examples of application as depicted in the following FIGS. 2 to 6.

[0059] Advantageously, analyzing the image data stream comprises detecting shadows 20 in the image frame, as they are shown in FIGS. 2 to 4.

[0060] Referring now to any of the FIGS. 2 to 4, 3D objects 10 will usually cast a shadow 20 in one direction or another. If contrast within the image data stream is high enough, these shadows can be detected by a camera system.

[0061] Depending on the angle of the light source to the object 10,11 that casts the shadow 20, the shadow 20 might be larger than the object 10,11 that casts it.

[0062] The shadows 21 of hidden traffic participants 11 can be used to anticipate their presence, although they themselves are not in the field of view 40 of the host vehicle already.

[0063] In FIG. 2, from the host vehicle's perspective, the pedestrian is not in the field of view 40 yet and therefore cannot directly be detected. The shadow 21 the pedestrian is casting, however, is already visible to the camera of the host vehicle 1 comprising the image sensor.

[0064] If the shadow is correctly identified, it can be used to anticipate that a pedestrian or other traffic participant 10 is about to cross the road in front of the host vehicle 1 in the near future. The host vehicle 1 can use this additional information to adapt its driving behaviour by a) slowing down or stopping if moving, or b) waiting until the other traffic participant has crossed the host vehicle's planned trajectory before launching.

[0065] Furthermore, as shadows can be rather unshapely and therefore their origin is sometimes hard to identify, a variant is to consider dynamic shadows 22 only. Again referring to FIG. 2, as the pedestrian 12 is moving, the shape of the shadow 22 in the host vehicle's field-of-view 40 will change accordingly. In this case, there is no need to identify the shadow's 22 origin, as the fact that the shadow 22 is dynamic already indicates that it originates from a moving traffic participant 12 rather than from infrastructure or other static obstacles.

[0066] Especially in complex environments such as parking lots with many obstructing objects, this can be helpful to better understand the traffic situation and react in time to other traffic participants 10.

[0067] In addition, dynamic shadows 22 are usually simpler to detect if movements and/or changes in size of the dynamic shadows 22 are referenced to the surfaces on which the respective shadows 22 are cast on.

[0068] For enabling the advanced driver assistance system according to aspects of the disclosure, the set of predetermined traffic situations used for recognizing the most probable traffic situation include traffic situations comprising out-of-sight traffic participants 11 casting a shadow 21 into the field-of-view 40 of the image sensor. Preferably, the set of predetermined traffic situations also include traffic situations having the shadow 21 being a moving shadow 22.

[0069] Turning to FIG. 3, the set of predetermined traffic situations also includes traffic situations comprising a row 15 of parking slots, wherein a plurality but not all of the parking slots are occupied by respective cars, the cars in the occupied parking slots respectively casting a shadow 20 into the field-of-view 40 of the image sensor. An unoccupied parking slot 16 can thus be identified by lack of a corresponding shadow even if the unoccupied parking slot 16 itself is still out-of-sight.

[0070] In other words, the general shadow-based approach of detecting other traffic participants 10 before they are in the host vehicle's field-of-view as described before with regard to FIG. 2, can also be used to identify empty parking slots 16 when searching in a parking lot. The contrast between a row 15 of vehicles 10 casting a shadow 20 vs. the gap without a shadow in-between could be used to identify an empty slot 16 when driving down an aisle even before reaching the particular slot.

[0071] In FIG. 4, a traffic situation comprising a ghost object 17 is shown. As previously stated, ghost objects are non-real objects, which falsely are detected by the respective algorithm.

[0072] Shadows, or rather the absence of shadows when they would be expected, can also be used to identify if an object that was detected by the advanced driver assistance system is a false positive, i.e., a ghost object 17. In the depicted traffic situation, the pedestrian that is entering the field-of-view 40 of the host vehicle 1 is casting a shadow 20 similar to all vehicles that are parked along the side. The street painting on the pavement directly in front of the host vehicle 1, however, does not cast any shadow as it is a mere 2D image painted on the road to raise driver's awareness of pedestrians. To increase confidence that the person drawn on the pavement is not a real person laying on the ground, the absence of a shadow could be used. Of course, this method is only applicable in case the illumination conditions support that 3D objects cast shadows.

[0073] Turning to FIGS. 5 and 6, analyzing the image data stream advantageously comprises detecting artificial lighting 30 in the image frame.

[0074] The set of predetermined traffic situations for recognizing the most probable traffic situation includes traffic situations comprising a traffic participant 13 with active vehicle lighting 30. The active vehicle lighting 30, for example, can comprise brake lights 33, reversing lights 34, direction indicator lights, hazard warning lights, low beams 35 and/or high beams, respectively emitting the artificial lighting 30 to be detected in the image frames.

[0075] As shown in FIG. 5, the contrast between light and shadow can be used once ambient illumination is low and other traffic participants (e.g., vehicles, motorcycles etc.) are driving with low beams 35 or high beams on.

[0076] In this case, the light beam 31 that the other traffic participant 13 is producing reaches far ahead of the traffic participant 13 itself. Especially in cases in which the other traffic participant 13 is not in the field-of-view of the host vehicle 1 yet, the low beams 35 can be detected early.

[0077] In the traffic situation depicted, the host vehicle 1 is approaching an intersection. The field of view 40 of the host vehicle 1 is empty but the low beams 35 of the other vehicle 13 are already visible within the field-of-view of the host vehicle 1, thus, indicating the presence of the other traffic participant 13.

[0078] A corresponding traffic situation of a predetermined set of traffic situations for anticipating an out-of-sight traffic participant 11 comprises the out-of-sight traffic participant 11 with active vehicle lighting 30. The active vehicle lighting 30 is detectable in the field-of-view 40 of the image sensor.

[0079] The active vehicle lighting 30 may illuminate objects (e.g., other cars), surfaces (e.g. the street), and/or airborne particles (e.g. fog, rain or snow) in the field-of-view 40 of the image sensor.

[0080] The artificial lighting 30 to be detected can be dynamic lighting 32, which is moving and/or changing size in relation to the surfaces and/or objects being illuminated. A misinterpretation of other non-relevant light sources that are part of the infrastructure can thereby be mitigated. In order to determine if the light beam is static or dynamic the illumination of subsequent captured images can be compared to each other. In case of a dynamic beam, the illuminated area of the image would move gradually along a trajectory in a specific direction.

[0081] Turning to FIG. 6, the set of predetermined traffic situations includes traffic situations for anticipating traffic participants, which may suddenly be moving into a path of the host vehicle 1. The respective traffic situations comprise a traffic participant 13 with active vehicle lighting 30 in the field-of-view of the image sensor. According to an aspect, the active vehicle lighting 30 is of a defined type, in this particular case the active vehicle lighting 30 comprises brake lights 33 and reversing lights 34.

[0082] In other words, the potential to anticipate the future behaviour of other traffic participants 13 that are already in the field-of-view can be enhanced by analyzing the traffic participants' 13 lighting. If the host vehicle 1 is driving down the aisle of a parking lot, it can sense, if any of the vehicles has turned on its vehicle lighting via camera. A vehicle with turned on vehicle lighting could then be assumed to be about to move, although being static at the moment.

[0083] This information may help avoid collisions, as the host vehicle 1 slows down or stops in case the driver of the other vehicle does not see it. Additionally, in crowded parking lots, it could be a strategy to wait until the other vehicle has parked out to use the empty slot.

[0084] The description of embodiments of the invention is not intended to limit the scope of protection to these embodiments. The scope of protection is defined in the following claims.