METHOD FOR MODELING THE SURROUNDINGS OF AN AUTOMATED VEHICLE

Abstract

A method for modeling the surroundings of an automated vehicle in which environment information is continuously received from currently available information sources. Each information source provides pieces of environment information. A formal assumption and a formal guarantee is associated with each piece of environment information in such a way that it is guaranteed, if the formal assumption associated with the respective piece of environment information is fulfilled, that the piece of environment information fulfills the formal guarantee associated with it. Each information source provides the associated formal assumptions and formal guarantees for the pieces of environment information it supplies. A piece of environment information is used for calculating the world model at a given point in time only if the formal assumption associated with this piece of environment information is fulfilled at this point in time.

Claims

1-17. (canceled)

18. A method for modeling surroundings of an automated vehicle in which environment information is continuously received by a plurality of currently available vehicle-based and/or vehicle-external information sources, the method comprising the following steps: providing, by each information source of the information sources, one or more pieces of environment information, wherein a formal assumption and a formal guarantee are associated with every respective piece of the pieces of environment information in such a way that it is guaranteed, when the formal assumption associated with the respective piece of environment information is fulfilled, that the respective piece of environment information fulfills the formal guarantee associated with the respective piece of environment information, wherein each of the information sources provides the associated formal assumptions and formal guarantees for the pieces of environment information it supplies; and calculating at least one world model of the surroundings of the automated vehicle, using the received pieces of environment information and the associated assumptions and guarantees, the at least one world model being calculated in such a way that a piece of environment information is used for calculating the world model at a given point in time only when the formal assumption associated with the piece of environment information is fulfilled at the point in time.

19. The method as recited in claim 18, wherein a check as to whether the formal assumptions associated with the pieces of environment information are fulfilled is performed during runtime.

20. The method as recited in claim 18, wherein the information sources include vehicle-internal and/or vehicle-external environment sensors, the respective environment information being generated from measuring data acquired by a corresponding one of the environment sensors.

21. The method as recited in claim 18, wherein one or several pieces of environment information include: (i) weather information, and/or (ii) information about sizes and/or positions and/or speeds and/or movement directions of objects in the surroundings of the vehicle, and/or (iii) object classes, and/or (iv) object lists, and/or information about open spaces.

22. The method as recited in claim 18, wherein a formal guarantee of a piece of environment information serves as a formal assumption for at least one other piece of environment information, and the world model is calculated in such a way that as many formal assumptions as possible are fulfilled.

23. The method as recited in claim 18, wherein a number of the pieces of environment information that were not used due to unfulfilled formal assumptions is determined and from the determined number, a quality measure for the world model is determined.

24. The method as recited in claim 23, wherein the world model is used further by the automated vehicle only when a specific quality measure is reached.

25. The method as recited in claim 23, wherein environment information of additional information sources is requested when a specific quality measure is not reached, each additional information source having a set of formal assumptions and formal guarantees associated with it.

26. The method as recited in claim 18, wherein a future world model is calculated based on a current world model.

27. The method as recited in claim 18, wherein one or more safe trajectories are calculated for the automated vehicle using at least a current world model.

28. The method as recited in claim 27, wherein the one or more safe trajectories are also calculated using a future world model.

29. The method as recited in claim 18, wherein at least one piece of environment information is provided, whose formal assumption is always fulfilled or assumed to be fulfilled.

30. The method as recited in claim 18, wherein those of the formal assumptions that are rarely or never fulfilled are identified, and additional information sources are provided based on the identification.

31. The method as recited in claim 30, wherein the additional information sources include infrastructure sensors that supply pieces of environment information capable of fulfilling the identified formal assumptions.

32. The method as recited in claim 18, wherein one or more of the information sources are configured as a camera and/or radar sensor and/or rain sensor and/or lidar sensor and/or pressure sensor and/or GPS receiver and/or as a data service for environment data for weather data and/or traffic information and/or traffic control information.

33. A device, comprising: a fusion component configured to continuously receive environment information from a plurality of vehicle-based and/or vehicle-external information sources and, depending on the received environment information, to calculate at least one world model of an automated vehicle, the fusion component being configured to use a piece of environment information of an information source at a given point in time for calculating the world model only if formal assumptions associated with the piece of environment information are fulfilled at the point in time, wherein a check whether the formal assumptions associated with a piece of environment information are fulfilled is performed during runtime.

34. The device as recited in claim 33, further comprising: a planner component configured to calculate one or more safe trajectories for the automated vehicle, based on at least one world model calculated by the fusion component, and to provide the trajectory/trajectories to the automated vehicle.

35. A vehicle configured for automated driving, comprising: at least one environment sensor system; and a device including a fusion component configured to continuously receive environment information from a plurality of vehicle-based and/or vehicle-external information sources and, depending on the received environment information, to calculate at least one world model of an automated vehicle, the fusion component being configured to use a piece of environment information of an information source at a given point in time for calculating the world model only if formal assumptions associated with the piece of environment information are fulfilled at the point in time, wherein a check whether the formal assumptions associated with a piece of environment information are fulfilled is performed during runtime.

36. A non-transitory computer-readable medium on which is stored a computer program including program code for modeling surroundings of an automated vehicle in which environment information is continuously received by a plurality of currently available vehicle-based and/or vehicle-external information sources, the program code, when executed by a computer, causing the computer to perform the following steps: providing, by each information source of the information sources, one or more pieces of environment information, wherein a formal assumption and a formal guarantee are associated with every respective piece of the environment information in such a way that it is guaranteed, when the formal assumption associated with the respective piece of environment information is fulfilled, that the respective piece of environment information fulfills the formal guarantee associated with the respective piece of environment information, wherein each of the information sources provides the associated formal assumptions and formal guarantees for the pieces of environment information it supplies; and calculating at least one world model of the surroundings of the automated vehicle, using the received pieces of environment information and the associated assumptions and guarantees, the at least one world model being calculated in such a way that a piece of environment information is used for calculating the world model at a given point in time only when the formal assumption associated with the piece of environment information is fulfilled at the point in time.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Specific embodiments of the present invention are described in detail with reference to the figures.

[0046] FIG. 1 shows a traffic setting with an automated vehicle in accordance with an exemplary embodiment of the present invention.

[0047] FIG. 2 in a schematic manner shows a device in accordance with an exemplary embodiment of the present invention.

[0048] FIG. 3 shows a flow chart of a method in accordance with an exemplary embodiment of the present invention.

[0049] Specific embodiments of the present invention are described in detail with reference to the figures.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0050] In the following description of specific embodiments of the present invention, identical elements are labeled with the same reference sign. Where applicable, these elements are not described repeatedly. The figures only represent the subject matter of the present invention in schematic form.

[0051] FIG. 1 shows an automated vehicle 10 driving in the right line 24 of a two-lane roadway 20. The vehicle comprises an environment sensor system that is configured to acquire measuring data about the surroundings of the vehicle and to generate environment information from these data. For example, the sensor system 12 may comprise one or a plurality of cameras and/or one or a plurality of radar sensors and/or one or a plurality of lidar sensors. The sensors of the sensor system 12 may, for example, acquire objects in the surroundings of vehicle 10 and localize and classify them based on the acquired measuring data. In the depicted situation, a bicycle 30, a pedestrian 32, and a further vehicle 34 in lane 22 are within the surroundings of the vehicle, in addition to various stationary objects.

[0052] Furthermore, the infrastructure of the roadway 20 comprises a plurality of stationary, vehicle-external, infrastructure-integrated sensors 14, 16, which also acquire and provide environment information about the current surroundings of vehicle 10 in the given situation. For example, sensors 14, 16 may include one or a plurality of cameras and/or one or a plurality of radar sensors and/or one or a plurality of rain sensors. In this example, sensors 14, 16 are configured to wirelessly transmit the environment information acquired by them to vehicle 10 and include appropriate communication modules for this purpose. Vehicle 10 in this example may receive the environment information wirelessly transmitted by sensors 14, 16 via a corresponding receiver module. Together with the ascertained pieces of environment information, sensors 14, 16 also transmit formal assumptions and guarantees associated with the pieces of environment information, it being guaranteed that, if the formal assumption associated with a piece of environment information is fulfilled, this piece of environment information fulfills the formal guarantee associated with it. The pieces of environment information generated by the vehicle-internal sensors of sensor system 12 also comprise such formal assumptions and guarantees.

[0053] Based on the received environment information and the associated assumptions and guarantees, it is now possible to calculate at least one world model of the surroundings of automated vehicle 10, for example by using an ASP solver implemented in a processing unit or a fusion component of vehicle 10, in such a way that a piece of environment information is only used for calculating the world model at a given point in time if the formal assumption associated with this piece of environment information is fulfilled at this point in time.

[0054] The world model generated in this manner in particular comprises all safety-relevant objects or open spaces in the surroundings of vehicle 10 so that, for example, a planner component of vehicle 10 is able to calculate one or more safe trajectories 40 for the automated vehicle 10 based on at least one world model calculated by the fusion component and to provide the trajectory/trajectories to the automated vehicle 10.

[0055] FIG. 2 in a schematic manner shows a device 50 for modeling the surroundings of an automated vehicle according to an exemplary embodiment of the present invention. Device 50 comprises a fusion component 54. The fusion component 54 continuously receives environment information from, in this example, three vehicle-based and/or vehicle-external information sources 1, 2, and 3. In this example, the first information source 1 is a vehicle-based camera. It is able to acquire, localize and classify objects in the vehicle environment. In addition, the camera is able to detect fog in the environment of the vehicle. The second information source 2 in the present example is configured as a vehicle-based radar sensor. It is able to acquire and localize objects in the vehicle environment. The third information source 3 in the present example is configured as a stationary rain sensor located outside of the vehicle. It is able to provide environment information to specify whether or not it is presently raining at its location. This environment information may, for example, be provided dynamically via car-to-infrastructure communication.

[0056] According to the present invention, a formal assumption and a formal guarantee is associated with each piece of environment information. Thus, each information source 1, 2, 3 transmits a set 61, 62, 63 of pieces of environment information and associated assumptions and guarantees, so-called contracts.

[0057] The table below lists examples of possible assumptions and guarantees of information sources 1, 2, 3 and of the pieces of environment information provided by information sources 1, 2, 3:

TABLE-US-00001 TABLE 1 Sensor 1: Camera (vehicle) Assumption S1-A1: true Guarantee S1 A1-G1: fog or no fog Assumption S1-A2: no other Guarantee S1 A2-G2: object + object in the vicinity of the object class at a specific detected object coordinate (x, y, z) Assumption S1-A3: no reflecting Guarantee S1 A3-G3: object + surface in the vicinity of the object class at a specific acquired object coordinate (x, y, z) Sensor 2: Radar (vehicle) Assumption S2-A1: no rain Guarantee S2 A1-G1: object at a specific coordinate (x, y, z) Assumption S2-A2: no rain & no Guarantee S2 A2-G2: fog object(s) at a specific coordinate (x, y, z) Sensor 3: Rain (weather transmitter) Assumption S3-A1: true Guarantee S3 A1-G1: rain or no rain

[0058] Depending on the received pieces of environment information and the assumptions and guarantees, fusion component 54 calculates at least one world model 51 of the automated vehicle during runtime, a piece of environment information of an information source 1, 2, 3 being used to calculate the world model at a given point in time only if the formal assumptions associated with this piece of environment information are fulfilled at this point in time. For this purpose, the attempt is made to fulfill as many of the present contracts as possible. This results in a cascade of dependencies, and it is possible, for example, to identify those pieces of environment information that cannot be included in the world model because their assumptions cannot be fulfilled.

[0059] In the above example it is noteworthy that the environment information as to whether fog is present or not, which is provided by the vehicle's own camera as information source 1, does not need to fulfill an assumption, i.e. the formal assumption S1-A1 is always “true”. As a result, it is guaranteed (guarantee S1 A1-G1) that a piece of environment information indicating whether fog is present or not is available in any case. Likewise, the environment information as to whether it is raining or not, which is provided by the external rain sensor as information source 3, does not need to fulfill an assumption, i.e. the formal assumption S3-A1 is always “true”. As a result, it is guaranteed (guarantee S3 A1-G1) that environment information indicating whether it is raining or not is available in any case. In contrast, the camera is only able to execute a reliable object classification and localization if no other object is acquired in the vicinity of a detected object (guarantee S1 A2-G2) and/or if no reflecting surface was detected in the vicinity of the acquired object (guarantee S1 A3-G3). The radar sensor is able to reliably acquire an object at the coordinate (x1,y1,z1) (guarantee S2 A2-G1) if the assumption that it is not raining is fulfilled. To reliably acquire one or a plurality of objects at the coordinate (x2,y2,z2) (guarantee S2 A2-G2), the assumption that it is not raining and that no fog is present must be fulfilled.

[0060] The accuracy/robustness of a measurement is in this connection highly dependent on the position of the objects relative to the sensor. This may mean for objects at a larger distance that stricter assumptions must be fulfilled. In this example, the position x2,y2,z2 is further removed from the radar sensor. This results in the additional assumption that no fog must be present to ensure that signals are not attenuated by fog, which would deteriorate the measuring accuracy to such an extent that individual objects are impossible to distinguish.

[0061] Different assumptions for objects at various distances may also apply to optical sensors, such as e.g. the camera (sensor 1). Different object classes may also have different assumptions, as some object classes are more difficult to detect than others for the corresponding image processing algorithms. To output such object classes as guarantees, it follows that stricter assumptions, e.g. regarding light conditions or other adjacent objects, must be fulfilled for reliable classification.

[0062] For example, the following pieces of environment information may be available at an exemplarily chosen point in time:

TABLE-US-00002 TABLE 2 Sensor 1: Camera (vehicle) Assumption S1-A1: true Guarantee S1 A1-G1: no fog Assumption S1-A2: no other Guarantee S1 A2-G2: object in the vicinity of bicyclist at (x1, y1, z1) the detected object Assumption S1-A3: no Guarantee S1 A3-G3: reflecting surface in the pedestrian at (x2, y2, z2) vicinity of the acquired object Sensor 2: Radar (vehicle) Assumption S2-A1: no rain Guarantee S2 A1-G1: single object at (x1, y1, z1) Assumption S2-A2: no rain & Guarantee S2 A2-G2: no fog object(s) at (x2, y2, z2) Sensor 3: Rain (weather transmitter) Assumption S3-A1: true Guarantee S3 A1-G1: no rain

[0063] For example, it may turn out that it is not possible to fulfill the contract for the camera having the assumption S1-A3: no reflecting surface in the vicinity of the acquired object, for the guarantee of environment information S1 A3-G3: pedestrian at (x2,y2,z2). As a result, this piece of environment information 52 is not considered when calculating world model 51. All other pieces of environment information allow for generating a consistent world model 51. Although it comprises an object at the coordinate x2,y2,z2, this object is not classified.

[0064] The contracts 61, 62, 63 of all pieces of environment information are now represented as clauses in a logic program by fusion component 54. A stable model indicating the currently existing guarantees may thus be derived with the aid of established methods such as ASP. Since these guarantees comprise the pieces of environment information, the stable model also represents a trustworthy world model. The guarantees of the individual sensors 1, 2, 3 are only considered valid for the world model if the corresponding assumptions can be fulfilled. In this example, this means that A3-G3 of sensor 1 (camera) cannot be used, and thus the pieces of information in this guarantee cannot be used because the assumption cannot be fulfilled. The other contract may be resolved by fusion block 54 and may contribute to world model 51. The resulting world model thus comprises the bicyclist detected at (x1,y1,z1) and an object at (x2,y2,z2). Since the guarantee S1 A3-G3 of camera 1 is not fulfilled, it is not possible to classify the object as a pedestrian.

[0065] The unfulfilled assumptions may be used as a performance indicator for an operative area. For example, if certain assumptions are frequently impossible to fulfill in certain situations (e.g., at a location, under weather conditions etc.), the analysis of these unfulfilled assumptions may be used during operation to assess and evaluate the entire distributed perception system. To improve the robustness of the distributed perception system in these situations, additional sensors may for example be installed to provide environment information that makes it possible to fulfill these assumptions for future events.

[0066] Device 50 additionally comprises a planner component 56 that is configured to calculate and transmit to the automated vehicle one or more safe trajectories for the automated vehicle based on at least one world model 51 calculated by fusion component 54.

[0067] FIG. 3 shows in a schematic manner a sequence of a method for modeling the surroundings of an automated vehicle according to one specific embodiment of the present invention.

[0068] In a first step 102, pieces of environment information are received from a plurality of currently available vehicle-based and/or vehicle-external information sources, each information source providing one or more pieces of environment information. A formal assumption and a formal guarantee are associated with every piece of environment information in such a way that it is guaranteed, if the formal assumption associated with the respective environment information is fulfilled, that the piece of environment information fulfills the formal guarantee associated with it. The information about the formal assumptions and guarantees (contracts) are also received by the vehicle-based and/or vehicle-external information sources.

[0069] In the subsequent step 104, at least one world model of the surroundings of the automated vehicle is calculated with the aid of the received pieces of environment information and the associated assumptions and guarantees. This occurs in such a way that a piece of environment information is used for calculating the world model at a given point in time only if the formal assumption associated with this environment information fulfilled at this point in time.

[0070] In step 106, the world model is checked to verify that it satisfies the safety requirements of the automated vehicle, i.e., whether the world model enables planning a safe trajectory for the vehicle. If this is not the case, pieces of environment information of additional information sources, for example further environment sensors outside of the vehicle, may be requested in step 108, including the associated assumptions and guarantees. Based on this additional environment information and the associated assumptions and guarantees, another attempt may be made in step 104 to calculate a stable world model.

[0071] Alternatively or in addition, certain unfulfilled assumptions may be gradually set to “true” in step 110 and another attempt may be made in step 104 to calculate a stable world model under these modified contract conditions. In particular, a plurality of stable world models may be derived for these modified assumptions. In view of the available pieces of information, these stable world models represent different possibilities of the real world. For a guaranteed safe behavior in the real world, the planner component must then find a trajectory that is safe in all of these stable world models.

[0072] If the check in step 106 reveals that the world model fulfills the requirements for the safety of an automated vehicle, one or several trajectories for the automated vehicle are generated or adapted in step 112 and made available to the automated vehicle.

[0073] The present invention thus describes the application of the concept of contract-based design to the application case of distributed perception. The present invention makes it possible to determine a stable world model during runtime even if the availability of information sources such as environment sensors changes dynamically during runtime and the inference to the world model must be drawn continuously.