METHOD, SYSTEM AND COMPUTER PROGRAM PRODUCT FOR CALIBRATING AND VALIDATING A DRIVER ASSISTANCE SYSTEM (ADAS) AND/OR AN AUTOMATED DRIVING SYSTEM (ADS)

Abstract

A method calibrates and validates a driver assistance system (ADAS) and/or an automated driving system (ADS) for a driving task in at least one scenario. The scenario represents a traffic event in a time sequence and is defined by selected parameters and associated parameter values. The method includes: creating first test cases by selecting scenarios, scenario parameters and calibration parameters using a test strategy for the driving task. The method proceeds by performing a simulation to determine simulation results; evaluating of the simulation results; adapting the test strategy to the evaluation results; creating second test cases using the adapted test strategy; starting a new simulation cycle; repeating the adaptation of the test strategy if an evaluation criterion is not met; or passing on the test cases of the last simulation cycle to an output module; outputting results of the test cases from the output module for calibration and validation.

Claims

1. A method for calibrating and validating at least one of a driver assistance system (ADAS), an automated driving system (ADS) and a driving function for a set driving task in at least one scenario (SZ.sub.i), wherein a scenario (SZ.sub.i) represents a traffic event in a temporal sequence and is defined by a selection of parameters (P.sub.1, P.sub.2, . . . , P.sub.n) and associated parameter values (PV.sub.1, PV.sub.2, . . . , PV.sub.n), and wherein in a parameterized scenario (SZp.sub.i) the parameters (P.sub.1, P.sub.2, . . . , P.sub.n) and associated parameter values (PV.sub.1, PV.sub.2, . . . , PV.sub.n) are freely selectable, and for a specific scenario (SZc.sub.i) the scenario parameters (Pc.sub.1, Pc.sub.2, . . . , P.sub.cn) and associated scenario parameter values (PVc.sub.1, PVc.sub.2, . . . , PVc.sub.n) are set, the method comprising: using a test agent (220) of a test agent module (220) for creating (S10) first test cases (T.sub.i) by selecting parameterized scenarios (SZp.sub.i), scenario parameters (Pc.sub.i) and calibration parameters (Pcal.sub.i) using a test strategy (230) for the driving task; passing on (S20) the first test cases (T.sub.i) to a simulation module (400); performing (S30) a simulation by the simulation module (400) to determine simulation results (450); passing on (S40) the simulation results (450) to an evaluation module (500); performing (S50) an evaluation of the simulation results (450) by the evaluation module (500) for determining evaluation results (550); adapting (S60) the test strategy (230) to the simulation results (450) and the evaluation results (550); using the test agent (220) for creating (S70) second test cases (T.sub.k) using the adapted test strategy (230); starting (S80) a new simulation cycle for the second test cases (T.sub.k); repeating (S90) the adaptation of the test strategy (230) for performing a further simulation cycle if a particular evaluation criterion is not met; or passing on (S100) the test cases (T.sub.k) of the last simulation cycle to an output module (700) if a particular evaluation criterion is met; generating and outputting (S110) output results (750) from the test cases (T.sub.k) for calibration and validation purposes by the output module (200), the output results including the calibration parameters (Pcal.sub.i) for at least one of the driver assistance system (ADAS), the automated driving system (ADS) and the driving function for performing the set driving task.

2. The method of claim 1, wherein the simulation module (400) comprises interchangeable sub-modules (410, 420, 430) that include a first sub-module (410) configured as an environmental model module, a second sub-module (420) configured as a driver model module, and a third sub-module (430) configured as a vehicle model module.

3. The method of claim 2, wherein the simulation module (400) and the sub-modules (410, 420, 430) are connected to at least one of sensors (470) and a database (450) to obtain further information for creating simulation models, wherein the simulation models are passed on to a driving function module (440) to perform the simulation of a driving assistance function.

4. The method of claim 1, wherein the assessment module (500) comprises a driving function evaluation module (520) for determining performance and safety of a driving function using performance indicators (KPIs) and a simulation evaluation module (520) for determining the quality of the simulation using simulation quality criteria (SQCs), and wherein the evaluation results (550) comprise the performance indicators (KPIs) and the simulation quality criteria (SC).

5. The method of claim 1, wherein the test cases (T.sub.i) are stored in a test database (300), the calibration parameters (Pcal.sub.i) are stored in a calibration parameter database (320), the parameterized scenarios (SZp.sub.i) and the scenario parameters (Pc.sub.i) are stored in a scenario database (330), and the evaluation results (550) are stored in an evaluation database (340).

6. The method of claim 1, wherein the test strategy (230) and the test agent (220) use at least one software application having calculation methods and algorithms of artificial intelligence.

7. The method of claim 6, wherein the algorithms and calculation methods are configured as at least one of mean values, minimum and maximum values, lookup tables, expected value models, linear regression methods, Gaussian processes, fast Fourier transforms, integral and differential calculations, Markov methods, probability methods, such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, and recurrent neural networks.

8. The method of claim 1, wherein the parameter (P.sub.i) comprises at least one of a physical variable, a chemical variable, a torque, a speed, a voltage, a current strength, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a movable object, a stationary object, a road configuration, a road sign, a traffic volume, a topographical structure, a time, a temperature, a precipitation value, a weather condition and a time of year.

9. The method of claim 1, wherein the sensors (470) are configured as at least one of a radar system, LI DAR optical distance and speed measurement system, image recording 2D/3D cameras, GPS systems, accelerometers, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, precipitation sensors, and temperature sensors.

10. A system (100) for calibrating and validating at least one of a driver assistance system (ADAS), an automated driving system (ADS) and a driving function for a set driving task in at least one scenario (SZ.sub.i) representing a traffic event in a temporal sequence and is defined by a selection of parameters (P.sub.1, P.sub.2, . . . , P.sub.n) and associated parameter values (PV.sub.1, PV.sub.2, . . . , PV.sub.n), and wherein, in a parameterized scenario (SZp.sub.i), the parameters (P.sub.1, P.sub.2, . . . , P.sub.n) and associated parameter values (PV.sub.1, PV.sub.2, . . . , PV.sub.n) are freely selectable, and in a specific scenario (SZc.sub.i) the parameters (Pc.sub.1, Pc.sub.2, . . . , P.sub.cn) and associated parameter values (PVc.sub.1, PV.sub.c2, . . . , PVc.sub.n) are defined, the system comprising: a test module (200) having a test agent (220) and a test strategy (230), a simulation module (400), an evaluation module (500) and an output module (700), wherein: the test agent (220) is configured to generate first test cases (T.sub.i) by selecting parameterized scenarios (SZp.sub.i), scenario parameters (Pc.sub.i) and calibration parameters (Pcal.sub.i) using the test strategy (230) for the driving task and passing on the selected first test cases (T.sub.i) to the simulation module (400); the simulation module (400) is configured to perform a simulation to determine simulation results (450) and to pass the simulation results (450) on to the evaluation module (500); the evaluation module (50) is configured to perform an evaluation of the simulation results (450) to determine evaluation results (550); the test module (200) is configured to adapt the test strategy (230) to the simulation results (450) and the evaluation results (550), to create second test cases (T.sub.k) from the test agent (220) using the adapted test strategy (230) and begin a new simulation cycle for the second test cases (T.sub.k), and to repeat the adaptation of the test strategy (230) for performing a further simulation cycle, if a particular evaluation criterion is not met, or pass the test cases (T.sub.k) of the at least last simulation cycle on to the output module (700), if a particular evaluation criterion is met; and the output module (700) is configured to generate and output results (750) outputted from the test cases (T.sub.k) of the at least last simulation cycle for calibration and validation purposes in the form of calibration parameters (Pcal.sub.i) for at least one of the driver assistance system (ADAS), the automated driving system (ADS) and the driving function for performing the determined driving task.

11. The system (100) of claim 10, wherein the simulation module (400) comprises interchangeable sub-modules (410, 420, 430) that include: a first sub-module (410) configured as an environmental model module, a second sub-module (420) configured as a driver model module, and a third sub-module (430) configured as a vehicle model module.

12. The system (100) of claim 10, wherein at least one of the simulation module (400) and the sub-modules (410, 420, 430) are connected to sensors (470) and/or a database (450) to obtain further information for creating simulation models, wherein the simulation models are passed on to a driving function module (440) to perform the simulation of a driving assistance function.

13. The system (100) of claim 10, wherein the evaluation module (500) comprises a driving function evaluation module (520) for determining performance and safety of a driving function using performance indicators (KPIs) and a simulation evaluation module (520) for determining quality of the simulation using simulation quality criteria (SQCs), and wherein the evaluation results (550) comprise the performance indicators (KPIs) and the simulation quality criteria (SQCs).

14. The system (100) of claim 10, wherein the test cases (T.sub.i) are stored in a test database (300), the calibration parameters (Pcal.sub.i) are stored in a calibration parameter database (320), the parameterized scenarios (SZp.sub.i) and the scenario parameters (Pc.sub.i) are stored in a scenario database (330) and the evaluation results (550) are stored in an evaluation database (340); and wherein at least one of the textual strategy (230) and the test agent (220) uses at least one software application having calculation methods or algorithms of artificial intelligence.

15. A computer program product (900) comprising an executable program code (950) that is configured to execute the method of claim 1.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG. 1 is a block diagram to explain an embodiment of a system according to the invention.

[0043] FIG. 2 is a schematic representation of a graphic.

[0044] FIG. 3 is a flow diagram to explain the individual method steps of a method according to the invention.

[0045] FIG. 4 is a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

DETAILED DESCRIPTION

[0046] Additional features, aspects, and advantages of the invention or its embodiments will be explained in the following detailed description in conjunction with the claims.

[0047] Increasingly simulated traffic scenarios created by programming are used for testing, training, and securing driver assistance systems (ADAS) and automated driving systems (ADS). A scenario in the context of this description is defined as a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning off on a turn-off lane, passing through a tunnel, turning onto a roundabout, or stopping in front of a pedestrian crossing. Moreover, specific visibility conditions, for example due to twilight or bright sunlight, as well as environmental conditions such as the weather and the time of year, traffic levels, and certain geographic topographical conditions, can affect a scenario. For example, an overtaking maneuver can be described as a scenario in which a first vehicle is initially behind another vehicle, then makes a lane change to the other roadway, and increases speed to overtake the other vehicle. Such a scenario is also referred to as a cut-in scenario. In addition, the time of year and weather play a role, as with heavy rain and ice, the road conditions look different than on a sunny summer day.

[0048] To use driver assistance systems (ADAS) and automated driving systems (ADS) in a motor vehicle, they must be calibrated and validated for reliable use. The calibration adapts the functions to the vehicle type as well as to the desired behavior of a driving function without changing the software code. For this purpose, the calibration parameters are modified and made available to the ADAS/ADS in a dataset. It is necessary to find a suitable dataset of calibration parameters for optimum behavior in as many operational design domain (ODD) scenarios as possible.

[0049] The goal of the validation is to comprehensively test the dataset obtained during the calibration, to demonstrate the reliability and robustness of a driver assistance system (ADAS) and/or an automated driving system (ADS) throughout the ODD, and then to release it.

[0050] According to the present invention, a modular simulation approach is used for the virtual calibration and validation of ADAS/ADS, and can be adapted fora variety of configurations of ADAS/ADS.

[0051] FIG. 1 shows a system 100 according to the invention for simulated calibration and validation of a driver assistance system (ADAS) and/or an automated driving system (ADS). The system 100 according to the invention comprises a test agent module 200 having a test agent 220 and a test strategy 230, a calibration parameter database 320, a scenario database 330, an evaluation database 340, a test database 300, a simulation module 400, an evaluation module 500, and an output module 700, which can each be provided with a processor and/or a memory unit.

[0052] A “module” can be understood in connection with the invention to mean a processor and/or a processor unit and/or a memory unit for storing program instructions. The processor is specifically configured to execute the program instructions to implement or realize the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated into a cloud computing infrastructure.

[0053] A “processor” can be understood in connection with the invention to mean a machine or an electronic circuit. In particular, a processor can be a main processor (central processing unit (CPU)), a microprocessor, or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc. A processor can also be understood to mean a virtualized processor, a virtual machine, or a soft CPU. For example, it can also be a programmable processor equipped with configuration steps for carrying out the above-mentioned method according to the invention or configured with configuration steps in such a way that the programmable processor realizes the features according to the invention of the method, the system, the modules, or other aspects and/or partial aspects of the invention. In particular, the processor can include highly parallelized computing units and high-performance graphics modules.

[0054] In connection with the invention, a volatile memory in the form of a working memory (random access memory, RAM), or a permanent memory such as a hard drive or a data carrier or, for example, a replaceable memory module can be understood. However, the storage module can also be a cloud-based storage solution.

[0055] The test agent 220 of the test agent module 200 creates a plurality of test cases T.sub.i for one or more driving tasks using a software application. The particular driving task is formulated prior to starting the simulation, for example, by an expert such as an engineer. However, a list of driving tasks can be generated by a software application. This list can then be successively edited by the system 100 according to the invention. An exemplary driving task is a lane change on a highway.

[0056] The test strategy 230 selects and designs the test cases T.sub.i and specifies how the test agent 220 creates the test cases T.sub.i. Various computational methods and algorithms, in particular algorithms of artificial intelligence, can be provided for establishing the test strategy 230. Thus, algorithms and calculation methods such as mean values, minimum and maximum values, lookup tables, expected value models, linear regression methods, Gaussian processes, fast Fourier transforms, integral and differential calculations, Markov method, probability methods, such as Monte Carlo methods, temporal difference learning, but also extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, and/or recurrent neural networks, which allow the strategy to be adapted through iteration procedures.

[0057] The required information for the creation of the test cases T.sub.i is taken by the test agent 220 from the calibration parameter database 320, the scenario database 330 and the evaluation database 340. In the context of the invention, it can be provided that further databases also are used. The created test cases T.sub.i are stored in the test database 300.

[0058] “Database” means both a storage algorithm and the also hardware in the form of a storage unit. In particular, the databases 300, 320, 330, 340 can be configured as part of a cloud computing infrastructure.

[0059] Data in connection with the invention should be understood as raw data as well as data already prepared from measurement results from sensors as well as from other data sources.

[0060] The creation of the test cases T.sub.i is based on parameterized scenarios, which are also referred to as logical scenarios. A parameterized scenario is referred to in the context of the invention as a scenario that is written in a machine-readable code. A parameterized scenario SZp.sub.i is defined by various parameters P.sub.1, P.sub.2, . . . , P.sub.n from a quantity of possible parameters P.sub.i and associated parameter values PV.sub.1, PV.sub.2, . . . , PV.sub.n from a quantity of possible parameter values PV.sub.i. Parameter values PV.sub.i define the range of values of a parameter P.sub.i. For example, a parameter P.sub.i sets a physical variable, a chemical variable, a torque, a rotary speed, a voltage, a current strength, an acceleration, a speed a braking value, a direction, an angle, a radius, a location, a number, a movable object such as a motor vehicle, a person or a cyclist, a stationary object such as a building or a tree, a road configuration such as a highway, a road sign, a traffic light, a tunnel, a roundabout, a turn-off lane, a traffic volume, a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a time of year. Parameters P.sub.i thus denote characteristics and features of a scenario in the context of the invention. An example of a parameter P.sub.i is the speed of an ego vehicle, and a range of values of the associated parameter value PV.sub.i can include the range of 100 km/h to 180 km/h for a scenario SZp.sub.i. For another scenario SZp.sub.k, the range of values of the parameter value PV.sub.i can range from 40 km/h to 70 km/h.

[0061] A parametrized scenario SZp comprises a time sequence of time intervals Δt.sub.1, Δt.sub.2, . . . , Δt.sub.n, in each of which different scenes and events take place. A parametrized scenario SZp starts with a start scene and then evolves through events that occur, resulting in new subsequent scenes over time. The start scene is thus amended by one or more events. An event can be a response actively triggered by a road user, such as an acceleration, as well as an event that recurs cyclically, for example, the switching operations of a traffic light. The start scene and the individual subsequent scenes thus each comprise only a small time interval Δt and/or a snapshot, while a parameterized scenario comprises SZp a longer period of time. In a graph representation of a scenario SZp, the events can be represented as edges and the individual scenes can be represented as nodes of a graph, as illustrated in FIG. 2

[0062] In the context of the present invention, a distinction is made between a parameterized scenario SZp and a specific scenario SZc. In the context of this invention, a scenario SZp is defined in which both the parameters P.sub.i and the associated parameter values PV.sub.i are not all defined. A specific scenario SZc is defined as a scenario SZ in which the scenario parameters Pc.sub.i and the associated scenario parameter values PVc.sub.i and/or value ranges of the scenario parameter values PVc.sub.i are set. In both scenarios, both the parameterized scenario SZp and the specific scenario SZc, are scenarios that are written in particular in a machine-readable code.

[0063] Various sources can be used to create parameterized scenarios SZp.sub.i, such as requirement specifications, expert knowledge, and/or measurements in public traffic or on test sites using sensors. The sensors used can be configured as radar systems, LIDAR systems for optical distance and speed measurement, 2D/3D image recording cameras in the visible, IR and/or UV range, and/or GPS systems. Furthermore, accelerometers, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, precipitation sensors, and/or temperature sensors, etc. can be used. Thus, a software application can derive a corresponding parameterized scenario SZp.sub.i from recorded data at a particular geographic location. In particular, the software application uses artificial intelligence algorithms to identify the parameterized scenarios SZp.sub.i. The artificial intelligence algorithms can be encoders and decoders with neural networks.

[0064] A neural network consists of neurons arranged in multiple layers and interconnected to one another differently. A neuron is able to receive information at its input from outside or from another neuron, evaluate the information in a particular manner, and pass it on in altered form at the neuron output to another neuron, or output it as a final result. Hidden neurons are located between the input neurons and the output neurons. Depending on the type of network, multiple layers of hidden neurons can be present. They provide for the forwarding and processing of the information. Output neurons finally deliver a result and issue it to the outside world. Neural networks can be trained through unsupervised or supervised learning.

[0065] The different arrangement and linking of the neurons result in different types of neural networks, in particular a feedforward neural network (FNN), a recurrent neural network (RNN) or a convolutional neural network (CNN). A convolutional neural network has multiple fold layers and is very well suited to machine learning and uses in the field of pattern recognition and image recognition applications. Because a large proportion of the data captured by the sensors is present as images, particularly convolutional neural networks (CNN) are used.

[0066] In the context of the invention, the test agent 220 creates, from the set of possible parameterized scenarios SZp.sub.i, those test cases T.sub.i that are of interest for the calibration and validation of ADAS/ADS. For this purpose, the test agent 220 selects suitable parameterized scenarios SZp.sub.i and scenario parameters Pc.sub.i from the scenario database 330 as well as calibration parameters Pcal.sub.i from the calibration parameter database 320. These relevant test cases T.sub.i created by the test agent 220 cover the relevant traffic situations set out in the operational design domain (ODD) that must be managed by driving assistance systems and automated driving functions. A test case T.sub.i is thus a relevant specific scenario SZc defined through the specification of scenario parameters Pc.sub.i and further calibration parameters Pcal.sub.i.

[0067] The test strategy 230 of the test agent 220 decides on the configuration of the test cases T.sub.i by selecting suitable parameterized scenarios Szp.sub.i and scenario parameters Pc.sub.i as well as further calibration parameters Pcal.sub.i. The test strategy 230 used is specified by way of the purpose of the simulation, i.e., which knowledge about the behavior of an ADAS/ADS and/or a specific driving function is to be obtained by a simulation when the specified driving task is performed. For example, for the virtual validation of an ADAS/ADS with a constant dataset of calibration parameters Pcal.sub.i, different parameterized scenarios Szp.sub.i and scenario parameter combinations can be created using a combinatorial method. Furthermore, an iterative strategy can be implemented based on mathematical optimizers to adapt the calibration parameters Pcal.sub.i for a smaller set of different parameterized scenarios Szp.sub.i.

[0068] The test agent 220 stores the test cases T.sub.i determined in this manner in the test database 300. The data for a test case T.sub.i can comprise a test identification number (test ID), a user name, a creation time, and the designation for the selected parameterized scenario Szp.sub.i. Furthermore, the scenario parameters Pc.sub.i that transform the parameterized scenario Szp.sub.i into a specific scenario Szc.sub.i are saved. Furthermore, calibration parameters Pcal.sub.i and other data such as performance indicators (KPIs) can be stored regarding the evaluation of the performance of the respective driving assistance function in the simulation as well as evaluation indicators for the evaluation of the simulation performed. Furthermore, information regarding the status of the simulation such as “performed or not performed” can be associated with a test case T.sub.i. Additionally, further text messages, video sequences and/or audio sequences, etc. with further information can be stored. Overall, all the relevant information relating to a test case T.sub.i can thus be stored in the test database 300.

[0069] To perform a simulation of the test cases T.sub.i created, the test agent 220 passes a test case T.sub.i or multiple test cases T.sub.i on to the simulation module 400. In addition, the simulation module 400 can retrieve information regarding the calibration parameters Pcal.sub.i from the calibration parameter database 320, as well as information about the parameterized scenarios SZp, and scenario parameters Pc.sub.i, such as machine-readable scripts from the scenario database 330, to perform the simulation. The simulation module 400 comprises various sub-modules 410, 420, 430 that are configured to simulate specific characteristics of the test cases T.sub.i for the driving task being tested and that can interact with each other. They are configured as interchangeable sub-modules and have the task of simulating individual aspects of a respective test case T.sub.i or a plurality of test cases T.sub.i. A first sub-module 410 can be configured as an ambient model module that relates to the environment and maps various environments of a motor vehicle 10. A second sub-module 420 can be configured as a driver model module that relates to the driving mode and/or driving style of a driver, such as dynamic or defensive. A third sub-module 430 can be configured as a vehicle model module that comprises various models of a motor vehicle, for example, regarding the different configuration of the powertrain, drive dynamics, and other sub-functions. The sub-modules 410, 420, 430 implement the description of the respective test case T.sub.i for a specific situation and/or configuration and provide the necessary input data for simulating a driving assistance function in a driving function module 440.

[0070] The simulation module 400 and/or the various sub-modules 410, 420, 430 are connected to one or more databases 480 in which sensor signals from sensors 470 and other data are stored. The sensor signals are measurement data of characteristics and features of a motor vehicle 10 and of objects and events in the vicinity of the motor vehicle 10 that have been recorded during driving along a travel path during a defined time window Δti and/or a defined travel path portion Δxi. The objects in the traffic environment of the motor vehicle 10 can be other road users, such as motor vehicles, pedestrians, cyclists, and the events can comprise an acceleration operation, a lane change, or a change-over of a traffic light. The sensors 470 can be configured as radar systems, LIDAR systems for optical distance and speed measurement, 2D/3D image recording cameras in the visible, IR and/or UV range, and/or GPS systems. Furthermore, accelerometers, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, precipitation sensors, and/or temperature sensors, etc. can be provided.

[0071] Further historical data can also be stored in the database 480 in the form of images, graphics, time series, characteristic values, etc. For example, target variables and target values that define a safety standard for the simulation can be stored in the database 480. The database 480 can also be integrated into a cloud computing infrastructure.

[0072] Furthermore, additional data sources or databases can be used, such as databases that contain data regarding the road network with road specifications, such as lanes and bridges, road infrastructure such as the road surface, edge construction, road route, etc., and are provided by authorities.

[0073] Data on traffic figures, such as real hourly traffic volumes at a particular traffic location, are of interest for certain scenarios types, such as a traffic jam. Aerial photographs are another source of data, for example, from Google Maps. Mapillary can also be used for road images. Mapillary collects user-generated road images with geo-tags recorded by dashcams and smartphones. These images are available under an open source license.

[0074] Weather conditions can also define a scenario type, and hence weather data is another data source. Weather data comprises historical weather measurements and future weather forecasts.

[0075] The calculation and storage of the geographic coordinates of the detected objects is preferably carried out in the EPSG 25832-coordinate system (Universal Transverse Mercator (UTM) zone 32N). This system is used by authorities in Germany. The lateral and longitudinal positions are represented in meters. Furthermore, global reference systems such as the “world geodetic system 1984 (WGS 84)”, which is also used in GPS receivers (global positioning system), can be used. For example, the entire map content can be imported from Germany.

[0076] The test cases T.sub.i selected by the test agent are integrated into the various sub-modules 410, 420, 430. Input signals, such as sensor data, vehicle health information, and driver inputs from the sub-modules 410, 420, 430, are provided to perform a simulation of the selected test cases T.sub.i. For this purpose, the sub-modules 410, 420, 430 use simulation algorithms such as X-in-the-loop (XiL). In particular, the modeling of the sensors 470 is of particular significance because inaccuracies and measurement errors in the sensor signals from the sensors 470 must be taken into account in the virtual calibration and validation. If a selected test case T.sub.i and/or a plurality of test cases T.sub.i cannot cover a defined driving task, further data is provided via the sub-modules 410, 420, 430 to map and simulate the entirety of a driving event.

[0077] The simulation module 400 performs a simulation of the test cases T.sub.i for a set driving task and passes the simulation results 450 to the evaluation module 500.

[0078] The evaluation module 500 evaluates the simulation results 450 with regard to the performance and functional capability of one or more functions or the overall performance of a driver assistance system (ADAS) or an automated driving system (ADS), in particular in the form of performance indicators (KPIs). In addition, the quality of the simulation procedure carried out is evaluated, in particular in the form of simulation quality criteria (SQCs). From these evaluations, the evaluation module 500 produces evaluation results 550.

[0079] KPIs are used to describe the performance of an ADAS/ADS to be tested, and different KPIs are established for different evaluation categories, such as comfort, safety, naturalness of travel and efficiency. Additionally, further KPIs can be implemented to verify the correct functionality of the ADAS/ADS being tested. An example of a KPI is the evaluation of a minimum distance from another vehicle or a mean acceleration in a deceleration scenario.

[0080] In addition to the KPls, SQCs are used to evaluate the quality of the simulation. For example, other road users can be represented by modeled simulation representatives that make different decisions depending on the reactions of the ADAS/ADS being tested. For this reason, an evaluation must be conducted to verify whether the simulated scenario matches the scenario description of the test case T.sub.i. For this purpose, metrics are defined for the SQCs that provide feedback on the correct execution of a scenario simulation. Thus, the manner of performing an overtaking maneuver can be evaluated by means of an SQC, for example, with regard to whether a collision could be avoided. The KPIs and SQCs can be represented by numeric values and also by Boolean values.

[0081] Both direct and indirect evaluation metrics can be used for the KPIs and SQCs. Direct evaluation metrics contain data directly from simulated measurement data. Indirect evaluation metrics use scenario parameters as a data source. An example of the application of indirect evaluation metrics is the implementation of simulation models that quantify and evaluate a subjective perception of a driver during the performance of a driving assistance function of an ADAS/ADS. Based on the KPIs and SQCs, the results of a simulated test case T.sub.i can thus be represented by numerical values or Boolean values.

[0082] The evaluation results 550 in the form of performance indicators (KPIs) and simulation quality criteria (SQCs) are passed to the test agent module 200. The test agent 220 can provide the calibration parameters Pcal.sub.i in the calibration parameter database 320, the parameterized scenarios Szp.sub.i and the scenario parameters Pc.sub.i in the scenario database 330, and the evaluation results 550 in the evaluation database 340 with labels such as weights or for identifying particular characteristics to create new test cases T.sub.k for the set driving task. In particular, the test strategy 230 can be adapted using the data labeled by the evaluation results 550 in the calibration parameter database 320, the scenario database 330, and the evaluation database 340. The test cases T.sub.k are in turn deposited in the test database 300 and can be used for a new simulation cycle in the simulation module 400 and a subsequent evaluation in the evaluation module 500.

[0083] To adapt the test cases T.sub.k for a new simulation run, the calibration parameters, Pcal.sub.i, are changed while the scenario parameters remain the Pc.sub.i same. The calibration parameters Pcal.sub.i may be changed on the basis of the performance indicators (KPIs) created by the function evaluation module 510. By iterating the calibration parameters Pcal.sub.i for new test cases T.sub.k, an optimal set of calibration parameters Pcal.sub.i can be determined.

[0084] Furthermore, given constant calibration parameters Pcal.sub.i, the test agent 220 can vary the scenario parameters Pc.sub.i to iteratively determine the critical specific scenarios Szc.sub.i.

[0085] The combination of the simulation module 400 and the evaluation module 500 forms the basis for processing test cases T.sub.i in the context of the virtual calibration and validation. Test cases T.sub.i created by the test agent 220 are simulated in the simulation module 400. The simulation module has access to the calibration parameter database 320 and the scenario database 330 to obtain the required information. After successful simulation, the simulation results 450 are transferred to the evaluation module 500. The assessment module 500 has access to the evaluation database 340 and the scenario database 330 to deposit the evaluation results 550 of the simulation results 450 therein. Both the simulation module 400 and the evaluation module 500 contain interchangeable sub-modules, thereby enabling an adaptation to different functions and application cases, as well as separate development of the sub-modules.

[0086] After completion of a first simulation cycle with first test cases T.sub.i, second test cases T.sub.k are created by the test agent 220 using the test strategy 230 adapted to the simulation results 450 and the calculation results of the first simulation run. A new simulation run is begun with these second test cases T.sub.k. The evaluation results 550 are used to decide whether the second test cases T.sub.k satisfy a particular evaluation criterion or whether a further adaptation of the test strategy 230 is required to perform a further simulation cycle. If the evaluation results 550 reveal that the simulation result 450 converges, the test cases T.sub.k of the last simulation cycle are passed on to the output module 700.

[0087] On the basis of the evaluation results 550 generated in the form of KPIs and SQCs by the evaluation module 500, an optimal dataset of calibration parameters Pcal.sub.i for the driver assistance system (ADAS) and/or the automated driving system (ADS) and/or the driving function for performing the set driving task can then be determined for the test cases T.sub.k of the last simulation cycle. This dataset of calibration parameters Pcal.sub.i defines the behavior of the driver assistance system (ADAS) and/or the automated driving system (ADS) and/or the driving function for the set driving task and can then be output as output results 750 from the output module 700. In addition, the output results 750 contain information about the simulation performed. The quality of the output results 750 thereby depends on the quality of the scenario description and the type of simulation as well as the definition of the evaluation metrics.

[0088] The simulation module 400 comprises different sub-modules 410, 420, 430. Thus, a simple adaptation to different ADAS and ADS can be made by replacing sub-modules. Furthermore, the sub-modules 410, 420, 430 can be developed and validated independently, resulting in higher quality of the overall process. The simulation module 400 can therefore be adapted to different application cases by means of the interchangeable sub-modules 410, 420, 430. Different driving tasks can be implemented simply by changing the test strategy 230 of the test agent 220.

[0089] In FIG. 3, the method steps for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a set driving task are shown in at least one scenario SZ.sub.i.

[0090] In a step S10, first test cases are created T.sub.i by a test agent 220 by selecting parameterized scenarios SZp.sub.i and scenario parameters Pc.sub.i as well as calibration parameters Pcal.sub.i using a test strategy 230 for the driving task.

[0091] In a step S20, the selected first test cases T.sub.i are passed on to a simulation module 400.

[0092] In a step S30, a simulation is performed from the simulation module 400 to determine simulation results 450.

[0093] In a step S40, the simulation results 450 are passed on to an evaluation module 500.

[0094] In a step S50, an evaluation of the simulation results 450 is performed by the evaluation module 500 for determining evaluation results 550.

[0095] In a step S60, the test strategy 230 is adapted to the simulation results 450 and the evaluation results 550.

[0096] In a step S70, second test cases T.sub.k are created by the test agent 220 using the adapted test strategy 230.

[0097] In a step S80, a new simulation cycle for the second test cases T.sub.k is started.

[0098] In a step S90, the adaptation of the test strategy 230 to perform a further simulation cycle is repeated if a particular evaluation criterion is not met.

[0099] In a step S100, the test cases T.sub.k of the last simulation cycle are passed on to an output module 700 if a particular evaluation criterion is met.

[0100] In a step S110, output results 750 from the test cases T.sub.k are generated and output by the output module 200 for calibration and validation purposes, particularly in the form of calibration parameters Pcal.sub.i for the driver assistance system (ADAS) and/or the automated driving system (ADS) and/or the driving function for performing the set driving task.

[0101] FIG. 4 schematically represents a computer program product 900 comprising an executable program code 950 that is configured to perform the method according to the first aspect of the present invention.

[0102] By means of the invention, an optimal set of calibration parameters for virtual calibration and validation of ADAS/ADS can be generated by the use of modular-structured modules. The modules can be developed and tested independently of each other, thereby increasing the quality of the overall process. By integrating and linking the necessary modules, the system 100 according to the invention can purposefully and efficiently perform a virtual calibration and validation of an ADAS/ADS. In this way, resources can be conserved, since actual departures from test routes both with standard traffic situations as well as with specific corner cases can be reduced.

REFERENCE NUMBERS

[0103] 10 Motor vehicle [0104] 100 System [0105] 200 Test agent module [0106] 220 Test agent [0107] 230 Test strategy [0108] 300 Test database [0109] 320 Calibration parameter database [0110] 330 Scenario database [0111] 340 Evaluation database [0112] 400 Simulation module [0113] 410 Environment model module [0114] 420 Driver model module [0115] 430 Vehicle model module [0116] 440 Driving function module [0117] 450 Simulation results [0118] 470 Sensors [0119] 480 Database [0120] 500 Evaluation module [0121] 510 Function evaluation module [0122] 520 Simulation evaluation module [0123] 550 Evaluation results [0124] 700 Output module [0125] 750 Output data [0126] 900 Computer program product [0127] 950 Program code