METHOD FOR DETERMINING SIMILAR SCENARIOS, TRAINING METHOD, AND TRAINING CONTROLLER

Abstract

A computer-implemented method for providing a machine learning algorithm for determining similar scenarios based on scenario data of a data set of sensor data, wherein an optimization algorithm is applied to the feature representation, output by the first machine learning algorithm, of the first augmentation of the data set of sensor data, wherein the optimization algorithm approximates the feature representation, output by the second machine learning algorithm, of the second augmentation of the data set of sensor data. The invention further relates to a method for determining similar scenarios based on scenario data of a data set of sensor data and to a training controller.

Claims

1. A computer-implemented method to provide a machine learning algorithm to determine similar scenarios based on scenario data of a data set of sensor data, the method comprising: providing the data set of sensor data of a drive, captured by a plurality of on-board environment detection sensors, by a vehicle; generating a first augmentation of the data set of sensor data and a second augmentation that is different from the first augmentation of the data set of sensor data; applying a first machine learning algorithm to the first augmentation of the data set of sensor data for generating a dimension-reduced feature representation of the first augmentation of the data set of sensor data and for determining a first class of a scenario covered by the first augmentation of the data set of sensor data; applying a second machine learning algorithm to the second augmentation of the data set of sensor data for generating an in particular dimension-reduced feature representation of the second augmentation of the data set of sensor data and for determining a second class of a scenario covered by the second augmentation of the data set of sensor data; and applying an optimization algorithm to the feature representation output by the first machine learning algorithm of the first augmentation of the data set of sensor data, the optimization algorithm approximating the feature representation output by the second machine learning algorithm of the second augmentation of the data set of sensor data.

2. The computer-implemented method according to claim 1, wherein a similarity loss between the first class output by the first machine learning algorithm of the scenario covered by the first augmentation of the data set of sensor data and the second class output by the second machine learning algorithm of the scenario covered by the second augmentation of the data set of sensor data, is minimized by the optimization algorithm.

3. The computer-implemented method according to claim 1, wherein the first machine learning algorithm has a first encoder, which receives trajectory and/or speed data of the vehicle of the first augmentation of the data set of sensor data, a second encoder, which receives trajectory, speed, and/or class ID data of at least one object of the first augmentation of the data set of sensor data, and a third encoder, which receives road information of the first augmentation of the data set of sensor data.

4. The computer-implemented method according to claim 1, wherein the second machine learning algorithm has a fourth encoder, which receives trajectory and/or speed data of the vehicle of the second augmentation of the data set of sensor data, a fifth encoder, which receives trajectory, speed, and/or class ID data of at least one object of the second augmentation of the data set of sensor data, and a sixth encoder, which receives road information of the second augmentation of the data set of sensor data.

5. The computer-implemented method according to claim 3, wherein the first encoder, the second encoder, and the third encoder each output a feature vector, which are concatenated into a first feature vector, and wherein the fourth encoder, the fifth encoder, and the sixth encoder each output a feature vector, which are concatenated into a second feature vector.

6. The computer-implemented method according to claim 5, wherein the first machine learning algorithm determines the first class of the scenario, covered by the first augmentation of the data set of sensor data, using the concatenated first feature vector, and wherein the second machine learning algorithm determines the second class of the scenario, covered by the second augmentation of the data set of sensor data, using the concatenated second feature vector.

7. The computer-implemented method according to claim 1, wherein the first to sixth encoders have LSTM layers.

8. The computer-implemented method according to claim 3, wherein trajectory data, covered by the data set of sensor data of the vehicle and/or of the object each have a different feature size depending on a number of time steps in which the object is located within a detection range of the plurality of on-board environment detection sensors.

9. The computer-implemented method according to claim 8, wherein the first machine learning algorithm and the second machine learning algorithm use ragged tensors to process the trajectory data covered by the data set of sensor data of the vehicle and/or the object.

10. The computer-implemented method according to claim 1, wherein the first augmentation and the second augmentation for creating different variants of the data set of sensor data are randomly generated.

11. The computer-implemented method according to claim 1, wherein the scenarios have driving maneuvers of the vehicle and/or a fellow vehicle and/or interaction maneuvers of the vehicle with the fellow vehicle and/or further objects.

12. The computer-implemented method according to claim 3, wherein the trajectory and/or speed data of the vehicle are captured by a GPS sensor, and wherein the trajectory, speed, and/or class ID data of the at least one object and the road information are captured by a camera sensor, LiDAR sensor, and/or radar sensor.

13. A computer-implemented method to determine similar scenarios based on scenario data of a data set of sensor data, the method comprising: providing the data set of sensor data of a drive, captured by a plurality of on-board environment detection sensors, by a vehicle; and applying a machine learning algorithm trained according to claim 1 to the data set of sensor data for determining clustering similar scenarios.

14. A training controller to provide a machine learning algorithm to determine similar scenarios based on scenario data of a data set of sensor data, the training controller comprising: a receiver to receive the data set of sensor data of a drive captured by a plurality of on-board environment detection sensors by a vehicle; a generator to generate a first augmentation of the data set of sensor data and a second augmentation, different from the first augmentation, of the data set of sensor data; a first applicator to apply a first machine learning algorithm to the first augmentation of the data set of sensor data for generating an in particular dimension-reduced feature representation of the first augmentation of the data set of sensor data and to determine a first class of a scenario covered by the first augmentation of the data set of sensor data; a second applicator to apply a second machine learning algorithm to the second augmentation of the data set of sensor data for generating a dimension-reduced feature representation of the second augmentation of the data set of sensor data and to determine a second class of a scenario covered by the second augmentation of the data set of sensor data; and a third applicator to apply an optimization algorithm to the feature representation output by the first machine learning algorithm of the first augmentation of the data set of sensor data, wherein the optimization algorithm approximates the feature representation output by the second machine learning algorithm of the second augmentation of the data set of sensor data.

15. A computer program with a program code to perform the method according to claim 1, when the computer program is executed on a computer.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The present invention will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only, and thus, are not limitive of the present invention, and wherein:

[0059] FIG. 1 shows a flowchart of a computer-implemented method for providing a machine learning algorithm for determining similar scenarios based on scenario data of a data set of sensor data according to an example of the invention;

[0060] FIG. 2 shows a flowchart of a method for determining similar scenarios based on scenario data of a data set of sensor data according to an example of the invention; and

[0061] FIG. 3 shows a training controller for providing a machine learning algorithm for determining similar scenarios based on scenario data of a data set of sensor data according to an example of the invention.

DETAILED DESCRIPTION

[0062] The method shown in FIG. 1 for providing a machine learning algorithm for determining similar scenarios based on scenario data of a data set D of sensor data comprises providing S1 the data set D of sensor data of a drive, captured by a plurality of on-board environment detection sensors 10, by an ego vehicle 12.

[0063] The method further comprises generating S2 a first augmentation 14 of the data set D of sensor data and a second augmentation 16, different from the first augmentation 14, of the data set D of sensor data and applying S3 a first machine learning algorithm A1 to the first augmentation 14 of the data set D of sensor data for generating an in particular dimension-reduced feature representation 18 of first augmentation 14 of the data set D of sensor data and for determining a first class K1 of a scenario covered by first augmentation 14 of the data set D of sensor data.

[0064] The method further comprises applying S5 a second machine learning algorithm A2 to second augmentation 16 of the data set D of sensor data for generating an in particular dimension-reduced feature representation 20 of second augmentation 16 of the data set D of sensor data and for determining S6 a second class K2 of a scenario covered by second augmentation 16 of the data set D of sensor data.

[0065] Moreover, the method comprises applying an optimization algorithm A3 to the feature representation 18, output by the first machine learning algorithm A1, of the first augmentation 14 of the data set D of sensor data, wherein optimization algorithm A3 approximates the feature representation 20, output by second machine learning algorithm A2, of second augmentation 16 of the data set D of sensor data.

[0066] Furthermore, a similarity loss V between the first class K1, output by first machine learning algorithm A1, of the scenario, covered by first augmentation 14 of the data set D of sensor data, and the second class K2, output by second machine learning algorithm A2, of the scenario, covered by second augmentation 16 of the data set D of sensor data, is minimized by optimization algorithm A3.

[0067] First machine learning algorithm A1 has a first encoder E1 that receives trajectory and/or speed data 22 of ego vehicle 12 of first augmentation 14 of the data set D of sensor data.

[0068] Further, first machine learning algorithm A1 has a second encoder E2 that receives trajectory, speed, and/or class ID data 24 of at least one object of the first augmentation 14 of the data set D of sensor data.

[0069] Moreover, first machine learning algorithm A1 has a third encoder E3 that receives road information 26 of first augmentation 14 of the data set D of sensor data.

[0070] Second machine learning algorithm A2 has a fourth encoder E4 that receives trajectory and/or speed data 28 of ego vehicle 12 of second augmentation 16 of the data set D of sensor data.

[0071] Further, second machine learning algorithm A2 has a fifth encoder E5 that receives trajectory, speed, and/or class ID data 30 of at least one object of second augmentation 16 of the data set D of sensor data.

[0072] Moreover, second machine learning algorithm A2 has a sixth encoder E6 that receives road information 32 of second augmentation 16 of the data set D of sensor data.

[0073] First encoder E1, second encoder E2, and third encoder E3 each output a feature vector, which are concatenated into a first feature vector MV1. Fourth encoder E4, fifth encoder E5, and sixth encoder E6 each also output a feature vector, which are concatenated into a second feature vector MV2.

[0074] First machine learning algorithm A1 determines the first class K1 of the scenario, covered by first augmentation 14 of the data set D of sensor data, using the concatenated first feature vector MV1. Second machine learning algorithm A2 determines the second class K2 of the scenario, covered by second augmentation 16 of the data set D of sensor data, using the concatenated second feature vector MV2. The first to sixth encoders E 1-E6 further have LSTM layers.

[0075] Trajectory data 22, 28, covered by the data set D of sensor data, of ego vehicle 12 and/or the object each have a different feature quantity depending on a number of time steps in which the object is within a detection range of the plurality of on-board environment detection sensors 10.

[0076] First machine learning algorithm A1 and second machine learning algorithm A2 further use ragged tensors for processing the trajectory data 22, 28, covered by the data set D of sensor data, of ego vehicle 12 and/or the object. First augmentation 14 and second augmentation 16 for creating different variants of the data set D of sensor data are thereby randomly generated.

[0077] The scenarios have driving maneuvers of ego vehicle 12 and/or a fellow vehicle and/or interaction maneuvers of ego vehicle 12 with the fellow vehicle and/or other objects.

[0078] The trajectory and/or speed data 22, 28 of ego vehicle 12 are captured by a GPS sensor. The trajectory, speed, and/or class ID data 24, 30 of the at least one object, as well as the road information, are captured by a camera sensor, LiDAR sensor, and/or radar sensor.

[0079] FIG. 2 shows a flowchart of a method for determining similar scenarios based on scenario data of a data set of sensor data according to the preferred embodiment of the invention.

[0080] The method comprises providing S1′ the data set D of sensor data of a drive, captured by a plurality of on-board environment detection sensors 10, by an ego vehicle 12 and applying S2′ a machine learning algorithm, trained according to the invention, to the data set D of sensor data for determining, in particular clustering, similar scenarios.

[0081] FIG. 3 shows a training controller 1 for providing a machine learning algorithm for determining similar scenarios based on scenario data of a data set of sensor data according to the preferred embodiment of the invention.

[0082] Training controller 1 comprises a receiver 34 for receiving the data set D of sensor data of a drive, captured by a plurality of on-board environment detection sensors 10, by an ego vehicle 12 and a generator 36 for generating a first augmentation 14 of the data set D of sensor data and a second augmentation 16, different from the first augmentation 14, of the data set D of sensor data.

[0083] Furthermore, training controller 1 has an applicator 38 for applying a first machine learning algorithm A1 to first augmentation 14 of the data set D of sensor data for generating an in particular dimension-reduced feature representation 18 of first augmentation 14 of the data set D of sensor data and for determining a first class K1 of a scenario covered by first augmentation 14 of the data set D of sensor data.

[0084] Training controller 1 further comprises a second applicator 40 for applying a second machine learning algorithm A2 to second augmentation 16 of the data set D of sensor data for generating an in particular a dimension-reduced feature representation 20 of second augmentation 16 of the data set D of sensor data and for determining a second class K2 of a scenario covered by second augmentation 16 of the data set D of sensor data.

[0085] Moreover, training controller 1 has a third applicator 42 for applying an optimization algorithm A3 to feature representation 18, output by first machine learning algorithm A1, of first augmentation 14 of the data set D of sensor data, wherein the optimization algorithm A3 approximates the feature representation 20, output by the second machine learning algorithm A2, of the second augmentation 16 of the data set D of sensor data.

[0086] The applicators may be software and/or hardware that are used in training an AI-algorithm, for example a personal computer, workstations or part of a cloud infrastructure. It can include a software that can be used in training process of an AI-algorithm and typically comprises one or more processing units, such as central processing units (CPU) or graphics processing unit (GPU).

[0087] The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are to be included within the scope of the following claims.