COMPUTER-IMPLEMENTED METHOD AND SYSTEM FOR CREATING A VIRTUAL ENVIRONMENT FOR A VEHICLE

Abstract

A computer-implemented method and system for generating a virtual environment for a vehicle for testing highly automated driving functions of a motor vehicle. The method comprises projecting the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data, wherein each point of the LiDAR point cloud, superimposed by classified pixels of the camera image data, in particular having the same image coordinates, is assigned an identical class and an instance segmentation of the classified LiDAR point cloud data for determining at least one real object comprised by a class. A computer program and a computer-readable data carrier are also provided.

Claims

1. A computer-implemented method for creating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle, the method comprising: providing pre-acquired camera image data and LiDAR point cloud data of a real vehicle environment; performing pixel-based classification of the pre-acquired camera image data using a machine learning algorithm, which outputs an associated class and a confidence value corresponding to the classification for each pixel; projecting the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data, wherein each point of the LiDAR point cloud, superimposed by classified pixels of the camera image data or points having the same image coordinates, is assigned an identical class; instance segmenting the classified LiDAR point cloud data to determine at least one real object comprised by a class; selecting and calling a stored, synthetically generated first object corresponding to the at least one real object or procedural generation of a synthetically generated second object corresponding to the at least one real object; and integrating the synthetically generated first object or the synthetically generated second object into a specified virtual vehicle environment.

2. The computer-implemented method according to claim 1, wherein for a specified first number of classes the selection and call of the stored, synthetically generated first object corresponding to the at least one real object is carried out and for a specified second number of classes, in particular the procedural generation of the synthetically generated second object corresponding to the at least one real object, is carried out.

3. The computer-implemented method according to claim 1, wherein, based on the instance segmentation of the classified LiDAR point cloud data for determining at least one real object comprised by a class, an extraction of features describing the at least one real object, in particular a size and/or a radius of the object, is performed.

4. The computer-implemented method according to claim 3, wherein based on the extracted features, the procedural generation of the synthetically generated second object corresponding to the at least one real object is carried out.

5. The computer-implemented method according to claim 3, wherein based on the extracted features, a comparison of the segmented, at least one real object of a class with a plurality of stored, synthetically generated objects is performed.

6. The computer-implemented method according to claim 5, wherein based on the comparison of the segmented, at least one real object of a class with a plurality of stored, synthetically generated objects, a stored, synthetically generated first object having a specified similarity measure is selected and called.

7. The computer-implemented method according to claim 1, wherein the classes determined by a machine learning algorithm represent buildings, vehicles, traffic signs, traffic lights, roadways, road markings, plantings, pedestrians and/or other objects.

8. The computer-implemented method according to claim 1, wherein respective points of the LiDAR point cloud, which are not superimposed by classified pixels of the camera image data, in particular having the same image coordinates, are removed from the LiDAR point cloud.

9. The computer-implemented method according to claim 1, wherein respective points of the LiDAR point cloud, which are superimposed by classified pixels of the camera image data, which pixels have a confidence value that is less than a predetermined first threshold, are removed in order to provide reduced LiDAR point cloud data.

10. The computer-implemented method according to claim 9, wherein the instance segmentation of the classified LiDAR point cloud data for determining the at least one real object comprised by a class is performed using the reduced LiDAR point cloud data.

11. The computer-implemented method according to claim 1, wherein the pre-acquired camera image data and LiDAR point cloud data represent the same real vehicle environment captured at the same time.

12. The computer-implemented method according to claim 3, wherein the features describing the at least one real object are extracted by a further machine learning algorithm.

13. A system to generate a virtual vehicle environment for testing highly automated driving functions of a motor vehicle using pre-acquired video image data, radar data and/or a LiDAR point cloud of a real vehicle environment, the system comprising: a data memory to provide pre-acquired camera image data and LiDAR point cloud data of a real vehicle environment; a calculation device for pixel-based classification of the pre-acquired camera image data using a machine learning algorithm which is configured to output for each pixel an associated class and a confidence value corresponding to the classification, wherein the calculation device is configured to project the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data and to assign an identical class to respective points of the LiDAR point cloud, each superimposed by classified pixels of the camera image data, in particular having the same image coordinates, wherein the calculation device is configured to perform an instance segmentation of the classified LiDAR point cloud data to determine at least one real object comprised by a class, wherein the calculation device is configured to select and call a stored, synthetically generated first object or a procedural generation of a synthetically generated second object corresponding to the at least one real object, and wherein the calculation device is configured to integrate the synthetically generated first or second object into a specified virtual vehicle environment.

14. A computer program with program code for performing the method according to claim 1 when the computer program is executed on a computer.

15. A computer-readable data carrier comprising program code of a computer program for performing the method according to claim 1 when the computer program is executed on a computer.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only, and thus, are not limitive of the present invention, and wherein:

[0061] FIG. 1 is a flowchart of a computer-implemented method for generating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle; and

[0062] FIG. 2 is a diagram of a system for generating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle using pre-acquired video image data, radar data and/or a LiDAR point cloud of a real vehicle environment according to the preferred embodiment of the invention.

DETAILED DESCRIPTION

[0063] FIG. 1 shows a flowchart of a method for generating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle.

[0064] The method comprises providing 51 pre-acquired camera image data D1 and LiDAR point cloud data D2 of a real vehicle environment and pixel-based classification S2 of the pre-acquired camera image data D1 using a machine learning algorithm A, which outputs an associated class K and a confidence value V corresponding to the classification.

[0065] The method further comprises a projection S3 of the pixel-based classified camera image data D1 onto the pre-captured LiDAR point cloud data D2, wherein respective points of the LiDAR point cloud superimposed by classified pixels of the camera image data D1, in particular having the same image coordinates, are assigned an identical class K.

[0066] Furthermore, the method comprises an instance segmentation S4 of the classified LiDAR point cloud data D2 for determining at least one real object 10 comprised by a class K and a selection and call S5a of a stored, synthetically generated first object 12 corresponding to the at least one real object 10 or procedural generation S5b of a synthetically generated second object 14 corresponding to the at least one real object 10.

[0067] In addition, the method comprises integrating S6 the synthetically generated first object 12 or the synthetically generated second object 14 into a specified virtual vehicle environment.

[0068] For a specified first number of classes K, the selection and call S5a of the stored, synthetically generated first object 12 corresponding to the at least one real object 10 is carried out. Furthermore, for a specified second number of classes K, in particular the procedural generation S4b of the synthetically generated second object 14 corresponding to the at least one real object 10 is carried out.

[0069] Based on the instance segmentation S4 of the classified LiDAR point cloud data D2 for determining at least one real object 10 comprised by a class K, an extraction of the features describing the at least one real object 10, in particular a size and/or a radius of the object, is carried out. Furthermore, based on the extracted features, the procedural generation S5b of the synthetically generated second object 14 corresponding to the at least one real object 10 is carried out.

[0070] Based on the extracted features, the segmented, at least one real object 10 of a class K is compared with a plurality of stored, synthetically generated objects.

[0071] In addition, based on the comparison of the segmented, at least one real object 10 of a class K with a plurality of stored, synthetically generated objects, a stored, synthetically generated first object having a specified similarity measure is selected and called.

[0072] The classes K determined by the machine learning algorithm A represent buildings, vehicles, traffic signs, traffic lights, roadways, road markings, plantings, pedestrians and/or other objects. Respective points of the LiDAR point cloud which are not superimposed by classified pixels of the camera image data D1, in particular having the same image coordinates, are removed from the LiDAR point cloud.

[0073] Respective points of the LiDAR point cloud, which are superimposed by classified pixels of the camera image data D1, which pixels have a confidence value V that is less than a specified first threshold value, are also removed to provide reduced LiDAR point cloud data D2.

[0074] The instance segmentation S4 of the classified LiDAR point cloud data D2 for determining the at least one real object 10 comprised by a class K is further performed using the reduced LiDAR point cloud data D2. The pre-acquired camera image data D1 and LiDAR point cloud data D2 represent the same real vehicle environment captured at the same time.

[0075] The features describing the at least one real object 10 are extracted by another machine learning algorithm.

[0076] FIG. 2 shows a diagram of a system 1 for generating a virtual vehicle environment for testing highly automated driving functions of a motor vehicle according to the preferred embodiment of the invention.

[0077] The system 1 comprises a data memory 16 for providing pre-acquired camera image data D1 and LiDAR point cloud data D2 of a real vehicle environment and a calculation device 18, which is configured to perform a pixel-based classification of the pre-acquired camera image data D1 using a machine learning algorithm A designed to output for each pixel an associated class K and a confidence value V corresponding to the classification.

[0078] The calculation device 18 is further configured to perform a projection of the pixel-based classified camera image data D1 onto the pre-acquired LiDAR point cloud data D2, wherein the calculation device 18 is configured to assign an identical class K to respective points of the LiDAR point cloud superimposed by classified pixels of the camera image data D1, in particular having the same image coordinates.

[0079] Furthermore, the calculation device 18 is configured to perform an instance segmentation of the classified LiDAR point cloud data D2 for determining at least one real object 10 comprised by a class K. The calculation device 18 is also configured to select and call a stored, synthetically generated first object 12 or perform a procedural generation of a synthetically generated second object 14 corresponding to the at least one real object 10.

[0080] Furthermore, the calculation device 18 is configured to integrate the synthetically generated first or second object 14 into a specified virtual vehicle environment.

[0081] Although specific embodiments have been illustrated and described herein, it is understood to the person skilled in the art that a variety of alternative and/or equivalent implementations exist. It should be noted that the exemplary embodiment or exemplary embodiments are examples only and are not intended to limit the scope, applicability or configuration in any way.

[0082] Rather, the above-mentioned summary and detailed description provides the skilled person with a convenient guide for implementing at least one exemplary embodiment, it being understood that various changes in the functionality and arrangement of the elements can be made without departing from the scope of the attached claims and their legal equivalents.

[0083] In general, this application is intended to cover changes or adaptations or variations of the embodiments presented herein. For example, a sequence of method steps can be modified. The method can also be carried out, at least in sections, sequentially or in parallel.

[0084] The invention being thus described, it will be obvious that the same may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are to be included within the scope of the following claims.