Method and Control Device for Training an Object Detector

Abstract

A method is for training an object detector configured to detect objects in sensor data of a sensor. The method includes providing first sensor data of the sensor, providing an object representation assigned to the first sensor data, and transmitting the object representation to a sensor model. The method further includes imaging object representations onto the first sensor data of the sensor with the sensor model, assigning the object representation to second sensor data with the sensor model, and training the object detector based on the second sensor data.

Claims

1. A method for training an object detector configured to detect objects in sensor data of a sensor, the method comprising: providing first sensor data of the sensor; providing an object representation assigned to the first sensor data; transmitting the object representation to a sensor model; imaging object representations onto the first sensor data of the sensor with the sensor model; assigning the object representation to second sensor data with the sensor model; and training the object detector based on the second sensor data.

2. The method according to claim 1, wherein training the object detector based on the second sensor data comprises: comparing the second sensor data with the first sensor data in order to determine a first cost function; and training the object detector based on the first cost function.

3. The method according to claim 1, wherein the object representation is an annotation.

4. The method according to claim 1, wherein the sensor model is an artificial neural network.

5. A method for controlling a driver assistance system of a motor vehicle, comprising: training an object detector configured to detect objects in sensor data of a sensor of the motor vehicle by: providing first sensor data of the sensor, providing an object representation assigned to the first sensor data, transmitting the object representation to a sensor model, imaging object representations onto the first sensor data of the sensor with the sensor model, assigning the object representation to second sensor data with the sensor model, and training the object detector based on the second sensor data; providing the trained object detector for the sensor of the motor vehicle; generating object detection results using the trained object detector; and controlling the driver assistance system based on the object detection results.

6. A control device for training an object detector for detecting objects in sensor data of a sensor, the control device comprising: a first provision unit configured to provide first sensor data of the sensor; a second provision unit configured to provide an object representation assigned to the first sensor data; a transmission unit configured to transmit the object representation to a sensor model configured to image object representations onto the first sensor data of the sensor; an assignment unit configured to assign the object representation to second sensor data using the sensor model; and a training unit configured to train the object detector based on the second sensor data.

7. The control device according to claim 6, further comprising: a first comparison unit configured to compare the second sensor data with the first sensor data in order to determine a first cost function, wherein the training unit is configured to train the object detector based on the first cost function.

8. The control device according to claim 6, wherein the object representation is an annotation.

9. The control device according to claim 6, wherein the sensor model is an artificial neural network.

10. The control device according to claim 6, wherein: the control device is a first control device, a second control device is configured to control a driver assistance system of a motor vehicle including the sensor, and the second control device comprises (i) a receiving unit configured to receive the object detector for the sensor of the motor vehicle, (ii) a generation unit configured to generate object detection results using the object detector, and (iii) a control unit configured to control the driver assistance system based on the generated object detection results.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are intended to impart further understanding of the embodiments of the disclosure. They illustrate embodiments and, in connection with the description, serve to explain principles and concepts of the disclosure.

[0053] Other embodiments and many of the mentioned advantages are apparent from the drawings. The illustrated elements of the drawings are not necessarily shown to scale relative to one another. In the drawings:

[0054] FIG. 1 is a flowchart of a method for training an object detector according to a first embodiment;

[0055] FIG. 2 is a flowchart of a method for training an object detector according to a second embodiment; and

[0056] FIG. 3 is a block diagram of a system for training an object detector according to embodiments of the disclosure.

DETAILED DESCRIPTION

[0057] In the figures of the drawings, identical reference signs denote identical or functionally identical elements, parts or components, unless stated otherwise.

[0058] FIG. 1 is a flowchart of a method for training an object detector 1 according to a first embodiment.

[0059] The starting point is an object detector which detects objects for at least one object class on the basis of sensor data. The detection can include both a classification with regard to the type of object and a regression of the corresponding object parameters.

[0060] According to the first embodiment, the aim is to train such an object detector on the basis of sensor data of a sensor for which no annotations are typically present. An example of this is a radar sensor which provides reflections as measurement data. However, since objects can only be annotated with difficulty due to reflections, LIDAR-based annotations are frequently used to train a radar-based object detector.

[0061] However, due to different measuring principles, different sensor specifications or the different installation positions of the sensors, it is not possible for LIDAR-based annotations to be transmitted to a radar sensor one-to-one. For example, different ranges or fields of view can restrict the suitability of the annotations. Likewise, some annotated object properties cannot be measured using the measuring principle of the target sensor, for example a radar sensor. For this reason, LIDAR-based annotations typically include a height of the annotated object, whereas many radar sensors have only a limited elevation resolution.

[0062] FIG. 1 shows a method 1, which comprises a step 2 of training or pretraining an object detector using training data, the object detector being designed to detect objects in sensor data of a sensor, a step 3 of providing first sensor data of the sensor, a step 4 of providing an object representation assigned to the first sensor data, a step 5 of transmitting the object representation to a sensor model which is designed to image object representations onto sensor data of the sensor, a step 6 of assigning the object representation to second sensor data, and a step 7 of training the object detector on the basis of the second sensor data.

[0063] The illustrated method 1 makes it possible for data records which have been annotated on the basis of particular sensors to be used for another sensor. For example, annotations which have been generated on the basis of a lidar sensor can be used to train a radar object detector. This increases the reusability of existing data records and thus reduces the need for additional annotations, resulting in time and cost savings.

[0064] In addition, however, the illustrated method 1 is also not dependent on the presence of annotations and can also be applied to unlabeled data. Consequently, for example, a significantly larger data volume can be used to train the object detector for the second sensor, which further positively affects the object detection quality or the quality of the object detector.

[0065] Overall, FIG. 1 thus shows an improved method for training an object detector for a sensor 1.

[0066] In particular, FIG. 1 shows a method 1 which uses a sensor model, the sensor model describing the measurement characteristic of a target sensor and supporting the transfer of existing annotations of a data record.

[0067] The sensor model is, in particular, designed in such a way, based on the object representation provided, for example a decision of the object detector for the first sensor data or a provided annotation, so as to determine a measured value to be expected for the sensor. In this case, the sensor model may be designed, for example, to provide a probability density over the expected positions of the measured values of the sensor. The probability density can be described, for example, by means of a parametric density function, for example a multivariate Gaussian distribution, and does not have to be limited to the expected positions of the measured values. Rather, the density function may also comprise further dimensions, for example with respect to a radial velocity in the case of a radar sensor.

[0068] Furthermore, however, the sensor model may also be described, for example, by means of discrete approximations, for example grid representations, or may be designed to determine specific values for the corresponding measured values of the sensor, it being possible for the sensor model to have, for example, a list of expected reflections in the case of a radar sensor or a 3D point cloud in the case of a lidar sensor.

[0069] As further shown in FIG. 1, the step 7 of training the object detector on the basis of the second sensor data has a step 8 of comparing the second sensor data with the first sensor data in order to determine a first cost function or a first loss, and a step 9 of training the object detector on the basis of the first cost function.

[0070] According to the first embodiment, the sensor model is differentiable, the sensor model, on the basis of the object representations provided, comparing expected sensor values with the actual conditions or values based on the sensor model. Errors in object detection by the object detector lead to expected sensor values which differ from the actual values. A first cost function calculated based on this difference can then be transmitted to the output of the object detector and used to retrain the object detector.

[0071] If the sensor model is an artificial neural network, the corresponding loss functions may be, for example, the negative log-likelihood of the real sensor measured values. If, on the other hand, the sensor model predicts specific sensor measured values, distances, for example the Euclidean distance or the Mahalanobis distance, can be used as the loss between the predicted and actual sensor measured values.

[0072] According to the first embodiment, the object representation is further an object detection result generated by the object detector on the basis of the first sensor data, the method 1 additionally comprising a step 10 of providing annotations, a step 11 of comparing the object detection result with the annotations provided in order to determine a second loss or a second cost function, and a step 12 of retraining the object detector on the basis of the second cost function.

[0073] For the training of the object detector, the outputs of the object detector are therefore compared with available annotations. On the basis of this comparison, a second cost function is then determined, which is used to retrain the object detector and, for example, propagates back through network layers of the object detector and can be used to adapt the corresponding network weights.

[0074] According to the first embodiment, the sensor model is also an artificial neural network. The sensor model can be trained simultaneously with the object detector. Furthermore, however, the sensor model can also be pretrained, for example.

[0075] The object detector for the first sensor can then be used, for example, to control a driver assistance system of a motor vehicle, for example an autonomously driving motor vehicle.

[0076] Furthermore, the trained sensor model can also be used independently of the object detector for other applications, for example to track objects or simulate sensor data.

[0077] FIG. 2 is a flowchart of a method for training an object detector 20 according to a second embodiment.

[0078] As FIG. 2 shows, the method 20 comprises a step 21 of training an object detector using training data, the object detector being designed to detect objects in sensor data of a sensor, a step 22 of providing first sensor data of the sensor, a step 23 of providing an object representation assigned to the first sensor data, a step 24 of transmitting the object representation to a sensor model which is designed to image object representations onto sensor data of the sensor, a step 25 of assigning the object representation to second sensor data, and a step 26 of training the object detector on the basis of the second sensor data.

[0079] The difference between the method shown in FIG. 2 according to the second embodiment and the method shown in FIG. 1 according to the first embodiment consists in that, according to the second embodiment, the object presentation is a provided annotation, the method also comprising a step 27 of estimating whether the provided annotation lies within a visible range of the first sensor, the first sensor data being discarded in a step 28 if the provided annotation does not lie within a visible range of the sensor.

[0080] FIG. 3 is a block diagram of a control device for training an object detector 30 according to embodiments of the disclosure.

[0081] The object detector is, in particular, designed to detect objects in sensor data of a sensor.

[0082] As FIG. 3 shows, the control device for training an object detector in this case comprises a first training unit 31, which is designed to pretrain the object detector on the basis of training data, a first provision unit 32, which is designed to provide first sensor data of the sensor, a second provision unit 33, which is designed to provide an object representation assigned to the first sensor data, a transmission unit 34, which is designed to transmit the object representation to a sensor model which is designed to image object representations onto sensor data of the sensor, an assignment unit 35, which is designed to assign the object representation to second sensor data by means of the sensor model, and a second training unit 36, which is designed to train the object detector on the basis of the second sensor data.

[0083] The first training unit, the assignment unit, and the second training unit can in each case be implemented, for example, on the basis of code stored in a memory and executable by a processor. The first provision unit can be implemented, for example, on the basis of a receiver which is designed to receive sensor data from the sensor, for example sensor data that are currently being recorded and/or stored in a memory. The second provision unit can be implemented, for example, on the basis of a receiver which is designed to receive an object detection result generated by the object detector on the basis of the first sensor data and/or annotations provided, for example, by other sensors. The transmission unit may also be, for example, a correspondingly designed transmitter.

[0084] According to the embodiments of FIG. 3, the control device 30 further comprises a first comparison unit 37, which is designed to compare the second sensor data with the first sensor data in order to determine a first cost function, the second training unit 36 being designed to train the object detector on the basis of the first cost function.

[0085] The comparison unit can in turn be implemented, for example, on the basis of code stored in a memory and executable by a processor.

[0086] A third provision unit 38, which is designed to provide annotations, can also be seen.

[0087] In this case, the control device 30 also comprises a second comparison unit 39, which is designed to compare the object representation with the provided annotations if the object representation is generated by means of the object detector on the basis of the first sensor data in order to determine a second cost function, the retraining unit 36 being designed to train the object detector on the basis of the second cost function.

[0088] The third provision unit 38 may be integrated into the second provision unit and have a corresponding receiver. The second comparison unit can again be implemented, for example, on the basis of code stored in a memory and executable by a processor.

[0089] The control device 30 shown also has an estimation unit 40, which is designed to estimate whether an annotation provided as an object representation lies within a visible range of the sensor, and a discarding unit 41, which is designed to discard the first sensor data or the corresponding object representation if the annotation does not lie within the visible range of the sensor.

[0090] The estimation unit and the discarding unit can again be implemented, for example, on the basis of code stored in a memory and executable by a processor.

[0091] According to the embodiments of FIG. 3, the sensor model is again an artificial neural network.