METHOD FOR CALCULATING A QUALITY MEASURE FOR ASSESSING AN OBJECT DETECTION ALGORITHM

Abstract

A method for calculating a quality measure of a computer-implemented object detection algorithm, which may be used, in particular, for enabling the object detection algorithm for semi-automated, highly-automated or fully-automated robots. The method includes: assigning ascertained object detections to annotations, the object detections and/or the annotations corresponding to bounding boxes; determining deviations, in particular, distances of the annotations with respect to their assigned object detections; calculating the quality measure of the object detection algorithm based on the determined deviations, the quality measure representing a probability with which a deviation of an object detection from the annotation assigned to it exceeds or falls below a predefined threshold value.

Claims

1-15. (canceled)

16. A method for calculating a quality measure of a computer-implemented object detection algorithm, for enabling the object detection algorithm for semi-automated, highly-automated or fully-automated robots, including the following steps: assigning ascertained object detections to annotations, the object detections and/or the annotations corresponding to bounding boxes; determining deviations including distances of the annotations with respect to their assigned object detections; calculating the quality measure of the object detection algorithm based on the determined deviations, the quality measure representing a probability with which a deviation of an object detection from an annotation assigned to it exceeds or falls below a predefined threshold value.

17. The method as recited in claim 16, wherein each deviation represents a shift between a point of the object detection to a point of its assigned annotation, the shift being a signed scalar, whose value represents a distance and its sign a direction, in which the point of the annotation is shifted from the point of the object detection.

18. The method as recited in claim 17, wherein the deviation represents a smallest shift from a set of ascertained shifts.

19. The method as recited in claim 18, wherein the set is made up of sides of the annotation to corresponding sides of the assigned object detection and the shifts being orthogonal to the respective side.

20. The method as recited in claim 16, wherein each deviation represents an area that corresponds to a part of the annotation that exhibits no overlap with the object detection.

21. The method as recited in claim 16, wherein the calculation of the quality measure representing the probability is based on a model which is ascertained based on the determined deviations.

22. The method as recited in claim 21, wherein the model is a parameterizable model, the parameterizable model being a parameterizable probability distribution, whose parameters are ascertained from the determined deviations.

23. A method for adapting a computer-implemented object detection algorithm for ascertaining object detections, comprising the following steps: a) ascertaining annotations of objects detected using the object detection algorithm; b) ascertaining object detections using the object detection algorithm; c) calculating a quality measure of the object detection algorithm by: assigning the object detections to the annotations, the object detections and/or the annotations corresponding to bounding boxes, determining deviations including distances of the annotations with respect to their assigned object detections, and calculating the quality measure of the object detection algorithm based on the determined deviations, the quality measure representing a probability with which a deviation of an object detection from an annotation assigned to it exceeds or falls below a predefined threshold value; d) adapting the object detection algorithm based on the calculated quality measure in such a way that a renewed execution of the object detection algorithm results in a scaling of the object detections ascertained using the object detection algorithm.

24. The method as recited in claim 23, wherein steps b through d are repeated using the respectively adapted object detection algorithm until the quality measure falls below or exceeds a predefined quality value and/or a predefined number of repetitions has been reached.

25. The method as recited in claim 23, wherein the scaling takes place based on properties of the ascertained object detection including size, and/or proportions and/or position in an image.

26. The method as recited in claim 23, wherein the scaling takes place independently of the ascertained object detections, and based on a predefined factor.

27. The method as recited in claim 23, wherein the object detection algorithm is based on a parameterizable model, the parameterizable model being a neural network, the adaptation being based on a change of parameters of the parameterizable model, including the steps: e. ascertaining scaled annotations, based on the ascertained annotations; f. ascertaining object detections using the object detection algorithm; g. assigning the object detections to the scaled annotations, based on the ascertained annotations; h. ascertaining an error between the object detections and the scaled annotations assigned to them; i. reducing the error by adapting the parameters.

28. The method as recited in claim 27, wherein the steps f through i are repeated using the respectively adapted parameters until a predefined error threshold value is fallen below and/or until a predefined number of repetitions is achieved.

29. A non-transitory machine-readable memory medium on which is stored a computer program for calculating a quality measure of a computer-implemented object detection algorithm, for enabling the object detection algorithm for semi-automated, highly-automated or fully-automated robots, the computer program, when executed by a computer, causing the computer to perform the following steps: assigning ascertained object detections to annotations, the object detections and/or the annotations corresponding to bounding boxes; determining deviations including distances of the annotations with respect to their assigned object detections; calculating the quality measure of the object detection algorithm based on the determined deviations, the quality measure representing a probability with which a deviation of an object detection from an annotation assigned to it exceeds or falls below a predefined threshold value.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG. 1 schematically shows a method diagram for determining a quality measure of the object detection algorithm, in accordance with an example embodiment of the present invention.

[0066] FIG. 2 shows by way of example the relationships between annotation, object detection and scaling of an object detection, in accordance with an example embodiment of the present invention.

[0067] FIG. 3 shows by way of example the determination of shifts of corresponding sides of an annotation and of the object detection assigned to it, in accordance with an example embodiment of the present invention.

[0068] FIG. 4 schematically shows a general extreme value distribution including a threshold value, in accordance with an example embodiment of the present invention.

[0069] FIG. 5 chematically shows the sequence for improving a quality measure of an object detection algorithm, in accordance with an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0070] In one first exemplary embodiment, a quality measure of an object detection algorithm is determined with the aid of a computer-implemented method. The object detection algorithm in this case is designed in such a way that it is able to recognize predefined objects by marking these objects with a bounding box in image data recorded with the aid of a camera. This is represented schematically, for example, in FIG. 2a, in which a vehicle including an annotation 201 and a bounding box 202a determined with the aid of the object detection algorithm are depicted.

[0071] In order to be able to determine a measure for the quality of the algorithm or of an accuracy of the object recognition, a set of images is used in this exemplary embodiment, in which objects are annotated, and the object detection algorithm has determined bounding boxes for the annotated objects. This data set is used for the method for determining a quality measure of the object recognition algorithm schematically represented in FIG. 1.

[0072] In step 101 of this method, the object detections, which have been determined with the aid of the object detection algorithm, are assigned to the annotations 201 encompassed by the image data. In this case, an annotation may in general project beyond an associated object detection 202a; this case is shown by way of example in FIG. 2a. The other possibility is that the annotation is completely enclosed by object detection 202b, which is schematically shown in FIG. 2b and FIG. 3. The specific case in which the annotation corresponds exactly to the object detection may be optionally assigned to one of the two categories shown in FIG. 2 for the following steps. The assignment of the annotation to an object detection takes place in this exemplary embodiment via the so-called Intersection over Union, i.e., the ratio of overlap of the two bounding boxes to the area of the union of the two bounding boxes.

[0073] (In alternative exemplary embodiments, at this point the distance of the midpoint of the two bounding boxes may also be used instead, in order to carry out the assignment.)

[0074] In step 102, the smallest deviation is ascertained for each pair of annotation and assigned object detection. The smallest deviation in this case is ascertained from a set of deviations of the object detections from the associated annotations, which is represented schematically in FIG. 3. The deviations in this exemplary embodiment are a respective shift of corresponding sides of an object detection and of the annotation assigned to it. This means that shifts for the left 301, upper 302, right 303 and lower 304 corresponding sides are ascertained. The shifts in this case are always in parallel to the corresponding side of annotation 201. In addition, the sign of a shift indicates the direction in which the object detection is shifted from annotation 201. In the event that annotation 201 projects at one side from the object detection, the corresponding shift is negative 301. Otherwise, the shift is positive 302, 303, 304. The smallest of the four shifts 301, 302, 303, 304 is subsequently ascertained.

[0075] In step 103, the quality measure is calculated. For this purpose, a model 401 that represents the distribution of the deviations is ascertained from the deviations ascertained in step 102. In this exemplary embodiment, a general extreme value distribution is used for this purpose.

[0076] The parameters of the general extreme value distribution are ascertained by using the method of Maximum Likelihood Estimation.

[0077] To calculate the quality measure, the cumulative distribution function of the general extreme value distribution is evaluated 402 at value 0. This step is schematically represented in FIG. 4. In the figure, the shift is plotted on the x-axis and the probability density of the extreme value distribution is plotted on the y-axis. The result of the evaluation corresponds to the probability that an annotation projects from the object detection assigned to it.

[0078] In one second exemplary embodiment, the same steps are carried out as in the first exemplary embodiment, in step 103, however, a Bayesian parameter estimation is carried out instead of the Maximum Likelihood Estimation.

[0079] In one third exemplary embodiment, which is schematically shown in FIG. 5, an object detection algorithm is changed in such a way that it becomes safer.

[0080] For this purpose, annotations are generated manually in step 501 for a data set of camera-based sensor data. Alternatively, the annotations may also be semi-automatically or fully-automatically generated.

[0081] In step 502, object detections are ascertained for the sensor data using the object detection algorithm, which are then assigned to the annotations in step 503. The assignment in this case takes place as in the first exemplary embodiment.

[0082] The quality measure of the object detection algorithm is determined in step 504. This takes place as in the first exemplary embodiment.

[0083] In step 505, the object detection algorithm is adapted in such a way that the probability of an annotation projecting from the object detection assigned to it becomes smaller. For this purpose, all object detections are scaled using a fixed factor in such a way that they enclose the annotation assigned to it.

[0084] In one fourth exemplary embodiment, the same steps are carried out as in the third exemplary embodiment, however, LIDAR-based sensor data are used instead of camera-based sensor data. The remaining steps proceed similarly.

[0085] In one fifth exemplary embodiment, the same steps proceed as in the third exemplary embodiment, step 505 being modified as follows: the object detections are scaled using a fixed factor and the quality measure for the scaled object detections is calculated. If the quality measure does not meet a predefined threshold value, the object detections already scaled are scaled using a factor in such a way that the object detection becomes greater. This adaptation of the size with the aid of a scaling factor is carried out until the quality measure falls below a predefined probability.

[0086] In one sixth exemplary embodiment, the object detection algorithm is based on a neural network. The same steps are carried out as in the third exemplary embodiment, step 505 being modified as follows: the neural network is trained using sensor data and annotations of a second data set in such a way that it outputs intrinsically larger object detections. For this purpose, the annotations of the second data set are scaled in such a way that they become larger. During subsequent training using the scaled annotations, the neural network then learns to predict the larger object detections. After the training, the changed neural network is applied again to the first data set and the quality measure is newly determined. If the quality measure is above a predefined probability value, the neural network is trained on the second data set using even larger scaled annotations. The adaptation of the neural network and evaluation of the quality measure is repeatedly carried out until the quality measure falls below the predefined probability value.