Method and a Device for Assigning a Bounding Box to an Object

Abstract

A method is provided for assigning a bounding box to an object in an environment of a vehicle. Data related to objects located in the environment of the vehicle are acquired via a sensor. Based on the data, a respective spatial location and a respective size of a plurality of preliminary bounding boxes are determined such that each preliminary bounding box covers one of the objects at least partly. A respective velocity of each preliminary bounding box is estimated based on the data. A subset of the plurality of preliminary bounding boxes being related to a respective one of the objects is selected, where the subset is selected based on the respective velocity of each of the preliminary bounding boxes. A final bounding box is assigned to the respective one of the objects by merging the preliminary bounding boxes of the corresponding subset.

Claims

1. A computer implemented method comprising: acquiring, via a sensor, sensor data related to objects located in an environment of a vehicle; determining, via a processing unit and based on the sensor data, a respective spatial location and a respective size of a plurality of preliminary bounding boxes such that each preliminary bounding box covers one of the objects at least partly; estimating a respective velocity of each preliminary bounding box based on the sensor data; selecting a subset of the plurality of preliminary bounding boxes that are related to a respective one of the objects, the subset selected based on the respective velocity of each of the preliminary bounding boxes; and assigning a final bounding box to the respective one of the objects by merging the preliminary bounding boxes included in the subset.

2. The method according to claim 1, wherein: a direction of the velocity is determined for each preliminary bounding box; a respective size-modified box is generated for each preliminary bounding box by: shrinking the preliminary bounding box in the direction of the velocity and in a lateral direction perpendicularly to the direction of the velocity; and extending the preliminary bounding box in the direction of the velocity based on an absolute value of the velocity estimated for the preliminary bounding box; and preliminary bounding boxes having overlapping size-modified boxes are selected for the respective subset.

3. The method according to claim 2, wherein the shrinking of the preliminary bounding box includes reducing a length and a width of the respective preliminary bounding box by a same factor.

4. The method according to claim 2, wherein the extending of the preliminary bounding box includes multiplying a length of the respective preliminary bounding box in the direction of the velocity by the absolute value of the velocity and by an extend factor.

5. The method according to claim 2, wherein selecting the subset of the plurality of preliminary bounding boxes includes applying a maximal connected subgraph algorithm to all preliminary bounding boxes of the plurality of preliminary bounding boxes.

6. The method according to claim 2, wherein two of the size-modified boxes are regarded as overlapping if their intersection over union is greater than a predefined threshold.

7. The method according to claim 1, wherein: a centroid is estimated for the preliminary bounding boxes of a respective subset; farthest vertices having a greatest distance to the centroid are determined for the preliminary bounding boxes of the subset; and the final bounding box is determined based on the farthest vertices.

8. The method according to claim 1, wherein a velocity of the final bounding box is determined by averaging estimated velocities over a corresponding subset of preliminary bounding boxes.

9. The method according to claim 1, further comprising: determining an attribute for each of the preliminary bounding boxes; and based on the attribute, determining whether a selected preliminary bounding box is excluded from the corresponding subset.

10. The method according to claim 9, wherein: a probability distribution is assigned to values of the attribute of the preliminary bounding boxes belonging to a respective subset; and the selected preliminary bounding box is excluded from the corresponding subset if a probability assigned to the value of the attribute of the selected preliminary bounding box is smaller than a predetermined threshold.

11. The method according to claim 1, wherein a machine learning algorithm is applied to the sensor data for determining the spatial location and the size of each preliminary bounding box.

12. A device comprising: a sensor configured to acquire sensor data related to objects located in an environment of a vehicle; and a processing unit configured to: determine, based on the sensor data, a respective spatial location and a respective size of a plurality of preliminary bounding boxes such that each preliminary bounding box covers one of the objects at least partly; estimate a respective velocity of each preliminary bounding box based on the sensor data; select a subset of the plurality of preliminary bounding boxes being related to a respective one of the objects, the subset selected based on the respective velocity of each of the preliminary bounding boxes; and assign a final bounding box to the respective one of the objects by merging the preliminary bounding boxes included in the subset.

13. The device according to claim 12, wherein the sensor includes at least one of a radar sensor or a Lidar sensor.

14. The device according to claim 12, wherein: a direction of the velocity is determined for each preliminary bounding box; a respective size-modified box is generated for each preliminary bounding box by: shrinking the preliminary bounding box in the direction of the velocity and in a lateral direction perpendicularly to the direction of the velocity; and extending the preliminary bounding box in the direction of the velocity based on an absolute value of the velocity estimated for the preliminary bounding box; and preliminary bounding boxes having overlapping size-modified boxes are selected for the respective subset.

15. The device according to claim 14, wherein the processing unit is configured to shrink preliminary bounding box by reducing a length and a width of the respective preliminary bounding box by a same factor.

16. The device according to claim 14, wherein the processing unit is configured to extend the preliminary bounding box by multiplying a length of the respective preliminary bounding box in the direction of the velocity by the absolute value of the velocity and by an extend factor.

17. The device according to claim 14, wherein the processing unit is configured to select the subset of the plurality of preliminary bounding boxes includes applying a maximal connected subgraph algorithm to all preliminary bounding boxes of the plurality of preliminary bounding boxes.

18. The device according to claim 14, wherein two of the size-modified boxes are regarded as overlapping if their intersection over union is greater than a predefined threshold.

19. The device according to claim 12, wherein the processing unit is configured to: estimate a centroid for the preliminary bounding boxes of a respective subset; determine farthest vertices having a greatest distance to the centroid for the preliminary bounding boxes of the subset; and determine the final bounding box based on the farthest vertices.

20. Non-transitory computer readable medium comprising instructions that, when executed by a processor, cause the processor to perform operations including: determining, based on sensor data acquired by a sensor and related to objects located in an environment of a vehicle, a respective spatial location and a respective size of a plurality of preliminary bounding boxes such that each preliminary bounding box covers one of the objects at least partly; estimating a respective velocity of each preliminary bounding box based on the sensor data; selecting a subset of the plurality of preliminary bounding boxes that are related to a respective one of the objects, the subset selected based on the respective velocity of each of the preliminary bounding boxes; and assigning a final bounding box to the respective one of the objects by merging the preliminary bounding boxes included the subset.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following drawings, showing schematically:

[0041] FIG. 1 depicts an overview of a device according to the disclosure;

[0042] FIG. 2 depicts an example for predicting bounding boxes according to the background art;

[0043] FIG. 3 depicts an example for clustering bounding boxes according to the disclosure;

[0044] FIGS. 4A, 4B, and 4C depict challenging scenarios for clustering bounding boxes;

[0045] FIGS. 5A, 5B, and 5C depict schemes for separating and merging bounding boxes according to the disclosure;

[0046] FIGS. 6A and 6B depict examples for removing outliers from a group of bounding boxes; and

[0047] FIG. 7 depicts a scheme for merging bounding boxes via farthest vertices.

DETAILED DESCRIPTION

[0048] FIG. 1 depicts a schematic overview of a device 11 according to the disclosure. In aspects, the device is installed in a vehicle 10. The device 11 includes a sensor 13 having an instrumental field of view 14 and a processing unit 15.

[0049] The sensor 13 is a radar sensor and/or a Lidar sensor and is configured for monitoring the environment of the vehicle 10. That is, the sensor 13 is configured to provide data related to objects 17, which are located in the environment of the vehicle 10. The data provided by the sensor 13 are transferred to the processing unit 15 that is configured to generate a respective bounding box 19 related to each object 17.

[0050] It is noted that the objects 17 are depicted in FIG. 1 in a sideview for illustration purposes only. Actually, the data provided by the radar sensor and/or Lidar sensor 13 are presented in bird's eye view, e.g. from a perspective when looking from a large height at the vehicle 10 and the objects 17 including their bounding boxes 19. In bird's eye view, the bounding boxes 19 are rectangles for each of which the spatial location is uniquely defined by a pair of vertices 21 that are positioned diagonally opposite to each other.

[0051] FIG. 2 depicts an example for predicting bounding boxes for large vehicles via a perception system according to the background art. The ground truth for the respective bounding boxes is denoted by 23 and illustrated by dashed lines. On the left side of FIG. 2, the dashed rectangle 23 represents the ground truth for a truck, whereas on the right side of FIG. 2, the two dashed rectangles 23 represent the ground truth, e.g. the rectangles that are to be predicted, for a bendy bus.

[0052] Usually, a conventional perception system is based on a machine learning algorithm trained by a ground truth comprising a majority of small vehicles, e.g. cars since the number of cars is usually much higher in a typical environment of the vehicle 10 than the number of trucks and busses. As a consequence, the conventional perception system predicts a plurality of small bounding boxes 25, each of which covers the ground truth for the bendy bus only partly. As can be seen in FIG. 2, each of the plurality of predicted bounding boxes 25 has a bad quality in comparison to the ground truth 23, where the bendy bus on the right side of FIG. 2 is even more difficult to predict. Some of the predicted bounding boxes 25 have an orientation that does not fit at all to the orientation of the ground truth rectangles 23.

[0053] Although each single predicted bounding box 25 has a bad quality when compared with the ground truth 23, the entire set of predicted bounding boxes 25 is able to cover the ground truth 23 much better. This is a motivation for one of the principles according to the present disclosure, e.g. merging a certain number of small bounding boxes 25 in order to generate one big bounding box.

[0054] This is illustrated in FIG. 3. The same ground truth 23 for a truck and for a bendy bus, respectively, is shown two times by dashed lines. On the respective left side, preliminary bounding boxes 27 are depicted, which correspond to the bounding boxes 25 as shown in FIG. 2 and which represent the output of a perception system according to the background art. That is, the preliminary bounding boxes 27 are the output of a machine learning algorithm, which has been trained with a ground truth including mostly smaller vehicles like cars.

[0055] Based on the group of preliminary bounding boxes 27, a final bounding box 29 is generated, which is shown on the respective right side in FIG. 3. In order to generate the final bounding boxes 29, the so-called maximal connected subgraph algorithm is applied to the entirety of preliminary bounding box 27, which are identified within the instrumental field of view 14 of the sensor 13 (see FIG. 1). Via this algorithm, clusters or subsets of preliminary bounding boxes 27 are identified that are assumed to belong to a certain object 17. If the truck and the bendy bus, which generate the ground truth 23 in FIGS. 2 and 3, were located in the same field of view 14 of the sensor 13, the maximal connected subgraph algorithm would generate two subsets or clusters of preliminary bounding boxes 27 for each of these objects 17, e.g. for the truck and for the bendy bus in the examples of FIGS. 2 and 3.

[0056] In the maximal connected subgraph algorithm, one important step is the computation of a so-called adjacency matrix, which represents the similarity between every pair of preliminary bounding boxes 27. The intersection over union (IoU), which is known in the art, is used as a measure for the similarity of two preliminary bounding boxes 27. Usually, a threshold is defined for the intersection over union in order to decide whether two objects, e.g. two preliminary bounding boxes 27, belong together or not. For the present case, e.g. for a plurality of preliminary bounding boxes 27, it is not easy to define a general threshold, as will be explained in detail below. One reason is that one big object has to be distinguished from two or more smaller objects, which are located closely to each other, e.g. having a small gap between them only such that the preliminary bounding boxes 27 overlap for both objects. After defining the clusters or subsets of the preliminary bounding boxes 27, which belong to a certain object 17 in the environment of the vehicle 10 (see FIG. 1), the final bounding boxes 29 are generated by determining farthest vertices 39 (see FIG. 7) for the respective cluster or subset of preliminary bounding boxes 27. This will also be explained in detail below.

[0057] Challenging scenarios for generating a proper final bounding box 29 are depicted in FIG. 4. In FIGS. 4A and 4B, the ground truth 23 (illustrated by dashed lines) includes the bounding boxes for two different smaller vehicles that are located close to each other in driving direction (FIG. 4A) and in lateral direction (FIG. 4B), respectively. In both cases, the preliminary bounding boxes 27 overlap for the two objects of the ground truth 23. As a consequence, the entire subset of preliminary bounding boxes 27 is regarded as one big cluster, and therefore one single big final bounding box 29 is generated. That is, the perception system is unable to resolve the two objects represented by the ground truth 23 for the examples in FIGS. 4A and 4B.

[0058] In the example of FIG. 4C, the predicted or preliminary bounding boxes 27 are not connected to each other in the middle of the subgraph. This might be caused by a low spatial resolution of the sensor 13, e.g. the radar sensor and/or the Lidar sensor (see FIG. 1). As a consequence, the preliminary bounding boxes 27 as shown in FIG. 4B is separated into two clusters or subsets although the ground truth 23 represents one big vehicle, e.g. a truck. Hence, two final bounding boxes 29 are generated in the example of FIG. 4C suggesting that two objects are present although the ground truth 23 represents one big object.

[0059] In order to overcome such concerns, the device 11 and a method according to the disclosure additionally consider the respective velocity of assumed objects that is reflected in the respective velocity of the preliminary bounding boxes 27. Generally, the velocity of the object 17 (see FIG. 1) can also be determined based on the data of the sensor 13, e.g. based on a sequence of radar and/or Lidar scans. Accordingly, a respective velocity is assigned to each of the preliminary bounding boxes 27, which are located in the region of the respective object 17.

[0060] Along the moving direction of the objects 17, e.g. along the direction of their velocity vector, it is quite unlikely for two fast moving vehicle that they are close to each other. In other words, it is more likely that preliminary bounding boxes 27, which are close to each other in the moving direction of an object, belong to one big object even if the preliminary bounding boxes do not overlap (as shown in FIG. 4C and FIG. 5C in the left subgraph). In contrast, in a lateral direction perpendicular to the moving direction of one or more vehicles, it is more likely that preliminary bounding boxes 27 belong to different objects although the preliminary bounding boxes may overlap (as shown in FIG. 4B and FIG. 5B in the respective left subgraph).

[0061] In order to consider the velocity of objects when generating bounding boxes, a shrink step and an extend step are applied to the preliminary bounding boxes 27. The shrink step is performed in moving direction and in lateral direction, whereas the extend step is performed along the moving direction only.

[0062] Before performing the shrink step and the extend step, a respective velocity vector 31 is estimated for each preliminary bounding box 27 based on the data from the sensor 13 (see FIG. 1). Examples for the velocity vector 31 are shown in FIG. 5. For FIG. 5A and FIG. 5B, it is assumed that the velocity vector 31 has a small absolute value, whereas for the example of FIG. 5C, the absolute value of the velocity vector 31 is assumed to be much larger than the absolute value of the velocity vector 31 of FIGS. 5A and 5B. The shrink step and the extend step of the respective preliminary bounding box 27 are defined as follows:

Shrink step: L=L*shrink factor,W=W*shrink factor;

Extend step: L=L*velocity*extend factor

[0063] L and W denote the length and the width of the respective preliminary bounding box 27, e.g. in longitudinal or moving direction and in lateral direction, respectively. The shrink factor is a number between 0 and 1. In the formula of the extend step, the term “velocity” denotes the absolute value of the respective velocity vector 31. The product of the velocity and the extend factor is a number greater than 1.

[0064] The shrink factor and the extend factor may be suitably defined during a training procedure for the entire device 11. That is, both factors are determined empirically by applying the device 11 and the method to realistic scenarios for which the device 11 and the method are intended to be used.

[0065] For the examples as shown in FIGS. 5A and 5B, the preliminary bounding boxes 27 as shown in the left subgraph are first shrinked in the moving direction, e.g. along the velocity vector 31, and in the lateral direction, e.g. perpendicularly to the velocity vector 31, by the same shrink factor. Thereafter, the preliminary bounding boxes 27 are extended in moving direction along the velocity vector 31 only, where the extend factor is multiplied by the absolute value of the velocity vector 31. As a result of the shrink step and the extend step, size-modified boxes 33 are generated, which are shown in the middle subgraph of FIGS. 5A, 5B and 5C. Based on the size-modified boxes 33, it is decided which of the preliminary bounding boxes 27 form a respective subset that is related to a certain assumed object.

[0066] In the examples as shown in FIGS. 5A and 5B, the plurality of preliminary bounding boxes 27 as shown in the respective left subgraph separates into two different subsets of size-modified boxes 33 as can be recognized in the middle subgraph. Due to the low absolute value of the velocity vector 31 for the examples of FIGS. 5A and 5B, the shrink step prevails over the extend step since the extend step depends on the absolute value of the velocity vector 31 and the shrink step is independent from the velocity. Due to the prevalence of the shrink step, the two different objects represented by the ground truth 23 (illustrated by dashed lines) can be resolved by the device 11 and the method, and final bounding boxes 29 can be assigned to the objects that cover these two objects properly.

[0067] In contrast, for the example as shown in FIG. 5C, the absolute value of the velocity vector 31 is assumed to be much larger, and therefore the extend step prevails over the shrink step. Although the preliminary bounding boxes 27 do not overlap in the middle of the subgraph as shown on the left side in FIG. 5C, the prevalence of the extend step leads to size-modified boxes 33 that overlap for the entire subset. In other words, there is no separation for the size-modified boxes 33 anymore, which was present for the preliminary bounding boxes 27. As a result, one final bounding box 29 is generated for the example as shown in FIG. 5C only which properly covers the ground truth 23 (illustrated by dashed lines), e.g. one large object like a truck.

[0068] It is noted that the size-modified boxes 33 in the middle subgraphs of FIG. 5 are used for identifying those preliminary bounding boxes 27 only which have to be grouped in a certain subset. A preliminary bounding box 27 is selected for a certain subset and assumed to belong to a certain object if the corresponding size-modified box 33 overlaps with further size-modified boxes 33 belonging to the same subset. In other words, preliminary bounding boxes 27 are grouped in a subset for generating one of the final bounding boxes 29 if their size-modified boxes 33 overlap at least partly. The size-modified boxes 33 are therefore used for the decision of which of the preliminary bounding boxes 27 belong to a certain subset, and they are not used for generating the final bounding boxes 29. Instead, the preliminary bounding boxes 27 of an identified subset are still used for generating the respective final bounding box 29, as will be explained below in context of FIG. 7.

[0069] The scheme for determining proper final bounding boxes 29 as described so far might be deteriorated by outliers of the prediction of the preliminary bounding boxes 27. in FIG. 6, two examples for such outliers are illustrated as outlier boxes 37. In the example of FIG. 6A, the outlier box 37 erroneously connects two groups of preliminary bounding boxes 27, which actually belong to two different objects represented by the respective ground truth 23 (illustrated by dashed lines). In the example of FIG. 6B, the outlier box 37 has an exceptional orientation in comparison to the further preliminary bounding boxes 27 related to the same object.

[0070] Therefore, an additional outlier removal procedure is applied to the preliminary bounding boxes 27, which are the result of the original prediction. For the outlier removal, the respective distribution is modeled for the attributes of the preliminary bounding boxes, where these attributes include the size, the orientation and/or the velocity of the preliminary bounding boxes 27. That is, based on values for these attributes, a probability distribution is modeled. Thereafter, a probability value can be assigned to each attribute of each preliminary bounding box 27. If the respective probability value of at least one of the attributes of a certain preliminary bounding box 27 is lower than a predetermined threshold, the corresponding preliminary bounding box 27 is considered as an outlier box 37. The outlier boxes 37 are excluded from any subset selection and are therefore not considered for determining the final bounding boxes 29.

[0071] When all clusters or subsets of preliminary bounding boxes 27 are identified, the preliminary bounding boxes 27 belonging to the respective cluster or subset are merged as is schematically illustrated in FIG. 7. For merging the subset of preliminary bounding boxes 27, the vertices 21 (see also FIG. 1) of the preliminary bounding boxes 27 are considered. Since the bounding boxes are represented as rectangles, each preliminary bounding box 27 includes four vertices 21.

[0072] In order to merge a subset of preliminary bounding boxes 27, so-called farthest vertices 39 are identified for each subset. The farthest vertices 39 have the largest distances, e.g. a larger distance than the further vertices 21, with respect to a centroid of the entire subset of preliminary bounding boxes 27. In the examples of FIG. 7, two farthest vertices 39 are sufficient for identifying the vertices of the respective final bounding box 29.

[0073] In addition, further attributes of the final bounding box 29 can be determined as an average over all preliminary bounding boxes 27 belonging to the subset that has been used for generating the final bounding box 29. For example, the velocity vector of the final bounding box 29 is determined as an average of all velocity vectors 31 determined for the preliminary bounding boxes 27, which have been used for generating the final bounding box 29.