METHOD FOR SPATIAL CHARACTERIZATION OF AT LEAST ONE VEHICLE IMAGE

Abstract

A method is provided for the spatial characterization of at least one vehicle image of image information, wherein the image information encompasses the vehicle image of an external vehicle and an environment image of an environment of the external vehicle. The method comprises: determining a bounding box for the vehicle image, in order to use the bounding box for a delimiting of the vehicle image from the environment image, determining a splitting line for the bounding box, in order to use the splitting line for a partitioning of the vehicle image into at least two vehicle sides, determining the spatial characterization with the aid of the bounding box and the splitting line, wherein at least one evaluation means based on machine learning, especially a neural network, is used for the determining of the bounding box and the splitting line.

Claims

1. A method for a spatial characterization of at least one vehicle image of image information, wherein the image information encompasses a vehicle image of an external vehicle and an environment image of an environment of the external vehicle, comprising: determining a bounding box for the vehicle image, in order to use the bounding box for delimiting the vehicle image from the environment image, determining a splitting line for the bounding box, in order to use the splitting line for partitioning the vehicle image into at least two vehicle sides, and determining the spatial characterization with the aid of the bounding box and the splitting line, wherein the determining of the bounding box and the splitting line is based on machine learning techniques.

2. The method according to claim 1, wherein the spatial characterization is performed as a three-dimensional reconstruction of the external vehicle from the vehicle image.

3. The method according to claim 1, wherein the bounding box is designed to separate the vehicle image rom the environment image in order to fully mask the external vehicle.

4. The method according to claim 1, wherein the splitting line is designed to divide the vehicle image into a front and/or rear view of the external vehicle and a side view of the external vehicle.

5. The method according to claim 1, wherein the splitting line is configured as a vertical line in relation to a ground surface on which the external vehicle is standing.

6. The method according to claim 1, wherein a classification of the external vehicle depicted by the vehicle image is performed, wherein a classification result of the classification is used for determining the spatial characterization.

7. The method according to claim 6, wherein the classification result is used to determine a ratio between width and length of the depicted external vehicle.

8. The method according to claim 1, wherein a three-dimensional back projection of the depicted external vehicle from the vehicle image is carried out for determining the spatial characterization.

9. The method according to claim 1, wherein determining the bounding box and the splitting line includes: applying the machine learning techniques, in the form of at least one neural network, with the image information as input for the neural network, using at least one result from the application of the at least one neural network as the bounding box and as the splitting line.

10. The method according to claim 9, wherein the result comprises information about the position of the splitting line in relation to the corresponding bounding box.

11. The method according to claim 9, wherein the result comprises information as to the position of a side view of the external vehicle relative to the splitting line.

12. The method according to claim 1, wherein the image information is recorded by a vehicle camera of another vehicle before determining the bounding box and the splitting line.

13. The method according to claim 12, wherein the recording of the image information is performed during a drive to monitor the environment of the other vehicle in order to subsequently detect the external vehicle in the environment with the aid of the bounding box and the splitting line.

14. The method according to claim 12, wherein the spatial characterization is performed in order to determine the orientation and/or position of the depicted external vehicle in relation to the other vehicle.

15. A non-transitory computer-readable medium having stored thereon instructions which, when executed by a processing device, cause the processing device to perform a method for a spatial characterization of at least one vehicle image of image information, wherein the image information encompasses a vehicle image of an external vehicle and an environment image of an environment of the external vehicle, the method comprising: determining a bounding box for the vehicle image, in order to use the bounding box for delimiting the vehicle image from the environment image, determining a splitting line for the bounding box, in order to use the splitting line for partitioning the vehicle image into at least two vehicle sides, and determining the spatial characterization with the aid of the bounding box and the splitting line, wherein the determining of the bounding box and the splitting line is based on machine learning techniques.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0034] Further advantages, features and details will emerge from the following specification, in which embodiments are described in detail, making reference to the drawings.

[0035] FIG. 1 shows a schematic representation of vehicles to illustrate methods described herein.

[0036] FIG. 2 shows a schematic representation of image information.

[0037] FIG. 3 shows a schematic representation of a bounding box and a splitting line.

[0038] FIGS. 4-8 show schematic representations to illustrate steps of methods described herein.

DETAILED DESCRIPTION

[0039] In the following figures, the same technical features, even in different embodiments, make use of the identical reference numbers.

[0040] FIG. 1 shows schematically a situation in which image information 200 can be recorded by a vehicle camera 5 of a vehicle 2 (hereinafter also called the ego vehicle 2, for better distinguishing). For this, the vehicle camera 5 can record the environment 6 with at least one other vehicle 1, i.e., an external vehicle 1. In the example, the rear region 20 of the external vehicle 1 is recorded with priority and only part of the side region 21 of the external vehicle 1 is recorded. The vehicles 1, 2 are located on a ground surface 8, so that this can be assumed, for simplicity, to be parallel to the horizontal. Next, the image information 200 can be transmitted digitally to a processing device 3 in order to carry out the method described herein. The method described herein may serve for performing a spatial characterization of at least one vehicle image 30 of the external vehicle 1 in the image information 200, where the image information 200 encompasses the vehicle image 30 of the external vehicle 1 and an environment image 40 of an environment 6 of the external vehicle 1.

[0041] According to FIG. 6, a first step 101 of the method may involve determining a bounding box 230 for the vehicle image 30, in order to use the bounding box 230 for a delimiting of the vehicle image 30 from the environment image 40. After this, in a second step 102 of the method, a determination of a splitting line 240 is done for the bounding box 230, in order to use the splitting line 240 for a partitioning of the vehicle image 30 into at least two vehicle sides 20, 21, especially into a front and/or rear region 20 of the external vehicle 1 and a side region 21 of the external vehicle 1. Next, in a third step 103 of the method, the spatial characterization is determined with the aid of the bounding box 230 and the splitting line 240.

[0042] Thus, with a method as described herein it is possible to estimate the 3D position of an external vehicle 1 from pictures (i.e., the image information 200), characterized by the bounding box 230 and the splitting line describing the borders between the front/rear and side view 20, 21 (if both are visible). A characterization of the image information 200 is shown in FIG. 2. For this, it is possible to use an existing traditional object detector or a traditional method for the object detection, supplemented by introducing an additional parameter dl.sub.x, expressing the position of a splitting line 240 in relation to its bounding box 230, in order to estimate the bounding box 230 with its corresponding splitting lines 240:

i dl.sub.x=pos.sub.x(dividing line)−pos.sub.x(bounding box center),

where dl.sub.x∈[pos.sub.x(left border),pos.sub.x(right border)] are the parameters for the corresponding bounding box 230.

[0043] FIGS. 7 and 8 show that at least one evaluation means 210 can be employed for determining the bounding box 230 and the splitting line 240—here for example in the form of a neural network 210. For this, first of all one can apply 104 the at least one neural network 210 using the image information 200 as input 200 for the neural network 210. After this, one will use 105 at least one result 220 from the application 104 of the at least one neural network 210 as the bounding box 230 and as the splitting line 240. In other words, the result 220 can be used to determine the bounding box 230 and the splitting line 240. It is possible for the result 220 to contain information dl.sub.x about the position of the splitting line 240 in relation to the corresponding bounding box 230. The result 220 can also contain information dl.sub.class as to the position of a side view 21 of the external vehicle 1 relative to the splitting line 240.

[0044] The ambiguity of whether the side 21 of the external vehicle 1, i.e., the side view 21, may lie on the left or right of the splitting line 240, can be resolved in various ways. On the one hand, a binary parameter can be used: [0045] dl.sub.class=0, if the side is on the left of the splitting line; or [0046] 1, if the side is on the right of the splitting line.

[0047] Another possibility is to encode the information about the position of the side view 21 in the information about the position of the splitting line 240 dl.sub.x: [0048] dl.sub.x= [0049] pos.sub.x (splitting line)−pos.sub.x (left border), if the side is at left of the splitting line; or [0050] pos.sub.x (splitting line)−pos.sub.x (right border), if the side is at right of the splitting line, where: dl.sub.x∈[−box.sub.width, +box.sub.width].

[0051] Here, zero represents the only visible front/rear side and the +/− width represents the only visible side. In order to achieve the same distribution of values for all object magnitudes, a normalization of dl.sub.x to the width of the corresponding objects can be done. With this definition, an object can be described by:

box=[class, pos.sub.x(center), pos.sub.y(center), width, height, dl.sub.x]

[0052] The described method can be used with a traditional object detector, such as is disclosed in Liu, Wei, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg, “SSD: Single Shot MultiBox Detector,” arXiv:1512.02325 [cs], 7 Dec. 2015 and Shaoqing Ren, Kaiming He, Ross Girshick, Jian Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” arXiv:1506.01497 [cs], 4 Jun. 2015.

[0053] FIG. 2 shows schematically image information 200 where the bounding box 230 and the splitting line 240 are indicated for better comprehension. It can be seen from this that the bounding box 230 can be designed to separate the vehicle image 30 from the environment image 40 in order to mask the external vehicle 1 entirely. It can likewise be seen that the splitting line 240 is designed to divide the vehicle image 30 into a front and/or rear view 20 of the external vehicle 1 and a side view 21 of the external vehicle 1. The splitting line 240 here is configured as a vertical line in relation to a ground surface 8 on which the external vehicle 1 is standing.

[0054] According to FIGS. 3 to 5, the spatial characterization can be performed as a three-dimensional reconstruction of the external vehicle 1 from the vehicle image 30. First of all, one may consider the general case where the front/rear view 20 and the side view 21 are visible in the image information 200. If the bounding box 230 and the corresponding splitting line 240 are indicated in distorted image coordinates, they are at first transformed into virtual camera coordinates. It follows, from the assumption of a parallel arrangement of the external vehicle 1 with respect to the ground surface 8, that for the resulting bounding box 230 the left side, the right side, and the splitting line 240 should be vertical (parallel to the y-axis). This is illustrated in FIG. 3, where the horizontal line 250 and the FOE (focus of expansion) are also indicated. The FOE here is located on different sides of the splitting line 240.

[0055] Furthermore, it is possible to perform a classification of the external vehicle 1 depicted by the vehicle image 30, wherein a classification result of the classification is used to determine the spatial characterization. Moreover, the classification result can be used to determine a ratio between the width and the length of the depicted external vehicle 1, i.e., the side ratio.

[0056] Next, a projection of the mentioned vertical left and right side and the splitting line 240 relative to the ground surface 8 can be done, as represented in FIGS. 4 and 5. From this, the following computations are possible, where α.sub.0 is indicated by reference 300, α.sub.1 by reference 301, β.sub.0 by reference 302, β.sub.1 by reference 303, c.sub.0 by reference 304 and c.sub.1 by reference 305:

[00001]$\begin{array}{cc}{c}_0/c_1=r,\mathrm{with}r\mathrm{being}\mathrm{the}\mathrm{estimated}\mathrm{side}\mathrm{ratio}& \left(1\right)\end{array}$$\begin{array}{cc}\sin \left({\alpha }_0\right)=c_0.Math.\cos \left({\beta }_0\right)& \left(2\right)\end{array}$$\begin{array}{cc}\sin \left({\alpha }_1\right)=c_1.Math.\cos \left({\beta }_1\right)& \left(3\right)\end{array}$$\begin{array}{cc}{\beta }_0+{\beta }_1-{\alpha }_0-{\alpha }_1=\frac{\pi }{2}& \left(4\right)\end{array}$$\mathrm{and}$${\beta }_0={\tan }^{-1}\left(\tan \left({\alpha }_0+{\alpha }_1\right)+r.Math.\frac{\sin \left({\alpha }_1\right)}{\sin \left({\alpha }_0\right).Math.\cos \left({\alpha }_0+{\alpha }_1\right)}\right).$

[0057] From β.sub.0 and the angle γ between the z-axis and the left viewing direction it is possible to calculate the absolute orientation δ of the left side by

[00002]$\gamma +{\beta }_0-\frac{\pi }{2}.$

[0058] Next, for the 3D reconstruction, the left FOE can be calculated by tan(δ) and the right FOE by tan

[00003]$\left(\delta +\frac{\pi }{2}\right).$

[0059] According to FIG. 5, for the determination of the spatial characterization it is possible to perform a three-dimensional back projection of the depicted external vehicle 1 from the vehicle image 30. It can be assumed here that the horizontal 250 runs through the bounding box 230. In order to ascertain the height of the back projection, it can be assumed that the bounding box 230 touches the external vehicle 1 on top or bottom at the splitting line 240, if both FOE lie on different sides of the splitting line 240, as in FIG. 3. On the other hand, if both FOE lie on the left or right side, it can be assumed that the bounding box 230 touches the external vehicle 1 at upper and lower right, or at upper and lower left. Moreover, FIG. 5 shows the special case where only one side (front region 20, rear region 20 or side region 21) of the external vehicle 1 is visible. In this case, there is uncertainty amounting to half of the angle 301. In this case, it can be assumed that the external vehicle 1 is situated perpendicular to the angle bisector (dotted line).

[0060] The foregoing explanation of the embodiments describes embodiments in the context of examples. Of course, individual features of the embodiments, if technically feasible, can be freely combined with each other, without leaving the scope of the present disclosure.

[0061] Aspects of the various embodiments described above can be combined to provide further embodiments. These and other changes can be made to the embodiments in light of the above-detailed description. In general, in the following claims, the terms used should not be construed to limit the claims to the specific embodiments disclosed in the specification and the claims, but should be construed to include all possible embodiments along with the full scope of equivalents to which such claims are entitled.