METHOD FOR ESTIMATING A RELATIVE POSITION OF AN OBJECT IN THE SURROUNDINGS OF A VEHICLE AND ELECTRONIC CONTROL UNIT FOR A VEHICLE AND VEHICLE

Abstract

A relative position of an object in the surroundings of a vehicle is estimated based on a two-dimensional camera image. A control unit determines an object contour of the object from the camera image and determines at least one digital object template that represents the object based on the object contour. The control unit forward projects the at least one object template from respective different positions onto an image plane of the camera image. Each forward-projected object template yields a respective two-dimensional contour proposal, and the control unit compares the contour proposals with the object contour of the object.

Claims

1-15. (canceled)

16. A method for determining a relative position of an object in a surroundings of a vehicle, the method comprising: determining, by an electronic control unit of the vehicle, an object contour of the object from a two-dimensional (2D) camera image taken by a camera of the vehicle; back-projecting the object contour into a three-dimensional virtual space that represents the surroundings, the back-projected object contour describing a virtual three-dimensional (3D) frustum reaching from the camera into the surroundings; determining at least one digital object template that represents the object; positioning the at least one digital object template at a plurality of predefined positions inside the frustum, the plurality of predefined positions being determined according to a predefined positioning rule; forward projecting the at least one digital object template from the plurality of predefined positions onto an image plane of the camera image, each forward-projected digital object template yielding a respective 2D contour proposal; comparing the 2D contour proposals with the object contour of the object; selecting, based on the comparing, at least one 2D contour proposal among the 2D contour proposals that fulfills a predefined matching criterion with respect to the object contour, as a respective best-fit contour proposal; and determining the relative position of the object based on the respective predefined position of each corresponding digital object template that leads to the at least one 2D contour proposal selected as the respective best-fit contour proposal.

17. The method according to claim 16, wherein the object contour corresponds to a 2D bounding box having a rectangular shape.

18. The method according to claim 16, wherein each digital object template represents a specific object type, an object size, and a spatial orientation.

19. The method according to claim 16, wherein determining the at least one digital object template comprises using a plurality of digital object templates, and at least two digital object templates among the plurality of digital object templates represent different object types and/or at least two digital object templates among the plurality of digital object templates represent different object sizes and/or at least two digital object templates among the plurality of digital object templates represent a same object type, but different potential spatial orientations of the object.

20. The method according to claim 16, wherein determining the at least one digital object template includes, determining, by an object classification module, an object type and/or an object size and/or a spatial orientation, of the object based on the camera image.

21. The method according to claim 16, wherein each digital object template is a 3D bounding box.

22. The method according to claim 16, wherein the predefined positioning rule comprises using a plurality of predefined positions for each digital object template, the plurality of predefined positions being arranged in a predefined pattern, and in the 3D virtual space a ground plane, on which the vehicle and/or the object are arranged, is represented and the plurality of predefined positions are arranged inside the frustum on the ground plane and/or on a plane parallel to the ground plane.

23. The method according to claim 16, wherein the predefined matching criterion comprises that the respective 2D contour proposal and the object contour overlap at least to a predefined minimum overlap value.

24. The method according to claim 16, wherein selecting the at least one 2D contour proposal among the 2D contour proposals comprises selecting only some of the 2D contour proposals that fulfill the predefined matching criterion, and selecting only some of the 2D contour proposals that fulfill the predefined matching criterion comprises applying a similarity criterion to the corresponding digital object templates and determining at least one group of similar corresponding digital object templates and only selecting one digital object template out of each group and selecting the associated best-fit contour proposal of each selected digital object template.

25. The method according to claim 16, wherein determining the relative position of the object comprises applying an artificial neural network (ANN) to each best-fit contour proposal, the ANN being trained to provide correction data for each predefined position of each corresponding digital object template that leads to the at least one 2D contour proposal, to increase a degree of matching between the best-fit contour proposal and the object contour.

26. The method according to claim 25, further comprising: calculating corrected positions for each predefined position of each corresponding digital object template that leads to the at least one 2D contour proposal based on the correction data; and calculating a mean value of the corrected positions as an estimate of the relative position of the object.

27. The method according to claim 16, wherein when none of the 2D contour proposals fulfills the predefined matching criterion based on the comparing, the method further comprises: providing, by an object estimation module, an estimate of an object type, an object size, and a spatial orientation of the object based on the camera image; back-projecting a digital object template of a corresponding object type, object size, and spatial orientation inside the frustum; varying a projection distance and generating, for each value of the projection distance, a 2D contour proposal by forward-projecting the back-projected digital object template onto an image plane of the camera image, until a value for the projection distance is found for which the 2D contour proposal fulfills the predefined matching criterion; and determining the relative position of the object based on the value for the projection distance found for which the 2D contour proposal fulfills the predefined matching criterion.

28. The method according to claim 27, wherein the object estimation module is configured as an artificial neural network.

29. An electronic control unit for a vehicle, comprising: a memory configured to store instructions; and a processor configured to execute the instructions to: determine an object contour of the object from a two-dimensional (2D) camera image taken by a camera of the vehicle, back-project the object contour into a three-dimensional virtual space that represents the surroundings, the back-projected object contour describing a virtual three-dimensional (3D) frustum reaching from the camera into the surroundings, determine at least one digital object template that represents the object, position the at least one digital object template at a plurality of predefined positions inside the frustum, the plurality of predefined positions being determined according to a predefined positioning rule, forward project the at least one digital object template from the plurality of predefined positions onto an image plane of the camera image, each forward-projected digital object template yielding a respective 2D contour proposal, compare the 2D contour proposals with the object contour of the object; select, based on the comparison, at least one 2D contour proposal among the 2D contour proposals that fulfills a predefined matching criterion with respect to the object contour, as a respective best-fit contour proposal, and determine a relative position of the object based on the respective predefined position of each corresponding digital object template that leads to the at least one 2D contour proposal selected as the respective best-fit contour proposal.

30. The electronic control unit according to claim 29, wherein the object contour corresponds to a 2D bounding box having a rectangular shape.

31. The electronic control unit according to claim 29, wherein each digital object template represents a specific object type, an object size, and a spatial orientation.

32. The electronic control unit according to claim 29, wherein the predefined matching criterion comprises that the respective 2D contour proposal and the object contour overlap at least to a predefined minimum overlap value.

33. A motor vehicle, comprising: a two-dimensional camera; and the electronic control unit according to claim 29.

34. The motor vehicle according to claim 33, wherein the object contour corresponds to a 2D bounding box having a rectangular shape, and each digital object template represents a specific object type, an object size, and a spatial orientation.

35. The motor vehicle according to claim 33, further comprising: a driver assistance system to receive the relative position of the object from the electronic control unit, and to control the vehicle based on the relative position of the object.

Description

[0034] In the following, an exemplary implementation of the invention is described. The figures show:

[0035] FIG. 1 a schematic illustration of a motor vehicle according to the invention;

[0036] FIG. 2 a flow diagram of an embodiment of the inventive method as it can be performed by an electronic control unit of the vehicle of FIG. 1;

[0037] FIG. 3 a 2D camera image as it may be taken by a camera of the vehicle of FIG. 1;

[0038] FIG. 4 a sketch of a three-dimensional virtual frustum and an object template positioned at a position inside the frustum;

[0039] FIG. 5 a schematic illustration of the object template;

[0040] FIG. 6 a schematic illustration of a two-dimensional contour proposal generated from the object template of FIG. 4 and FIG. 5;

[0041] FIG. 7 a schematic illustration of several two-dimensional contour proposals as they might be generated by at least one object template at different positions;

[0042] FIG. 8 a schematic illustration of best-fit contour proposals that were chosen from all contour proposals by means of a matching criterion;

[0043] FIG. 9 a schematic illustration of a selection of best-fit contour proposals that may result from applying a selection criterion; and

[0044] FIG. 10 a schematic illustration of all those object templates at their respective positions inside the frustum that belong to the best-fit contour proposals of FIG. 9 and an illustration of correction data.

[0045] The embodiment explained in the following is a preferred embodiment of the invention. However, in the embodiment, the described components of the embodiment each represent individual features of the invention which are to be considered independently of each other and which each develop the invention also independently of each other and thereby are also to be regarded as a component of the invention in individual manner or in another than the shown combination. Furthermore, the described embodiment can also be supplemented by further features of the invention already described.

[0046] In the figures identical reference signs indicate elements that provide the same function.

[0047] FIG. 1 shows a motor vehicle 10 that can be, e.g., a passenger vehicle or a truck. The vehicle 10 may drive on a ground plane 11, which can be, e.g., a road. While driving on the ground plane 11, an object 12 may appear in the surroundings 13 of the vehicle 10. For example, as an object 12 another vehicle may approach or pass by the vehicle 10. The vehicle 10 may comprise an electronic control unit 14 which may be designed to estimate a relative position 15 of the object 12 with regard to vehicle 10. FIG. 1 illustrates that the relative position 15 can be, e.g., the centre of object 12 or (not shown) the front end or the back end of the object 12. The relative position 15 may be described by coordinates. For better orientation, a coordinate system 16 is illustrated in FIG. 1 and the following figures. A longitudinal axis of vehicle 10 may correspond to the x-axis, a lateral axis of vehicle 10 may correspond to the y-axis and a vertical axis of vehicle 10 may correspond to the z-axis.

[0048] For estimating the relative position 15 of object 12, control unit 14 may receive from a 2D camera 17 one or more 2D camera images 18. In the following, it is assumed that the estimation of the relative position 15 is performed on a single 2D camera image. From camera 17, an image sensor 19 is shown in order to illustrate an image plane 20 on which an optical lens 21 forward-projects the light from the surroundings 13 onto the image sensor 19. On image sensor 19, the 2D camera image is generated. Based on 2D camera image 18, the electronic control unit may perform a method 22 in order to provide an estimate 23 of the relative position 15. The estimate 23 can be forwarded to, e.g., a driver assistance system which may, e.g., autonomously drive vehicle 10. FIG. 1 also illustrates the field of coverage C of the camera 17. As can be seen from FIG. 1, object 12 is within the field of coverage C and thus camera image 18 will also comprise an image of object 12. However, for estimating the relative position 15, a distance 25, which can be measured or defined along the x-axis or the longitudinal axis of vehicle 10, cannot be directly measured from the 2D camera image 18.

[0049] In order to estimate the relative position 15, electronic control unit 14 may perform method 22. For performing method 22, electronic control unit 14 may comprise a processing unit CPU, which may be based on one or more microprocessors and/or graphical processing units and/or microcontrollers. By means of the processing unit CPU, electronic control unit 14 may operate one or more artificial neural networks ANN.

[0050] FIG. 2 illustrates possible steps of method 22. The single steps of method 22 will be explained in connection with FIGS. 3 to 10.

[0051] In a first step S10, the control unit may determine an object contour of the object 12 from the camera image 18.

[0052] FIG. 3 illustrates how in the camera image 18 the object 12 can be detected. FIG. 3 illustrates that object 12 can be another vehicle which may be driving on the ground plane 11 towards vehicle 10. As an object contour 24, a 2D bounding box BB may be determined. Object 12 can be detected in image 18 on the basis of state of art image processing algorithms which can be, e.g., based on at least one artificial neural network. Additionally or alternatively, an optical flow analysis of several consecutive camera images 18 can be used.

[0053] FIG. 3 and FIG. 4 illustrate step S11 (FIG. 2) which may be performed by the control unit 14.

[0054] FIG. 4 shows a three-dimensional virtual space 26 which may be defined on the basis of the coordinate system 16, which is illustrated by the y-axis, the x-axis and the z-axis. FIG. 4 shows a top view. Starting from the image plane 20, the object contour 24 may be back-projected by a back-projection 27 into the virtual space 26. The virtual space 26 represents the surroundings 13. As the object contour 24 may describe the outer bounds of object 12, the back-projection 27 defines a three-dimensional frustum 28 which encompasses or comprises all possible positions of object 12. FIG. 4 illustrates the true position 15 of object 12 together with the true spatial orientation and true size of object 12. FIG. 3 illustrates another view of the back-projection of the object contour 24 into the surroundings 13 together with the resulting frustum 28. For purposes of orientation, FIG. 3 and FIG. 4 show a virtual backplane 29 of the three-dimensional virtual frustum 28.

[0055] FIG. 4 also illustrates a step S12 of method 22. An object classification module 30′ of electronic control unit may determine at least one digital object template 30 which represents object 12.

[0056] A potential position 31 of for the determined object template 30 may be defined in a step S13 for positioning object template 30. Optionally, for object template 30 and/or for at least one other object template, further potential or possible positions 32 inside frustum 28 may be set. In FIG. 4, from all the possible positions 32 only three are indicated by a reference sign for the sake of clarity. The positions 32 may be chosen on the basis of a predefined positioning rule 33 which may define the positions 32 in a predefined raster or pattern. Additionally, positioning rule 33 may define that positions 32 shall be arranged on the ground plane 11 and/or on at least one parallel plane parallel to ground plane 11. From positions 32, at least some differ in regard to distance 25 to the vehicle 10. A parallel plane may compensate for bumps and/or hills on a road.

[0057] FIG. 5 illustrates steps S12 and S13 further. FIG. 5 shows an object template 30 at one potential position 31. Object template 30 can be, e.g., a 3D bounding box 34. By object template 30, an object type (e.g. vehicle, pedestrian, cyclist) and/or an object size (width 35, height 36, length 37) and/or a spatial orientation 38 (e.g. defined as rotational angle around a vertical axis 39) can be represented or defined.

[0058] FIG. 4 additionally illustrates a step S14 of method 22. The object template 30 may be forward-projected by a forward-projection 40 onto the image plane 20. In other words, an artificial image of the object template 30 at its position 31 may be generated. If the object template 30 is a 3D-bounding box 34, the forward-projection 40 may be performed by only forward-projecting corners 41 of bounding box 34. For the sake of clarity, FIGS. 4 and 5 only show two reference signs for corners 41. The forward-projections 40 may consider optical properties of lens 21 and/or so-called extrinsic and intrinsic parameters of camera 17.

[0059] FIG. 6 illustrates the result of step S14. FIG. 6 shows that additionally to object contour 24, for object template 30 at its position 31 the result of forward-projecting 40 is a two-dimensional contour proposal 42 which shows the outer lines of the artificial or virtual image of object template 30 on image plane 20.

[0060] FIG. 6 also illustrates a step S15 of method 22: The contour proposal 42 may be compared with the object contour 24 of object 12. The comparison may be performed by comparing overlapping areas 43, which is illustrated in FIG. 6 by a hatching pattern. The degree of overlap may be expressed by an overlap value (e.g. percentage of overlap). It may be verified, if the contour proposal 42 fulfils a matching criterion 44 may demand that the overlapping area 43 must be larger than a minimum percentage of the area of object contour 24 and/or contour proposal 42.

[0061] So far, method 22 has been described on the basis of one single object template 30 and its resulting contour proposal 42, if object template 30 is positioned at position 31. However, each single object template 30 is positioned at more than one position 31, as illustrated in FIG. 4 by the additional positions 32.

[0062] FIG. 7 illustrates how forward-projection 40 yields several contour proposals in step S14, one for each object template 30 at each possible position 31, 32. For each contour proposal 42, the matching criterion 44 can be applied. Those contour proposals 42 which fulfil the matching criterion 44 are denoted here as best-fit contour proposals 45′.

[0063] FIG. 8 illustrates that for each best-fit contour proposal 45 the corresponding object template 30 at its respective position 31, 32 can be determined. Each best-fit contour proposal 45 is associated with one object template 30 at a specific position 31, 32. FIG. 8 also illustrates that some object templates 30 may be very similar in size and position (note that FIG. 8 is a top view onto the virtual space 26). To this end, FIG. 8 illustrates two groups 46, wherein each group 46 consists of object templates 30, which match a predefined similarity criterion 47 with regard to position and/or size and/or object type.

[0064] FIG. 9 illustrates how, by means of the similarity criterion 47, the number of best-fit contour proposals 45 can be reduced to the selected best-fit contour proposals 45′, each representing one of the groups 46. As is shown in FIG. 9, not only a single selected best-fit contour proposal 45′ may be left, but more than one.

[0065] FIG. 10 illustrates a step S16 of method 22. FIG. 10 illustrates how the object templates 30 that belong to the selected best-fit contour proposals 45 may indicate two possible positions 31, 32 of object 12. By means of an artificial neural network that may be applied to image 18 and especially the regions of the selected best-fit contour proposals 45 and the object contour 24, in order to determine correction data 48 for each of positions 31, 32 of the object templates 30. By means of correction data 48, for the positions 31, 32 a respective corrected position 49 may be calculated. The corrected position 49 can be used as the estimate 23 of position 15. For generating the correction data 48, the artificial neural network ANN can be trained to associate images inside the bounds of the selected best-fit contour proposals 45′ with corresponding correction data 48 in order to result in a better match of the image data encompassed by the selected best-fit contour proposals 45′ in comparison to the image data encompassed by the object contour 24.

[0066] Method 22 may comprise an additional step S17, which is provided for the case that (see FIG. 7) none of the contour proposals 42 matches the matching criterion 44 in step S15, such that no best-fit contour proposal 45 can be determined in step S15. In this case, instead of step S16, step S17 can be performed. In step S17, from the image area of camera image 18 that is inside the object contour 24, by means of an artificial neural network ANN, an object estimation module 50 may be provided which directly estimates an object template that provides an estimate for object type and object size and spatial orientation of the object. This object template may be back-projected in the same way as is illustrated in FIG. 3 and FIG. 4 by the back-projection 27 for the object contour 24. The back-projection 27 may be performed for different distance values of distance 25 (see FIG. 1). This also yields several possible positions 32. From each of these possible positions 32, again, a forward-projection 40 may be performed, as is illustrated in FIG. 4 and FIG. 6. This yields contour proposals 42 in the same way as illustrated in FIG. 6. The forward-projection for the distance value of distance 25 which yields the best-matching contour proposal 45 can then be chosen as an estimate of the distance 25. This also allows for an estimate 23 of the position 15 of object 12.

[0067] The general background of the method therefore can be a monocular camera (2D camera) 3D (non-temporal) object detection. This means trying to estimate the position of objects in 3-dimensional space, including the pose, dimensions, location and class of the object based solely on a single camera frame, and known camera intrinsic and extrinsic parameters.

[0068] As part of the environment model use by an autonomous vehicle, one needs to capture dynamic objects and differentiate them from static objects. Potentially mobile objects include cars, trucks, buses, cyclists, pedestrians, animals etc. These should not only be detected in image coordinates (pixel region of interest) but also in real-world coordinates. This method described herein seeks to solve this problem in the context of mono-cameras (i.e. not using additional sensors like lidar, radar or addition stereo-vision cameras), using a single frame.

[0069] In the method described here, a 2D image detector or camera can be used to give areas of interest in the image. A rectangular area of interest (bounding box) creates a frustum in 3D, in which points from many potential (but known) locations of the 3D world can fall in, provided we know the camera instrinsic/extrinsic parameters (i.e. the properties of the lens). Given that we know already the class of the object (i.e. car, truck, pedestrian) we can select prior learnt templates for such classes which maximise the likelihood of an accurate size estimate and place them in 3D space. We then re-project in 2D and check the re-projection error of such template to the originally detected 2D bounding box. For example, within the frustum, we can have a point coming from 400 meters away; however, if we place a “car” template 3D box at this location and re-project into 2D and find that the 2D box is much smaller than the original 2D detection, we know that this location is too far. The best matching templates+locations are then refined by a neural network, which learns to correct the orientation, dimensions, and location of the template to minimize the localization loss and re-projection loss to ground truths, purely based on the camera input. For efficiency, we also make an assumption that objects are not in mid air, but relatively close to the ground plane, thereby reducing further the potential templates.

[0070] There are some caveats with this approach above—for example, one might not be able to find any templates with the right threshold. This might be because, especially at large distances, small deviations in pose and orientation might shift the 3D box so much that the re-projection error is very high, and thus no templates are proposed (all are filtered out). In these cases, or as an additional check for all cases, rather than look for templates in the frustum, we can ask another network to estimate pose R and size S of the object. Given these, we can solve for T (translation) by assuming that the 3D box fits snugly in the 2D image detection. One of the 8 points in the 3D image could potentially define either xmin, ymin, xmax or ymax of the 2D bounding box in image space, so 8**4 combinations (4096). We can solve these combinations exhaustively and pick the translation that gives the lowest re-projection error.

[0071] Overall, the example shows how a position of an external object can be estimated on the basis of a single 2D camera image by the invention.