Method for Determining a Semantic Free Space

Abstract

A method for determining a semantic free space in an environment of a vehicle comprises capturing a two dimensional visual image from the environment of the vehicle via a camera and determining a limitation of a free space within the visual image. Via a sensor, distance data of objects are captured and assigned to the visual image, and the limitation of the free space is transferred to a bird's-eye view based on the assigned distance data. For objects identified in the visual image a respective bounding box and a respective classification are determined. Objects limiting the free space are selected, and their bounding box is assigned to the limitation of the free space in the bird's-eye view. Finally, segments of the limitation of the free space are classified according to the classification of each bounding box of the selected objects.

Claims

1. A method, comprising: capturing, via a camera of a vehicle in an environment, a two dimensional visual image of the environment; determining, with a processor of the vehicle, within the visual image, a limitation of a free space with respect to the vehicle; capturing, via a sensor of the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle; based on assigning the distance data to the visual image, transferring the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle; determining, for each object identified in the visual image, a bounding box and a classification; selecting objects identified in the visual image that are limiting the free space; assigning the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and according to the classification of each of the objects selected, classifying segments of the limitation of the free space transferred to the bird's-eye view, thereby determining a semantic free space in the environment of the vehicle.

2. The method according to claim 1, wherein determining the limitation of the free space comprises determining the limitation of the free space based on applying a first neural network to the visual image.

3. The method according to claim 1, wherein determining the limitation of the free space comprises determining the limitation of the free space based on executing a border following algorithm to determine the limitation of the free space as being a continuously extending contour of the free space.

4. The method according to claim 1, wherein assigning the distance data to the limitation of the free space comprises applying a projecting transformation of the three dimensional distance data to the two dimensional visual image.

5. The method according to claim 4, wherein transferring the limitation of the free space from the visual image to the bird's-eye view comprises applying an inverse transformation of the projecting transformation to the limitation of the free space.

6. The method according to claim 1, wherein the limitation of the free space comprises a plurality of limitation points represented in the visual image, and transferring the limitation of the free space from the visual image to the bird's-eye view comprises: selecting a predetermined number of closest points from distance data for each of the limitation points; and estimating a distance, with respect to the vehicle, for each of the limitation points as an average of distances, with respect to the vehicle, for each of the closest points.

7. The method according to claim 1, wherein determining the bounding box and the classification for each of the objects selected comprises applying a second neural network to the visual image.

8. The method according to claim 7, wherein determining the classification comprises estimating a class probability for the bounding box for each of the objects selected when applying the second neural network.

9. The method according to claim 8, further comprising: determining a certainty score for each segment of the limitation of the free space based on the class probability for the bounding box for each of the objects selected.

10. The method according to claim 1, further comprising: determining, with respect to the vehicle, based on the three dimensional distance data, a distance of a center for the bounding box for each of the objects selected; and assigning the distance of the center for the bounding box for each of the objects selected to the limitation of the free space in the bird's-eye view.

11. The method according to claim 10, wherein determining the distance of the center for the bounding box for each of the objects selected comprises determining the distance of the center for the bounding box for each of the objects selected based on a predetermined number of assigned distance data being closest to the center of that bounding box.

12. The method according to claim 11, wherein assigning the distance of the center for the bounding box for each of the objects selected to the limitation of the free space in the bird's-eye view comprises assigning the distance of the center for the bounding box for each of the objects selected only if the distance of the center of the bounding box to at least one of the assigned distance data being less than a predetermined distance.

13. The method according to claim 1, further comprising: dividing the segments of the limitation of the free space equally by a fixed and predetermined azimuth angle with respect to the vehicle.

14. A system, comprising: a processor for a vehicle, the processor configured to: capture, via a camera for the vehicle, a two dimensional visual image of an environment; determine, within the visual image, a limitation of a free space with respect to the vehicle; capture, via a sensor for the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle; based on assigning the distance data to the visual image, transfer the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle; determine, for each object identified in the visual image, a bounding box and a classification; select objects identified in the visual image that are limiting the free space; assign the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and classify segments of the limitation of the free space transferred to the bird's-eye view according to the classification of each of the objects selected to determine a semantic free space in the environment of the vehicle.

15. The system of claim 14, further comprising: the camera for the vehicle; and the sensor for the vehicle.

16. The system of claim 15, further comprising the vehicle, wherein the processor is configured as a free space module to output the semantic free space determined in the environment of the vehicle.

17. A non-transitory computer readable storage medium comprising instructions, that when executed, configure a processor for a vehicle to: capture, via a camera for the vehicle, a two dimensional visual image of an environment; determine, within the visual image, a limitation of a free space with respect to the vehicle; capture, via a sensor for the vehicle, for objects in the environment, three dimensional distance data with respect to the vehicle; based on assigning the distance data to the visual image, transfer the limitation of the free space from the visual image to a bird's-eye view, with respect to the vehicle; determine, for each object identified in the visual image, a bounding box and a classification; select objects identified in the visual image that are limiting the free space; assign the bounding box of each of the objects selected to the limitation of the free space transferred to the bird's-eye view; and classify segments of the limitation of the free space transferred to the bird's-eye view according to the classification of each of the objects selected to determine a semantic free space in the environment of the vehicle.

18. The computer readable storage medium according to claim 17, wherein the instructions, when executed, configure the processor to determine the limitation of the free space by determining the limitation of the free space based on applying a first neural network to the visual image.

19. The computer readable storage medium according to claim 17, wherein the instructions, when executed, configure the processor to determine the limitation of the free space based on executing a border following algorithm to determine the limitation of the free space as being a continuously extending contour of the free space.

20. The computer readable storage medium according to claim 17, wherein the instructions, when executed, configure the processor to assign the distance data to the limitation of the free space by applying a projecting transformation of the three dimensional distance data to the two dimensional visual image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following drawings, showing schematically:

[0038] FIG. 1 depicts a schematic diagram of a vehicle comprising a system according to the disclosure,

[0039] FIG. 2 depicts a visual image taken by a camera of the system as shown in FIG. 1,

[0040] FIG. 3 is a representation of a free space determined from the visual image as shown in FIG. 2,

[0041] FIG. 4 depicts a contour for the free space of FIG. 3,

[0042] FIG. 5 depicts three-dimensional distance data projected to the visual image as shown in FIG. 2,

[0043] FIG. 6 depicts the three-dimensional distance data from FIG. 5 assigned to the contour of the free space from FIG. 4,

[0044] FIG. 7 depicts a representation of the free space as shown in FIGS. 3 and 4 transferred to a bird's-eye view coordinate system,

[0045] FIG. 8 depicts bounding boxes for objects identified in the visual image from FIG. 2,

[0046] FIG. 9 depicts the assignment of the three-dimensional distance data to the bounding boxes as shown in FIG. 8,

[0047] FIG. 10 depicts the assignment of the bounding boxes from FIGS. 8 and 9 to the contour of the free space in the bird's-eye view coordinate system, and

[0048] FIG. 11 depicts classified segments for the contour of the free space as shown in FIG. 10.

DETAILED DESCRIPTION

[0049] FIG. 1 schematically depicts a host vehicle 11 which includes a camera 13 and a sensor 14 for monitoring the environment of the vehicle 11. The camera 13 and the sensor 14 belong to a system 12 for determining a semantic free space in the environment of the vehicle 11 according to the disclosure. The system 12 further includes a free space module 15 and a classification module 16 which are connected to each other and which are provided with data from the camera 13 and from the sensor 14.

[0050] The camera 13 is a monocular camera providing a two-dimensional visual image as shown e.g. as camera image 17 in FIG. 2. The sensor 14 (see FIG. 1) is generally a sensor providing three-dimensional distance data for objects in the environment of the vehicle 11. In the present example, the sensor 14 is a LIDAR sensor. However, as an alternative, a radar sensor may also be used for the sensor 14, or any suitable sensor providing distance or depth information.

[0051] FIG. 2 depicts the visual image 17 which is provided by the camera 13 (see FIG. 1). The visual image 17 is a two-dimensional representation of the environment in front of the host vehicle 11. The visual image 17 includes a road 18 in front of the vehicle 11 including three lanes 19 and a boundary 20 of the road 18 which is represented by guide rails. In addition, the visual image 17 includes further objects or obstacles in front of the vehicle 11, i.e. other passenger cars 21 and trucks 23 driving in front of the vehicle 11 or in an opposite lane. The boundaries 20 of the road 18 and the other vehicles, i.e. the passenger cars 21 and the trucks 23, limit the drivable area which is available for the host vehicle 11.

[0052] FIG. 3 depicts a representation of a free space 25 which is derived from the visual image 17 as shown in FIG. 2. In detail, a first convolutional neural network is applied to the two-dimensional visual image 17 as shown in FIG. 2 which is available in the form of pixel data provided by the camera 13 (see FIG. 1). Via the convolutional neural network, a limitation 29 (see FIG. 3) is determined for the free space 25. Therefore, the upper part of FIG. 3 denoted by 27 represents a non-drivable area which is currently not available for the navigation of the host vehicle 11. The limitation 29 of the free space 25 is determined by the free space module 15 (see FIG. 1) which includes the first neural network.

[0053] FIG. 4 additionally depicts a contour 30 which extends continuously along the limitation 29 between the free space 25 and the non-drivable area 27 as also shown in FIG. 3. The contour 30 is determined by applying a border following algorithm, for which examples are known in the art.

[0054] FIG. 5 shows the visual image 17 from FIG. 2 together with a projection of three-dimensional data points 33 which are provided by the sensor 14. That is, the three-dimensional data 33 captured by the sensor 14 are transformed to the plane of the visual image 17 via a projecting transformation in order to provide a representation of the data points 33 as shown in FIG. 5. In detail, the data points 33 are provided by the LIDAR system and represent the respective shortest distance of the next obstacle or object with respect to the host vehicle 11.

[0055] As may be recognized in FIG. 5, a part of the distance data points 33 are assigned to a respective passenger car 21, whereas another part of the distance points 33 is assigned to the boundaries 20 of the road 18.

[0056] FIG. 6 depicts the assignment of the three-dimensional distance data 33 to the representation of the free space 25 as shown in FIGS. 3 and 4. In detail, the distance data 33 provided by the sensor 14 are related to respective points of the continuous contour 30 of the free space 25 as shown in FIG. 4.

[0057] For the assignment of the projected distance data 33 to the points of the limitation 29 or contour 30, a fixed number (e.g. one to five) of closest points from the projected distance data points 33 is determined for each point of the contour 30. The closest points from the distance data 33 with respect to the points of the contour 30 are those points which have the shortest distance within the representation of FIG. 6 with respect to a certain point of the contour 30. For determining the closest points 33 or nearest neighbor points 33 with respect of the points of the contour 30, efficient methods like k-d trees may be used which are known in the art.

[0058] In order to determine a distance or “depth” with respect to the host vehicle 11 for each point of the contour 30, an average over the measured distances of the closest distance data points 33 is estimated for each point of the contour 30. This average may be a weighted average wherein each weight depends on the respective distance to the point of the contour 30 under consideration.

[0059] The estimated distance or depth of the points of the contour 30 is used for transforming the contour 30 to a bird's-eye view coordinate system 35 as shown in FIG. 7. For this transformation, the inverse transform of the projecting transform is used which is applied for projecting the three-dimensional distance data 33 provided by the sensor 14 to the visual image 17, as shown in FIG. 5.

[0060] As shown in FIG. 7, the bird's-eye view coordinate system 35 comprises an x-axis 38 and a y-axis 39 which are located in a plane parallel to a tangent to the lane in which the host vehicle 11 is currently driving. In other words, in FIG. 7 one is looking from above at the environment in front of the host vehicle 11. In the bird's-eye view coordinate system 35 of FIG. 7, the free space 25 is shown again which is limited by the limitation 29 or transformed contour 31 which has been transformed from the representation according to the visual image 17 (see FIGS. 3 and 4) to the bird's-eye view of FIG. 7 using the distance information provided by the three-dimensional distance data 33 from the sensor 14. A smoothing by applying a moving average is used for the x- and y-coordinates of the contour 31 in the bird's-eye view coordinate system 35 in order to achieve a smooth representation of the transformed contour 31 of the free space 25.

[0061] In the area close to the host vehicle 11, the free space 25 is limited according to an angle 37 representing the instrumental field of view of the camera 13. In addition, the projection of the three-dimensional distance data 33 from the sensor 14 is shown within the bird's-eye view coordinate system 35. Since the respective distance of the points belonging to the contour 31 is determined based on the three-dimensional distance data 33 from the sensor 14, the projection of the distance data 33 is positioned at the contour 31 in the bird's-eye view coordinate system 35.

[0062] The free space 25 as shown in FIG. 7 represents the drivable area in front of the host vehicle 11. However, it is not yet known for the points of the contour 31 as shown in FIG. 7 by which type of object the free space or drivable area 25 is restricted. For actions regarding the navigation of the host vehicle 11 it is desirable, however, to have additional information regarding the objects restricting the free space 25, i.e. whether the free space 25 is limited by another vehicle like the cars 21 as shown in FIG. 2, or by a pedestrian or by debris located on one of the lanes 19 (see FIG. 2), for example. Depending on the type of the object limiting the free space 25, different decisions or actions may be taken for the navigation of the host vehicle 11. In other words, additional “semantic” information is desired regarding the objects in the environment of the host vehicle 11.

[0063] In order to provide such semantic information, objects are identified within the visual image 17 provided by the camera 13, as shown in FIG. 8. For each identified object, a two-dimensional bounding box 41 is determined together with a classification 43 of the object. In detail, a respective minimum and maximum value for an x- and a y-coordinate is determined within the visual image 17 in order to determine the frame of respective bounding box 41 which encloses the respective object within the visual image 17. In addition, the classification 43 is determined for each bounding box 41, i.e. whether the respective bounding box encloses a car or a truck, as shown in FIG. 8. That is, the items “car” and “truck” describe the respective detection class or classification 43 for each of the identified objects. Moreover, a detection certainty or class probability 45 is determined for each bounding box 41.

[0064] In order to determine the position of each bounding box 41, i.e. its coordinate within the visual image 17, the classification 43 and the class probability 45, a second convolutional neural network is applied to the pixel data of the visual image 17. The determination of the bounding boxes 41 together with the classification 43 and the class probability 45 based on the visual image 17 using a convolutional neural network is also referred to as single shot multi-box detection (SSD) since no segmentation of the visual image 17 is previously performed. The second convolutional neural network is included in the classification module 16 (see FIG. 1). The module also performs all further method steps as described below.

[0065] In order to relate the bounding boxes 41 as well as their classification 43 and class probability 45 to the free space 25 as shown in FIG. 7, a center is determined for each bounding box 41, and for each center of the respective bounding box 41 the closest points 33 are determined from the projection of the three-dimensional distance data provided by the sensor 14. The points 33 representing the projection of the three-dimensional distance data 33 are shown together with the bounding boxes 41 in FIG. 9.

[0066] The projection of the dimensional distance data points 33 to the visual image 17 is the same as shown in FIG. 5. Therefore, no additional calculation is required regarding the distance data points 33 before assigning these points to the respective centers of the bounding boxes 41. For each center of the bounding boxes 41, a fixed number (e.g. one to five) of closest data points 33 is determined, in the same manner as for the points of the contour 30 as shown in FIGS. 4 and 6. That is, the fixed number of data points 33 is determined for each center of the bounding boxes 41 which have the shortest distance to the respective center of the bounding boxes 41. Again, an average or a weighted average for the distances of the closest data points 33 is calculated for each center of the bounding boxes 41 in order to determine a depth or distance of each center of the bounding boxes 41 with respect to the host vehicle 11.

[0067] However, only those centers of the bounding boxes 41 are selected for the further procedure, i.e. for a transform to the bird's-eye view coordinate system 35, for which the distance to the closest distance data point 33 within the visual image 17 is less than a predetermined distance. By this means, only those centers of the bounding boxes 41 are selected which are “reasonably” close to at least one of the distance data points 33.

[0068] For example, the bounding boxes 41 of the passenger cars 21 as shown in FIG. 9 each comprise distance points 33 which are quite close to the center of the respective bounding boxes 41. In contrast, the bounding boxes 41 of the trucks 23 do not include distance data points 33 which are close to the center of the respective bounding box 41. Therefore, the bounding boxes 41 of the passenger cars as shown in FIG. 9 are selected for the further procedure only and transformed to the bird's-eye view coordinate system 35 as shown in FIG. 10.

[0069] FIG. 10 depicts the free space 25 in front of the vehicle 11 in the bird's-eye view coordinate system 35, which has already been shown in FIG. 6. The free space 25 is limited by the transformed contour 31 on which the free dimensional distance data 33 are also shown.

[0070] In addition, FIG. 10 depicts transformed bounding boxes 51 which are assigned to the contour 31. The bounding boxes 41 as shown in FIGS. 8 and 9 are transformed from the visual image 17 to the bird's-eye view coordinate system 35 via the same transform as for the contour 30 (see FIGS. 4 and 6), i.e. the inverse transform of the projecting transform which is used for projecting the three-dimensional distance data 33 to the visual image 17.

[0071] In detail, the respective center of the bounding boxes 41 (see FIGS. 8 and 9) is transformed from the visual image 17 to a respective transformed center 53 of the bounding box 41 in the bird's-eye view coordinate system 35. In addition, the semantics of the bounding boxes 41 are also known from applying the second neural network to the visual image 17. That is, the type or class of the respective object is known for the representation as shown in FIG. 10 for each bounding box 41. For the present example, it is known that the three bounding boxes 51 each represent a car 21 in FIG. 10.

[0072] Finally, the contour 31 of the free space 25 in front of the vehicle 11 is equally divided by a predetermined azimuth angle with respect to the vehicle 11, and each segment 55 of the contour 31 is classified by assigning the respective segment 55 to the respective classification of the center 53 of the bounding box 51 (see FIG. 10) if this segment comprises the center 53 of the bounding box 51. Classified segments 55 are shown in FIG. 11 which includes the same representation of the free space 25 as FIG. 10. That is, the segments 55 of the contour 31 are classified as in “passenger car” 21 which means that the free space 25 in front of the vehicle 11 is limited by a passenger car 21 (see FIG. 2) at the classified segments 55.

[0073] In summary, the method according to the disclosure determines the limits or the contour 31 of the free space 25 in front of the vehicle in bird's-eye view via the first neural network, and in addition, the semantic of the segments of the contour 31 is determined via the second neural network such that it will be known which part of the free space 25 is limited by which kind of object. In case that no center of a bounding box can be assigned to a specific segment of the contour 31, a default classification may be assumed for these segments, e.g. a classification as boundary 20 of the road 18 (see FIG. 2) or a limitation by the instrumental field of view of the camera 13.

[0074] In addition, a certainty score is estimated for each classified segment 55 of the contour 31 based on the class probability 45 which is determined for each bounding box 41 via the second neural network. The certainty score and the semantic free space represented by the segments of the contour 31 (see FIG. 11) can be used for the navigation of the host vehicle 11, e.g. in autonomous driving.