Method and Device for Classifying Pixels of an Image

Abstract

A method is provided for classifying pixels of an image. An image comprising a plurality of pixels is captured by a sensor device. A neural network is used for estimating probability values for each pixel, each probability value indicating the probability for the respective pixel being associated with one of a plurality of predetermined classes. One of the classes is assigned to each pixel of the image based on the respective probability values to create a predicted segmentation map. For training the neural network, a loss function is generated by relating the predicted segmentation map to ground truth labels. Furthermore, an edge detection algorithm is applied to at least one of the predicted segmentation maps and the ground truth labels, wherein the edge detection algorithm predicts boundaries between objects. Generating the loss function is based on a result of the edge detection algorithm.

Claims

1. A computer-implemented method comprising: receiving an image captured by a sensor device, the image comprising a plurality of pixels; estimating, using a neural network implemented on a processing device, a respective probability value for each pixel of the plurality of pixels, wherein each respective probability value indicates a probability for a respective pixel being associated with one of a plurality of predetermined classes; assigning one of the plurality of predetermined classes to each pixel of the image based on the respective probability value for each pixel to create a predicted segmentation map for the image; generating a loss function for training the neural network by relating the predicted segmentation map to ground truth labels; and applying an edge detection algorithm to at least one of the predicted segmentation map and the ground truth labels, the edge detection algorithm predicting boundaries between objects in the predicted segmentation map and the ground truth labels, generation of the loss function being based on a result of the edge detection algorithm.

2. The method of claim 1, wherein the edge detection algorithm is applied to the predicted segmentation map only.

3. The method of claim 1, wherein the edge detection algorithm is applied to the ground truth labels only.

4. The method of claim 1, wherein the edge detection algorithm is applied to the predicted segmentation map and the ground truth labels.

5. The method of claim 4, wherein a result of applying the edge detection algorithm to the predicted segmentation map and a result of applying the edge detection algorithm to the ground truth labels are merged by selecting a maximum value of the respective results for each pixel.

6. The method of claim 1, wherein the result of the edge detection algorithm is applied to a result of relating the predicted segmentation map to the ground truth labels for generating the loss function.

7. The method of claim 1, wherein the result of the edge detection algorithm is a mask that covers the pixels of the predicted boundaries.

8. The method of claim 7, wherein: the mask includes a respective element for each pixel; a loss matrix including a matrix element for each pixel is calculated by relating the predicted segmentation map to the ground truth labels; and each element of the mask is multiplied by the corresponding matrix element of the loss matrix for each pixel when generating the loss function.

9. The method of claim 8, wherein the elements of the mask whose pixels are outside the predicted boundaries are assigned a value equal to or about equal to zero.

10. The method of claim 1, wherein the edge detection algorithm includes a Sobel operator comprising two predefined convolutional kernels and two additional kernels which are generated by rotating the two predefined convolutional kernels.

11. The method of claim 10, wherein the edge detection algorithm includes a consequential convolutional kernel for increasing a width of the predicted boundaries between objects.

12. The method of claim 11, wherein the additional convolutional kernel is a bivariate Gaussian kernel.

13. A system comprising: a sensor device configured to capture an image comprising a plurality of pixels; and a processing device configured to: receive the image from the sensor device; implement a neural network to estimate a respective probability value for each pixel, each respective probability value indicating a probability for a respective pixel being associated with one of a plurality of predetermined classes; assign one of the plurality of pre-determined classes to each pixel of the image based on the respective probability values for each pixel to create a predicted segmentation map for the image; generate a loss function for training the neural network by relating the predicted segmentation map to ground truth labels; and apply an edge detection algorithm to at least one of the predicted segmentation map and the ground truth labels, the edge detection algorithm predicting boundaries between objects in the predicted segmentation map and the ground truth labels, generation of the loss function being based on a result of the edge detection algorithm.

14. The system of claim 13, wherein the edge detection algorithm is applied to the predicted segmentation map only.

15. The system of claim 13, wherein the edge detection algorithm is applied to the ground truth labels only.

16. The system of claim 13, wherein the edge detection algorithm is applied to the predicted segmentation map and the ground truth labels.

17. The system of claim 16, wherein a result of applying the edge detection algorithm to the predicted segmentation map and a result of applying the edge detection algorithm to the ground truth labels are merged by selecting a maximum value of the respective results for each pixel.

18. The system of claim 13, wherein the result of the edge detection algorithm is applied to a result of relating the predicted segmentation map to the ground truth labels for generating the loss function.

19. The system of claim 13, wherein the result of the edge detection algorithm is a mask that covers the pixels of the predicted boundaries.

20. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processing device, cause the processing device to: receive an image from a sensor device, the image including a plurality of pixels; implement a neural network to estimate a respective probability value for each pixel, each respective probability value indicating a probability for a respective pixel being associated with one of a plurality of predetermined classes; assign one of the plurality of pre-determined classes to each pixel of the image based on the respective probability values for each pixel to create a predicted segmentation map for the image; generate a loss function for training the neural network by relating the predicted segmentation map to ground truth labels; and apply an edge detection algorithm to at least one of the predicted segmentation map and the ground truth labels, the edge detection algorithm predicting boundaries between objects in the predicted segmentation map and the ground truth labels, generation of the loss function being based on a result of the edge detection algorithm.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Exemplary embodiments and functions of the present disclosure are described herein in conjunction with the following drawings, showing schematically:

[0033] FIG. 1 depicts a high-level diagram of a system according to the disclosure,

[0034] FIG. 2 depicts results of a semantic segmentation according to the background art,

[0035] FIG. 3 depicts a flow diagram for a method according to the disclosure,

[0036] FIGS. 4A-4C depict edge detection results and their application to ground truth labels,

[0037] FIGS. 5 and 6 compare a respective semantic segmentation based on background art and based on the method according to the disclosure, and

[0038] FIGS. 7A-7C depict a visualization of a validation of the semantic segmentation including a border intersection over union and its complement.

DETAILED DESCRIPTION

[0039] FIG. 1 schematically depicts a high-level diagram of a system 10 for classifying pixels of an image, i.e., for performing a semantic segmentation of the picture. The system includes a camera 11 as a sensor device and processing device 15. The camera 11 is mounted to a vehicle and can capture an image 13 of the environment of the vehicle, e.g., in front of the vehicle. Alternatively, a Lidar system may be used for capturing the image 13 which may be in this case a three-dimensional matrix of pixels.

[0040] The image 13 captured by the camera 11 is used as an input for the processing device 15 which is configured to generate a convolutional neural network 33 (see FIG. 3) for performing the semantic segmentation of the image 13. For the semantic segmentation, the processing device 15 uses a method 31 which will be described in context of FIG. 3 below. The output of the processing device 15, i.e., a semantic segmentation 17 of the picture 13, is provided as an input for further devices of the vehicle which are part e.g., of a driver assistance system.

[0041] FIG. 2 depicts results of a semantic segmentation according to the background art. That is, areas within pictures 13 captured by the camera 11 are assigned to predefined types of objects or object class, e.g., to one of the object classes “other passenger car”, “truck”, “road”, “sidewalk”, “pedestrian”, etc. The area occupied by a certain object type is represented by a unique color in FIG. 2.

[0042] On the left side of FIG. 2, ground truth labels are shown for two different scenes. For these scenes, it is known which area belongs to which object type. The ground truth labels depicted on the left side of FIG. 2 are used for training a convolutional neural network which provides a network prediction shown on the right side, respectively, of FIG. 2 for the semantic segmentation of the images 13. In detail, the semantic segmentation or classification of objects is performed based on pixels of the respective image 13. For example, each image 13 includes r=2048 rows and c=1024 columns, i.e., almost 2.1 million of pixels. For each pixel, the convolutional neural network provides a prediction regarding the object class, i.e., it assigns an indicator for a certain object class to each pixel. Therefore, each area as shown on the left side of FIG. 2 and having a certain color includes a plurality of pixels which are classified as a certain type of object by the convolutional neural network.

[0043] As can be recognized on the left side of FIG. 2, the respective ground truth labels for the pictures 13 include many small objects 21 and boundaries 23 between objects. As can be seen on the right side of FIG. 2, the respective network prediction according to the background art fails to classify the small objects 21 and the boundaries 23 properly. In detail, in the areas designated by 25 the small objects 21 are missing or almost missing. In addition, the sharp boundaries 23 between the objects as shown on the left side of FIG. 2 are also not properly recognizable in the network prediction, as is indicated by 27. The sharp boundary 23 between the objects appears to be corrupted in the output of the convolutional neural network.

[0044] A reason for the erroneous or not satisfying classification by the network prediction according to the background art is related to a loss function which is usually used for training the convolutional neural network. A robust loss function is generally essential for a learning or training process of any neural network. The loss function generally includes a certain comparison of a predicted output of the neural network and a desired output, i.e., the ground truth labels. Typically, a regression is performed by assigning large loss values to undesired values within the network prediction, and the total loss is minimized thereafter during training of the neural network.

[0045] For evaluating the output or prediction of a neural network, a so-called cross-entropy function is commonly used as a loss function and defined as

loss=−Σ.sub.i=0.sup.ny.sub.i*log(ŷ.sub.i)  (1)

[0046] wherein y.sub.i represents the output or prediction of the neural network, whereas ŷ.sub.i represents the desired or ground truth label. The total loss is then defined as the mean over all pixels within the image 13. Since for an image having 2048 rows and 1024 columns and therefore almost 2.1 million of pixels, the contribution of a single pixel to the loss function or total loss is approximately weighted by 1/(2.1*10.sup.6). As a consequence, small objects and boundaries between objects which include quite a small number of pixels are not properly represented in the loss function according to the background art in which all pixels have the same weight for their contribution to the loss function. Therefore, it is desirable to have a loss function for training the convolutional neural network in which the representation of small objects and boundaries between objects is improved.

[0047] FIG. 3 depicts a flow diagram for a method 31 according to the disclosure. The method 31 is provided for classifying pixels of the image 13 which is captured by the camera 11 (see FIG. 1). That is, the image 13 including a plurality of pixels is provided as input for the method, i.e., for a convolutional neural network 33 which is generated by the processing device 15 (see FIG. 1). The convolutional neural network 33 (see FIG. 3) includes a plurality of layers which are based on an underlying model.

[0048] The neural network 33 estimates probability values for each pixel. Each probability value indicates a probability for the respective pixel for being associated with one of a plurality of predetermined classes. Based on the respective probability values for each pixel, each pixel of the image 13 is assigned to one class of the plurality of classes, i.e., by selecting the class having the highest probability value. The predetermined classes include predetermined types of objects visible for the camera 11, i.e., other vehicles, the road, the sidewalk, pedestrians, etc. As an output, the convolutional neural network 33 creates a predicted segmentation map 35 which includes a predicted class for each pixel. In the predicted segmentation map 35 as shown in FIG. 3 and in similar predicted segmentation maps as shown in FIGS. 5 and 6, different types of objects are represented by different colors.

[0049] In order to train the neural network 33, the predicted segmentation map 35 is related to ground truth labels 39, i.e., the desired output of the convolutional neural network 33. For the ground truth labels 39, the correct assignment to the respective class or object type is known for each pixel of the image 13. The predicted segmentation map 35 is related to ground truth labels 39 via a loss function 37 which is based on the cross-entropy function as described above in context of FIG. 2. During training of the convolutional neural network 33, a total loss provided by the loss function 37 is to be minimized.

[0050] As mentioned above, the commonly used loss function which is based on the cross-entropy function as the disadvantage that the contribution of all pixels is the same which leads to an underestimation of small objects and object boundaries when estimating the loss function and therefore to an erroneous or missing classification of the small objects and the object boundaries.

[0051] In order to overcome this disadvantage, the method includes providing a modified version 37 of the loss function in which the small objects and object boundaries are provided with a greater weight in order to increase their contribution to the total loss.

[0052] In order to increase the contribution of small objects and object boundaries to the loss function 37, an edge detection algorithm 41 is applied to the predicted segmentation map 35 and to the ground truth labels 39. The output of the edge detection algorithm 41 is a prediction mask 43 when applied to the predicted segmentation map 35, and a ground truth map 44 when applied to the ground truth labels 39. Within the masks 43, 44, all boundaries between objects are highlighted in FIG. 3 since the masks assign much higher weights to the pixels of the detected boundaries or edges than to the pixels which do not belong to the boundaries.

[0053] The edge detection algorithm 41 is based on a so-called Sobel operator which can extract boundaries between the predicted classes. The standard Sobel operator includes two predefined and constant convolutional kernels:

[00001]$\begin{array}{cc}{s}_y=\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right],s_x=\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right]& \left(2\right)\end{array}$

[0054] By using these kernels, color gradients in the image 13 can be detected.

[0055] However, it turned out that a standard Sobel operator based on the predefined constant convolutional kernels is not sufficient for successfully weighting the loss function 37. In detail, the Sobel kernels as defined in (2) show a minor performance when trying to detect diagonal edges. Therefore, two additional convolutional kernels have been added for which the original Sobel kernels are rotated by +/−45°.

[0056] In addition, the pixels not belonging to the detected edges are suppressed by setting their value close to zero in the mask. Therefore, almost no loss values are assigned to a major portion in the predicted segmentation map 35 to create the masks 43, 44. However, since a prediction is still to be provided for all pixels of the image 13, all pixels in the respective mask 43 or 44 have at least a small value of e.g., 0.1 to account for loss values of all pixels.

[0057] Moreover, it turned out that the resulting edges provided by the edge detection algorithm 41 so far are not sufficient to support a successful learning or training of the convolutional network 33. Hence, a consequential convolutional kernel is applied which is a bivariate Gaussian kernel having a predefined size. In detail, a 3×3 padding kernel is used which is given by

[00002]$\begin{array}{cc}{k}_p=\left[\begin{array}{ccc}0.0625& 0.125& 0.0625\\ 0.125& 0.25& 0.125\\ 0.0625& 0.125& 0.0625\end{array}\right]& \left(3\right)\end{array}$

[0058] When the convolution is performed within the edge detection algorithm 41, the additional kernel adds a padding of three pixels before and of three pixels after the original line or boundary between respective objects. Therefore, the widths of the boundaries as represented in the masks 43, 44 are increased. In practice, additional padding kernels of roughly 30×30 pixels are used. In contrast to the classical Gaussian kernel which is normalized such that the sum over all its elements is equal to one, the method according to the disclosure normalizes the bivariate Gaussian kernel such that the mean value over its elements is one.

[0059] After a respective mask 43, 44 is generated based on the modified or extended Sobel operator including additional kernels, the two masks 43,44 are merged by using a pixel-wise maximum operator at 45. That is, for each pixel the greater value of the respective masks 43, 44 is selected to generate a final or merged mask 47. For each pixel, the final mask 47 is included in the generation of the loss function 37. That is, for each pixel the contribution to the loss is calculated according to the cross-entropy function as defined in (1) such that a “preliminary loss” or loss matrix is generated for which each element includes the contribution of the respective pixel. Thereafter, the final mask 47 is applied to the preliminary loss or loss matrix. In detail, for each pixel the preliminary loss is multiplied by the corresponding element of the final mask 47. Due to this, object boundaries and small objects have a greater contribution to the loss function 37 than areas which do not belong to the detected boundaries between objects.

[0060] According to the embodiment as shown in FIG. 3, the method includes generating the prediction mask 43 for which the edge detection algorithm 41 is applied to the prediction segmentation map 35, generating the ground truth mask 44 for which the edge detection algorithm 41 is applied to ground truth labels 39, and generating the final mask 47 for which both masks 43, 44 are merged and which is applied when generating the loss function 37. For another embodiment, however, the prediction mask 43 is applied to the ground truth labels 39 only. That is, the final mask 47 is identical with the prediction mask 43 for this embodiment since no merging of the two masks 43, 44 is performed. For an alternative embodiment, the ground truth mask 44 is used only to generate the final mask 47, i.e., the prediction mask 43 is omitted and no merging of the two masks 43, 44 is performed. In other words, the ground truth mask 44 is applied directly to the loss function 37, i.e., to the preliminary loss, for this alternative embodiment.

[0061] FIG. 4A depicts an enlarged example for the prediction mask 43. FIG. 4B depicts the result for applying the prediction mask 43 from FIG. 4A to the ground truth labels 39. By using the prediction mask 43, wrongly classified or “hallucinated” objects may be suppressed. Such a hallucinated object can be recognized in the bottom left corner of FIG. 4B. Since the weight or contribution of this hallucinated object is increased due to the prediction mask 43 when calculating the total loss, such objects will be suppressed by the trained neural network after the minimization of the loss function during the training. In addition, FIG. 4B also represents the masked ground truth labels for the embodiment of the method for which the prediction mask 43 is considered for generating the loss function 37 only.

[0062] In contrast, FIG. 4C depicts the masked ground truth labels for which the merged final mask 47 has been applied to the original ground truth labels 39. That is, FIG. 4C represents the masked ground truth labels for the embodiment as shown in FIG. 3 for which the final mask 47 is generated by merging both masks 43, 44. By taking the ground truth mask 44 additionally into account, it is ensured that the most important parts of the image are properly respected. As can be seen in the upper right corner of FIG. 4C, the contribution of the boundaries which belong to a pink element 50 is strongly enhanced since the ground truth mask 44 is additionally considered. Weights for the pink element 50 are almost missing in the prediction mask 43 as can be recognized in the upper right corner of FIG. 4A.

[0063] It is noted that the prediction mask 43 as shown in FIG. 4A and the merged final mask 47 (see FIG. 3) are applied to or multiplied by the ground truth labels 39 for illustration purposes only, i.e., for demonstrating the effect of the masks 43, 47 on individual objects within the image 13. For generating the loss function 37 and estimating the total loss, however, the masks are applied to (or multiplied by) the “preliminary loss”, i.e., the loss matrix which is calculated based on the cross-entropy function, but not to the ground truth labels 39 directly.

[0064] FIGS. 5 and 6 depict a comparison of the semantic segmentation for a respective image performed by a method according to the background art (upper part of FIGS. 5 and 6) and performed by a method according to the disclosure (lower part of FIGS. 5 and 6). As can be seen in the encircled regions 51, the boundaries between the objects are much clearer in the segmentation result for the method according to the disclosure. In addition, some small objects are almost completely classified in a wrong manner by the method according to the background art. For example, the pole on the right side in the image of FIG. 6 cannot be recognized in the segmentation result according to the background art, whereas is clearly comes out when the segmentation is performed by the method according to the disclosure. The same holds true for the traffic lights as shown at 55. In addition, the recognition of pedestrians 57 is improved by the method according to the disclosure, as can be seen on the right side of FIG. 6.

[0065] The improved segmentation results as shown on the respective lower part of FIGS. 5 and 6 are due to the fact that the edge detection algorithm 41 (see FIG. 3) is incorporated into the method 31 when generating the loss function 37. Since the result of the edge detection algorithm 41 is applied to the ground truth labels 39 before relating the predicted segmentation map 35 to the ground truth labels 39 via the loss function 37, the contribution of object boundaries and small objects to the loss function 37 is enhanced during the training procedure of the neural network 33. Since small objects and object boundaries are better recognizable in the semantic segmentation results as shown in FIGS. 5 and 6 (respective lower part), the safety for a vehicle may be improved if the method according to the disclosure is applied e.g., in connection with driver assistance systems and autonomous driving.

[0066] For a detailed validation of the segmentation results, a so-called border intersection over union (border IoU) has been estimated. The border IoU is defined and applied e.g., in Milioto, A. et al.: “RangeNet++: Fast and accurate LiDAR semantic segmentation”, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 42134220. IEEE, 2019. The intersection over union (IOU) is commonly used for describing a similarity of sets, vectors and objects. In semantic segmentation, the IoU is generally used as a metric to assess the labeling performance which relates the true positives of a network prediction to the sum over true positive, false positive and false negative predictions of the neural network. For the border IoU, this assessment key is applied to boundaries between objects only.

[0067] This is visualized in FIG. 7. FIG. 7A depicts an original picture 13 captured by the camera 11 (see FIG. 1) being installed in a vehicle. FIG. 7B depicts the border IoU wherein the evaluated pixels of the image 13 are masked with their respective color which is assigned to the respective object class. In FIG. 7C a so-called interior IoU is depicted which is evaluated as the complement of the border IoU shown in FIG. 7B. For almost all object classes, it turned out that all metrics, i.e., the “normal” IoU, the interior IoU and especially the border IOU, provided better values for the semantic segmentation based on the method according to the disclosure in comparison to the semantic segmentation based on the method according to the background art.