METHOD AND SYSTEM FOR FREQUENCY CODING IMAGE DATA

Abstract

A computer-implemented method for frequency coding of image data from an imaging sensor. The method includes: supplying first image data of an individual image recorded by an imaging sensor, the first image data having depth values of the individual image coded as a whole number or as a floating-point number; receiving the first image data by an algorithm, which frequency codes the depth values of the individual image by a predefined number of periodic functions; and outputting second image data by the algorithm, the second image data having frequency coded depth values of the individual image. A computer-implemented method is described for supplying an algorithm of machine learning for the classification of objects included in image data of an individual image from an imaging sensor. A system for the frequency coding of image data from an imaging sensor, a computer program, and a computer-readable data carrier, are also described.

Claims

1. A computer-implemented method for frequency coding of image data from an imaging sensor, the method comprising the following steps: supplying first image data of an individual image recorded by the imaging sensor, the first image data having depth values of the individual image that are coded as a whole number or a floating-point number; receiving the first image data by an algorithm which frequency codes the depth values of the individual image by a predefined number of periodic functions; and outputting second image data by the algorithm, the second image data having frequency coded depth values of the individual image.

2. The computer-implemented method as recited in claim 1, wherein the predefined number of periodic functions is given by sine functions or cosine functions of a constant frequency, and each one of the periodic functions has a frequency that is defined within a predefined spectrum and differs from a frequency of the other ones of the predefined number of periodic functions.

3. The computer-implemented method as recited in claim 1, wherein the depth value of each image pixel of the individual image is frequency coded by the algorithm using the predefined number of periodic functions.

4. The computer-implemented method as recited in claim 1, wherein the imaging sensor is a camera sensor or a lidar sensor or a radar sensor or an ultrasonic sensor.

5. The computer-implemented method as recited in claim 1, wherein the algorithm is a set of all periodic functions, and the predefined number of periodic functions amounts to 10-50.

6. The computer-implemented method as recited in claim 5, wherein the predefine number of periodic functions amount to 15-25.

7. The computer-implemented method as recited in claim 1, wherein the algorithm converts the frequency coded depth values of the individual image into a vector representation.

8. The computer-implemented method as recited in claim 1, wherein the second image data, which are output by the algorithm and have the frequency coded depth values of the individual image, are provided as input data for training and/or an inference of an algorithm of machine learning for the classification of objects included in the image data of the imaging sensor.

9. A computer-implemented method for supplying an algorithm of machine learning for classification of objects included in image data of an individual image of an imaging sensor, the method including the steps: supplying first image data of an individual image recorded by an imaging sensor, the first image data having depth values of the individual image that are coded as a whole number or a floating-point number; receiving the first image data by an algorithm which frequency codes the depth values of the individual image by a predefined number of periodic functions; outputting second image data by the algorithm, the second image data having frequency coded depth values of the individual image receiving the second image data output by the algorithm and have frequency coded depth values of the individual image; receiving a classification result allocated to an object included in the individual image; and training of an algorithm of machine learning using the image data and the classification result allocated to the object included in the individual image, by an optimization algorithm, which calculates an extreme value of a loss function.

10. A system for frequency coding image data of an imaging sensor, comprising: an imaging sensor configured to supply first image data of a recorded individual image, the first image data having depth values of the individual image coded as a whole number or as a floating-point number; and a computing device configured to: receive the first image data using an algorithm which frequency codes the depth values of the individual image by a predefined number of periodic functions; and output second image data using the algorithm, the second image data having frequency coded depth values of the individual image.

11. A non-transitory computer-readable data carrier on which is stored program code of a computer program for frequency coding of image data from an imaging sensor, the program code, when executed by a computer, causing the computer to perform the following steps: supplying first image data of an individual image recorded by the imaging sensor, the first image data having depth values of the individual image that are coded as a whole number or a floating-point number; receiving the first image data by an algorithm which frequency codes the depth values of the individual image by a predefined number of periodic functions; and outputting second image data by the algorithm, the second image data having frequency coded depth values of the individual image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0046] FIG. 1 shows a flow diagram of a computer-implemented method for frequency coding of image data from an imaging sensor according to a preferred embodiment of the present invention.

[0047] FIG. 2 shows a flow diagram of the computer-implemented method for frequency coding of image data from the imaging sensor according to the preferred embodiment of the present invention.

[0048] FIG. 3 shows a schematic representation of a system for frequency coding of image data from the imaging sensor according to the preferred embodiment of the present invention.

[0049] FIG. 4 shows a flow diagram of a computer-implemented method for supplying an algorithm of machine learning for the classification of objects included in image data of an individual image from an imaging sensor according to the preferred embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0050] Unless stated to the contrary, matching reference numerals in the figures of the drawings refer to the same or functionally equivalent elements, parts or components.

[0051] FIG. 1 shows a flow diagram of a computer-implemented method for frequency coding of image data from an imaging sensor according to a preferred embodiment of the present invention.

[0052] The imaging sensor is preferably installed in a motor vehicle. As an alternative, the imaging sensor is able to be used in a public space, e.g., in the region of a road intersection, in order to monitor traffic. As a further alternative, the imaging sensor may be provided in a building, such as in an entry area of the building, for the purpose of monitoring the environment.

[0053] The imaging sensor preferably is an environment sensor of a motor vehicle. As an alternative, for example, the imaging sensor may be embodied as an interior sensor installed in the motor vehicle or as an environment sensor installed in a public space and/or on or in buildings.

[0054] The present method includes the supply S1 of first image data BD1 of an individual image E recorded by an imaging sensor 10, the first image data BD1 having depth values TW of individual image E coded as a whole number or as a floating-point number. In addition, the present method includes the receiving S2 of the first image data BD1 by an algorithm A1 which frequency codes depth values TW of individual image E by a predefined number of periodic functions F.

[0055] In addition, the present method includes an output S3 of second image data BD2 by algorithm A1, the second image data BD2 having frequency coded depth values TW of individual image E. Second image data BD2 have the format of a width B, a height H and a number of channels K which include the frequency-coded depth values TW.

[0056] The predefined number of periodic functions F is given by sine functions F or cosine functions F of a constant frequency. Each one of periodic functions F has a frequency which is defined within a predefined spectrum and differs from the frequency of the other ones of the predefined number of periodic functions F.

[0057] The depth value of each coordinate, in particular of each image pixel, of the input image is frequency coded by the algorithm using the predefined number of periodic functions. This is able to be realized pixel by pixel of a respective line of individual image E, for example.

[0058] Each one of periodic functions F stores depth values TW of individual image E in a channel K which is allocated to function F.

[0059] First image data BD1 of individual image E are coded in gray scales, in particular using one channel K per pixel for color information and one channel per pixel for depth value TW.

[0060] As an alternative, first image data BD1 of individual image E are able to be coded as an RGB color image, in particular using three channels K per pixel for the color information and one channel K per pixel for depth value TW.

[0061] Imaging sensor 10 is preferably embodied as a camera sensor. Alternatively, imaging sensor 10 is able to be embodied as a lidar sensor, a radar sensor or an ultrasonic sensor, for example.

[0062] Algorithm Al is a set of all periodic functions F. The predefined number of periodic functions F amounts to 10 to 50, in particular to 15 to 25, and to 20 functions in this particular exemplary embodiment.

[0063] FIG. 2 shows a flow diagram of the computer-implemented method for frequency coding of image data from the imaging sensor according to the preferred embodiment of the present invention.

[0064] Algorithm A1 converts frequency coded depth values TW of individual image E into a vector representation.

[0065] Second image data BD2, which are output by algorithm A1 and have frequency coded depth values TW of individual image E, are supplied as input data for the training and/or inference of an algorithm A2 of machine learning.

[0066] Algorithm A2 of machine learning preferably is an artificial neural network. Other trained algorithms suitable for the image classification, for instance, may be used as an alternative.

[0067] Algorithm A2 of machine learning may alternatively be used for a semantic segmentation, for instance, that is to say, a region-wise, in particular pixel-wise, classification. In addition, algorithm A2 of machine learning is alternatively usable for the detection, i.e., a classification, as to whether or not the object is present.

[0068] In this particular exemplary embodiment, algorithm A2 of machine learning carries out a classification KL of objects included in second image data BD2. As an alternative, algorithm A2 of machine learning is able to be used for a regression task or an image-segmentation task, for instance.

[0069] FIG. 3 shows a schematic representation of a system for frequency coding of image data from the imaging sensor according to a preferred embodiment of the present invention.

[0070] System 1 includes an imaging sensor 10, which supplies first image data BD1 of a recorded individual image, first image data BD1 having depth values of the individual image that are coded as a whole number or a floating-point number.

[0071] In addition, the system has a computing device 20, which is designed to carry out the method according to the present invention.

[0072] Computing device 20 is set up to receive first image data BD1, first image data BD1 being processed by an algorithm which is executed on computing device 20 and frequency codes the depth values of the individual image using a predefined number of periodic functions.

[0073] Computing device 20 is furthermore set up for the output of second image data by the algorithm, the second image data having frequency coded depth values of the individual image.

[0074] The present invention was described in the context of a method and system for frequency coding image data from the imaging sensor, but it is not restricted thereto.

[0075] Further alternative possible uses are the determination of a stereo depth within the framework of an indoor robotics navigation, for example.

[0076] Robot arms, for instance used for sorting objects or for automated screw-fitting and welding, act in a three-dimensional space, but often only see the 2D projection of the space as an RGB image. This does not permit decisions about free trajectories of the robot arm because the depth information gets lost in the projection. Depth images in which each pixel of the image corresponds to the distance from the object surface, on the other hand, enable decisions about the collision-free navigation or the grasping of objects.

[0077] Neural networks have the capability of generating very good depth images (regression of the distance values per pixel) from two RGB images. A partial task which the networks solve in the process is the allocation of pixel values in the two images, that is to say, which pixel in the left image and which pixel in the right image is imaging the same point of the 3D world.

[0078] Small differences such as those caused by a different illumination or reflections make the matching of pixels in both images more difficult. The standard representation in RGB therefore varies. Here, too, the frequency coding according to the present invention is helpful in that it decodes the factors of the variation in more than three channels and simultaneously standardizes them, which should allow better training of the neural network.

[0079] A further alternative application case is a 2D LIDAR as a sensor which should help the robot arm in finding the correct point for grasping the object. A neural network ascertains (classifies) the correct point for grasping the object based on the sensor data.

[0080] The LIDAR sensor supplies distance values with respect to the object surface in the form of floating-point numbers. The distance of the object represents the dominant portion in the floating-point number, surface features take up a small portion but are essential for the grasping decision, e.g., a handle of a cup that has a length of two centimeters and is situated at a distance of one meter from the sensor.

[0081] The proposed frequency coding of the depth values is helpful here in redundantly representing portions of the floating-point number corresponding to the handle and thereby making them more accessible to the neural network.

[0082] A further alternative application case is the optical inspection on the production line. In the optical inspection, RGB/depth images are analyzed or depth measurements are carried out via LIDAR. Neural networks are used for classifying, localizing or regressing errors. As a consequence, the method and system according to the present invention are able to be applied in this context as well.

[0083] FIG. 4 shows a flow diagram of a computer-implemented method for supplying an algorithm of machine learning for the classification of objects included in image data of an individual image from an imaging sensor according to the preferred embodiment of the present invention.

[0084] The method includes the receiving S1′ of image data BD2 which are output by algorithm Al according to the present invention and have frequency coded depth values TW of individual image E.

[0085] The present invention furthermore includes the receiving S2′ of a classification result allocated to an object included in a respective individual image E.

[0086] Moreover, the present method includes training S3′ of algorithm A2 of machine learning using image data BD2 and the classification result allocated to the object included in respective individual image E, the training being implemented by an optimization algorithm A3, which calculates an extreme value of a loss function.