Abstract
A method for deriving at least one item of information from images of a stereo camera. A resource-saving and at the same time accurate derivation of information from the images is achieved by alternately using and processing the images with methods of monocular image processing and deriving at least one item of information from the results. A motor vehicle comprising a stereo camera and comprising a control device which carries out the method, are also described.
Claims
1. A method for deriving at least one item of information from images of a stereo camera including a first camera and a second camera, the method comprising the following steps: capturing, by the first camera, first images, and capturing, by the second camera, second images; in a monocular step, alternately using and processing the first images and the second images with methods of monocular image processing, and outputting results of the monocular step; and in a derivation step, deriving at least one item of information from results of the monocular step.
2. The method according to claim 1, wherein in the monocular step, an optical flow of the alternately used first images and second images is determined and output as a result.
3. The method according to claim 1, wherein, in the monocular step, at least one image feature of the alternately used first images and second images is tracked over time and output as a result.
4. The method according to claim 1, wherein monocular deep learning methods are used in the monocular step.
5. The method according to claim 1, wherein at least one monocular method for deriving at least one of the at least one item of information is used in the derivation step.
6. The method according to claim 1, wherein, in the derivation step, a movement of the stereo camera itself is derived as information.
7. The method according to claim 1, wherein the derivation step includes warping.
8. The method according to claim 1, wherein the derivation step derives an estimation of depths and/or detection of self-moving objects, in an environment of the stereo camera as information.
9. A non-transitory computer-readable storage medium on which is stored a computer program for deriving at least one item of information from images of a stereo camera including a first camera and a second camera, the computer program, when executed by a computer, causing the computer to perform the following steps: capturing, by the first camera, first images, and capturing, by the second camera, second images; in a monocular step, alternately and processing the first images and the second images with methods of monocular image processing, and outputting results of the monocular step; and in a derivation step, deriving at least one item of information from results of the monocular step.
10. A motor vehicle, comprising: a stereo camera including a first camera which captures first images during operation, and a second camera which captures second images during operation; and control device configured to derive at least one item of information from the first and second images of the stereo camera, the control device configured to: in a monocular step, alternately use and process the first images and the second images with methods of monocular image processing, and outputting results of the monocular step; and in a derivation step, derive at least one item of information from results of the monocular step.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0040] FIG. 1 shows a greatly simplified circuit diagram-like illustration of a stereo camera in a motor vehicle,
[0041] FIG. 2 shows a symbolic illustration of images captured with the stereo camera,
[0042] FIG. 3 shows a flow chart to explain a method for deriving at least one item of information from images of the stereo camera, according to an example embodiment of the present invention.
[0043] FIG. 4 shows images of the camera.
[0044] FIG. 5 shows a difference between images of FIG. 4 according to the related art.
[0045] FIG. 6 shows a difference between images of FIG. 4 according to the present invention.
DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS
[0046] A stereo camera 3 comprising a first camera 1 and a second camera 2, as shown greatly simplified in FIG. 1, is in particular used in a motor vehicle 100, as shown as an example and greatly simplified and in the manner of a circuit diagram in FIG. 1. During operation, the first camera 1 captures first images 4 and the second camera 2 captures second images 5. The images 4, 5 are shown in FIG. 2, wherein the first images 4 are shown symbolically as rectangles and the second images 5 as triangles. In FIG. 4, examples of images 4, 5 are shown greatly simplified.
[0047] As is typical for stereo cameras, the two cameras 1, 2 have an intrinsic alignment to one another, wherein the distance between the cameras 1, 2 is preferably known and is taken into account in the method described in the following. The cameras 1, 2 furthermore have an at least similar alignment and similar overlapping fields of view. The cameras 1, 2 are preferably also configured in the same way, in particular identically.
[0048] As indicated in FIG. 2 and shown as an example by the flow chart in FIG. 3, at least one item of information, in particular about the environment of the stereo camera 3 and thus the motor vehicle 100, is derived from the images 4, 5. For this purpose, the first images 4 and the second images 5 are used alternately and processed with methods of monocular image processing. A time axis is shown as an arrow in FIG. 2. In the shown example, therefore, both cameras 1, 2 respectively capture successive images 4, 5. In other words, at successive points in time tN, wherein N is a natural number greater than or equal to 0, the first camera 1 captures a first image 4 and the second camera 2 captures a second image 5. As indicated in FIG. 2 with dashed lines, not both captured images 4, 5 are used at the points in time tN; rather a first image 4 of the first camera 1 and a second image 5 of the second camera 2 are alternately used and processed with methods of monocular image processing.
[0049] According to the flow chart shown in FIG. 3, this is carried out in a step 20, which hereinafter is also referred to as a monocular step 20. In the monocular step 20, therefore, the first images 4 and the second images 5 are alternately used and processed with methods of monocular image processing. A new image 4, 5 is thus processed at each point in time tN. Compared to stereo methods available in the related art (not shown), in which both new images 4, 5 have to be processed at each point in time tN, this results in a significant reduction in the required resources.
[0050] The result of this image processing is transmitted to a subsequent step 21. In the step 21, at least one item of information is derived from results of the monocular step 20. The step 21 is also referred to hereinafter as the derivation step 21.
[0051] As can further be seen in FIG. 3, the monocular step 20 in the shown embodiment example includes a step 22, in which the images 4, 5 are used alternately as described, and a step 23, in which the respective currently being used image 4, 5 is processed with methods of monocular image processing. The step 22 is hereinafter also referred to as alternating step 22 and the step 23 is referred to as the processing step 23. As can further be seen in FIG. 3, after the derivation step 21, the method can return to the monocular step 20, in particular the alternating step 22.
[0052] The advantages of the method according to the invention over a monocular system consisting of only one camera are explained in the following with reference to FIGS. 4 to 6 as examples.
[0053] FIG. 4 shows the time axis with an arrow, wherein images 4, 5 are shown at two successive points in time t0 and t1. Images 4 of the first camera 1, which can be a left camera 1, for example, are shown at both points in time t0, t1, and an image 5 of the second camera 2, which can be a right camera 2, for example, is shown at the point in time t1. If only one camera 1, 2 were provided, the system would be a monocular system. In the following, it is assumed purely as an example that the monocular system consists of the first camera 1. This results in the difference indicated in FIG. 5 between the first images 4 of the monocular system consisting of the first camera 1 which are captured at the point in time t0 and t1. The method according to the invention, on the other hand, is used to create the difference between the first image 4 captured by the first camera 1 at the point in time t0 and the second images 5 captured by the second camera 2 at time t1, as indicated as an example in FIG. 6. As a comparison between FIGS. 5 and 6 shows, the difference in FIG. 6 and thus in the method according to the invention is significantly more pronounced than the difference in FIG. 5 and thus according to the related art. The method according to the invention therefore makes it possible to derive information from the images 4, 5 more reliably, more accurately and in a significantly simplified manner. It is in principle also possible to capture only one image 4, 5 with the camera 1, 2 relevant to the time tN at any time tN.
[0054] The monocular step 20, in particular the processing step 23, preferably includes determining an optical flow from the alternately used first images 4 and second images 5, i.e. similar to the difference shown in FIG. 6. Alternatively or additionally, monocular deep learning methods are used in the monocular step 20, in particular in the processing step 21.
[0055] Advantageously, at least one monocular method for deriving at least one of the at least one item of information is used in the derivation step 21.
[0056] In the derivation step 21, for example, a movement of the stereo camera 3 itself and thus of the motor vehicle 100 is derived as information. Alternatively or additionally, the derivation step 21 can include warping. This involves, for example, warping the images 4, 5, depth measurement values, predications, features or certainties, preferably in accordance with a depth map and another type of predication which can be both forward and backward in time. The derivation step 21 alternatively or additionally includes the estimation of depths and/or the detection of objects 10 (see FIG. 4), in particular self-moving objects 11, in the environment of the stereo camera 3 and thus the motor vehicle 100 as derived information. The derivation step 21 can also include estimating a change in the images 4, 5 as information.
[0057] As indicated in FIGS. 1 and 3, the at least one derived item of information can be provided to a driving assistance system 102 of the motor vehicle 100. The driving assistance system 102 is configured to provide assistance when driving the motor vehicle 100 and/or for at least partially autonomous driving of the motor vehicle 100, wherein the driving assistance system 102 takes the at least one derived item of information into account for this purpose.
[0058] As can be seen in FIG. 1, for carrying out the method, the motor vehicle 100 comprises a control device 101 which is connected to the stereo camera 3 and is configured accordingly. The control device 101 can comprise a not depicted computer system, for example, or it can be a component of the computer system. The control device 101 or the computer system can include an appropriately configured computer program product for carrying out the method.
[0059] In the embodiment example shown in FIG. 1, the stereo camera 3 is mounted in the front in a Z-direction 103 of the motor vehicle 100, for example on a not depicted windshield. Thus, during operation, the cameras 1, 2 capture images 4, 5 from the front area, in particular from the front area in the forward direction of travel, of the motor vehicle 100. In a driving situation shown in FIG. 4, objects 10, such as self-moving objects 11, can be captured in the images 4, 5 as explained above and their depths, in particular their relative position to the stereo camera 3 and consequently to the motor vehicle 100, can be derived.
