METHOD FOR DETECTING AT LEAST ONE ROAD USER

Abstract

The invention relates to a method for detecting at least one road user on a traffic route by means of a radar sensor and an optical detector, wherein with said method radar radiation is emitted by at least one radar transmitter of the radar sensor and reflected by the at least one road user, the reflected radar radiation is detected by means of at least one radar receiver of the radar sensor, the detected radar radiation is evaluated in such a way that at least one distance and one radial velocity of the at least one road user relative to the radar sensor is determined, an optical image of the at least one road user is detected by means of the optical detector, and the optical image is evaluation,
wherein at least one parameter of the at least one road user is determined both from the detected radar radiation and the optical image.

Claims

1. A method for detecting at least one road user on a traffic route by a radar sensor and an optical detector, comprising: emitting radar radiation by at least one radar transmitter of the radar sensor, wherein said radar radiation is reflected by the at least one road user, detecting the reflected radar radiation by at least one radar receiver of the radar sensor, evaluating the detected radar radiation in such a way that at least a distance and a radial velocity of the at least one road user relative to the radar sensor is determined, detecting an optical image of the at least one road user by the optical detector, and evaluating the optical image detected, wherein at least one parameter of the at least one road user is determined both from the detected radar radiation and the optical image.

2. The method according to claim 1, further comprising determining at least one detection matrix from the detected reflected radar radiation, from which a distance and radial velocity are determined.

3. The method according to claim 2, wherein the at least one detection matrix comprises multiple detection matrices which contain information about road users in different directions, and information about a direction in which the at least one road user is located relative to the radar sensor.

4. The method according to claim 1 further comprising using the optical image to determine a direction in which the at least one road user is located relative to the optical detector.

5. The method according to claim 1 wherein the reflected radar radiation and the optical image of the at least one road user are detected several times in succession at different points in time.

6. The method according to claim 5, wherein when evaluating the detected reflected radar radiation, at least one parameter is taken into account which was determined during at least one previous evaluation of the optical image and/or, when evaluating the optical image, at least one parameter is taken into account which was determined during at least one previous evaluation of the detected radar radiation.

7. The method according to claim 5 further comprising determining a velocity of the at least one road user and/or a direction in which the at least one road user is located relative to the optical detector.

8. The method according to claim 1 further comprising detecting a component of the at least one road user in the detected optical image, wherein a dimension of the component is known in at least one direction, and wherein a distance of the at least one road user from the optical detector is determined from the detected optical image.

9. The method according to claim 1 further comprising determining from the detected reflected radar radiation and the optical image a relative offset between the radar sensor and the optical detector and/or a relative orientation of the radar sensor to the optical detector.

10. The method according to claim 9, wherein a distance and/or an orientation is determined multiple times during the method.

11. The method according to claim 1 wherein during evaluation of the optical image, determining an extent of the at least one road user in the azimuth direction and/or in the elevation direction and, during evaluation of the detected reflected radar radiation, determining a distance of the at least one road user, and determining a position and/or size of the at least one road user from the extent and distance of the at least one road user.

12. The method according to claim 1 wherein when evaluating the optical image, determining an orientation of the at least one road user, determining with the radial velocity and distance of the at least one road user from the detected reflected radar radiation, and determining a complete velocity and/or direction of movement and/or a longitudinal acceleration and/or a transverse acceleration of the at least one road user.

13. The method according to claim 1 wherein the at least one road user is classified on a basis of parameters, at least one of which was determined during evaluation of the detected reflected radar radiation and at least one of which was determined during the evaluation of the optical image.

14. A device for detecting at least one road user on a traffic route, comprising: at least one radar sensor with at least one radar transmitter and at least one radar receiver, at least one optical detector, and an electric control unit which is configured to carry out a method according to claim 1.

15. The device according to claim 14, wherein the at least one radar transmitter comprises multiple transmission antennas and/or the at least one radar receiver has multiple receiving antennas.

16. The method according to claim 2 wherein the at least one detection matrix is a Range-Doppler-Matrix.

17. The method according to claim 5 further comprising determining an optical flow from different successively detected optical images of the at least one road user.

18. The method according to claim 7 further comprising determining a radial velocity from the detected reflected radar radiation.

19. The method according to claim 10 further comprising comparing the distance and/or the orientation to predetermined nominal values.

Description

[0031] In the following, some examples of embodiments of the present invention will be explained in more detail by way of the attached figures: They show

[0032] FIGS. 1 to 3—various stages in the evaluation of the radar radiation,

[0033] FIG. 4—the schematic arrangement of multiple radar receivers,

[0034] FIG. 5—the schematic representation of multiple detection matrices, in particular range-Doppler matrices,

[0035] FIG. 6—a further step in the evaluation of the direction-depending detection matrices,

[0036] FIGS. 7 and 8—different road users and their parameters and

[0037] FIG. 9—the schematic representation of the determination of range-Doppler matrices for various spatial directions.

[0038] FIGS. 1 to 3 schematically depict three steps for determining a detection matrix, in particular a range-Doppler matrix. The outgoing radar radiation is preferably emitted in the form of frequency ramps, which are referred to as “ramp”. In the matrix shown in FIG. 1, the respective received or detected radar radiation is stored as complex amplitude values under the terms “Sample 0”, “Sample 1”, etc., which designate the columns of the matrix shown. The different columns thus relate to different points in time, which are preferably equidistant, at which the detected radar radiation is read. This is done several times for a plurality of frequency ramps emitted one after the other. The respective measured values are entered in the different rows, each row representing one frequency ramp.

[0039] In fact, the detected radar radiation is mixed with the emitted radar radiation. This mixed radiation is transferred to the matrix shown in FIG. 1 and fed to the evaluation. From the frequency offset between the transmitted radar radiation and the detected radar radiation, which is essentially defined by the amount of time radar radiation takes to travel from the radar transmitter to the road user and back to the radar receiver, a conclusion can be reached about the distance of the road user from the radar sensor.

[0040] To this end, a first Fourier transformation is performed in the example of an embodiment shown, whereby the Fourier transformation is performed for each row of the matrix in FIG. 1. Consequently, the measured values within each individual frequency ramp are transferred to a Fourier transform. The result is the matrix shown schematically in FIG. 2, in which the columns are now defined by different radial distances from the radar sensor.

[0041] In order to arrive at the matrix shown in FIG. 3, a second Fourier transformation is performed, which is now not performed over the entries of a row, but over the entries of a column of the matrix. This is the so-called Doppler transformation, which is performed over a so-called “range gate”, i.e. a column of the matrix shown in FIG. 2. This results in Doppler frequencies which can be converted into radial velocities, i.e. velocities leading towards or away from the radar sensor. In this way, the range-Doppler matrix shown in FIG. 3 is created.

[0042] FIG. 4 schematically shows an embodiment of a radar receiver which can be part of a radar sensor. It has a plurality of receiving antennas 2 which are equidistantly arranged at a distance d from each other. An equidistant arrangement is advantageous, but not necessarily required. The arrows 4 represent radar radiation that has been reflected by a road user in the example of an embodiment shown. The angle between the extension direction in which the receiving antennas 2 are arranged next to each other and the direction of the arrows 4 from which the reflected radar radiation strikes the receiving antennas 2 is an angle other than 90°. The dashed lines 6 schematically represent wave fronts of plane waves of the radar radiation. Due to the angle, the reflected radar radiation in the example of an embodiment shown first strikes the receiving antennas 2 arranged furthest to the right and only strikes the receiving antennas 2 arranged further to the left at later points in time.

[0043] If the radar radiation detected in this way is to be evaluated according to FIGS. 1 to 3, the detected radar radiation can be mixed with the transmitted radiation for each of the receiving antennas 2 and fed to the evaluation. In this way, a large number of range-Doppler matrices are created. In particular, a separate range-Doppler matrix can be calculated for each desired spatial direction by feeding the complex amplitude values for the respective desired spatial direction to the evaluation procedure, such values having been phase corrected. This method is known as digital beam forming (DBF). This procedure is shown systematically in FIG. 9. The number of possible M spatial directions, for each of which a range-Doppler matrix is calculated, can be significantly higher than the number of N (virtual) receiving antennas. Even if in principle M could be enlarged further and further, the computational effort and the maximum achievable spatial resolution provide arguments against it. In particular, however, a range-Doppler matrix can be calculated explicitly for certain spatial directions which turn out to be interesting based on the analysis of the camera image, and this result can be fed to the fusion.

[0044] While in FIG. 4 the schematically depicted receiving antennas 2 are arranged in a row or line, but this is of course not necessary in practice. Of course, it is also possible to arrange receiving antennas 2 in a two-dimensional arrangement, for example a rectangular grid. This arrangement thus enables the calculation of a range-Doppler matrix for arbitrary spatial directions through the application of digital beam forming. An example of range-Doppler matrices calculated in this way for various spatial directions is shown schematically in FIG. 5. Each of the shaded rectangles represents a range-Doppler matrix 8 calculated according to the methodology in FIG. 9.

[0045] By way of known calculations of beam shaping, different range-Doppler matrices 8 can be added in a weighted way to obtain an angular resolution. FIG. 6 shows schematically that the nine range-Doppler matrices 8 shown in FIG. 5 are combined. This is of course also possible with more or fewer range-Doppler matrices 8. By combining the information obtained and calculated from the different range-Doppler matrices 8, a depth image 10 can be determined. This contains not only the radial distance of a road user from the radar sensor and their radial velocity in relation to the radar sensor, but in particular also angular information about the azimuth direction and, if necessary, the elevation direction. The direction in which an angular resolution can be achieved depends in particular on how the pattern arrangement of the receiving antennas 2 is constructed.

[0046] FIG. 7 shows schematically different road users which can be distinguished by a classification. A truck 12, a motor vehicle 14, a motorcycle 16, a cyclist 18 and a pedestrian 20 are depicted. These different road users can be distinguished from each other on the basis of the parameters to be determined. This applies to velocities as well as stray cross sections, dimensions and behavior on the respective traffic route.

[0047] FIG. 8 shows—schematically and as an example—the truck 12 with different parameters. It has a height 22, a width 24 and a length 26, which represent some of the parameters to be determined. In addition, it moves forward along the velocity arrow 28 and has an orientation and direction, which may also be referred to as a pose.

[0048] For example, the dimensions of height 22, width 24, and length 26 can be determined from the optical image captured by means of the at least one optical detector.

[0049] This is particularly the case when elements that are also captured in the optical image are known, such as buildings, signs, dimensions of displayed elements, or other variables of the displayed objects. In this case, the dimensions of the road user contained in the optical image can be related to the known dimensions of another displayed object, so that an estimate of the dimensions is possible. However, a much better result is obtained if the dimensions contained in the optical image are related to the distance data determined from the evaluated radar radiation. The width 24 of the truck 12 can be determined via the distance of various points of the road user depicted in the optical image, for example the two exterior mirrors of a truck 12, if it is known how far away the truck 12 is from the radar sensor. This information can be determined from the range-Doppler matrix 8 or otherwise from the detected and evaluated radar beams.

[0050] A complete location of the road user, in particular of the truck 12 shown, can also be determined from the evaluated optical image alone, provided that the road geometry, i.e. in particular the course, the number of lanes, and other structural conditions of the traffic route, are known. In this case, too, the location can at least be estimated by evaluating only the optical image. However, a better result is also achieved with these parameters if this data determined from the optical image are combined with range data determined from the evaluation of the detected radar radiation. Particularly in the case of so-called radar tracking, in which a road user is tracked over a period of time and their movement is tracked by evaluating a plurality, in particular many, detection matrices generated in succession over time, it is particularly easy to determine the distance and the change in distance of the road user to the radar sensor. In combination with the significantly better angular resolution of the optical image compared to the radar sensor, the actual location of the road user can be determined very accurately.

[0051] The same applies to the pose, i.e. in particular the orientation of the road user. This can also be determined, for example, by means of image recognition software when evaluating the optical image. However, the combination with the information determined from the detected radar radiation also improves the quality of the determined parameter in this case and renders it possible to compare and check the determined values. If, for example, the orientation of a road user determined from the optical image, for example of the truck 12 shown, does not match the velocity determined from the detected radar radiation and, in particular, the direction of this velocity, this indicates an incorrect evaluation of the data.

[0052] The complete velocity, i.e. the determination of the velocity as a vectorial quantity with direction and magnitude, can also be improved by combining the data from the evaluation of the optical image with the determined parameters from the detected radar radiation. While an evaluation of several optical images taken in succession determines an optical flow and thus, for example, the change in the range of a road user in the optical image can be inferred from the change in the distance, the evaluation of radar radiation detected in succession is significantly more accurate, particularly with regard to the distance and the change in the distance to the radar sensor.

[0053] FIG. 9 schematically depicts the way to calculate range-Doppler matrices 8 for different spatial directions. Different receiving antennas 2, of which only two are shown for the sake of clarity, are used for this purpose. The received measurement signals are first used in a conventional manner to determine a range-Doppler matrix 8 in each case. This contains complex values, which in particular have an intensity I and a phase. For example, if the receiving antennas 2 are arranged next to each other, there is an offset between them. As a result, the radar radiation reflected by a road user reaches the different receiving antennas 2 at different times, as shown schematically in FIG. 4. As a result, the individual entries of the range-Doppler matrices generated separately for each receiving antenna 2 also differ in the phase Q.

[0054] If a range-Doppler matrix 8 is to be calculated for a certain spatial direction, the individual range-Doppler matrices 8 are summed up in a summer 30. A separate summer 30 is depicted for each desired spatial direction. The individual range-Doppler matrices are previously subjected to a phase shifter 32 that has been individually determined for the desired spatial direction. The phase Q of each individual matrix element of the range-Doppler matrices 8 to be summed is changed by means of these phase shifters 32 in the course of digital beam forming in order to achieve the desired spatial direction.

REFERENCE LIST

[0055] 2 receiving antenna [0056] 4 arrow [0057] 6 dashed line [0058] 8 range-Doppler matrix [0059] 10 depth image [0060] 12 truck [0061] 14 motor vehicle [0062] 16 motorbike [0063] 18 cyclist [0064] 20 pedestrian [0065] 22 height [0066] 24 width [0067] 26 length [0068] 28 velocity arrow [0069] 30 summer [0070] 32 phase shifter