DETERMINING THE RADIAL VELOCITY OF OBJECTS IN VEHICLE ENVIRONMENTS

Abstract

The present disclosure relates to a computer-implemented method for determining a radial velocity of an object in a surrounding of a vehicle, the method comprising the steps of: obtaining measurement data from a radar, the measurement data comprising signal data indicative of a measured radial velocity of the object, mapping the measured radial velocity of the object to a plurality of radial velocity intervals, determining, using an artificial intelligence (AI) engine, a probability value for each interval of the plurality of radial velocity intervals based on supplemental measurement data, and determining the radial velocity of the object by selecting an interval of the plurality of radial velocity intervals based on the probability value. The disclosure further relates to a corresponding apparatus, computer program and vehicle.

Claims

1. A computer-implemented method for determining a radial velocity of an object in a surrounding of a vehicle, the method comprising the steps of: obtaining measurement data from a radar, the measurement data comprising signal data indicative of a measured radial velocity of the object; mapping the measured radial velocity of the object to a plurality of radial velocity intervals; determining, using an artificial intelligence, AI, engine, a probability value for each interval of the plurality of radial velocity intervals based on supplemental measurement data; and determining the radial velocity of the object by selecting an interval of the plurality of radial velocity intervals based on the probability value.

2. The method according to claim 1, wherein determining the probability value is based on one or more of the following values, included in the supplemental measurement data: reflection angle data associated with the object; environmental context data; and the signal data indicative of a measured radial velocity of the object.

3. The method according to claim 2, wherein selecting the interval further comprising: estimating a radial velocity value of the object based on the supplemental measurement data, preferably included in the measurement data; and considering the estimated radial velocity value for determining the probability value assigned to each interval of the plurality of radial velocity intervals.

4. The method according to claim 1, wherein the radial velocity of the object is determined by selecting the interval of the plurality of radial velocity intervals having the highest probability value.

5. The method according to claim 1, wherein determining the radial velocity of the object further comprises: defining the measured radial velocity mapped to the selected interval as the radial velocity of the object.

6. The method according to claim 1, further comprising: generating the plurality of radial velocity intervals by shifting one unambiguous radial velocity interval to form at least one ambiguous radial velocity interval; wherein each mapped measured radial velocity interval represents a potential radial velocity of the object.

7. The method according to claim 3, further comprising: deriving the probability value assigned to each interval based on a difference between the potential measured radial velocities and the estimated radial velocity value.

8. The method according to claim 7, wherein a smallest difference between the potential measured radial velocity and the estimated radial velocity values corresponds to the highest probability value.

9. The method according to claim 1, wherein the intervals are allocated adjacent to each other in the Doppler velocity dimension.

10. The method according to claim 1, wherein the intervals have a common width in the Doppler velocity dimension, wherein, preferably, the common width depends on characteristics of the radar.

11. The method according to claim 1, wherein during a training process of the AI engine, the AI engine maps feature embeddings of the environmental context data, the reflection angle data and measured radial velocities to the probability values assigned to each interval of the plurality of radial velocity intervals.

12. The method according to claim 1, further comprising: determining an operating instruction for the vehicle based on the determined velocity of the object affecting a function of a vehicle assistance system, wherein the function comprises at least one of: displaying the object and preferably the velocity of the object on a display of the vehicle; conducting a vehicle path planning; triggering a warning; affecting control of the vehicle during a parking process and/or during driving.

13. An apparatus comprising means configured to perform the method of claim 1.

14. A vehicle comprising an apparatus according to claim 13.

15. A non-transitory computer readable medium storing computer executable instructions, which, when executed by a computing system, cause the computing system to perform the method of claim 1.

Description

DRAWINGS

[0046] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0047] Various aspects of the present invention are described in more detail in the following by reference to the accompanying figures without the present invention being limited to the embodiments of these figures.

[0048] FIG. 1 illustrates potential ambiguities of radial velocity measurements using a radar.

[0049] FIG. 2 illustrates mapped measured radial velocities of an object mapped to a plurality of intervals.

[0050] FIG. 3 illustrates a flow chart of a method for disambiguating radial velocity measurements.

[0051] FIG. 4 illustrates mapped measured radial velocities of an object mapped to the plurality of intervals and an estimated velocity of the object.

[0052] FIG. 5 illustrates a flow chart of an exemplary implementation of the method for determining the actual radial velocity of an object based on a radar measurement.

[0053] FIG. 6 illustrates an exemplary implementation of the method using a machine learning algorithm.

[0054] FIG. 7 illustrates the training and inference of an exemplary artificial intelligence engine application of the method for determining the radial velocity of an object.

[0055] FIG. 8 illustrates a vehicle according to aspects of the present invention.

[0056] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings.

DETAILED DESCRIPTION

[0057] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0058] In FIG. 1, the ambiguity of measured radial velocity data is illustrated. Due to the constraints related to the maximum unambiguous velocity and the maximum resolvable velocity, radars in the art are typically configured to unambiguously cover a certain velocity/distance range/interval. An exemplary radar known in the art is a FMCW radar. The FMCW radar typically provides measurement data with Doppler velocity information of an object. An exemplary FMCW radar may have an unambiguous interval of [?30 m/s; 30 m/s] (corresponding to [?108 km/h; 108 km/h]). The velocities provided with the interval are not the total velocities of objects, but radial velocities with respect to the radar or the vehicle on which the radar is mounted. The term unambiguous interval means that the radar can reliably detect the actual velocity of the object in this range, wherein velocities outside of this range are not covered reliably. That is, radial velocities of the object measured outside this interval are typically undersampled such that the actual velocity of the object are difficult to obtain in an unambiguous manner. In particular, velocities outside the range violate the Nyquist-Shannon theorem such that reliable velocity information cannot be obtained from the radar data without further processing. In fact, a radar configured with a certain unambiguous interval would always provide a velocity within the range, no matter how fast the object actually is with respect to the radar (radial velocity). For example, a radar having the unambiguous interval of [?82.8 km/h; 37.8 km/h] would be able to reliably determine the radial velocity of an object that is 20 km/h faster than the radar (i.e., the vehicle that carries the radar) itself. However, it would not be able to reliably determine the radial velocity of an object that is 60 km/h faster. In this case, the measurement data from the radar would not feedback the correct radial velocity but a radial velocity from the range of [?82.8 km/h; 37.8 km/h]. In the following, the (maximum) interval of the radar is denoted as I.sub.D.

[0059] In FIG. 1, the unambiguous interval will be described in more detail. In the plot shown in FIG. 1, the x-axis indicates the actual radial velocity of the object, and the y-axis indicates the corresponding Doppler velocity measurement provided by the radar. It can be seen that the measured radial velocity of the vehicle will only be provided within the range of I.sub.D, no matter how fast the object actually is. As already indicated above, this is due to aliasing effects due to violation of the Nyquist-Shannon theorem at high/low velocities outside the unambiguous interval. The central line 110 indicates the unambiguous interval. In this range, any measured radial velocity v.sub.r,measure corresponds to the actual radial velocity of the object v.sub.r,object. However, object radial velocities above the unambiguous interval (lines to the right) are measured at least one interval of I.sub.D below their actual value. For example, an object having a radial velocity of v.sub.r+1I.sub.D would be measured as having a radial velocity of v.sub.r. Similarly, object radial velocities below the unambiguous interval are measured at least one interval of I.sub.D above their actual value. For example, an object having a radial velocity of v.sub.r?2I.sub.D would also be measured as having a radial velocity of v.sub.r.

Disambiguating Module

[0060] Referring to FIGS. 2 and 3, an exemplary method for disambiguating the ambiguous measured radial velocity from radar measurement data is shown. As described above, from the radar, a measured radial velocity of the object is provided/obtained. While this measured radial velocity may resemble a value within the unambiguous interval of the radar, this radial velocity value may not be the actual radial velocity of the object, but an ambiguous radial velocity in an area outside the unambiguous interval.

[0061] In a first step of the disambiguation method, the measured radial velocity of the object may be mapped, as shown in FIG. 2. From the measured radial velocity potential radial velocities 210, 212, 214, 216, 218 that may cause the measured radial velocity waveform (when violation of the Nyquist-Shannon theorem is assumed) are mapped in the frequency domain and mapped to the unambiguous interval 201c and one or more ambiguous intervals 201a, 201b, 201d, 201e. Since, in the case of an FMCW radar, the radial velocities are determined by the phase shift of the transmitted radar signal, the potential radial velocities correspond to the measured radial velocity shifted by multiples of the interval. For example, a measured radial velocity of v.sub.r results in multiple potential measured radial velocities of v.sub.r, v.sub.r+I.sub.D, v.sub.r+2I.sub.D, v.sub.r?I.sub.D and v.sub.r?2I.sub.D. In this example, only two intervals 201d, 201e above and two intervals 201a, 201b below the unambiguous interval 201c are considered. However, the number of intervals may be changed based on the application. From the potential measured radial velocities, one potential measured radial velocity corresponds to the actual radial velocity of the object. Similarly, one of the intervals 201a, 201b, 201c, 201d, 201e corresponds to the radial velocity range in which the object is moving. Disambiguating the interval allows to determine subsequent radial velocities more effective, as it is likely that the measured radial velocity is still allocated in (or at least close to) the previous interval.

[0062] To determine which of the intervals (i.e., the measured radial velocity) is most likely the interval comprising the actual radial velocity of the object, a radial velocity of the object is estimated.

[0063] In FIG. 3, a disambiguation module (Doppler disambiguation module) 310 for defining the interval that has the highest probability of comprising the actual radial velocity of the object is illustrated. The disambiguation module 310 obtains the (unfolded) intervals 201a, 201b, 201c, 201d,201e (also denoted as D.sub.prop) obtained in the previous step as well as probability values att.sub.prop for each proposal to be the actual radial velocity of the object.

[0064] While the term highest probability is used in this context, performing a weighted sum between Doppler proposals and the probability values att.sub.prop may be sufficient to reliably determine the actual radial velocity of the object.

[0065] The probability values att.sub.prop may be determined by means of an AI engine. For example, neural network layers within the AI engine may be used to determine/predict att.sub.prop. In an AI-based (machine learning) approach, att.sub.prop may be determined/predicted as a function using dense DL-layers of supplemental measurement data, such as environmental context, the reflection angle of the object and the measured radial velocity of each detection. The reflection angle of the object relates to the angle at which the radar signal is reflected back from the object to the radar. The environmental context describes feature encodings that may be extracted by the neural network based on the radar input data. The environmental context may be obtained from an output of a Recurrent Neural Network (RNN) module (described in more detail with reference to FIG. 5 below). These outputs contain encodings that could be used to extract a velocity estimate of the object (by incorporating the reflection angle information related to the object, the radial velocity can be obtained). Exemplary machine learning methods for the above engine may be soft attention (softmax or temperature-softmax interval probabilities) and/or hard attention.

[0066] While the estimated radial velocity of the object may only be a rough estimate, it is well below the potential error caused by the ambiguously measured radial velocity values. Thus, in a next step, the estimated measured radial velocity 420 is mapped to a corresponding interval of the plurality of intervals and the distance between the mapped potential measured radial velocity values mapped to the individual intervals are measured. The respective distances obtained this way are then utilized within the disambiguation module to determine/predict probabilities for the intervals each detection belongs to.

[0067] In the example shown in FIG. 4, the estimated measured radial velocity corresponds to the interval 201d, including the (potential) measured radial velocity value v.sub.r+I.sub.D. In this case, the distance between v.sub.r+I.sub.D and the estimated radial velocity of the object 412a has the smallest difference with respect to the other mapped radial velocities. Accordingly, the interval 201d including the measured radial velocity value v.sub.r+I.sub.D will be assigned the highest probability of comprising the actual radial velocity of the object. Since in this example the potential measured radial velocity value included in the unambiguous interval 201c has the second smallest difference to the estimated radial velocity of the object 410a, the unambiguous interval 201c will also be provided with a relatively high probability of comprising the actual radial velocity of the object, but smaller than the probability associated with the interval including the measured radial velocity value v.sub.r+I.sub.D. Each of the intervals 201a-e may obtain a probability value this way.

[0068] With this information, the most probable interval will be selected and the (potential) measured radial velocity in this interval will be considered as the actual radial velocity of the object. According to this approach, the benefits of estimating the radial velocity and determining the radial velocity precisely by means of a radar are combined and the negative effects mentioned above are avoided.

Exemplary Implementation of the Method for Determining the Radial Velocity of an Object

[0069] In FIG. 5, an implementation example of the above-described method is shown. For example, such an implementation may be included in a radar-based system of an ADAS.

[0070] In step S510, according to this exemplary implementation, the measurement data (radar data) is obtained from the radar. The measurement data comprises a signal indicative of a measured radial velocity of the object, which corresponds to the actual radial velocity of the object and/or to the ambiguous radial velocity outside the unambiguous interval of the radar. The actual radial velocity and the radial velocities outside the unambiguous interval of the radar are considered the potential radial velocities of the object. The measurement data may be provided in a data cube comprising range, angle, and azimuth information. In the following, the focus will be on the radial velocity of the object. However, the skilled person understands that the radial velocity may be associated with further range and angle information.

[0071] In step S520, the measurement data may then be processed by neural network layers to extract spatial local correlation within the radar data. Afterwards, this data may then be transformed into a cartesian system comprising X?Y grid cells (Step S522). While transforming the radar data into cartesian grid cells is generally preferred, the transformation step may also be omitted, and the measurement data may further be processed in the range-azimuth-Doppler-(RAD)-dimension. After the (optional) transformation of the measurement data, the data may then be processed by a RNN for aggregation and to extract temporal patterns within the data across consecutive frames (Step S524). The processed data may provide the environmental context of the object, which may include or may be used to obtain the estimated radial velocity. The environmental context may comprise information on the object. For example, the environmental context may comprise information that is characteristic for the object, preferably one or more of the radial velocity of the object determined at a previous point in time, a most probable interval determined at a previous point in time, a correlation between a location of the object in relation to the vehicle and/or a determination of the type of object and preferably a predetermined radial velocity to be expected for that type of object. For example, if it was determined during a previous radial velocity in a measurement interval before the current measurement that the actual radial velocity is likely in the interval comprising the measured radial velocity of v.sub.r+I.sub.D, it is likely that the current measured radial velocity is also in this interval. In another example, it may be determined that the object is likely a tree or a vehicle and a typical estimated radial velocity may be associated with the respective object. Moreover, information on the surrounding may also be obtained and considered for estimating the radial velocity. For example, information on the surrounding may include determining that an area of the surrounding is a road or that an area of the surrounding is a building and estimate radial velocities of objects in the corresponding areas.

[0072] In parallel to steps S520 to S524, in step S530, the radial velocity information of the measurement data is extracted, and the unfolded intervals are obtained. Similar to above, the radial velocity information is provided in the RAD dimension, initially. Thus, in Step S532, the RAD dimensional data may be transformed into cartesian X?Y grid cells.

[0073] As described above, the measured radar data may comprise the ambiguous potential measured radial velocities due to the radial velocity of the object being outside of the unambiguous interval of the radar. Thus, in step S540, the disambiguation module is used to disambiguate the Doppler velocity information and obtain/determine the actual radial velocity of the object, as described above.

[0074] Furthermore, in step S550, the information and/or the resulting feature maps obtained by the RNN in step 530 may be processed by further external modules, for example by an object detection or grid segmentation module (S250) for use in the ADAS.

[0075] For example, based on the object detection, the grid segmentation and/or the radial velocity determination of the object, the ADAS may display the object and preferably also the velocity of the object on a display of the vehicle to a user, conduct a vehicle path planning, triggering a warning to the user and/or affect the control of the vehicle during operation, preferably in a parking process.

[0076] While the above method was mainly described with reference to a FMCW radar, other radar sensors and technologies are also possible.

[0077] Moreover, the above-described method may be implemented in an apparatus, which may be comprised by a vehicle. The above method may also be implemented in the form of a computer program. That is, a computer program may comprise instructions, which when executed by a computing system cause the system to perform the above-described method.

Exemplary Application of the Method Using a Machine Learning Algorithm

[0078] The basic functionality of this module is depicted in FIG. 6. Doppler velocity measurements vr of radar data located in a grid map are unfolded. Various methods like soft attention (softmax or temperature-softmax interval probabilities), hard attention, etc. may be utilized to determine probabilities attprop and define resulting radial velocities.

[0079] Neural network layers within Define Interval, as shown in FIG. 6, then determine/predict probabilities attprop for each proposed interval to comprise the true radial velocity of the object. This determination/prediction may be designed as a function utilizing dense DL-layers of the environmental context, the reflection angle and the initially radial velocity measurement. The environmental context describes feature encodings that are extracted by a neural network based on the radar data. The environmental context may stem from the output of a Recurrent Neural Network (RNN) module. These outputs contain encodings that may be used to extract velocity estimates of captured objects with a weak accuracy compared to unambiguous radar velocity measurements. However, it can be assumed that the error of these velocity estimates remains well below the potential error of ambiguous radar velocity measurements which is determined by a multitude of the span of one interval of the plurality of intervals. The discrepancy between velocities with a consistently poor accuracy estimate based on environmental context and highly accurate radial velocity measurements from the radar sensor with an alternating error which potentially is magnitudes higher due to aliasing effects.

[0080] As velocity estimates from environmental context don't necessary relate to radial velocity components, operations like neural network layers within Define Interval, as shown in FIG. 5, may interpret relationships between total velocities and their corresponding radial velocity components. For this, the model may use the reflection angles under which objects were detected related to the longitudinal axis of the host vehicle for disambiguation.

[0081] Finally, the actually measured radial velocity is incorporated to calculate the discrepancy to radial velocities estimated by the model based on environmental context and reflection angles. This discrepancy can then be utilized within Define Interval to determine/predict probabilities of intervals each detection belongs to. That discrepancy can then be utilized by the neural network layers to determine the shift each radial velocity measurement needs to be projected by for disambiguation.

Training and Inference of the Artificial Intelligence Engine

[0082] During Training, class probabilities att.sub.prop of each potential measured radial velocity belonging to one of n.sub.prop intervals of the plurality of intervals that are determined/predicted by consecutive neural network layers based on aforementioned input features environmental context, reflection angle and measured radial speed are propagated as shown in FIG. 3.

[0083] For Ground Truth (GT), the radial velocities of objects are calculated from total velocities that are projected to the radial axis for each radar sensor mounted on the host vehicle as varying mounting positions result in varying radial velocities for the same absolute target position a radial velocity measurement was obtained from. Within the loss function, in FIG. 7 a classification loss function (Classification) is shown as an exemplary loss function the right radial velocity GT for each detection is then extracted based on the knowledge about the radar sensor obtaining this detection. Resulting radial velocity scalar values are then transferred to the corresponding Doppler intervals. Finally, in the example of the classification loss function, a cross entropy loss function determines the error between determined/predicted Doppler bin probabilities for measured radial velocities vr and the Doppler bin each radial velocity in the GT is assigned to.

[0084] While FIG. 7 shows the classification loss function as an example, other loss functions may also be used. For example, approaches based on soft attenuation or hard attenuation may also be used and employed accordingly.

[0085] During inference, the module then extracts the interval n.sub.prop with the highest responding value across all probabilities att.sub.prop and shifts v.sub.r by the corresponding offsets. That is, if it is determined that the interval including the (potential) measured radial velocity v.sub.r+2I.sub.D has the highest probability, the measured radial velocity is shifted by 2I.sub.D to obtain the actual radial velocity of the object.

[0086] In other examples, during inference, a weighted sum between Doppler proposals and probabilities att.sub.prop may be performed to determine the actual radial velocity of the object

[0087] By applying the above method in an ADAS, the ADAS may perform the determination of the radial velocity of on object in its surrounding, as described above. Accordingly, the vehicle assistance system may trigger one or more actions. Such actions may be any action performed by ADAS, in particular, in view of the determined radial velocity. That is, the ADAS may determine an operating instruction for the vehicle based on the determined radial velocity of the object, affecting the function of the ADAS such as one or more of: displaying the object and preferably also the velocity of the object on a display of the vehicle; conducting a vehicle path planning; triggering a warning; affecting control of the vehicle during a parking process and/or during driving.

[0088] An example vehicle 800 comprising an ADAS as mentioned above, and corresponding applications are depicted in FIG. 8. In this example, the vehicle 800 is a car. However, in other examples, the vehicle 800 may be any one of a ship, a drone, a robot or other vehicles that may benefit from considering the environment in their respective actions/operations.

[0089] In an example, the ADAS may be configured to execute the method described above. For example, the ADAS may consider the information obtained by the above method, for example by means of the radar, such as the determined radial velocity of the objects, to control the vehicle. One exemplary application is an adaptive cruise control (ACC) function for the vehicle, for example, for a robot in a manufacturing plant such that collisions with other robots or persons are avoided. Furthermore, the vehicle 800 may comprise a collision warning or mitigation feature that may benefit from the above-described methods. Another potential application for the above is a pre-crash warning. This may be particularly critical in a vehicle such as a car, for example in a traffic accident situation where the pre-crash warning may trigger security relevant applications such as an airbag to avoid harm to the driver. Similarly, a rear crash collision avoidance could be implemented. Moreover, another potential application is to aid a driver of a car in a parking operation. For this, the information obtained may be used for blind spot detection or as a general parking aid, to facilitate the parking. Moreover, the information may also be used to assist a driver in changing the lane by triggering a warning, if an object is detected that may be accidentally hit during a lane change. This may also be used for realizing fully automatic lane changes.

[0090] The method according to the present invention may be implemented in terms of a computer program which may be executed on any suitable data processing device comprising means (e.g., a memory and one or more processors operatively coupled to the memory) being configured accordingly. The computer program may be stored as computer-executable instructions on a non-transitory computer-readable medium.

[0091] Embodiments of the present disclosure may be realized in any of various forms. For example, in some embodiments, the present invention may be realized as a computer-implemented method, a computer-readable memory medium, or a computer system.

[0092] In some embodiments, a non-transitory computer-readable memory medium may be configured so that it stores program instructions and/or data, where the program instructions, if executed by a computer system, cause the computer system to perform a method, e.g., any of the method embodiments described herein, or, any combination of the method embodiments described herein, or, any subset of any of the method embodiments described herein, or, any combination of such subsets.

[0093] In some embodiments, a computing device may be configured to include a processor (or a set of processors) and a memory medium, where the memory medium stores program instructions, where the processor is configured to read and execute the program instructions from the memory medium, where the program instructions are executable to implement any of the various method embodiments described herein (or, any combination of the method embodiments described herein, or, any subset of any of the method embodiments described herein, or, any combination of such subsets). The device may be realized in any of various forms.

[0094] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the present disclosure, even where only a single embodiment is described with respect to a particular feature. Examples of features provided in the disclosure are intended to be illustrative rather than restrictive unless stated otherwise. The above description is intended to cover such alternatives, modifications, and equivalents as would be apparent to a person skilled in the art having the benefit of this disclosure.

[0095] The scope of the present disclosure includes any feature or combination of features disclosed herein (either explicitly or implicitly), or any generalization thereof, whether or not it mitigates any or all of the problems addressed herein. In particular, with reference to the appended claims, features from dependent claims may be combined with those of the independent claims and features from respective independent claims may be combined in any appropriate manner and not merely in the specific combinations enumerated in the appended claims.