METHOD FOR ASCERTAINING SIGNAL PROPAGATION TIMES, AND SYSTEM FOR FUSING SENSOR DATA FROM AT LEAST TWO SENSORS FOR AN OBJECT DETECTION

Abstract

A method for synchronizing at least two environment sensors of a multi-sensor system using a central processing unit. In the method, the environment sensors acquire sensor signals that represent at least one item of environment information. The respective environment sensors generate data packets which include the respective acquired sensor signals and/or measured variables derived from the sensor signals in each case. These data packets are received by the central processing unit via a data network. Signal propagation times of the data packets for each environment sensor are ascertained using an algorithm, and a mean signal propagation time of data packets from a respective environment sensor is determined based on a content comparison of the measured variables included by the data packets with corresponding measured variables from data packets of at least one other environment sensor, and the determined mean signal propagation time is assigned to the respective environment sensor.

Claims

1. A method for synchronizing at least two environment sensors of a multi-sensor system using a central processing unit, the method comprising the following steps: acquiring, by the environment sensors, sensor signals which represent at least one item of environment information; generating data packets which include the acquired sensor signals and/or measured variables derived from the sensor signals by the environment sensors; receiving the data packets from the environment sensors by the central processing unit via a data network; ascertaining signal propagation times of the data packets for each respective sensor of the environment sensors using an algorithm, a mean signal propagation time of data packets of each respective environment sensor being determined based on a content comparison of the measured variables included in the data packets of the respective environment sensor with corresponding measured variables of data packets from at least one other environment sensor, and the determined mean signal propagation time being assigned to the respective environment sensor.

2. The method as recited in claim 1, wherein each data packet of the data packets is provided with a corrected time stamp, and the corrected time stamp is determined using the ascertained mean signal propagation time for the respective environment sensor transmitting the data packet.

3. The method as recited in claim 1, wherein one of the environment sensors is selected as a reference sensor, and the signal propagation times of the data packets are determined based on a content comparison of measured variables with corresponding measured variables acquired by the reference sensor, and corrected time stamps of the data packets are determined relative to the reference sensor in each case.

4. The method as recited in claim 3, wherein a signal propagation time between the reference sensor and the central processing unit is determined in advance using a Network Time Protocol (NTP), and an absolute time stamp based on the determined signal propagation time is assigned to the data packets of the reference sensor.

5. The method as recited in claim 1, wherein the ascertaining of the mean signal propagation times of the data packets for the environment sensors takes place using a matching algorithm, and for each of the environment sensors, the signal propagation time is ascertained using incoming data packets by a comparison with a reference environment sensor.

6. The method as recited in claim 1, wherein the measured variables include a distance from an object and/or an object position and/or a velocity of an object and/or an object class and/or an object form and/or an object height.

7. The method as recited in claim 1, wherein the ascertainment of the signal propagation times is implemented in that for at least one pair of environment sensors, a set of quadruples is determined from a measured variable acquired by a first environment sensor, associated uncorrected first measurement time, measured variable acquired by a second environment sensor, associated uncorrected second measurement time, for which it holds that the first measured variable and the second measured variable match, and that a difference between the uncorrected first measurement time and the uncorrected second measurement time is less than a predefined limit value, and that at the first measurement time, no further sensor signals were acquired by the first environment sensor that result in a measured variable close to the first measured variable, and that at the second measurement time, no further sensor signals were acquired by the second environment sensor that result in a measured variable close to the second measured variable; wherein for each one of the quadruples, a signal propagation time difference is generated from the first measurement time and the second measurement time, and a mean propagation time difference is calculated as a mean value of the signal propagation time differences, and the first environment sensor is selected as the reference sensor and the mean propagation time difference is determined as the signal propagation time of the second environment sensor.

8. A method for fusing sensor signals from at least two environment sensors using a central processing unit, the method comprising the following steps: acquiring, by the environment sensors, sensor signals that represent at least one item of environment information; generating data packets which include the acquired sensor signals and/or measured variables derived from the sensor signals by the environment sensors, and for each respective environment sensor of the environment sensors, determining a mean signal propagation time; receiving the data packets from the environment sensors by the central processing unit via a data network, a time stamp being assigned to each of the data packets based on the mean signal propagation time of the respective environment sensor; and fusing the sensor data included in the data packets based on the respective time stamps.

9. The method as recited in claim 8, wherein, for each respective sensor, the mean signal propagation time is determined based on a content comparison of the measured variables included in the data packets of the respective environment sensor with corresponding measured variables of data packets from at least one other environment sensor.

10. A multi-sensor system for fusing sensor data for an object detection, comprising: at least two environment sensors; a central processing unit configured to synchronize the environment sensors by: acquiring, by the environment sensors, sensor signals which represent at least one item of environment information; generating data packets which include the acquired sensor signals and/or measured variables derived from the sensor signals by the environment sensors; receiving the data packets from the environment sensors by the central processing unit via a data network; ascertaining signal propagation times of the data packets for each respective sensor of the environment sensors using an algorithm, a mean signal propagation time of data packets of each respective environment sensor being determined based on a content comparison of the measured variables included in the data packets of the respective environment sensor with corresponding measured variables of data packets from at least one other environment sensor, and the determined mean signal propagation time being assigned to the respective environment sensor; wherein the central processing unit is configured to fuse sensor data included in the data packets transmitted by the environment sensors based on the ascertained signal propagation times.

11. The multi-sensor system as recited in claim 10, wherein at least one of the environment sensors is developed as a radar sensor and/or a lidar sensor and/or a camera.

12. The multi-sensory system as recited in claim 11, wherein at least two of the environment sensors have a different development.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Example embodiments of the present invention are described in detail in the following text with reference to the figures.

[0044] FIG. 1 schematically shows a multi-sensor system for fusing sensor data for an object detection according to an exemplary embodiment of the present invention.

[0045] FIG. 2 exemplarily shows a diagram of measured variables acquired over time with the aid of a multi-sensor system for fusing sensor data for an object detection according to the present invention.

[0046] FIG. 3 shows a flow diagram of a method for synchronizing two environment sensors of a multi-sensor system according to an exemplary embodiment of the present invention.

[0047] FIG. 4 shows a flow diagram of a method for fusing sensor signals from two environment sensors according to an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0048] In the following description of the exemplary embodiments of the present invention, identical elements are denoted by matching reference numerals, and a repeated description of these elements is omitted, as the case may be. The figures illustrate the subject matter of the present invention merely schematically.

[0049] FIG. 1 shows a multi-sensor system 10 for fusing sensor data for an object detection according to an exemplary embodiment of the present invention. Multi-sensor system 10 in this example includes two environment sensors 12, 14, first environment sensor 12 being embodied as a camera sensor and second environment sensor 14 being embodied as a radar sensor. In this particular example, environment sensors 12, 14 are permanently situated within a street infrastructure and detect objects 50, e.g., road users moving within a certain region of a street or traffic lane. For instance, the environment sensors may acquire an entrance region of a tunnel and/or the interior of a tunnel. As an alternative, however, it is also possible within the framework of the present invention that one or more environment sensors of the multi-sensor system are not stationary, i.e., are situated on board a vehicle, for example.

[0050] The sensor data acquired by environment sensors 12 and 14 represent environment information, e.g., information pertaining to positions of objects within the respective acquisition region of environment sensors 12 and 14. Based on the acquired sensor signals, measured variables are able to be derived that characterize the environment information such as a distance between respective environment sensor 12, 14 and an object 50. The sensor signals and/or the measured variables derived therefrom are transmitted from respective environment sensor 12, 14 to a central processing unit 30 in the form of data packets 22, 24 via a data network 40. Central processing unit 30 is developed to merge (fuse) the environment information included by data packets 22, 24 and in this way prepare a comprehensive and reliable environment model, for instance.

[0051] Data packets 22, 24 usually arrive at central processing unit 30 with different time delays dt.sub.Radar and dt.sub.camera due to the specific network architecture (cable length, switches, firewalls, VPN,. . . ) of data network 40,. In order to further process the data after its arrival in central processing unit 30, the data must be converted into identical time stamps prior to the fusion.

[0052] In this particular example, first environment sensor 12 (camera sensor) has been synchronized in advance per NTP (Network Time Protocol) so that data packets 22 bear an absolute time stamp t.sub.stamp. Second environment sensor 14 in this particular image has not been synchronized in advance so that data network 40 must first ascertain a precise signal propagation time of data packets 24.

[0053] According to the present invention, signal propagation times of data packets 24 for second environment sensor 14 are ascertained with the aid of an algorithm in that the measured variables included by data packets 24 are compared in terms of their content to the measured variables included by data packets 22 of first environment sensor 12. In the present example, both environment sensors 12 and 14 determine a time- dependent distance d to an object 50 as a measured variable. After the conversion to a uniform coordinate system, the measured distances are able to be compared and a mean signal propagation time be determined relative to absolute time stamp t.sub.stamp of first environment sensor 12.

[0054] This procedure is described in greater detail in FIG. 2. FIG. 2 shows a diagram of possible measured variables acquired by environment sensors 12 and 14.

[0055] Here, data packets including measured variables transmitted by environment sensors 12 and 14 arrive at central processing unit 30, the time stamps for each measured value initially simply corresponding to the arrival instant of the data packets in central processing unit 30. Received from first environment sensor 12 (the camera sensor) are measurement curves 212′, 212″, 212″', which represent the time-dependent distances to three detected objects in the detection range of environment sensors 12, 14. Each measuring point has the coordinate (dx.sub.Camera, t.sub.StampCamera, Receive) Received from second environment sensor 14 (the radar sensor) are measurement curves 214′, 214″, 214″', which likewise represent time-dependent distances to the same three detected objects in the detection range of environment sensors 12, 14. Each measuring point has the coordinate (dX.sub.Radar, t.sub.StampRadar, Receive).

[0056] In this example, the three detected objects, which may involve vehicles, for example, move away from the sensor system at a constant velocity (in other words, dx increases). The time offset dt', dt“, dt”' between the arrival of the camera data and the considerably later radar data is shown in greatly emphasized form. The signal propagation times of environment sensors 12 and 14 are denoted by dt.sub.camera and dt.sub.Radar.

[0057] The following applies per definition:

t.sub.StampCamera +dt.sub.Camera =t .sub.StampCamera, Receive

t.sub.StampRadar +dt.sub.Radar =t.sub.StampRadar, Receive

where t.sub.StampCamera , dt.sub.Camera , t.sub.StampRadar and dt.sub.Radar are unknown to begin with. In the first step, no absolute time stamps are taken into account yet, which means that environment sensors 12 and 14 are first synchronized with one another. For this purpose, the respective signal propagation times dt.sub.Radar and dt.sub.Camera are determined except for an additive constant. Searched for is time offset dt.sub.Radar, Camera =dt.sub.Radar dt.sub.Camera.

[0058] A conventional matching algorithm in conjunction with an outlier detection may be used for this purpose, and a method based on a cross-correlation or something similar may be used as an alternative. The precise selection of the algorithm is not meant to be limited here, but the following possibility serves as an illustration.

[0059] To begin with, a sufficiently large set of quadruples (dX.sub.Radar, t.sub.camera, Receive, dx.sub.Camera, t.sub.Radar, Receive) is determined with the aid of the matching algorithm, for which the following conditions are satisfied: [0060] dx.sub.Radar =dx.sub.camera (if no matching measured values are available for distance d, they may be ascertained by an interpolation, for example) [0061] |t.sub.Radar, Receive −t.sub.Camera, Receive |<c, [0062] at the same instant, NO further measurements take place in the physical environment in order to avoid mismatches, that is to say [0063] neither (dx .sub.'Radar, t.sub.Radar, Receive) with |dX.sub.Radar −dx .sub.'Radar |<d nor (dx .sub.'Camera, t.sub.Camera, Receive) with |dX.sub.Camera −dx.sub.'Camera |<d
where c and d describe previously determined threshold values.

[0064] In a next step, the respective signal propagation time differences dt.sub.Radar, Camera =dt.sub.Radar −dt.sub.camera =t.sub.Radar −t.sub.Camera are calculated for each one of these quadruples (matches).

[0065] Now, the signal propagation time difference dt.sub.Radar,Camera is able to be calculated by averaging, e.g., as a median, a set of individual results, sufficiently distributed over the time, of individual results from the preceding step.

[0066] Thereafter, first environment sensor 12 (camera sensor) is able to be selected as the reference. For the time being, it is hypothetically assumed that the signal propagation time dt.sub.camera =0. The data packets from second environment sensor (radar data) arriving at instant t.sub.Now are now able to be provided with a time stamp t.sub.Radar, Camera corrected relative to first environment sensor 12:

t.sub.Radar, Camera : =t.sub.Now −dt.sub.Radar, Camera.

[0067] With the aid of NTP or similar mechanisms it can be ensured that first environment sensor 12 (or randomly any other reference sensor in the system) is synchronized with the fusion server. As a result, the correct time stamps for all incoming sensor packets 22, 24 are finally able to be specified in the time basis of the fusion server. To this end, dt.sub.camera is determined to begin with by assigning an internal time stamp t.sub.Camera,send in first environment sensor 12 when data packets 22 are transmitted. Thanks to the synchronized timer of first environment sensor 12 and central processing unit 30, dt.sub.camera is then able to be directly “read out” as dt.sub.camera :=t.sub.Camera, Receive −t.sub.Camera, Send.

[0068] Here, too, it is advantageous to view multiple measurements that are averaged, e.g., via a median, in order to achieve greater stability.

[0069] It is now possible to determine the actual, absolute signal propagation time also for environment sensor 14 not synchronized via NTP, i.e., the radar sensor in the example: dt.sub.Radar =dt.sub.Radar, Camera +dt.sub.camera.

[0070] Thanks to the now known signal propagation times, the correct time stamp is able to be specified in the time basis of central processing unit 30 upon arrival of each data packet in central processing unit 30:

[0071] t.sub.Radar, Measured:=t.sub.Radar, Received −dt.sub.Radar tCamera, Measured:=t.sub.Camera, Received −dt.sub.camera (accordingly t.sub.Camera,Send, with the exception of minimal fluctuations if there are changes in the signal propagation times).

[0072] Within the framework of the present invention, this principle can be transferred to general scenarios featuring more than two environment sensors. If multiple environment sensors are involved, the previously described determination of signal propagation time difference dt.sub.Radar,Camera is applied to each individual environment sensor, that is to say, the signal propagation time difference with respect to a first selected environment sensor used as the reference sensor is always applied. As an alternative, randomized matching with randomly selected pairs of environment sensors or complete matching of all environment sensors with one another is possible. This may require a greater processing effort but can increase the stability of the system.

[0073] In FIG. 3, an exemplary embodiment of a method according to the present invention for synchronizing two environment sensors of a multi-sensor system with the aid of a central processing unit is shown in the form of a flow diagram. In a first step 310, sensor signals representing at least one item of environment information are acquired by the environment sensors. In a second, following step 320, data packets are generated which include the respectively acquired sensor signals and/or measured variables derived from the sensor variables by the environment sensors. In a subsequent, third step 330, the data packets from the environment sensors are received by the central processing unit via a data network. In a fourth step 340, signal propagation times of the data packets are ascertained for each environment sensor with the aid of an algorithm, and a relative mean signal propagation time of data packets from one of the environment sensors is determined based on a content comparison of the measured variables included by the data packets with corresponding measured variables from data packets of the other environment sensor, and the determined relative mean signal propagation time is assigned to the respective environment sensor.

[0074] FIG. 4 shows, in the form of a flow diagram, an exemplary embodiment of a method according to the present invention for fusing sensor signals from two environment sensors with the aid of a central processing unit. In a first step 410, sensor signals representing at least one item of environment information are acquired by the environment sensors. In a second step 420, data packets including the acquired sensor signals and/or measured variables derived from the sensor signals are generated by the environment sensors, and a mean signal propagation time is determined for each environment sensor according to a method according to the first aspect of the present invention. In a following step 430, data packets of the environment sensors are received by the central processing unit via a data network, and a time stamp is assigned to each data packet based on the previously ascertained mean signal propagation time of the respective environment sensor. In a subsequent step 440, the central processing unit fuses the sensor data included by the data packets on the basis of the respective time stamp.