METHOD AND DEVICE FOR OPERATING AN INFRASTRUCTURE SENSOR SYSTEM

Abstract

A method for operating an infrastructure sensor system, wherein the infrastructure sensor system comprises a plurality of networked infrastructure sensors arranged on a shared mounting device. First, data are transmitted to a sway estimation module by at least one of the infrastructure sensors, wherein the data comprise at least pre-processed data, for example environmental information and/or current measurement data, in particular raw data, detected by the respective infrastructure sensor. The transmitted data are further processed in the next step and the sway estimation module determines therefrom a motion function for the mounting device. Based on the motion function, correction information, for example for at least one of the infrastructure sensors, is now determined by the sway estimation module. The correction information and/or motion function can now be provided for further use, for example, to the respective infrastructure sensors or to a central computing unit.

Claims

1. A method for operating an infrastructure sensor system, wherein the infrastructure sensor system has a plurality of infrastructure sensors arranged on a shared mounting device, the method comprising the following steps: transmitting data by each respective infrastructure sensor of at least one of the infrastructure sensors to a sway estimation module, wherein the data include: pre-processed data including environmental information determined by the respective infrastructure sensor, and/or current measurement data including raw data detected by the respective infrastructure sensor; processing the transmitted data, and determining from the processed transmitted data, using the sway estimation module, a motion function for the mounting device; ascertaining, using the sway estimation module, correction information based on the motion function; and providing the correction information and/or the motion function.

2. The method according to claim 1, wherein the pre-processed data include a position of the respective infrastructure sensor and/or an orientation of the respective infrastructure sensor and/or a motion vector according to a previously performed calibration of the respective infrastructure sensor and/or measurement data of the respective infrastructure sensor.

3. The method according to claim 1, wherein at least one of the infrastructure sensors of the infrastructure sensor system is configured as an environment sensor including an imaging sensor, and the transmitted data include a first sway estimate which is ascertained using environmental information detected by the environment sensor, wherein the first sway estimate is provided to the sway estimation module and is used in determining the motion function and/or in ascertaining the correction information.

4. The method according to claim 3, wherein the processing of the data and the determining of the motion function for the mounting device is performed by the sway estimation module, additionally as a function of the first sway estimation.

5. The method according to claim 3, wherein the environmental information includes raw data, wherein the sway estimation module determines, based on the raw data, the first sway estimate and/or a second sway estimate, wherein the determination of the motion function for the mounting device is performed by the sway estimation module, additionally as a function the first sway estimate and/or of the second sway estimate.

6. The method according to claim 3, wherein the first sway estimate is determined by optical flow analysis of image data detected by an infrastructure sensor configured as an imaging sensor.

7. The method according to claim 3, wherein the first sway estimate is determined by an analysis of landmarks or point clouds in comparison to a map, which were detected by an infrastructure sensor configured as an imaging sensor.

8. The method according to claim 1, wherein the environmental information includes object features of objects in the environment of the infrastructure sensor system, wherein the object features are provided to the sway estimation module and are used in determining the motion function and/or in ascertaining the correction information.

9. The method according to claim 1, wherein the correction information includes functional parameters of the motion function, wherein using the functional parameters of the motion function, an updated position and orientation and/or a motion vector for at least one of the infrastructure is determined.

10. The method according to claim 1, wherein sensor-specific motion vectors are determined as the correction information, wherein using a sensor-specific motion vector, an updated position and/or orientation for at least one of the infrastructure sensors is determined.

11. The method according to claim 1, wherein the correction information is transmitted to at least one of the infrastructure sensors of the infrastructure sensor system so that measurement data of the at least one of the infrastructure sensors can be corrected using the correction information and/or can subsequently be marked as inaccurate.

12. The method according to claim 1, wherein the correction information is transmitted to a computing unit and measurement data and/or environmental information are transmitted from the infrastructure sensors to the computing unit, wherein the computing unit calculates an environmental model of the infrastructure sensor system using the correction information and the measurement data and/or environmental information.

13. The method according to claim 1, wherein: for information exchange between each infrastructure sensor and the sway estimation module, a message is used, which includes the data and the correction information, wherein the correction information includes motion function and/or parameters of the motion function and/or a sensor-specific motion vector and/or a corrected sensor position and/or a corrected sensor orientation; the message includes information relating to the infrastructure sensor including a sensor type of the infrastructure sensor and/or information as to whether the infrastructure sensor has its own sway detection and/or information as to whether the infrastructure sensor requires the correction information.

14. The method according to claim 13, wherein the message further includes a signature and further optionally a certificate for validating the signature.

15. A device for operating an infrastructure sensor system, comprising: a sway estimation module; and a communication unit, which is used to receive data, the data including pre-processed data including environmental information and/or current measurement data including raw data, from infrastructure sensors of the infrastructure sensor system, wherein the infrastructure sensors are arranged on a shared mounting device; wherein the sway estimation module is configured to process the data received by the communication unit and to determine therefrom a motion function for the mounting device, to ascertain correction information based on the motion function, and to provide the correction information and/or the motion function.

16. An infrastructure sensor system, comprising: a plurality of infrastructure sensors arranged on a shared mounting device; and a device for operating an infrastructure sensor system, including: a sway estimation module, and a communication unit, which is used to receive data, the data including pre-processed data including environmental information and/or current measurement data including raw data, from the infrastructure sensors; wherein the sway estimation module is configured to process the data received by the communication unit and to determine therefrom a motion function for the mounting device, to ascertain correction information based on the motion function, and to provide the correction information and/or the motion function.

17. The infrastructure sensor system according to claim 16, wherein at least one of the infrastructure sensors is configured as an imaging sensor including a camera sensor, and/or as a radar sensor and/or as a lidar sensor.

18. The infrastructure sensor system according to claim 16, wherein the infrastructure sensors include at least one strain sensor and/or at least one accelerometer and/or at least one eddy current sensor and/or at least one travel sensor.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIGS. 1A-1C show different embodiments of mounting devices for infrastructure sensors for use in infrastructure sensor systems according to the present invention.

[0045] FIG. 2 shows an infrastructure sensor system according to a first embodiment example of the third aspect of the present invention.

[0046] FIG. 3 shows an infrastructure sensor system according to a second embodiment example of the third aspect of the present invention.

[0047] FIG. 4 shows an example of a motion function determined according to an embodiment example of a method according to the present invention.

[0048] FIG. 5 shows a flow diagram according to an embodiment example of a method according to the present invention.

[0049] FIG. 6 schematically shows the architecture of an infrastructure sensor system according to a third embodiment example of the third aspect of the present invention.

[0050] FIG. 7 schematically shows the architecture of an infrastructure sensor system according to a fourth embodiment example of the third aspect of the present invention.

[0051] FIG. 8 schematically shows a possible format for a message for exchanging information between an infrastructure sensor and the sway estimation module according to another embodiment of the present invention.

[0052] Example embodiments of the present invention are described in detail with reference to the figures.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0053] In the following description of the embodiment examples of the present invention, identical elements are denoted by identical reference signs, a repeated description of these elements being dispensed with, where appropriate. The figures show the subject matter of the present invention only schematically.

[0054] FIG. 1A shows an embodiment example of a mounting device 12 for infrastructure sensors for use in an infrastructure sensor system according to the present invention in a perspective view. The mounting device 12 includes a base 13 and a mast 15 anchored in the base that is oriented substantially perpendicular to a ground plane 11 on which the mounting device 12 stands. The mast 15 is surrounded by and connected to a support structure 14. The support structure comprises several elements aligned perpendicular and parallel to the ground plane 11 and connected to one another. The support structure 14 is configured to impart mechanical stability to the mast 15. A boom 16 is arranged on the mast 15 and extends substantially perpendicular to the mast 15 and substantially parallel to the ground plane 11. Infrastructure sensors (not shown) may be situated, that is, fastened, on the boom 16 and/or on the mast 15, which can detect, for example, the environment of the mounting device 12.

[0055] In case of strong winds, for example, or by other environmental factors, the boom 16 of the mounting device 12 can be excited to mechanical oscillations or swaying, as indicated by arrows 18.

[0056] The infrastructure sensors are typically calibrated for a specific position and orientation, which typically corresponds to a stationary boom 16. In order to be able to use the measured values of the infrastructure sensors even in case of a swaying boom 16, a method according to the first aspect of the present invention can be employed.

[0057] FIG. 1B shows an alternative configuration of a mounting device 22 for infrastructure sensors in a plan view. The mounting device 22 has a substantially vertically extending construction 23, as well as a substantially horizontally extending boom 26 arranged elevated above the ground on the construction 23. The boom 26 can be configured as a mount for several infrastructure sensors. In this example, in order to detect a swaying of the boom 26, strain sensors 27 are attached along the boom 26. In the illustration i), the boom 26 is deflected in the positive y-direction. In Figure ii), the boom 26 is in a center position or a rest position, where no correction is necessary. In Figure iii), the boom 26 is deflected in the negative y-direction. In the positions according to i) and iii) of the boom 26, there is a strain so that the strain sensors 27 provide corresponding measured values. For example, on the basis of the sign of the measured values and/or the difference of the measured values of two strain sensors, a distinction can be made between an orientation in the positive y-direction according to i) and an orientation in the negative y-direction according to iii).

[0058] In FIG. 1C, another alternative configuration of a mounting device 32 for infrastructure sensors 36, 37, 38, 39 is shown in a plan view. The mounting device 32 includes a base 33 and a mast anchored in the base that is oriented substantially perpendicular to a ground plane on which the mounting device 32 is stands (not visible in this view). The mast has a boom 35 that is substantially parallel to the ground plane. The boom 35 has two transverse struts 34 disposed in different positions along the boom 35, which are short relative to the length of the boom 35 and to each of which two infrastructure sensors 36 and 38 and 37 and 39, respectively, are attached.

[0059] For example, in case of strong winds or by other environmental factors, the boom 35 of the mounting device 32 can be excited to mechanical oscillations or swaying, as indicated by arrows 31.

[0060] Infrastructure sensors 36 and 39, in this example, are configured as camera sensors that can record images of the environment of mounting device 32 in at regular time intervals. The image data generated in this manner can be pre-processed by being evaluated, for example by optical flow analysis, and based on this evaluation, a first sway estimate can be ascertained based on which a motion function for the mounting device 32 can be calculated. Infrastructure sensors 37 and 38, in this example, are configured as radar sensors that are able to detect distances to objects in the environment of mounting device 32 with high accuracy. The thus generated object data can be additionally evaluated and/or corrected based on the calculated motion function for mounting device 32.

[0061] FIG. 2 shows an infrastructure sensor system 200 according to a first embodiment example of the third aspect of the present invention as a block diagram. The infrastructure sensor system 200 includes a plurality of infrastructure sensors 212, 214, 216, 218, 220, 222, 223 wherein the infrastructure sensors 212, 214, 216, 218, 220, 222, 223 are arranged on a shared mounting device (not shown). Infrastructure sensor system 200 includes a device 100 for operating infrastructure sensor system 200, comprising a sway estimation module 110 and a communication unit 120 for receiving data transmitted by infrastructure sensors 212, 214, 216, 218, 220, 222, 223. Communication unit 120 can further be configured to send information to at least one of infrastructure sensors 212, 214, 216, 218, 220, 222, 223 such that data exchange with infrastructure sensors 212, 214, 216, 218, 220, 222, 223 is enabled.

[0062] The sway estimation module 110 is configured to process the data received from the communication unit 120 and to determine a motion function therefrom for the mounting device, to ascertain correction information for at least one of the infrastructure sensors based on the motion function, and to provide the correction information and/or the motion function.

[0063] In the example of FIG. 2, the infrastructure sensors 220 and 216 are configured as camera sensors. Infrastructure sensor 216 transmits pre-processed data comprising a motion vector 236 associated with the infrastructure sensor 216 itself to the communication unit 120. The motion vector 236 was previously ascertained by the infrastructure sensor 216 itself, for example, by an analysis of the optical flow of the image data detected by the infrastructure sensor 216 and represents a first sway estimate. The motion vector 236 can indicate, for example, speed components and/or acceleration components of a motion of the infrastructure sensor 216 for all three spatial directions.

[0064] The infrastructure sensor 220 transmits pre-processed data comprising a motion vector 240 associated with the infrastructure sensor 220 itself and its own position to the communication unit 120. For example, the position of infrastructure sensor 220 can include a position determined from the motion vector and/or a position determined during an initial calibration of the infrastructure sensor 220. Alternatively or additionally, the infrastructure sensor 220 can transmit raw data, for example, in the form of a video stream 224, to the communication unit 120.

[0065] Infrastructure sensors 214 and 218 are configured as radar sensors in this example, and are representative of any sensor type that initially is not capable of performing its own sway estimate in sufficient quality. Infrastructure sensors 214 and 218 transmit their own respective positions 239, 234 to communication unit 120. For example, the infrastructure sensor's own position may be a position determined upon initial calibration of the respective infrastructure sensor 214, 218.

[0066] For example, a position of an infrastructure sensor 214, 216, 218, 220 may be represented by a global or relative coordinate. The position can additionally comprise at least one angle information (e.g., pitch angle, yaw angle, roll angle) describing an orientation of the respective infrastructure sensor 214, 216, 218, 220 and thus the detection range of the respective infrastructure sensor 214, 216, 218, 220.

[0067] The illustrated infrastructure sensor system 200 also includes an accelerometer 212 that is also arranged on the mounting device. The accelerometer 212 may be configured as a MEMS sensor, for example, and, in addition to an acceleration vector for the three spatial directions, may also detect angular accelerations and/or gravitation and/or the Earth's magnetic field and out of these generate highly accurate absolute orientation and motion information. The measurement data generated by the accelerometer 212 may be transmitted to the communication unit 120 as pre-processed data 232 and/or as raw data along with the current and/or initial position of the accelerometer 212.

[0068] Optionally, an strain sensor 222 can be provided on the mounting device or even several strain sensors 223. By means of the strain sensors 222, 223, as shown for example in FIG. 1B, a bending or deflection of an element of the mounting device can be recognized and the direction and magnitude of the bending or deflection can be detected. The strain sensors can transmit the information thus detected to the communication unit 120 in the form of raw measurement data 242, for example.

[0069] All data transmitted to the communication unit 120 are made available to the sway estimation module 110, which determines a motion function for the mounting device based on the data. Based on the motion function, the sway estimation module can determine correction information for at least one of the infrastructure sensors.

[0070] In this example, the sway estimation module 110 is configured to calculate motion vectors at least for the infrastructure sensors 214 and 218 based on the ascertained motion function for the mounting device. Such a motion vector describes the specific motion for each individual infrastructure sensor 214 and 218. For this purpose, it is necessary for infrastructure sensors 214 and 218 to communicate their own position in advance. Accordingly, based on the transmitted position of the infrastructure sensor 214 and the motion function for the mounting device, a first motion vector 235 is determined and transmitted to the infrastructure sensor 214 by the communication unit 120. Infrastructure sensor 214 can use the first motion vector 235 as correction information, for example, to correct detected environmental information or, for example, if the deviations are too large, to request recalibration. Analogously, based on the transmitted position of the infrastructure sensor 218 and the motion function for the mounting device, a second motion vector 238 can be determined and transmitted to the infrastructure sensor 218 by the communication unit 120. Infrastructure sensor 218 can utilize the second motion vector 238 as correction information.

[0071] Determining correction information for infrastructure sensors 216 and 220 configured as camera sensors is not absolutely necessary because the infrastructure sensors 216 and 220 can already perform a sway estimate by evaluating the respective detected image data, for example, by optical flow analysis. Nevertheless, it can be advantageous to calculate also a motion vector for each infrastructure sensor 216 and 220 based on a known position of the respective infrastructure sensor 216 and 220 and the motion function for the mounting device and to transmit it to the respective infrastructure sensor 216 and 220, for example, in order to check the plausibility and improve the accuracy of the sway estimates of the infrastructure sensors 216 and 220.

[0072] FIG. 3 shows an infrastructure sensor system 400 according to a second, alternative embodiment example of the third aspect of the present invention as a block diagram. The infrastructure sensor system 400 includes a plurality of infrastructure sensors 412, 414, 416, 418, 420, 422, 423, wherein the infrastructure sensors 412, 414, 416, 418, 420, 422, 423 are arranged on a shared mounting device (not shown). The infrastructure sensor system 400 includes a device 300 for operating the infrastructure sensor system 400, comprising a sway estimation module 310 and a communication unit 320 configured to receive data transmitted from the infrastructure sensors 412, 414, 416, 418, 420, 422, 423.

[0073] The sway estimation module 310 is configured to process the data received from the communication unit 320 and to determine a motion function for the mounting device therefrom, to ascertain correction information for at least one of the infrastructure sensors based on the motion function, and to provide the correction information and/or the motion function.

[0074] In the example of FIG. 3, the infrastructure sensors 420 and 416 are configured as camera sensors and the infrastructure sensors 414 and 418 are configured as radar sensors. Furthermore, one or more strain sensors 433, 423 and at least one inertial sensor 412 are provided.

[0075] As in the example of FIG. 2, the inertial sensor 412 transmits its position and measurement data 432 that characterize at least one motion and/or acceleration of the mounting device to the communication unit 320. The camera sensors 416 and 420 each transmit their own position and a first sway estimate 436 and 440 determined by the respective camera sensor 416 and 420, respectively, to the communication unit 320, for example in the form of a motion vector. Alternatively or additionally, the camera sensors 416 and 420 can transmit raw data, for example, in the form of a video stream, to the communication unit 320. The strain sensors 422, 423 transmit raw data, which, for example, denote a deflection of an element of the mounting device in a specific spatial direction, to the communication unit 320.

[0076] All data transmitted to the communication unit 320 are made available to the sway estimation module 310, which determines a motion function for the mounting device based on the data. Based on the motion function, the sway estimation module can ascertain correction information for at least one of the infrastructure sensors.

[0077] As a result 435, 438, in this example, the motion function is passed on to infrastructure sensors 414 and 418 that do not have a sway estimation of their own. Subsequently, infrastructure sensors 414 and 418 can determine their current motion vector using the motion function and their initially calibrated position. Alternatively or additionally (not shown here), the result can also be distributed to infrastructure sensors 416 and 420 to improve or overwrite their in-sensor sway estimate. The advantage of this variant is that the motion function is not dependent on the respective sensor position and can therefore be distributed equally to all infrastructure sensors. By using multicast/broadcast mechanisms in IP-based networks, the network load can be kept small.

[0078] The difference between the two variants according to FIGS. 2 and 3 is shown again separately in FIG. 4. The vertical line 115 illustrates a mast of an mounting device at rest without external influences. The line 116 describes the deformation of the mast at a specific point in time and thus represents the motion function. Positions 114, 117, and 118 are exemplary sensor positions that have changed according to the motion function or a motion vector based on the motion function for the respective sensor rest position.

[0079] A flowchart 500 of an example of a sway estimation module 510 (as a hardware or software component) employed in a device configured according to an embodiment example of the present invention is shown in FIG. 5. As input, two infrastructure sensors 512 and 514 are illustrated here by way of example. The relevant infrastructure sensor system, in which the illustrated sway estimation module 510 is used, can additionally have further sensors. The first infrastructure sensor 512 provides as output 502 pre-processed data that already include a first sway estimate, e.g., in the form of a current position and a motion vector and/or in the form of extrinsic calibration parameters.

[0080] The second infrastructure sensor 514 provides a raw data signal as output 504. The pre-processed data 502 can be easily read in by a read-in module 522. The raw data 504 are post-processed by a motion detection module 524, e.g., by optical flow analysis. Subsequently, information obtained from the transmitted data of sensors 512 and 514 is combined in module 530 and converted into a common representation, if necessary, such as a common local coordinate system. Thereafter, a motion function is ascertained in module 540 with the information thus combined. The motion function can be characterized for example by various functional parameters. The motion function, or data determining the motion function, are passed on to an output interface 550. The output interface 550 generates data 552, 554, which are specific to the sensors 512, 514 and are transmitted as sensor outputs 562, 564 to the respective sensor 512, 514 and/or other sensors of the infrastructure sensor system. For example, a motion function 554 is passed on, which is transmitted as output 564 to the sensor 514 and/or other sensors of the infrastructure sensor system and by which a receiving sensor can specify its current position and/or motion. Using a known sensor position of the sensor 512 and/or of another sensor of the infrastructure sensor system, a specific motion vector 552 for the sensor 512 or for another sensor of the infrastructure sensor system can be determined and transmitted as output 562 to the respective sensor 512.

[0081] Architectural diagrams of infrastructure sensor systems 600 and 700 are shown in FIGS. 6 and 7, respectively. In these two embodiment examples, the correction information is not fed back into the infrastructure sensors, but rather flows into the post-processing of the environmental information detected by the infrastructure sensors of the respective infrastructure sensor system 600 and 700. In this way, the infrastructure sensors involved do not have to be compatible with the sensor sway detection and, for example, older or conventional sensors can also be used and their measurement data quality can be improved.

[0082] FIG. 6 shows an infrastructure sensor system 600 having a plurality of infrastructure sensors 612, 614, 615, 616, 618, 619 arranged on a shared mounting device (not shown), for example, according to FIG. 1, and which transmits data 622, 624, 626, 628, 629 to a device 610 for operating the infrastructure sensor system 600. The data are in this case received and further distributed by a communication unit (not shown). The device 610 in this example is configured as a computing unit comprising various modules (hardware and/or software) for further processing of the transmitted data. The device 610, for example, can be part of a so-called roadside unit (RSU), that is, an infrastructure-side computer system arranged in spatial proximity of the mounting device with the infrastructure sensors 612, 614, 615, 616, 618, 619 and configured to receive and further process data generated by the infrastructure sensors 612, 614, 615, 616, 618, 619, and to make the further processed data available, for example, to networked vehicles. Alternatively, the device 610 can be configured as part of a cloud system.

[0083] Infrastructure sensor system 600 includes infrastructure sensors 616, 618, and 619 that are configured as environmental sensors. In the example shown, the infrastructure sensor 616 is configured as a camera, the infrastructure sensor 618 is configured as a radar sensor, and the infrastructure sensor 619 is configured as a LIDAR sensor. The use of other or additional types of environmental sensors is possible. Infrastructure sensors 616, 618 and 619 are configured to detect their environment and to generate, from the detected environmental data, object lists that include characteristics, e.g., position, speed, acceleration, object size, object type, . . . of moving and/or stationary objects in the environment of the respective sensor. Infrastructure sensors 616, 618, and 619 each transmit such lists of objects as pre-processed data 626, 628, 629 to device 610 for further processing and evaluation.

[0084] Infrastructure sensor system 600 further includes infrastructure sensors 612, 614, 615 configured to detect a sway of the mounting device. In this case, the infrastructure sensor 612 is configured as an strain sensor, in particular as strain gage, and provides a measurement signal when a strain of a mechanical element of the mounting device occurs. The infrastructure sensor 614 is configured as an inertial sensor or accelerometer and is able to detect a motion for example, in particular an acceleration, of a mechanical element of the mounting device and output a corresponding measurement signal. Infrastructure sensor system 600 can include several strain sensors and/or accelerometers 615. Infrastructure sensors 612, 614, 615 transmit data 622, 624 as raw measurement data and/or as pre-processed motion and/or strain information to the device 610 for further processing and/or evaluation. The data 622, 624 transmitted by the infrastructure sensors 612, 614, 615 are fed to a sway estimation module 630 of the device 610 that processes the data 622, 624 and determines from this a motion function for the mounting device and/or provides correction information for the infrastructure sensors 616, 618, and 619 based on the motion function and provides the correction information and/or the motion function for sway compensation. The object lists transmitted from the infrastructure sensors 616, 618 and 619 are now corrected in a sensor-specific manner using the correction information and/or motion function provided to a respective sway compensation module 646, 648, 649 of the device 610, that is, the object properties included in the object lists are adjusted using the provided correction information and/or the motion function in such a way that this results in corrected object properties, for example corrected positions, speeds, accelerations, . . . of the objects in the object lists. The thus corrected object lists are fed to a sensor fusion module 650 that creates an environmental model based on the corrected object lists. The environmental model can be provided, e.g., to networked vehicles or other road users by way of a radio module 660.

[0085] In the example shown, infrastructure sensors 616, 618, and 619 can each already perform their own, first sway detection based on the data they themselves collect and can transmit the result to the sway detection module 630, which can take it into account in determining the motion function. In this case, it should be ensured that the infrastructure sensors 616, 618 and 619 do not already perform any additional sway compensation internally as the compensation steps can otherwise interfere with each other.

[0086] In FIG. 7, sensors without pre-processing are illustrated, by way of example, which provide raw data for example (e.g., video stream) and/or feature data (e.g., prominent lines, points, . . . ). These data must first be processed in the infrastructure system, for example, as a detection and tracking of objects. In addition, prominent points can be detected or read in and also fed into the sway detection.

[0087] FIG. 7 illustrates an infrastructure sensor system 700 having a plurality of infrastructure sensors 712, 714, 715, 716, 718, 719 disposed on a shared mounting device (not shown), for example, according to FIGS. 1A-1C, and respectively transmitting data 722, 724, 726, 728, 729 to a device 710 for operating the infrastructure sensor system 700. The data are received and further distributed by a communication unit (not shown). In this example, the device 710 is configured as a computing unit comprising various modules (hardware and/or software) for further processing the transmitted data.

[0088] The infrastructure sensor system 700 includes infrastructure sensors 716, 718, and 719 that are configured as environmental sensors. In the example shown, the infrastructure sensor 716 is configured as a camera, the infrastructure sensor 718 is configured as a radar sensor, and the infrastructure sensor 719 is configured as a LIDAR sensor. Infrastructure sensors 716, 718, and 719 are configured to detect their environment by, for example, recording image data of the environment, or measuring distances to objects in the environment. Infrastructure sensors 716, 718 and 719 can be configured to extract specific prominent environmental features from the detected raw data, for example, fixed structures such as guardrails or walls. Infrastructure sensors 716, 718, 719 transmit as data 726, 728, 729 respectively raw measurement data and/or, in the case of camera sensor 716, video stream data and/or information about detected prominent environmental characteristics (“feature data”) to the device 710 for further processing and/or evaluation. For each of the infrastructure sensors 716, 718, 719, the device 710 includes a pre-processing module 746, 748, 749. In the pre-processing module 746, the data 726 transmitted by the camera sensor 716 are processed, for example, by methods of digital image processing. In the process, in particular objects can be recognized and tracked. In the pre-processing module 748, the data 728 transmitted by the radar sensor 718 are processed, for example, by determining object distances and/or relative velocities from transmitted raw data. If the data 728 additionally or alternatively include feature data, the features can be associated with known features. In the pre-processing module 749, the data 729 transmitted by the LIDAR sensor 719 are processed, for example, by determining object distances from transmitted raw data. Thus, for each of the infrastructure sensors 716, 718, 719, pre-processed data are obtained that are provided to a sway detection module 730.

[0089] Optionally, the infrastructure sensor system 700 further includes infrastructure sensors 712, 714, 715 configured to detect a sway of the mounting device. The infrastructure sensor 712 is here configured as a strain sensor, in particular as a strain gage, and provides a measurement signal when a strain of a mechanical element of the mounting device occurs. The infrastructure sensor 714 is configured as an inertial sensor or accelerometer and can detect, for example, a motion, in particular an acceleration, of a mechanical element of the mounting device and output a corresponding measurement signal. Infrastructure sensor system 700 can include several strain sensors and/or accelerometers 715. Infrastructure sensors 712, 714, 715 transmit data 722, 724 as raw measurement data and/or as pre-processed motion and/or strain information to the device 610 for further processing and/or evaluation. The data 722, 724 transmitted by the infrastructure sensors 712, 714, 715 are provided to the sway estimation module 730 of the device 710.

[0090] The sway estimation module 730 can now determine from the optional data of infrastructure sensors 712, 714, 715 and from the data provided by pre-processing modules 746, 748, 749 a motion function for the mounting device, and/or ascertain correction information for infrastructure sensors 716, 718, and 719 based on the motion function, and provide the correction information and/or the motion function for sway compensation. In respective sway compensation modules 756, 758, and 759, the pre-processed data of infrastructure sensors 716, 718, and 719 can now be corrected for swaying of the shared mounting device. The thus corrected data, which can include, for example, object information, are fed to a sensor data fusion module 760 that creates an environment model based on the corrected data. The environmental model can be provided by means of a radio module 770, e.g., to networked vehicles or other road users.

[0091] In both the example shown in FIG. 6 as well as in the example shown in FIG. 7, the result of the sway detection must be applied to the pre-processed data streams of each individual infrastructure sensor 616, 618, 619, and 716, 718, 719, respectively, so that all pieces of object information in the sensor data fusion refer to the same coordinate system. In the sway compensation, individual sensors of infrastructure sensors 616, 618, 619, and 716, 718, 719, respectively, can also be excluded or assigned a lesser confidence or weight if specific thresholds of swaying have been exceeded.

[0092] FIG. 8 schematically shows a data format for a message 800 for exchanging information between an infrastructure sensor and the sway estimation module according to a possible embodiment of the present invention. The message 800 here includes the data transmitted from the sensor to the infrastructure sensor system and the correction information and/or motion function sent from the infrastructure sensor system to a sensor. The message 800 includes a message header 810 for this purpose. Furthermore, the message 800 includes a content portion 820. The content portion 820 includes a first data block 830 comprising sensor characteristics. In this example, the sensor characteristics include information about the sensor type 832 (e.g., camera/IMU/radar/lidar/ . . . ), the information 834 whether the sensor itself has sway detection, and the information 836 whether the sensor would like to receive sway information (e.g., a motion function) or not. The content portion 820 includes a second data block 840 comprising sensor output data, for example, directly the raw data 842 (e.g., pixel values in the case of a video sensor or point clouds in the case of a radar/LIDAR point clouds) or a link to the data, e.g., by a link to a RTSP stream (Real-Time Streaming Protocol) by which the video stream of the camera can be retrieved. Furthermore, the sensor output data can include a position 844, e.g., in the form of coordinates in a defined coordinate system and/or a motion vector and/or a sensor alignment or sensor orientation (yaw angle, pitch angle, roll angle). The content portion 820 comprises a third data block 850 comprising sensor input data, for example a motion function 852 and/or a corrected sensor position 854 and/or a corrected motion vector and/or a corrected sensor alignment or sensor orientation (yaw angle, pitch angle, roll angle). The message 800 further optionally includes a signature 860 and further optionally a certificate 870 for validating the signature.