Apparatus for Validating a Position or Orientation of a Sensor of an Autonomous Vehicle

Abstract

An apparatus for validating a position or an orientation of one or more sensors of an autonomous vehicle is provided. The one or more sensors provide consecutive sensor data of surroundings of the vehicle. The apparatus includes a validation module, which is configured to compare the consecutive sensor data and to validate a position or an orientation of at least one sensor of the one or more sensors, based on a deviation in the consecutive sensor data.

Claims

1.-14. (canceled)

15. An apparatus for validating a position or an orientation of one or more sensors of an autonomous vehicle, the one or more sensors providing consecutive sensor data of surroundings of the vehicle, the apparatus comprising: a validation module configured to compare the consecutive sensor data, and to validate a position or an orientation of at least one sensor of the one or more sensors based on a deviation in the consecutive sensor data.

16. The apparatus of claim 15, wherein: the consecutive sensor data includes calibration data indicating a correct position or orientation of the at least one sensor, and the validation module is configured to compare present sensor data with the calibration data.

17. The apparatus of claim 15, wherein: the validation module is configured to compare the consecutive sensor data by localizing a pattern in surroundings of the at least one sensor.

18. The apparatus of claim 17, wherein: the validation module is configured to localize as the pattern a structural feature of the vehicle.

19. The apparatus of claim 15, wherein: the vehicle is a commercial vehicle comprising a cabin on a vehicle frame, the cabin and/or the vehicle frame includes a structural feature, the one or more sensors include at least one sensor on the cabin and/or at least one sensor on the vehicle frame, and the validation module is configured: to localize the structural feature on the cabin when the at least one sensor is positioned on the cabin, and/or to localize the structural feature on the vehicle frame when the at least one sensor is positioned on the vehicle frame.

20. The apparatus of claim 15, wherein: the vehicle is a commercial vehicle combination comprising a towing vehicle and a trailer, the towing vehicle and/or the trailer include a structural feature, the one or more sensors include at least one sensor on the towing vehicle and/or at least one sensor on the trailer, and the validation module is configured: to localize the structural feature on the towing vehicle when the at least one sensor is positioned on the towing vehicle, and/or to localize the structural feature on the trailer when the at least one sensor is positioned on the trailer.

21. The apparatus of claim 17, wherein: the validation module is configured to take into account a driving situation of the vehicle, and the pattern is one of: a road marking, a traffic barrier, a kerb, or a traffic light.

22. The apparatus of claim 15, wherein: the one or more sensors comprise at least two sensors, and the validation module is configured to compare consecutive sensor data provided by one of the at least two sensors with consecutive sensor data provided by another of the at least two sensors.

23. The apparatus of claim 15, wherein: the one or more sensors are multiple sensors, and the validation module is configured to validate a position or an orientation of one sensor of the multiple sensors by using sensor data of another sensor of the multiple sensors.

24. A vehicle comprising: the one or more sensors for providing the consecutive sensor data of the surroundings of the vehicle in order to perform automated driving tasks, and the apparatus according to claim 15.

25. The vehicle of claim 24, wherein: the vehicle is a commercial vehicle with or without a trailer.

26. The vehicle of claim 24, wherein: the one or more sensors comprise one or more of: a mirror replacement sensor, a downward-facing camera, a front-facing camera, a surround-view camera, a radar sensor, a lidar sensor, or an ultrasonic sensor.

27. A method for validating a sensor position of an autonomous vehicle, the vehicle comprising one or more sensors for providing consecutive sensor data of surroundings of the vehicle, the method comprising: comparing the consecutive sensor data; and validating a position or an orientation of at least one of the one or more sensors based on a deviation in the consecutive sensor data.

28. A computer product comprising a non-transitory computer readable medium having stored thereon program code which, when executed on a processor, carries out the method of claim 27.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG. 1 depicts an apparatus for validating a position or orientation of a sensor of an autonomous vehicle.

[0045] FIG. 2 depicts a more detailed embodiment of an apparatus for validating positions or orientations of several sensors present on an autonomous commercial vehicle.

[0046] FIG. 3 illustrates an overlaying of features in sensor data for comparison and validation of a position or orientation of a sensor.

[0047] FIG. 4 illustrates a comparison of sensor data by template matching for online validation from correlation changes in the feed of a sensor.

[0048] FIG. 5 shows steps of a method for validating a position or orientation of a sensor of an autonomous vehicle.

DETAILED DESCRIPTION OF THE DRAWINGS

[0049] FIG. 1 depicts an apparatus 100 for validating a position or orientation of a sensor of an autonomous commercial trailing vehicle 50. The vehicle 50 comprises a sensor 510, which in the present embodiment is a rear-facing camera whose field of view is indicated by emerging dashed lines. The sensor 510 provides consecutive sensor data—in this case, a camera image—of surroundings of the vehicle 50. The sensor data in particular comprises a pattern originating from an edge 54 of the vehicle. The apparatus 100 is configured to access the sensor data, and comprises a validation module 110 configured to perform a comparison 113 of the consecutive sensor data, and a validation 117 of a position or orientation of the sensor based on a deviation in the consecutive sensor data, which in this embodiment is a location of the edge 54 within the image conveyed by the sensor data.

[0050] FIG. 2 shows an embodiment of an apparatus 100 for validating positions or orientations of sensors 510, 520, . . . , typically present on an autonomous commercial vehicle 50. The vehicle 50 comprises a towing vehicle with a cabin 53 and a vehicle frame 55, and a trailer 57. The cabin comprises in particular a rear-facing camera 510, a forward-facing camera and radar 520, and a surround-view fish-eye camera 530, among others. The vehicle frame in particular hosts a wide-angle radar sensor 540 centered around a forward direction, and a similar radar sensor 550 for a lateral direction, with an overlap of their respective fields of view. The trailer comprises several more fish-eye cameras 560, 570, which together with the fish-eye cameras 530 on the cabin provide a surround view of the vehicle in multiple coverings. Furthermore, a couple of rear-facing lidar sensors 580, 590 are depicted on the trailer. Trailer and/or trailing vehicle may comprise further sensors not detailed in this figure.

[0051] The sensors on the vehicle, including the mentioned sensors 510, 520, . . . repeatedly provide respective sensor data for a variety of autonomous and automatized driving tasks. The sensor data are tapped by the apparatus 100 and provided to a validation module 110. The validation module 110 in this embodiment is configured as a single data processing unit, but may also be spread over several such units, or to various degrees combined with existing electronic control units of the vehicle 50.

[0052] In the present embodiment, the validation module 110 compares and validates sensor data in several ways. One way is by storing data of an individual sensor, possibly including sensor calibration information, and comparing present data of that sensor with said stored data in order to recognize deviations attributable to sensor misalignment or displacements. For example, images of the rear-facing camera 510 installed on the cabin 53 may be compared with previous such images, and a localization of a structural pattern of the cabin 53, like e.g. the position of an edge 54, may be evaluated. If the position of the edge 54 in the compared images deviates to a degree above a certain threshold, a warning about an inaccurate orientation of the rear-facing camera 510 may be issued e.g. to a driver, to a superordinate system of the autonomous vehicle 50, or to a place remote from the vehicle 50, like e.g. a workshop. The structural pattern, here the vehicle edge 54, is observed on a part of the vehicle 50 (the cabin 53) which is in rigid connection with the sensor, here the rear-facing camera 510.

[0053] As another way, positions or orientations of sensors directly observed by other sensors on the vehicle may be validated directly. In the depicted embodiment, the fish-eye camera 530 appears in the image of the rear-facing camera 510, and the image of the rear-facing camera 510 may therefore serve for directly detecting deviations in the position or orientation of the fish-eye camera 530. Again, a rigid connection between the sensors, as here for the rear-facing camera 510 and the fish-eye camera 530, is advantageous.

[0054] Another way involves features either on or off the ego vehicle 50 which are simultaneously observed by at least two different sensors. Relative localizations of such features in the sensor data of different sensors can be compared and validated. In such cases, a suitable pattern recognition could even allow an exploitation of merely transiently visible features.

[0055] For example, the two fish-eye cameras 560, 570 may both provide an aspect of a rear mudguard of the vehicle frame 55 of the trailing vehicle, e.g. in a curve or in other situations where there is an angle between trailing vehicle and trailer 57. A difference between localizations of an edge of the mudguard in the images provided by the two respective cameras 560, 570 could be compared to a previously determined such difference, with a change signaling at least a potential displacement of one of the two cameras 560, 570. Other features of the vehicle may be employed either alternatively or as a cross-check. The fish-eye cameras 560, 570 are rigidly connected to the trailer, and an angle under which the two cameras 560, 570 observe structures of the trailer 57, such as e.g. a point on the lateral underrun protection 58, may serve as a more steady mark for a monitoring of sensor calibration than structure on the mudguard of the trailing vehicle.

[0056] The validation module may also be configured to employ objects or markings away from the vehicle. This can be particularly advantageous for sensors which do not feature vehicle structure in their respective sensor data. In the present embodiment, a situation is depicted in which two of the rear lidar sensors 580, 590 both localize a particular street bollard 70 in their respective field of view, which may be employed for determining a difference in the angle between the lines of sight of the two lidar sensors 580, 590. Comparing this to previous or calibration data for the two lines of sight, which in particular may have been obtained from a different object than the depicted pollar, can serve for monitoring the calibration of the two lidar sensors 580, 590. Objects serving for orientation in traffic may be particularly advantageous for such a validation, especially if their recognition in the sensor feed is already effectuated in the context of other automatic driving tasks, and may merely be exploited for calibration validation.

[0057] Similar checks of position or orientation as those described above may furthermore be configured for the forward-facing camera 520, for the wide angle radar sensors 530, 540, and for other sensors on the vehicle 50.

[0058] FIG. 3 illustrates an overlaying of features in sensor data as an embodiment for comparison and validation of a position or orientation of a sensor (e.g. the rear-facing camera 510) on a commercial vehicle 50. On the left of this figure, an image captured by an exteroceptive rear-facing sensor, such as e.g. a mirror replacement sensor, is displayed, including parts of the commercial vehicle 50 with a cabin 53 and a trailer 55.

[0059] According to embodiments patterns 517 of the cabin 53 (or the trailer 55) are used for the validation of the correct position or orientation of the exemplary camera. In this embodiment, patterns 517 include various edges/grooves of the cabin 53 and an inscription or a label (e.g. an advertisement writing or logotype), which can be stored as a pattern structure with a correct orientation. The validation module may in particular be configured to perform a pattern recognition, for which several methods or algorithms are known. The validation can then be performed by overlaying a current with the stored pattern structure, and to verify whether the images match, e.g. by verifying that the various edges/grooves in the captured current image and in stored pattern structure are at the same position. This is shown on the right-hand side, wherein only an excerpt of the image with highlighted structural features 517 is displayed originating from grooves on the vehicle cabin and advertisement writing. Positions of these structural features 517 can be compared with corresponding positions also in subsequent images in order to validate the alignment of the exemplary camera. Validation may involve a threshold or a tolerance, in translational and/or in rotational degrees of freedom, for the deviation, and the validation module may be configured to send in information and/or trigger a warning for e.g. a driver, a superordinate system, or a destination remote from the vehicle, like e.g. a workshop.

[0060] FIG. 4 illustrates a comparison of sensor data in the validation module by template matching for online validation from correlation changes in the feed of the exteroceptive rear-facing sensor 510 of FIG. 3. Methods of pattern matching can include matching sensor data with data from a template, for which the validation module can obtain input data e.g. via vehicle-to-infrastructure or mobile communication. In the present figure, a pattern matching algorithm performed by the validation module recognizes a logotype 518 by way of a template 80 obtained by vehicle-to-infrastructure or mobile communication. The term pattern matching is understood broadly, and may also include methods for thermal patterns. The right-hand side of the figure shows a matching intensity map of a particular structure in the image. Points or pixels on such a map are drawn according to a color or intensity code, where higher values represent a better matching of two compared images. Depicted is an exemplary case in which parts of the logotype 518 are matched for a current sensor image (the middle image) and a stored calibration reference structure (the image to the left), according to a code displayed on the right of the figure.

[0061] FIG. 5 shows steps of a method for validating a sensor position of an autonomous vehicle 50, the vehicle 50 comprising one or more sensors for providing consecutive sensor data of surroundings of the vehicle. A first step of this method comprises comparing S110 consecutive sensor data. This comparison S110 can be performed for sensor data provided by an individual sensor at different points in time, and/or for sensor data provided by different sensors both at the same time and at different times. Comparing S110 consecutive sensor data can in particular also involve calibration data. On-vehicle environment perception sensors, as e.g. a video camera, may be employed to crosscheck proper position and orientation of the sensors and of other sensors on the vehicle by using own/ego vehicle geometry and texture to find features and compare them to the earlier calibrated sensor signals in order to detect changes.

[0062] This method may also be a computer-implemented method. A person of skill in the art would readily recognize that steps of various above-described methods may be performed by programmed computers. Embodiments are also intended to cover program storage devices, e.g., digital data storage media, which are machine or computer readable and encode machine-executable or computer-executable programs of instructions, wherein the instructions perform some or all of the acts of the above-described methods, when executed on the computer or processor.

[0063] The description and drawings merely illustrate the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.

[0064] Furthermore, while each embodiment may stand on its own as a separate example, it is to be noted that in other embodiments the defined features can be combined differently, i.e. a particular feature described in one embodiment may also be realized in other embodiments. Such combinations are covered by the disclosure herein, unless it is stated that a specific combination is not intended.

[0065] Functions of various elements shown in the figures, when provided by a processor, may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term “processor” or “electronic control unit” should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional and/or custom, may also be included.

LIST OF REFERENCE SIGNS

[0066] 50 vehicle [0067] 53 cabin of a towing vehicle (tractor) [0068] 54 edge [0069] 55 vehicle frame of a towing vehicle [0070] 56 rear mudguard of vehicle frame [0071] 57 trailer [0072] 58 underrun protection of trailer [0073] 70 street bollard [0074] 80 logotype template [0075] 100 apparatus for validating a position or orientation of a sensor [0076] 110 validation module [0077] 113 comparison of consecutive sensor data [0078] 117 validation of a sensor position [0079] 510 rear-facing camera [0080] 517 structural feature of vehicle in sensor data [0081] 518 logotype on vehicle in sensor data [0082] 520 forward-facing camera and radar [0083] 530 surround-view fish-eye camera on cabin [0084] 540, 550 wide-angle radar sensors on towing vehicle frame [0085] 560, 570 surround-view fish-eye cameras on trailer [0086] 580, 590 rear-facing lidar sensors on trailer [0087] S110, S120 steps of a method