Method and Device for Processing Sensor Data

Abstract

A method for processing sensor data includes assessing the sensor data of a sensor using metadata of the sensor as well as sensor data of at least one additional sensor using the metadata of the additional sensor, in order to receive assessed sensor data of the sensors. The method further includes merging the assessed sensor data in order to receive merged sensor data.

Claims

1. A method for processing sensor data, comprising: assessing sensor data of a sensor using metadata of the sensor; assessing sensor data of at least one additional sensor using metadata of the additional sensor; receiving the assessed sensor data of the sensor and the at least one additional sensor; merging the assessed sensor data; and receiving the merged sensor data.

2. The method according to claim 1, wherein the metadata of the sensor are read out from a memory of the sensor.

3. The method according to claim 1, wherein the metadata of the sensor are stored in a memory of a data processing apparatus and read out from the memory.

4. The method according to claim 1, further comprising: inputting the metadata via a metadata interface.

5. The method according to claim 1, wherein the metadata map static properties of the sensor.

6. The method according to claim 1, wherein: the metadata map variable properties of the sensor, and at least one parameter currently influencing the sensor is detected and the metadata are parametrized using the at least one parameter.

7. The method according to claim 6, further comprising: detecting at least one current environmental condition at the sensor and/or within a detection range of the sensor as the at least one parameter.

8. The method according to claim 1, wherein: the sensor data are coordinate-based, and an item of information assigned to a coordinate of the sensor data is assigned an item of metainformation stored in the metadata in relation to the coordinate, in order to assess the item of information.

9. A device configured to carry out, implement, and/or actuate in corresponding apparatuses the method according to claim 1.

10. The method according to claim 1, wherein a computer program product is configured to instruct a processor, when the computer program product is executed, to carry out, implement, and/or actuate the method.

11. A non-transitory machine-readable storage medium on which the computer program product according to claim 10 is stored.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Embodiments of the invention are described below with reference to the accompanying drawings, and neither the drawings nor the description should be construed as limiting the invention.

[0024] FIG. 1 is an illustration of an information system comprising a device according to an exemplary embodiment; and

[0025] FIG. 2 is an illustration of metadata according to an exemplary embodiment.

[0026] The figures are merely schematic and not true to scale. In the figures, identical reference signs refer to identical or identically acting features.

EMBODIMENTS OF THE INVENTION

[0027] FIG. 1 is an illustration of an information system 100 comprising a device 102 according to an exemplary embodiment. The information system 100 is, for example, a sensor system of a vehicle. The information system 100 has a plurality of sensors 104 and a plurality of data sources 106. In this case, the information system comprises 1 to n sensors 104 and 1 to m data sources 106. Sensor data 108 of the sensors 104 and data 110 of the data sources are input by the device 102. In addition, metadata 112 of the sensors 104 are input by the device 102. In the process, metadata 112 of the data sources 106 may also be input. In the process, metadata 112 may also not be input for each sensor 104. For this reason, no metadata 112 may be present for some of the sensors 104. At least the sensor data 108 are assessed in the device 102 using the metadata 112 in order to receive assessed sensor data 114. During assessment, quality assessments are added to descriptions of objects imaged in the sensor data 108.

[0028] In one exemplary embodiment, the assessed sensor data 114 are combined or merged in order to receive merged sensor data 116. In this case, a description of an object imaged in more than the assessed sensor data 114 of one of the sensors 104 is supplemented by descriptions from the assessed sensor data 114 of at least one other of the sensors 104. During the combining to form the merged sensor data 116, the descriptions having the higher quality assessments are given greater weighting than the descriptions having the lower quality assessments.

[0029] In one exemplary embodiment, the sensor data 108 of a sensor are assessed in a location-dependent manner. In this case, coordinates 118 of an object imaged in the sensor data 108 are determined and assessed using metadata 112 assigned to the coordinates 118. For this reason, objects recognized at different coordinates 118 may also be assessed using different metadata 112.

[0030] In one exemplary embodiment, the metadata 112 are parametrized before the sensor data 108 are assessed. To parametrize the metadata 112 of a sensor 104, at least one parameter 120 influencing the sensor 104 is determined. The parameter 120 may, for example, map environmental conditions at the sensor 104.

[0031] FIG. 2 is an illustration of metadata 112 of a sensor 104. The metadata 112 may be used for processing sensor data according to the approach presented here. The metadata 112 describe a detection quality of the sensor 104 over a detection range 200 of the sensor. The detection range 200 here is thus described three-dimensionally, i.e. spatially. Alternatively, the detection range may also be described two-dimensionally, i.e. in a planar manner.

[0032] The detection range 200 is divided into small portions 202. In this case, the portions 202 are cubic and substantially all of the same size. For each portion 202, metainformation 204 is stored in the metadata. The metadata 204 describe a detection quality of the sensor 104 for that portion 202.

[0033] In one exemplary embodiment, the metainformation 204 can be parametrized. In this case, the metainformation 204 is dependent on at least one current condition at the sensor and/or in the detection range 200.

[0034] In other words, what is presented is a safety interface for flexibly and dynamically using sensors in a safe sensor merging for automated driving.

[0035] The integrity class of a sensor (for example ASIL, SIL, PL) relates to specific functions of the sensor, not to the sensor as a component. This may lead to misunderstandings, such as an erroneous adaptation of the requirements.

[0036] ISO/PAS 21448 (SOTIF) points out that integrity alone is not decisive for whether sensor signals are usable in safety-related functions. Rather, the performance or insufficiencies of the sensor (based on individual measurement functions) may be taken into account in the safety concept or in the signal merging. Critical errors may, for example, be incorrect measurements, false positives (FP) or false negatives (FN).

[0037] In the approach presented here, a sensor supplies information about its capabilities, its safety integrity for specific functions (ISO 26262, IEC 61508, ISO 13849, IEC 62061, ISO 25119, etc.) and its insufficiencies (SOTIF, ISO/PAS 21448) via a standardized interface. This information can be referred to as metadata or “safety metadata.”

[0038] The capabilities and insufficiencies can be described on the basis of a geometric grid (e.g. a 3D cube grid) around the sensor; for this grid, quality classes per measurement function (color, object position, speed, etc.) can be defined depending on further parameters (e.g. environmental conditions, sensor state).

[0039] By means of the approach presented here, sensors can be combined more easily to form a sensor set that demonstrably achieves a required safety integrity and sufficient safety for the functionality (SOTIF, ISO/PAS 21448).

[0040] Sensors (or data sources) equipped with this standard interface can thus also be integrated into an existing sensor set on an ad hoc basis. For example, sensors in the road infrastructure (traffic control systems, systems of a mobility data marketplace) can be (temporarily) integrated into the merging (sensor/information) of a passing automated vehicle. In a similar scenario, sensors installed on construction sites can be integrated into the perception/merging of a construction site vehicle.

[0041] An integrity (QM-ASIL D, SIL 1-4, PLa-PLe) and a quality class (e.g. 1-4) can be stored as safety information of the interface (“safety metadata”) for all measurements, per measurement attribute (color, position, dynamics) and/or per grid element.

[0042] ODD-specific factors may be taken into account as influencing factors, for example internal and/or external use (protected or unprotected); stationary operation and/or mobile operation; temperature and humidity; precipitation (rain, hail or snow) and wind; pressure (of ambient air, water, etc.); solar irradiation and heat radiation; condensation and icing; fog, dust, sand and salt mist; vibrations and shaking; fauna and flora (e.g. mold formation); chemical influences; electrical and electromagnetic influences; mechanical loads; sound.

[0043] A current sensor state may also be taken into account as an influencing factor. In this respect, internal errors, heating and/or occlusion may be taken into account.

[0044] Known points of interest (POI), in the form of a GPS position with a direction, at which deterioration in the sensor performance or an increase in sensor errors caused by external effects has been detected during validation (“triggering events” as per ISO/PAS 21448) may also be taken into account as influencing factors.

[0045] The interface may be designed, for example, as a multi-dimensional characteristic matrix, a safety contract, as Conditional Safety Certificates (ConSerts) or as Conditional Dependability Certificates (DDI).

[0046] As in FIG. 2, the characteristic matrix may have 3D cubes as a grid geometry, for example. The cubes can be arranged, for example, in accordance with a grid used in the merging. An occupancy grid map may measure 10×10×10 cm, for example. The characteristic matrix may also have concentric circles as a grid geometry; these can be arranged equidistantly or at increasing distances from one another. The grid geometry may also be a mixed form of the two options. For each 3D grid element, the capabilities of the sensor are specified in a matrix together with safety attributes (integrity, conditional insufficiencies).

[0047] By means of the approach presented here, any sensors or information sources that are intended to be integrated into an existing sensor set of a system can be used (e.g. retrofitting, also “off-the-shelf” sensors). Sensors of other vehicles located in the vicinity can be used together with additional information about a (global) position and an orientation for transforming the 3D grid into the coordinates of the ego-vehicle. Sensors in the infrastructure, in traffic control systems or in systems installed in the context of a mobility data marketplace by third parties at the road edge, on buildings, etc. (smart cities), can also be used.

[0048] To supply the information, an intelligent sensor can itself determine the current capabilities and insufficiencies (possibly also the safety integrities, depending on transient errors) (preprocessing) and output them at the interface. Alternatively or additionally, the sensor can in advance supply specific validated and calibrated information as a characteristic matrix of an “expert system” in the above-described standardized format, so that the subsequent modules can evaluate this information according to their requirements.

[0049] From the data, the merging generates measures for the integrity and reliability or insufficiency (SOTIF) of specific information (“safety metadata”); these are taken into account in the later behavior and trajectory planning for the vehicle and may lead to limitations in behavior (slower driving, prohibition of specific maneuvers), for example if the information has low integrity or low reliability.

[0050] Sensors and further data sources deliver their data, including the information about the quality of the data, depending on conditions (e.g. environmental conditions, position). Use is made of these data in the merging to optimize the performance and integrity thereof. The merging result is forwarded to the subsequent modules together with an aggregated assessment of quality with regard to safety (integrity, reliability, confidence).

[0051] The data can, for example, also be centrally collected and supplied via an expert system on board the vehicle.

[0052] Finally, it should be pointed out that terms like “having,” “comprising,” etc. do not exclude other elements or steps and terms like “a” or “an” do not exclude a plurality. Reference signs in the claims are not to be considered as limiting.