Method for detecting inconsistencies in the outputs of perception systems of autonomous vehicles

Abstract

A system and method for the detection of inconsistencies in perception systems of autonomous vehicles is described. The system receives the observations of objects in the surrounding environment from one or more sensors or perception systems of an automated vehicle. At actual time, the system estimates the consistency of the currently observed elements of the perception system according to the previous inputs received. This consistency is decided by calculating the boundaries of possible states of the previously observed elements, based on the received information and on assumptions.

Claims

1. A computer-implemented method for detecting inconsistencies in the observations from perception systems (1.2) and perception sensors (1.1) of an automated vehicle (1) and running on an electronic control unit (1.3), which comprises the steps of: a. receiving and storing the observed states of the scene and environment from perception systems (1.2) and sensors (1.1), b. calculating the boundaries of one or more possible states of a previously observed object at a given timestamp based on the previous states of the object, the assumptions on the behavior of the object, or the type of scene, or the environmental conditions, or a combination thereof, c. checking whether an estimated state of a scene or object stays within the calculated boundaries obtained in step b, d. sending a notification when an estimated state does not stay within a calculated boundary to the electronic control unit (1.3), e. optionally having the electronic control unit perform safety actions based on the notification of step d.

2. The inconsistency detection method of claim 1, wherein the observed states of the scene are objects, road shapes, or environmental conditions or combinations thereof.

3. The inconsistency detection method of claim 1, wherein the observed and estimated states of scenes, objects, road shapes, or environmental conditions or combinations thereof are stored and then used to calculate the boundaries of states of objects in the future or to match current observed states with future observed states.

4. The inconsistency detection method of claim 1, wherein the observed and estimated states are stored for fixed period of time, wherein the fixed period is comprised between 0.1 seconds and 10 seconds, preferably between 1 second and 5 seconds, even more preferably between 1.5 seconds and 3 seconds.

5. The inconsistency detection method of claim 1, wherein the observed and estimated states are stored until new information about the same object is received.

6. The inconsistency detection method of claim 1, wherein boundaries of possible states of an object or a scene are calculated at a given timestamp.

7. The inconsistency detection method of claim 1, wherein the boundaries are calculated based on one or more of: a. the previous bounding box of the object, b. the previous velocity of the object, c. the previous acceleration of the object, d. the previous heading of the object, e. the shapes of the road or the lane markings, f. the assumption on the maximum acceleration of the object, g. the assumption on the minimum acceleration of the object, which can be negative, h. the assumption on the maximum velocity of the object, i. the assumption on the minimum velocity of the object, j. the assumption on the space boundary that the object could reach, k. the assumption on the environment conditions fluctuation.

8. The inconsistency detection method of claim 1, wherein the calculated boundaries are one or more of the following values: a. the maximum and minimum velocity of the object, b. the occupancy space of the object, represented by the maximum and minimum on each axis of a coordinate system.

9. The inconsistency detection method of claim 1, wherein the coordinate systems comprise: a. A 2D cartesian coordinate system, b. A 3D cartesian coordinate system, or c. A 2D or a 3D Frenet coordinate system.

10. The inconsistency detection method of claim 1, wherein the assumption about the acceleration of the objects are calculated as follows:
v_max=previous_v+a_max*delta_t  a.
v_min=previous_v+a_min*delta_t  b.
p_max=previous_p_max+previous_v*delta_t+0.5*a_max*delta_t{circumflex over ( )}2  c.
p_min=previous_p_min+previous_v*delta_t+0.5*a_min*delta_t{circumflex over ( )}2.  d.

11. The inconsistency detection method of claim 1, wherein the assumption about the acceleration and velocity of the objects are calculated as follows:
v_max=min(previous_v+a_max*delta_t,v_assumption_max);  a.
v_min=max(previous_v+a_min*delta_t,v_assumption_min)  b.

Description

SHORT DESCRIPTION OF THE DRAWINGS

[0078] FIG. 1 is a schematic flow chart of the inconsistency detector system of the present invention in an autonomous vehicle.

[0079] FIG. 2 is a flow chart of the method for the detection of inconsistencies in autonomous vehicles of the present invention.

DETAILED DESCRIPTION OF THE DRAWINGS

[0080] FIG. 1 is a schematic flow chart of the inconsistency detector system of the present invention in an autonomous vehicle system (1). The information from the environment measured by sensors (1.1) is directed to the perception systems (1.2) of the automated vehicle. Examples of sensors include: [0081] Cameras, [0082] Light Detection And Ranging, also referred to as LiDAR, [0083] Radars, or [0084] Global Navigation Satellite System positioning, also referred to as GNSS positioning.

[0085] The perception systems (1.2) of the vehicle interpret the raw information from the sensors (1.2) and extract observations on the scene. Such observations include one or more of the existing elements, their position, or environmental conditions.

[0086] The vehicle central board (1.3) is capable of performing several vehicle processes, such as vehicle control and decision making units that perform tasks such as path planning. The outputs of the vehicle central board (1.3) are executed by the vehicle actuators (1.4).

[0087] The inconsistency detector system (1.5) of the present invention monitors information from the sensors (1.1) and the perception systems (1.2), hereinafter jointly referred to as observations. The inconsistency detector system (1.5) informs the vehicle central board (1.3) about the reliability of those observations.

[0088] The system is running on an electronic control unit including one or more processors and a memory. The memory may include one or more instructions which can be executed by one or more processors causing the detection of inconsistencies from the input observations received at the electronic control unit.

[0089] The system receives observations of the scene and objects in the surrounding environment from one or more sensors (1.1) or from one or more perception systems (1.2) in the vehicle. The system may receive additional input from road information such as shape of the road, curvature, traffic status, or surface condition or a combination thereof. The system may also receive additional input such as environmental conditions including the position of the sun, weather, or humidity.

[0090] In another embodiment, the system receives the observations at a single time or during an interval comprising consecutive times.

In another embodiment, the system receives previously stated input, observations and times.

[0091] In general, each observation obtained from the real scene observed by the sensors (1.1) or the perception systems (1.2) generates one or several observation states in the inconsistency system associated to a time. Each observed state is stored for a fixed period of time. For example, an observed state may be stored for 2 seconds.

[0092] In subsequent times, the current observed state system inputs are updated to estimated states. The estimated states are obtained by calculating the boundaries of the possible states of the objects or the scene, hereinafter referred to as state boundaries.

[0093] The calculation of the state boundaries is based on one or more of the following parameters and features: [0094] the previously received observations, [0095] the assumptions on the behavior or aspect of the object, and [0096] the road information and environmental conditions received.

[0097] Once the current observations are received, the inconsistency detection system (1.5) evaluates their consistency as shown in FIG. 2.

[0098] In a first step, the inconsistency detector system (1.5) checks for each new observed state, whether there exists previously stored estimated states of the same object or scene.

[0099] If there are no previously stored estimated states of the same object or full or partial scene, the system does not perform an inconsistency check.

[0100] If there are previously stored estimated states of the same object or full or partial scene, the system performs an inconsistency check. The inconsistency check consists of assessing whether or not the current observed state lies in the estimated state boundaries. If the new observed state is outside of the calculated boundaries, the inconsistency detection system (1.5) will consider the output of the perception system (1.2) or of the sensors (1.1) as inconsistent.

[0101] If an inconsistency is detected, the inconsistency detection system sends a notification to the control units in the vehicle to act accordingly and safely. With this signalization, the control units can perform appropriate actions to mitigate the inconsistency such as informing other subsequent systems as for example systems responsible for planning, decision making and control on the autonomous vehicle.

[0102] In one embodiment, the actions taken by the control system (1.4) that receives the inconsistency system signals of sensor or perception inconsistencies are not under the scope of this invention.

TABLE-US-00001 TABLE 1 English expressions used in the drawings for translation purposes: Autonomous vehicle Autonoom voertuig Sensor Sensor Perception component Perceptie-element Inconsistency detector Inconsistentiedetector Planning and control components Onderdelen voor planning en regeling Actuators Actuatoren Receive a new observation Ontvang een nieuwe waarneming Check if there exists previous Controleer of er eerdere observations of the same object waarnemingen van hetzelfde object bestaan Yes Ja No Nee Exit Exit Calculate boundaries from each Bereken de grenzen van iedere previous observation voorgaande waarneming Check whether the new Controleer of de nieuwe waarneming observation stays inside binnen alle grenzen blijft all boundaries Notify other systems about the Informeer andere systemen over de inconsistency inconsistentie