LOCALIZATION AND MAPPING METHOD AND SYSTEM

Abstract

The invention relates to a localisation and mapping method used by a mobile vehicle in an environment, said method comprising the following steps: the determination of the type of an object located in an area of the environment, on the basis of the data received from an on-board sensor in the mobile vehicle; and implementation of a localisation algorithm using detection data, without taking into account the detection data relating to said area or said object when the determined type is a type of mobile object.

Claims

1. A localization and mapping method used by a mobile machine in an environment, comprising: determining, on the basis of data received from a sensor on board the mobile machine, a type of an object located in an area of the environment; and executing a localization algorithm using detection data, without taking into account the detection data relating to said area or to said object when if the determined type is a mobile object type.

2. The method as claimed in claim 1, wherein the sensor is a lidar sensor.

3. The method as claimed in claim 2, wherein said determination is performed by recognition of a shape or a signature in the received data.

4. The method as claimed in claim 1, wherein the sensor is an image sensor.

5. The method as claimed in claim 4, wherein said determination is performed by recognition of a shape in at least one image represented by the received data.

6. The method as claimed in claim 1, wherein the detection data are obtained from the on-board sensor.

7. The method as claimed in claim 1, wherein the detection data are obtained from another sensor separate from said on-board sensor.

8. The method as claimed in claim 1, wherein the localization algorithm uses said object as a reference point if the determined type is a fixed object type.

9. The method as claimed in claim 1, wherein the localization algorithm uses the detection data relating to a given area when no object located in said given area is detected with a type corresponding to a mobile object type.

10. The method as claimed in claim 1, wherein the localization algorithm that is executed constructs a map of the environment.

11. The method as claimed in claim 10, further comprising: saving the constructed map; and at a later time, loading and re-using the map constructed by the localization algorithm.

12. A localization and mapping system to be fitted to a mobile machine in an environment, comprising: a module for determining, on the basis of data received from a sensor on board the mobile machine, a type of an object located in an area of the environment; and a localization module configured to localize the mobile machine on the basis of detection data, without taking into account the detection data relating to said area or to said object when if the determined type is a mobile object type.

Description

DETAILED DESCRIPTION OF AN EXEMPLARY EMBODIMENT

[0029] The following description, referring to the attached drawings which are provided by way of non-limiting example, will make the nature and application of the invention clear.

[0030] In the attached drawings:

[0031] FIG. 1 shows a motor vehicle equipped with a localization and mapping system according to the invention;

[0032] FIG. 2 shows an example of a particular context that may be encountered by the vehicle of FIG. 1;

[0033] FIG. 3 shows schematically a first example of a localization and mapping system according to the invention;

[0034] FIG. 4 shows a table of data used in the system of FIG. 3;

[0035] FIG. 5 shows schematically a second example of a localization and mapping system according to the invention; and

[0036] FIG. 6 shows the main steps of a localization and mapping method according to the invention.

[0037] FIG. 1 shows a motor vehicle V equipped with a localization and mapping system S.

[0038] In this case, the localization and mapping system S is constructed in the form of a microprocessor-based processing device.

[0039] This processing device comprises a memory (for example a read-only memory or a rewritable non-volatile memory, or any random access memory in general) adapted to store computer program instructions, the execution of which by the microprocessor of the processing device causes the execution of the methods and processes described below.

[0040] The motor vehicle V comprises one or more on-board sensors, for example a visual sensor such as a video camera CAM and/or a distance sensor such as a laser remote sensor or lidar (an acronym for “light detection and ranging”) sensor LID.

[0041] The localization and mapping system S receives the data INFO.sub.CAM, INFO.sub.LID generated by the on-board sensor(s), and processes them for the purposes of constructing a map C of the environment in which the motor vehicle V maneuvers and establishing the localization of the vehicle V in the constructed map C.

[0042] FIG. 2 shows an example of a context that may be encountered by the vehicle V.

[0043] In this example, the vehicle V maneuvers in a two-way road R, bordered on either side of the carriageway by a sidewalk TR, and then by houses H beyond the sidewalk TR.

[0044] A third-party vehicle V′ is parked in the part of the road R located in front of the vehicle V, partly on the carriageway of the road R and partly on the sidewalk TR.

[0045] FIG. 3 shows schematically a first example of a localization and mapping system according to the invention. In this example, the localization and mapping system S uses the data INFO.sub.CAM, INFO.sub.LID delivered by two sensors (in this case, the video camera CAM and the lidar sensor LID).

[0046] FIG. 3 shows functional modules, each of which corresponds to a particular process carried out by the localization and processing system S. In the example described here, the processes are carried out, as mentioned above, as a result of the execution by the microprocessor of system S of computer program instructions stored in a memory of the system S. In a variant, the processes carried out by one or more functional modules could be executed by a dedicated integrated circuit, for example an application specific integrated circuit (or ASIC).

[0047] The system of FIG. 3 comprises a detection module 10 which receives the data INFO.sub.CAM generated by a first sensor, in this case the video camera CAM, and generates, for each detected object OBJ.sub.i, information on the localization L.sub.i of the object concerned. The localization information L.sub.i is, for example, stored in a table TAB stored in the memory of the system S, as shown schematically in FIG. 4.

[0048] In the example described here, the data INFO.sub.CAM represent images successively taken by the video camera CAM; the objects OBJ.sub.i are detected and located with respect to the motor vehicle V by the analysis of these images, as described, for example, in patent application WO 2004/059 900, mentioned in the introduction above.

[0049] In the context of FIG. 2, the detection module 10 detects, for example, the third-party vehicle V′ as an object OBJ.sub.1, and determines its localization (defined by the localization information L.sub.1) with respect to the vehicle V by analysis of the images supplied by the video camera CAM.

[0050] The system of FIG. 3 also comprises a classification module 12 which receives at its input the data INFO.sub.CAM generated by the first sensor (in this case the video camera CAM) and a designation of the detected objects OBJ.sub.i (including, for example, their position in the image received from the video camera CAM).

[0051] The classification module 12 is designed to identify the type T.sub.i of each object OBJ.sub.i on the basis of the data INFO.sub.CAM received from the first sensor, for example, in the case described here where the data INFO.sub.CAM represent an image, by means of a shape recognition algorithm.

[0052] In a variant, the first sensor could be the lidar sensor LID, in which case the identification of the type of an object OBJ.sub.i could be carried out, for example, on the basis of the signature of the signal received by the lidar sensor LID by reflection from the object OBJ.sub.i.

[0053] The identification of type T.sub.i of the objet OBJ.sub.i enables it to be classified among various object types (for example, vehicle, pedestrian, cyclist, house, road lighting or signaling equipment, etc.), so that it can be determined whether this object OBJ.sub.i is of a mobile or a fixed type. It should be noted that the classification according to the object type is performed regardless of whether the object concerned is actually fixed or mobile during the passage of the vehicle V.

[0054] For example, in the context de FIG. 2, the classification module 12 determines, by shape recognition, that the object OBJ.sub.1 (that is to say, the third-party vehicle V′ as explained above) is of the vehicle type.

[0055] The object type T.sub.i is stored, in relation to the object concerned OBJ.sub.i, in the aforesaid table TAB, as shown in FIG. 4. In a variant, the stored information could be limited to an indication of the mobile or fixed nature of the object concerned OBJ.sub.i, this indication being determined on the basis of the type T.sub.i, identified as mentioned above.

[0056] For clarity of description, the detection module 10 and the classification module 12 have been described as two separate modules. However, it would be feasible for the detection of an object OBJ.sub.i and the identification of its type T, (enabling it to be classified as a mobile or fixed object) to be performed during the same step, for example by means of an algorithm for shape recognition in the images delivered by the video camera CAM.

[0057] The system S comprises a filtering module 14 which receives the data INFO.sub.LID received from the second sensor, in this case the lidar sensor LID. The filtering module 14 also uses the localization L.sub.1 of each object OBJ, detected by the detection module 10 and the type Ti of each object determined by the classification module 12 (this information may be received from the module concerned or read from the stored table TAB).

[0058] In the example described here, the data INFO.sub.LID delivered by the lidar sensor represent, for example, a set of values of detection distance d(a) associated, respectively, with angles a over the whole angular range from 0° to 360°.

[0059] From among the data INFOLID, the filtering module 14 transmits only the data INFO.sub.FIX which correspond to areas for which no object has been detected, or for which an object OBJ.sub.i has been detected with a fixed object type T.sub.i, according to the information generated by the detection and classification modules 10, 12 as described above. In other words, the filtering module 14 does not transmit the data INFO.sub.FIX relating to areas for which an object OBJ.sub.i has been detected with a mobile object type T.sub.i.

[0060] In the context of FIG. 2, the object OBJ.sub.1 (a third-party vehicle V′) detected with a mobile object type T.sub.1 (vehicle) covers the angular range α.sub.1-α.sub.2, according to the localization information L.sub.1, so that in the absence of any other object identified with a mobile object type, the filtering module 14 transmits only the data INFO.sub.FIX associated with the angular ranges [0°, α.sub.1 [and] α.sub.2, 360° [(that is to say, the data representing the values of d(α) only for 0≦α<α.sub.1 and α.sub.2<α<360°).

[0061] The transmitted data INFO.sub.FIX are received, after filtering by the filtering module 14, by a localization module 16, which uses these data INFO.sub.FIX for the execution of an algorithm for simultaneous localization and mapping, such as that described in the paper “A real-time robust SLAM for large-scale outdoor environments” by J. Xie, F. Nashashibi, M. N. Parent and 0. Garcia-Favrot, in ITS World Congr. 2010.

[0062] The localization module 16 may be used, on the basis of the data INFO.sub.FIX obtained from the second sensor (the lidar sensor LID in the example described), after filtering in this case, and using a map C constructed by the localization module 16 in the preceding iterations, to determine the current position (or localization) LOC of the vehicle V on the map C, and also to enrich the map C, notably as a result of the presence among the data INFO.sub.FIX of data relating to areas not reached by the lidar sensor in the preceding iterations.

[0063] However, it should be noted that, owing to the rejection (by the filtering module 14) of the data INFO.sub.LID relating to areas where an object OBJ.sub.i has been detected with a mobile object type T.sub.i, only the data relating to objects permanently present are processed by the localization module 16, thus preventing the processing of data which are actually of no use (thereby speeding up the processing), while also allowing the construction of a map containing no object that might be moved subsequently: such a map is more robust and easy to re-use.

[0064] FIG. 5 shows schematically a second example of a localization and mapping system according to the invention. In this example, the localization and mapping system S uses the data DAT delivered by a single sensor, in this case the video camera CAM.

[0065] As in FIG. 3, FIG. 5 shows functional modules, each of which corresponds to a particular process carried out by the localization and processing system S, in this case as a result of the execution, by the microprocessor of the system S, of computer program instructions stored in a memory of the system S. In a variant, the processes carried out by one or more functional modules could be executed by a dedicated integrated circuit, for example an application specific integrated circuit (or ASIC).

[0066] The system of FIG. 5 comprises a detection module 20 which receives the data DAT generated by the sensor, in this case data representative of images taken by the video camera CAM, and generates, for each object OBJ.sub.i that is detected (by image analysis in this case), information on the localization L.sub.i of the object concerned. The localization information L.sub.i is, for example, stored in a table TAB stored in the memory of the system S, as shown schematically in FIG. 4.

[0067] The system S of FIG. 5 comprises a classification module 22 which receives at its input the data DAT generated by the sensor (in this case the video camera CAM) and a designation of the detected objects OBJ.sub.i (including, for example, their position in the image received from the video camera CAM).

[0068] The classification module 12 is designed to identify the type T.sub.i of each object OBJ.sub.i on the basis of the data DAT received from the sensor, for example, in the case described here where the data DAT represent an image, by means of a shape recognition algorithm.

[0069] As mentioned above with reference to FIG. 3, the identification of type T.sub.i of the objet OBJ.sub.i enables it to be classified among various object types, so that it can be determined whether this object OBJ.sub.i is of a mobile or a fixed type, regardless of whether or not the object is actually fixed or mobile during the passage of the vehicle V.

[0070] The object type T.sub.i is stored, in relation to the object concerned OBJ.sub.i, in the aforesaid table TAB, as shown in FIG. 4. In a variant, the stored information could be limited to an indication of the mobile or fixed nature of the object type T.sub.i recognized for the object concerned OBJ.sub.i.

[0071] As mentioned above in relation to FIG. 3, it would be feasible, in a variant, for the detection of an object OBJ.sub.i and the identification of its type T.sub.i (enabling it to be classified as a mobile or fixed object) to be performed during the same processing step (that is to say, by the same functional module).

[0072] For each object detected by the detection module 20, a localization module 26 receives the description of this object OBJ.sub.i, its localization L.sub.i and its identified type T.sub.i, and, on the basis of this information, executes a simultaneous mapping and localization algorithm, also using a map C constructed in preceding iterations of the algorithm.

[0073] The map C includes, for example, a set of reference points (or “landmarks”, as they are known in English), each corresponding to an object detected in a preceding iteration.

[0074] The localization module 26 is designed so that the processing carried out by it takes into account only the objects OBJ.sub.i for which the associated type T.sub.i does not correspond to a mobile object type. For example, before taking localization information L.sub.j of an object OBJ.sub.j into account in the simultaneous mapping and localization algorithm, the localization module 26 checks the type T.sub.j of the object OBJ.sub.j (in this case by consulting the table TAB stored in the memory of the system S), and will only actually use the localization information L.sub.j in the algorithm if the type T.sub.j is a fixed object type and not a mobile object type.

[0075] On the basis of the localization information L.sub.i for objects whose type T.sub.i corresponds to a fixed object type (but without taking into account the localization information L.sub.i for objects whose type T.sub.i corresponds to a mobile object type), the localization module 26 determines the current position (or localization) LOC of the vehicle V on the map C (typically by comparing each detected object OBJ.sub.i with the reference points included in the map C) and enriches the map C (typically by adding to the map C the detected objects OBJ.sub.i which do not correspond to any reference point, so that each of them forms a new reference point in the completed map C).

[0076] As mentioned above in relation to the first example of a localization and mapping system, the constructed map C is robust and easily re-usable, since it is constructed on the basis of objects that are not subject to movement.

[0077] FIG. 6 shows the main steps of a localization and mapping method according to the invention.

[0078] This method starts with a step E30 or receiving data generated by an on-board sensor in the vehicle V, in this case the video camera CAM or the lidar sensor LID.

[0079] The method continues with a step E32 of detecting objects present in the environment in which the vehicle V maneuvers, by analyzing the data received from the on-board sensor in step E30. This step is executed, in the examples described above, by the detection module 10, 20.

[0080] The method then comprises a step E34 of determining, for each object detected in E32, and on the basis of the data received from the on-board sensor in step E30, the type of the object concerned, for example by means of a shape recognition algorithm (if the data received from the on-board sensor represent an image) or by means of a signature recognition algorithm (if the data received from the on-board sensor represent a signal). This step is executed, in the examples described above, by the classification module 12, 22.

[0081] It should be noted that it would be possible, in a variant, to use the data obtained from a plurality of sensors to classify the objects according to their type, after a step of merging the data obtained from the different sensors if necessary.

[0082] If two sensors are for the classification of the objects and for the localization of the mobile machine respectively, as in the first example given above with reference to FIG. 3, the method comprises a step E36 of receiving data generated by the second sensor.

[0083] The method may then include, if necessary, a step E38 of filtering the data received in step E36, in order to reject the data relating to the objects whose type determined in step E34 is a mobile object type, or relating to areas of the environment where an object has been detected with a type (determined in step E34) corresponding to a mobile object type. The filtering module 14 used in the first example described above with reference to FIG. 3 executes a step of this kind.

[0084] In a variant, as in the case of the second example described with reference to FIG. 5, the method does not specifically include a filtering step; in this case, the step of localization described below is designed to operate without taking into account the data relating to the objects whose type determined in step E34 is a mobile object type, or relating to areas of the environment where an object has been detected with a type (determined in step E34) corresponding to a mobile object type.

[0085] The method continues with a step E40 of localizing the mobile machine (in this case the motor vehicle V) on the basis of detection data, which may be the data received in step E30 and/or the data received in step E36 (if such a step is executed), and on the basis of a map constructed in previous iterations of the method.

[0086] This step E40 comprises the execution of a simultaneous localization and mapping algorithm, making it possible not only to localize the machine but also to enrich the map.

[0087] It should be noted that, as mentioned above, the data used by the localization and mapping algorithm may be obtained from a plurality of on-board sensors, after a step of merging the data obtained from the different sensors if necessary.

[0088] In this case, provision is usually made for the method to loop back to step E30 for the execution, at a later instant, of a new iteration of steps E30 to E40.

[0089] The map constructed during the process described above is kept permanently so that it may be re-used subsequently, for example during the passage of the mobile machine (in this case the motor vehicle V) in the environment at a later time (for example on another day, after the day on which the map was constructed).

[0090] For this purpose, the localization and mapping system S incorporates, for example, a mechanism for comparing the map being constructed with the maps previously constructed (and stored) with a view to their re-use. Thus, if the mobile machine again travels in the same environment at said later time, the comparison mechanism may be used to recognize the neighboring environment as that represented in the previously constructed map, and to use the previously constructed map (by loading this map into memory and using it in the localization and mapping algorithm).

[0091] The comparison mechanism operates particular well if the stored card has been constructed by the method described above, since a map of this kind contains only the information relating to objects which remain fixed, and contains no information relating to objects which will no longer be present during the later passage. Thus the invention provides mapping which may be used in the long term for the localization of the mobile machine.