METHOD AND SYSTEM TO SHARE SCENE MAPS

Abstract

A method carried out in a system including at least a first imaging unit mounted on a first entity or first vehicle, at least a second imaging unit mounted on a second entity, the method including: —building, from the first imaging unit, a first map, formed as a floating map, through a simultaneous localization and mapping process; —building, from the second imaging unit, a second map; —establishing a data channel between first and second entities; —determining if there is at least an overlapping portion between first and second maps; —receiving, at the first entity, part or all the elements of the second map, from the second entity; —identifying matching candidate solutions to register the second map into the first map; and —registering and appending the second map to the first map of first vehicle.

Claims

1. A method carried out in a system comprising at least a first imaging unit mounted on a first entity, the first entity being formed as a first vehicle, at least a second imaging unit mounted on a second entity independent from the first vehicle, the method comprising: a1—building, from the first imaging unit, a first map, formed as a floating map, independently from any absolute geolocation, through a simultaneous localization and mapping process, a2—building, from the second imaging unit, a second map, b2—establishing a data channel between first and second entities, c—determining if there is at least an overlapping portion between first and second maps, and whenever at least an overlapping portion is determined or likely, d1—receiving, at the first entity, part or all the elements of the second map, from the second entity, e1—identifying matching candidate solutions to register the second map into the first map, f1—registering and appending the second map to the first map of first vehicle.

2. The method according to claim 1, wherein the second entity is a second vehicle, moving independently from the first vehicle, wherein the second map is formed as a floating map build through a simultaneous localization and mapping process, the method comprising: d2—receiving, at the second entity, part or all the elements of the first map, from the first vehicle.

3. The method according to claim 2, wherein the method further comprises: e2—identifying matching candidate solutions to register the first map into the second map, f2—registering and appending the first map to the second map of the second vehicle.

4. The method according to claim 1, further comprising, before step b2-, b1—determining a short-list of entities or vehicles located in the vicinity of the first vehicle, through a basic geolocation function.

5. The method according to claim 1, wherein the data channel is a direct communication link.

6. The method according to claim 1, wherein the data channel is an indirect communication link via a third entity.

7. The method according to claim 1, wherein the first imaging unit and/or the second imaging unit is a Lidar scanner.

8. The method according to claim 1, wherein the first and second entities has no common clock and respectively build first and second maps asynchronously with regard to one another.

9. The method according to claim 2, wherein the first vehicle is travelling on a first lane which is not parallel to a second lane where the second vehicle is travelling.

10. The method according to claim 2, the first vehicle is travelling on a first lane which is parallel and in opposite direction with regard to a second lane where the second vehicle is travelling.

11. The method according to claim 1, wherein one or more geo-located landmarks are used to determine at least an overlapping portion between first and second maps.

12. The method according to claim 1, wherein steps d— to f— are repeated until the first and second vehicles are away from one another, by at least a predetermined distance.

13. The method according to claim 1, wherein steps d— to f— are repeated until the first and second vehicles are located on different floors in a multi-floor building.

14. The method according to claim 1, wherein there is provided at least a third imaging unit mounted on a third vehicle or mounted on a road/street equipment, and the method comprises a registration of the images outputted by the third imaging unit into the shared map between first and second maps.

15. A system comprising at least a first imaging unit mounted on a first entity, the first entity being formed as a first vehicle, at least a second imaging unit mounted on a second entity independent from the first vehicle, the method being configured to carry out the method according to claim 1.

16. A vehicle comprising a system according to claim 15.

17. The method according to claim 2, further comprising, before step b2—, b1—determine a short-list of entities or vehicles located in the vicinity of the first vehicle, through a basic geolocation function.

18. The method according to claim 3, further comprising, before step b2—, b1— determining a short-list of entities or vehicles located in the vicinity of the first vehicle, through a basic geolocation function.

19. The method according to claim 2, wherein the data channel is a direct communication link.

20. The method according to claim 3, wherein the data channel is a direct communication link.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features and advantages of the invention appear from the following detailed description of two of its embodiments, given by way of non-limiting example, and with reference to the accompanying drawings, in which:

[0049] FIG. 1 illustrates a diagrammatical top view of one or more vehicle(s) circulating on a road,

[0050] FIG. 2 illustrates another diagrammatical top view of one or more vehicle(s) circulating on a city street with a crossroad/junction,

[0051] FIG. 3 shows a diagrammatical block diagram of the system promoted in the present disclosure, involving two vehicles,

[0052] FIG. 4 illustrates a diagrammatical elevation view of a vehicle of interest circulating on a road,

[0053] FIG. 5 is a chart illustrating the rolling maps sharing process, involving two vehicles.

[0054] FIG. 6 is a chart illustrating the management of a list of current communication links,

[0055] FIG. 7 is a chart illustrating an extended map built from a sharing process among a plurality of vehicles.

DETAILED DESCRIPTION OF THE DISCLOSURE

[0056] In the figures, the same references denote identical or similar elements. For sake of clarity, various elements may not be represented at scale.

[0057] General Context

[0058] FIG. 1 shows diagrammatically a top view of a road where several vehicles are moving. The first vehicle denoted Vh1 is of particular interest, since it is equipped with at least two imaging units. The second vehicle denoted Vh2 moves in the same direction as per Vh1, slightly ahead of Vh1.

[0059] In a broader scope of the present disclosure, the first vehicle Vh1 is called a first entity denoted ET1, and the second vehicle Vh2 is called a second entity denoted ET2. However, the second entity ET2 can be a completely different type of device like a public video surveillance camera system, or a traffic monitoring camera system.

[0060] Besides, although road vehicles are represented in the figures, the term “vehicle” herein encompasses not only road vehicle but also drones, aircrafts, boats, mobile robots, . . . .

[0061] A third vehicle denoted Vh3 moves in the same direction as per Vh1, behind Vh1. Additionally, there may be among other things: road/traffic signs on the side of the road or above the road, trees, bushes, etc. . . .

[0062] There may be also buildings in dense city areas, those buildings can be close to the street border. There may be also street furniture like a bus stop shelter, a food truck, . . . .

[0063] Besides fixed entities, there may be also possibly moving entities likes animals, people 95, trash bins, objects blown by the wind, etc.

[0064] Besides, it is to be considered any kind of users of the road like bicycles C1, C2, scooters, motorcycles, trucks, buses, vehicles with trailers, not to mention also pedestrians 95,96. Some of them are moving while others can be stationary either at the side of the road or on traffic lane.

[0065] The vehicles of interest Vh1, Vh2, Vh3 travel in an environment also named the ‘scene’.

[0066] Some objects or entities present in the scene may serve as landmarks for the mapping process and sharing process to be detailed later on.

[0067] FIG. 2 shows other vehicles of interest Vh4, Vh5 which are travelling on a road which is perpendicular to the road where Vh1 travels.

[0068] Also, there may be provided also one or more vehicles of interest which travel in a direction opposite to the direction travelled by vehicle Vh1.

[0069] As illustrated in FIG. 2, the street environment may comprise fixed object like traffic lights denoted 84, 85.

[0070] There is provided a first imaging unit 31 mounted on the first entity ET1. The first imaging unit 31 exhibits a first filed-of-view FOV1. The first imaging unit 31 is coupled to a first control unit 41 arranged aboard the first entity ET1. The first control unit 41, via the data supplied by the first imaging unit 31 is configured to build a map denoted herein ‘first map’ 61. The data about the first map 61 is stored in a memory 71 included or associated with the first control unit 41.

[0071] In the case the first entity ET1 is a vehicle (i.e. Vh1), the first map is a floating map. The floating map is built through a simultaneous localization and mapping process (‘SLAM’ in short).

[0072] The imaging unit 31 can be an imager such as a Lidar system or to a 3D camera/video system.

[0073] The imaging unit 31 is used to compute a relative (or floating) localization of the environment, along with the travel of the mobile entity. These systems and methods are known in the art as Simultaneous Localization And Mapping (‘SLAM’ in short).

[0074] Basically, the imaging unit is configured to generate at least a plurality of successive point cloud frames F(j), and construct therefrom, via the above-mentioned simultaneous localization and mapping process (e.g. SLAM process), a floating map FM of a scene travelled by the mobile entity.

[0075] A tridimensional scanner (or imager) acquires sets of data points, called point clouds F(j), that are representative of the objects located in a local volume of the environment surrounding said scanner/imager, also called a ‘scene’. One example of a commonly used imaging unit is a laser rangefinder such as a light detection and ranging (LIDAR) module which periodically scans its environment using a rotating laser beam. The term “lidar-type scanner” may be construed as a scanner using bursts of electromagnetic waves and echoes on objects therefrom, said electromagnetic waves being generally in the near infra-red domain, for example having a wavelength comprised between 600 nanometer and 2000 nanometer, more preferably in the range 1400-1600 nm. Also alternatively, some special Lidars are able to acquire their environment from a common simultaneous illumination, they are known as “flash lidars”.

[0076] With reference to FIG. 4, The imager unit 31 computes a range, corresponding to a distance Dis from the imager 31 to a point M of reflection of the initial signal on a surface of an object located in the scene. Said range is computed by comparing the timings features of respective transmitted signal and reflected signal, for instance by comparing the time or the phases of emission and reception.

[0077] The imager unit 31 exhibits an available field of view denoted FOV1.

[0078] In one example, the imager unit 1 comprises a laser emitting light pulses with a constant time rate, said light pulses being deflected by a two moving mirrors rotating θ, φ along two respective directions.

[0079] The scanning processes performed in real-time, i.e., controllable mirrors are rotated in the space (θ, φ) simultaneously with the firing of burst of electromagnetic waves (Tx) θ, φ, along the firing line 57, to scan the field a view, FOV1=from θmin, φmin to θmax, φmax. Firing period is denoted Tb. Tb is as small as a few nanoseconds, or even less.

[0080] As soon as all the field of view FOV1 has been scanned, the first scanner unit issues a point cloud frame which can be represented by a matrix/tensor Mx(t.sub.z), namely an array of points with (θ, φ, Dis). t.sub.z is considered as a sampling time, which can be stored as the timestamp for the frame F(j). Scanning or sweeping all the field of view takes a short time, let's say less than 100 ms, preferably less than 50 ms, possibly even less.

[0081] As stated above, the imager unit acquires (collects) point cloud frames F(j) of the scene, each point cloud frame comprising an array of points, each point having as attribute angles and distance (θ, φ, Dis) with regard to the imager unit point of view SLP. In addition, the imager unit transmits each point cloud frame F(j) to the computing unit 6 as soon as they are available, such that the second point cloud frame can be registered into the floating 3D map 61.

[0082] Each point cloud frame F(j) has a pose which is here the position of the focal point of the imaging unit, or a base reference optical/sensing point of the imaging unit. A floating trajectory is determined from the successive poses of the of the imaging unit.

[0083] Registration process involves a geometrical transformation function TR cause a point cloud frame of interest to match into the floating map of the scene, i.e. find the best possible match into the floating 3D map of the scene.

[0084] The process is carried out without geolocation information such GPS. Said otherwise, the promoted registration process does not require absolute coordinates or GPS data.

[0085] In practice, there may be a substantial overlap between a newly received frame and the floating 3D map, and this is enough to allow reliable registration and then incrementing the content of the floating 3D map.

[0086] The registration process causes the point cloud frame of interest (the latest received) to find the best possible match into the floating 3D map 61 of the scene, which implies mathematical transformation(s) to shift, orientate, spread-in spread-out the array of points of the point cloud frame of interest.

[0087] Find the best possible match into the floating 3D map can be done by scanning transformation noted TRi, and searching from the best match with an interactive closest points process [TRi]×[F(j)] to be compared to portions of [RMAP(tk)] (full floating 3D map).

[0088] Once the best match TRi=TR.sub.best is found, the relevant data [TR.sub.best]×[F (i)] imported into the floating 3D map 61, which is summarised by the symbolic formula:

[RMAP(tk)]<=[TR]×[F(j)]. TR is a tensor-to-tensor transform or a tensor-to-matrix transform.

[0089] One example of general registration technique can be found in document EP3078935.

[0090] Any type of ICP registration technique or the like can also be used.

[0091] Advantageously, there is no need to know coordinates and/or absolute geolocation for the promoted process.

[0092] Each time a new frame F(j) is imported into the general floating map, the corresponding position of the reference point, or pose SLP, is stored in memory.

[0093] Therefore the floating map is associated with a series of poses.

[0094] The registration and the determination and storage of poses can be performed either locally at the imaging unit 31 or can be performed at the computing unit 6.

[0095] Also, additionally or alternatively, one can use one or more video camera(s), either with a plurality of 2D camera and/or one or more TOF 3D camera (TOF means “time of flight”).

[0096] The successively acquired point clouds F(j) can be used to generate 3D maps of the environment seen by the mobile entities during a travel for mapping purposes, for example for identifying fixed objects like a tree, or a road signal, or moving objects like pedestrian, animals.

[0097] Optionally, each point cloud F(j) can be timestamped, the timestamp corresponding to the moment in time when the point cloud is acquired.

[0098] There is provided a second imaging unit 32 mounted on the second entity ET2. The second imaging unit 32 exhibits a second filed-of-view FOV2. The second imaging unit 32 is coupled to a second control unit 42 arranged aboard the second entity ET2. The second control unit 42, via the data supplied by the second imaging unit 32 is configured to build a map denoted herein ‘second map’ 62. The data about the second map 62 is stored in a memory 72 included or associated with the second control unit 42.

[0099] In the case the second entity ET2 is a stationary equipment, the second map is a map updated along the time. It takes into account evolution of the scene, for example roadworks, evolution of the building construction, growing vegetation, parked vehicle.

[0100] In the case the second entity ET2 is a vehicle (i.e. Vh2), the second map is a floating map. The floating map is built through a simultaneous localization and mapping process in a way similar or identical to what is been explained above for the first entity.

[0101] Up to this point of the specification, first and second entities ET1, ET2 build, independently from one another, an environmental map, i.e. a floating map whenever the entity is a moving entity (i.e. vehicle).

[0102] Advantageously, said maps built on their own will be supplemented by a sharing process as it will be set forth below. Steps denoted a1 and a2 are map building processes, respectively for first entity ET1 and second entity ET2 as set forth above.

[0103] The proposed method can comprise:

[0104] Step b1 consists in determining a short-list of entities or vehicles located in the vicinity of the first vehicle Vh1, through a basic geolocation function. GPS is one available solution to identify entities or vehicles located in the vicinity of the first vehicle Vh1. However, other like systems are also considered, like cellular phone triangulation method versus ground antennas. The so-called basic geolocation function needs not to be precise. An uncertainty of several meters or dozens of meters is accepted.

[0105] We understand by “vicinity” of first vehicle, and area with a radius, between 500 m to 1 km from the first vehicle. Any entity situated in this vicinity and that carries communication feature and imaging feature is of interest from the first vehicle standpoint.

[0106] The radius of the vicinity can be made adaptive to the traffic condition for example it can be made smaller in urban areas, and larger on motorways.

[0107] There may be provided a filter to eliminate vehicles which are not located on the same floor as per vehicles in a multi-story park building. Altitude delivered by GPS can be taken as determining criterion to disregard other vehicle(s)/entities that are located in the geo vicinity (latitude & longitude) but at different altitudes.

[0108] The proposed method can comprise a step of having the communication between first and second entities. There may be provided the following: Step b2— consists in establishing a data channel between first and second entities ET1,ET2. Step b2— is involved for nearby entities or vehicles determined as an output of step b1-.

[0109] According to a first possibility, a direct data channel 15 is established between first and second entities ET1,ET2.

[0110] According to a second possibility, the data channel is established between first and second entities ET1,ET2 with an indirect data channel 16. This case arises when both vehicles Vh1,Vh2 are each connected to a remote server, same or distinct, and in such case the data transit by one or more remote servers when going from Vh1 to Vh2 or conversely.

[0111] Stated otherwise, the data channel can be an indirect communication link via a third entity. It is not excluded to the third entity be itself a vehicle.

[0112] Four direct and indirect data channels, data flow capacity must be sufficient to transmit relevant data in a short time, but it does imply necessarily high capacity data link.

[0113] Communication between entities and especially vehicles can be based on Vehicule-to-Vehicule communication, likewise called ‘V2V’.

[0114] Communication is maintained as long as it is useful for sharing data between two approximate entities, also shown graphically at FIG. 6.

[0115] Communication is ceased under some circumstances, for example when first and second vehicles are away from one another, by at least a predetermined distance (let's say more than 1 km), or when first and second vehicles are moving away from one another, with at least a predetermined relative velocity.

[0116] One special case arises when the first and second vehicles are located on different floors in a multi-floor building, like a multi-story park lot. As already stated, altitude discrimination can also lead to ceasing communication from one vehicle to another.

[0117] The proposed method can comprise a step of overlapping determination, i.e. determining if the first map 61 and the second map 62 have in common part of the scene. This is denoted step c— of the promoted method.

[0118] Since first and second entities ET1,ET2 have now established communication, communication between the two entities allowed one of them (or both) to determine whether there might be an overlapping portion in their respective maps 61,62.

[0119] As illustrated at FIG. 7, an overlapping portion Ov12 between first and second map A,B, is found. Conversely, if there is no intersection between first and second maps, therefore no registration will be possible. We note here that the overlapping determination step is repeated as a looped process as long as the two entities ET1, ET2 remain in communication.

[0120] This is apparent from FIG. 5 where step 104 is looped. Indeed, when talking about vehicles, along the time, the first vehicle Vh1 can get closer to the second entity (or second vehicle), or conversely can go away from it.

[0121] The determination of the presence of overlapping portion may not be a binary decision, this may be the likelihood index, computed from respective geolocation of the two entities. We note here that we don't need precise geolocation, only a rough estimate is enough, the proposed solution can work with the jitter of GPS systems.

[0122] Whenever an overlapping portion Ov12 is likely, the proposed method can comprise a step of receiving, at the first entity ET1, part or all the elements of the second map 62, from the second entity ET2. This is denoted step d1— of the promoted method.

[0123] In other words, the second entity transmit to the first entity at least a portion of its own updated floating map 62, notably concerning meaningful points, like those relative to moving object in the scene. In a variant, all the latest acquired cloud is transmitted from the second entity to the first entity.

[0124] The proposed method can comprise a step of identifying one or more matching candidate solution(s) to register the second map 62 into the first map 61. We note here that the first map or the second map can already constitute a merge of floating maps from more than one vehicle.

[0125] Registration of the second map into the first map is a process similar to what have been described earlier for registering point cloud frame into rolling map.

[0126] The registration process causes the point cloud set of the second map to find the best possible match into the first map 61 of the scene, which implies mathematical transformation(s) to shift, orientate, spread-in spread-out the array of points of point cloud set of interest.

[0127] Find the best possible match of part or all the second map 62 into the first map 61 can be done by scanning a transformation noted TRF, and searching from the best match with an interactive closest points process [TRF]×[F(62)] to be compared to portions the first map 61.

[0128] Once the best match TRF=TRFbest is found, the relevant data [TR.sub.best]×[F(62)] imported into the first map 61, in a similar way as above-mentioned for registration.

[0129] We note here that the use of landmarks or special features is not necessary in the above process.

[0130] Of course it is not excluded to use landmarks if available to facilitate the registration of the second map into the first map. In particular any road equipment or street ancillaries can be used as landmarks. Also it is not excluded to use another vehicle can be used as a landmark.

[0131] Further, when talking about vehicles sharing maps, the registration, even without landmarks, can work irrespective of the respective travelling direction of the vehicle as illustrated at FIGS. 1 and 2. The second vehicle can travel perpendicularly to the first one; second vehicle can travel opposite the first one, there is no constraint in the main direction of of the fields of view of the respective imager units. Therefore, map sharing can occur between vehicles travelling in different directions and having complementary fields-of-view. This is a key advantage to suppress pseudo blind zones and suppressing mask effect for some vehicles.

[0132] With reference to FIG. 5, ref 101 designates the functional block “Identify vehicle(s) or entities in the vicinity/surroundings”, ref 102 designates the functional block “Establish data channel/wireless link if functionally relevant”.

[0133] Ref 103 designates the functional block “Share (send and/or receive) floating 3D maps between Veh1 and Veh2/ET2”.

[0134] Ref 104 designates the functional block “Check if match/registration is possible (overlapping portion)” which gives as a result the above-mentioned overlapping likelihood index, and the registration process is carried out when a likelihood index is above a certain threshold.

[0135] Ref 105 designates the functional block “Append 3D map of Veh1 into 3D map of Veh2 And/or vice versa” which give an extended map for first vehicle Vh1 or first entity ET1.

[0136] Ref 106 designates the mobile entity control block, in which any control system benefits from the accurate geolocations. We note here that this step 106 is an option in the sense of the present disclosure.

[0137] Ref 107 denotes the general iteration of the process.

[0138] With reference to FIG. 6, the process of establishing communication between vehicles, updating a shortlist of connected vehicles, and terminating, negation between two vehicles is summarized.

[0139] Ref 201 designates the functional block “Identify vehicle(s) or entities in the vicinity/surroundings”, ref 202 designates the functional block “Establish data channel/wireless link if functionally relevant”.

[0140] Ref 203 designates the functional block “Include a new V2V communication”, ref 205 designates the functional block “Discard a previously active V2V communication” ref 204 designates the functional block “Maintain a list of connected′ vehicles”.

[0141] As illustrated at FIG. 7, in a network logic, it is not required that all the entities share a common overlap of the rolling map, but a continuity between all the rolling maps (ie. Vehicle

[0142] A and B have an overlap, A doesn't overlap with C, but B overlaps with C, so we can build the A+B+C rolling map). Looking at FIG. 6, a plurality of proximity pairs is established: A+B, B+C, C+D, D+E, D+F, D+G. We form therefrom a kind of a daisychain and obtain therefrom the global map covering A+B+C+D+E+F+G.

[0143] There may be provided a third imaging unit 37 mounted on the third entity ET3. The third imaging unit 37 exhibits a third filed-of-view FOV3. The third imaging unit 33 is coupled to a third control unit 43 arranged aboard the third entity ET3. The third control unit 43, via the data supplied by the third imaging unit 37 is configured to build a map denoted herein ‘first map’ 63.

[0144] We note here that the first imaging unit 31 and/or the second imaging unit 32 can be a Lidar scanner. Also the third imaging unit 37 mounted can be a Lidar scanner.

[0145] The field of view of the imaging unit(s) can have forward depth of at least 100 m, preferably at least 150 m. Since the floating maps are cumulative maps they extend over a length which is greater than the depth of the imaging units; for example for moving vehicle, the length of the floating map can be as long as 500 m, or even up to 1 Km.

[0146] In the case the third entity ET3 is a vehicle (i.e. Vh3), the third map is a floating map. As per first and second entities, the floating map is built through a simultaneous localization and mapping process.

[0147] There may be a limit to the depth of the first and second rolling maps, due to memory and processing constraints. Each rolling map can contain several thousands points for example. A way to keep the more interesting points can be a proximity criterion rather than a recentness criterion, i.e. we keep points that are located at a distance below a certain threshold. However, points belonging to an assumed moving object can also be retained in the rolling map even though there are at a distance above the threshold.

[0148] Typical frame rate for imagers 31,32 is comprised between 10 Hz and 30 Hz, it can be around 20 Hz. Angular resolution for the imager unit like a Lidar can typically be comprised between 0.05° and 0.5°, although other resolution is not excluded.

[0149] There may be provided additional sensors 33,34 of any type to supplement data with regard to already mentioned imagers 31,32.

[0150] first and second entities (ET1,ET2) has no common clock, i.e. first and second entities (ET1,ET2) has no common time reference. Each of them has a general clock which needs not to be precise. An uncertainty of one second does not affect proper operation of the proposed system.