METHOD FOR PREDICTING AT LEAST ONE FUTURE VELOCITY VECTOR AND/OR A FUTURE POSE OF A PEDESTRIAN

Abstract

A method for predicting at least one future velocity vector and/or a future pose of a pedestrian in an area of prediction. A map of a surrounding environment of the pedestrian and current velocity vectors of other pedestrians in the area of prediction are taken into account in the prediction.

Claims

1-13. (canceled)

14. A method for predicting at least one future velocity vector and/or a future pose of a pedestrian of a plurality of pedestrians in a prediction area, the method comprising: predicting the at least one future velocity vector and/or the future pose of the pedestrian; and taking into account, in the prediction, a map of the prediction area and current velocity vectors of further pedestrians of the plurality of pedestrians in the prediction area.

15. The method as recited in claim 14, wherein for the prediction, the pedestrians in the prediction area are combined into groups, a current pose and a current velocity vector of the each group being ascertained from current poses and current velocity vectors of all the members of the group.

16. The method as recited in claim 14, wherein in each case, social interactions between the pedestrians are taken into account in the prediction.

17. The method as recited in claim 14, wherein a respective destination is assigned to each of the pedestrians, and it is taken into account in the prediction that each of the pedestrians is moving towards the respective destination.

18. The method as recited in claim 14, wherein the prediction is carried out in a plurality of temporal substeps, all values taken into account in the prediction of the velocity vectors being recalculated for each substep.

19. The method as recited in claim 18, wherein in each of the substeps, the future pose of the pedestrian is ascertained, and from all ascertained future poses, a movement map of the pedestrian is created.

20. The method as recited in claim 14, wherein current poses and current velocity vectors of the pedestrians are ascertained using at least one sensor, the at least one sensor including at least one of the following: a monocular sensor, and/or a stereo sensor, and/or a depth sensor.

21. The method as recited in claim 14, further comprising: using a result of the prediction to control a self-driving vehicle or a robot in such a way that a collision with the pedestrian is avoided.

22. A non-transitory machine-readable storage medium on which is stored a computer program for predicting at least one future velocity vector and/or a future pose of a pedestrian of a plurality of pedestrians in a prediction area, the computer program, when executed by a computer, causing the computer to perform: predicting the at least one future velocity vector and/or the future pose of the pedestrian; and taking into account, in the prediction, a map of the prediction area and current velocity vectors of further pedestrians of the plurality of pedestrians in the prediction area

23. The non-transitory machine-readable storage medium as recited in claim 22, wherein the computer program includes a prediction module having a movement plan, tracklets, and social context information, which are used for the prediction.

24. An electronic control device configured to predict at least one future velocity vector and/or a future pose of a pedestrian of a plurality of pedestrians in a prediction area, the electronic control device configured to: predict the at least one future velocity vector and/or the future pose of the pedestrian; and take into account, in the prediction, a map of the prediction area and current velocity vectors of further pedestrians of the plurality of pedestrians in the prediction area.

25. The electronic control device as recited in claim 24, wherein the electronic control device is further configured to control a self-driving vehicle or a robot, in such a way that a collision with the pedestrian is avoided.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Exemplary embodiments of the present invention are shown in the figures and are explained in more detail below.

[0018] FIG. 1 schematically shows a system architecture of a self-driving vehicle that, using an exemplary embodiment of the method according to the present invention, carries out a prediction of at least one future velocity vector and of a future pose of a pedestrian.

[0019] FIG. 2 schematically shows the sequence of a prediction of a future velocity vector and of a future pose of a pedestrian in an exemplary embodiment of the method according to the present invention.

[0020] FIG. 3 schematically shows social interactions between a plurality of pedestrians that, in an exemplary embodiment of the method according to the present invention, are used for the prediction of at least one future velocity vector and of a future pose of a pedestrian.

[0021] FIG. 4 schematically shows the movement of a group of pedestrians relative to a self-driving vehicle, in a method according to an exemplary embodiment of the present invention.

[0022] FIG. 5 shows, in a diagram, the negative log probability as a function of the prediction duration in simulations according to comparative examples and an exemplary embodiment of the present invention.

[0023] FIG. 6 shows, in a diagram, the modified Hausdorff distance as a function of the prediction duration in simulations according to comparative examples and an exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0024] In an exemplary embodiment of the method for predicting at least one future velocity vector and a future pose of a pedestrian in order to avoid a collision of the pedestrian with a self-driving vehicle, sensors of the vehicle create a map 11 of an area through which the vehicle will move in the future. Map 11 contains all geographical features of the area. A prediction is to be made as to whether, within this area, pedestrians could move into the path of movement of the vehicle, so that there is a risk of collision. Therefore, in the following this area is referred to as the area of prediction. Using the sensors, images of the area of prediction are created, and a recognition 12 is carried out of persons in the images. In a prediction module 20, from map 11 a movement plan 21 is now created in which it can be recognized in which segments of the area of prediction, which is not blocked by obstacles, pedestrians can move. The data acquired concerning the individual pedestrians is divided into tracklets 22 and social context information 23. Tracklets 22 acquire the current velocity vectors and the current poses of the pedestrians. Here, each velocity vector contains information about the direction of movement of a pedestrian and about his or her movement speed. The pose indicates his or her orientation. The social context information 23 is obtained through image analysis, and enables a conclusion as to which individual pedestrians are part of a group having a common destination. Movement plan 21, tracklets 22, and social context information 23 are made available for the prediction 30. The result of the prediction is on the one hand provided to a control unit 40 of the self-driving vehicle in order to avoid a collision with pedestrians. On the other hand, it is also used the next time recognition 12 is carried out of pedestrians in the recorded images, in order to facilitate this recognition 12.

[0025] FIG. 2 shows in detail how prediction 30 runs. In a step 31, movement plan 21, tracklets 22, and social context information 23 are acquired, individual pedestrians being combined into groups on the basis of the social context information 23. In the next step 32, the destinations of the pedestrians are extrapolated on the basis of their current velocity vectors. For this purpose, a plurality of possible movement routes and destinations of the pedestrians are taken into account. In the next step 33, these are corrected, using movement plan 21, in such a way that only destinations remain that are plausible on the basis of map 11. The collected data about current velocity vectors, current poses, and destinations of each pedestrian are provided to a prediction function 50 in the next step 34. This has four substeps. In a first substep 51, it is ascertained which path each pedestrian can be expected to select in the area of prediction in a specified time interval, based on his or her destination, current velocity vector, and current pose. In a second substep 52, this path is corrected by taking into account social interactions of each pedestrian with adjacent pedestrians. These social interactions represent probable paths of avoidance of collision with other pedestrians, based on typical movement patterns of pedestrians. In a third substep 53, a new correction is carried out based on social interactions between group members. The social interactions influence the movement in such a way that groups are not broken up. In a fourth substep 54, based on the results of the first three substeps 51, 52, 53, the current poses and current velocity vectors are replaced with predicted future poses and future velocity vectors at the end of the time interval. The future poses ascertained in this way are entered in a movement map 35, and there they are linked with the time interval on which the prediction is based. Substeps 51 through 54 are then carried out again, the future velocity vectors and future poses of the pedestrians ascertained in step 54 now being used instead of the current velocity vectors and current poses provided from step 34. However, the destinations of the pedestrians from step 34 are retained. Each time substeps 51 through 54 are run through, another entry is created in movement map 35 for a new time interval. Movement map 35 is finally provided to control unit 40, so that this control unit can read out, for each time interval, the most probable locations of all persons in the area of prediction.

[0026] The sequence of substeps 51 to 53 is further illustrated in FIG. 3. There, the movement of six persons 61 through 66 is shown. The first three persons 61 through 63 form a group. This group is grouped around a midpoint 70. In area of prediction 80, the members of the group are currently moving vertically upward in FIG. 3. The current velocity vectors v.sub.64-66.sup.cur of the further pedestrians 64 through 66 are shown by arrows in FIG. 3. Inside the group, attractive social interactions F.sub.61-63.sup.att act on the group members, through which these numbers move towards midpoint 70. These attractive social interactions F.sub.61-63.sup.att can be calculated for each pedestrian i according to Equation 1:

F.sub.i.sup.att=α.Math.U.sub.i.Math.q  (Equation 1)

[0027] Here, α designates the strength of a group attraction effect, and U.sub.i is a unit vector that points from pedestrian i to midpoint 70. Value q is a threshold value that indicates whether the attractive social interaction is acting or not. If the distance between pedestrian i and midpoint 70 is below the threshold value, then F.sub.i.sup.att=0. The group attraction effect is thus acting only if pedestrian i is moving away from midpoint 70 by more than the threshold value.

[0028] Further social interactions F.sub.61-62.sup.vis have the result that the first two pedestrians 61, 62 of the group reduce their speed so that third pedestrian 63 will not lose the connection to the group. These further social interactions F.sub.61-62.sup.vis can be calculated for each pedestrian i according to Equation 2:

F.sub.i.sup.vis=β.Math.v.sub.i.sup.cur.Math.γ.sub.i  (Equation 2)

[0029] Here, β designates the strength of an interaction within the group. The angle between the current velocity vector v.sub.i.sup.cur of pedestrian i and his or her direction of view is designated γ.sub.i.

[0030] A further movement of all pedestrians 61 through 66 along their current velocity vectors is not possible without a collision occurring. Therefore, repelling social interactions F.sub.61-66.sup.soc act, which move each of the pedestrians 61 through 66 so as to avoid the other pedestrians. These repelling social interactions F.sub.i,j.sup.soc can be calculated, for each pedestrian i relative to another pedestrian j, according to Equation 3:

[00001]$\begin{array}{cc}{F}_{i,j}^{\mathrm{soc}}=a_j.Math.e^{\frac{r_{i,j}-d_{i,j}}{b_j}}.Math.n_{i,j}.Math.\left(\lambda +\left(1-\lambda \right).Math.\frac{1+\cos {\phi }_{i,j}}{2}\right)& \left(\mathrm{Equation}3\right)\end{array}$

[0031] Here a.sub.j≥0 designates the strength and b.sub.j>0 designates the direction of the repellent social interaction. The distance between the two pedestrians i, j is designated d.sub.i,j, and r.sub.i,j designates the sum of their radii. An anisotropy factor λϵ[0, 1] scales the repelling social interaction in the direction of the movement of pedestrian i. The interaction reaches its maximum value when the angle φ.sub.i,j between a normalized vector n.sub.i,j, which points from pedestrian i to pedestrian j, and the current velocity vector v.sub.i.sup.cur of pedestrian i is zero. It assumes its minimum when φ.sub.i,j=π.

[0032] Taking all these factors into account, future velocity vectors v.sub.61-66 of pedestrians 61 through 66 are ascertained that differ from their current velocity vectors. The future movement of all pedestrians 61 through 66 is shown by broken lines. It will be seen that the future velocity vectors v.sub.61-66.sup.fut will change over the individual time intervals in such a way that each pedestrian 61 through 66, after passing the other pedestrians, will again move towards his or her original destination.

[0033] FIG. 4 shows how the current movement of pedestrians 61 through 63, who are members of the group, can be represented by a common current velocity vector v.sub.70.sup.cur that goes out from midpoint 70 of the group. A self-driving vehicle 90, which, in the situation shown in FIG. 4, has planned a horizontal movement to the left, can recognize pedestrians 61 through 63 using a sensor 91, here realized as an RGB-D camera. In its electronic control device 92, using the method according to the present invention the vehicle predicts the movement of the group members in order to adapt its direction of movement, in order to reduce its speed if warranted, or in order to stop, and thus to avoid a collision with the group members. Further pedestrians, who themselves could not move into the direction of movement of vehicle 90, are nonetheless acquired by sensor 91 and are taken into account in the prediction, because they may influence the future movement of the group members through social interactions.

[0034] Based on the simulated environment of the ATC department store in Osaka, Japan, described in D. Brscic, T. Kanda, T. Ikeda, T. Miyashita, “Person position and body direction tracking in large public spaces using 3D range sensors,” IEEE Transactions on Human-Machine Systems, Vol. 43, No. 6, pp. 522-534, 2013, a prediction B1 according to the present invention, and to predictions according to comparative examples VB1, VB2, were carried out. For comparative example VB1, a prediction method was used according to V. Karasev, A. Ayvaci, B. Heisele, S. Soatto, “Intent-aware longterm prediction of pedestrian motion,” in 2016 IEEE International Conference on Robotics and Automation (ICRA), May 2016. In this prediction method, a map 11 of area of prediction 80 is taken into account. However, current velocity vectors (v.sub.61-66.sup.cur) of further pedestrians 61-66 in area of prediction 80 are not taken into account in the prediction. For comparative example VB2, a prediction method was used according to J. Elfring, R. Van De Molengraft, M. Steinbuch, “Learning intentions for improved human motion prediction,” Robotics and Autonomous Systems, vol. 62, no. 4, pp. 591-602, 2014. In this method, current velocity vectors (v.sub.61-66.sup.cur) of further pedestrians 61-66 in area of prediction 80 are taken into account in the prediction. However, no map 11 of area prediction 80 is taken into account.

[0035] In Example B1 according to the present invention, and in the comparative examples VB1, VB2, 21 scenarios were simulated with a total of 172 persons, of which 90 pedestrians were in groups, having a total of 15 different possible destinations. A prediction was made over a time period t of 12 seconds. The average negative log probability NLP obtained in the respective simulations is shown in FIG. 6, and the average modified Hausdorff distance MHD obtained is shown in FIG. 7. It can be seen that the method according to the present invention supplies lower values for NLP and MHD than do comparative examples VB1, VB2. Consequently, the method according to the present invention is more accurate.