METHOD FOR DETERMINING A LOCALIZATION POSE OF AN AT LEAST PARTIALLY AUTOMATED MOBILE PLATFORM

Abstract

A method for determining a localization pose of an at least partially automated mobile platform, the mobile platform being equipped to generate ground images of an area surrounding the mobile platform, and being equipped to receive aerial images of the area surrounding the mobile platform from an aerial-image system. The method includes: providing a digital ground image of the area surrounding the mobile platform; receiving an aerial image of the area surrounding the mobile platform; generating the localization pose of the mobile platform with the aid of a trained convolutional neural network, which has a first trained encoder convolutional-neural-network part and a second trained encoder convolutional-neural-network part.

Claims

1. A method for determining a localization pose of an at least partially automated mobile platform, the mobile platform being equipped to generate ground images of an area surrounding the mobile platform, and being equipped to receive aerial images of the area surrounding the mobile platform from an aerial-image system, the method comprising the following steps: providing a digital ground image of the area surrounding the mobile platform; receiving an aerial image of the area surrounding the mobile platform; and generating the localization pose of the mobile platform using a trained convolutional neural network, which has a first trained encoder convolutional-neural-network part and a second trained encoder convolutional-neural-network part.

2. The method as recited in claim 1, wherein the generating of the localization pose includes the following steps: inputting the ground image as an input signal to the first trained encoder convolutional-neural-network part to form a first encoding vector; inputting the aerial image as an input signal to the second trained encoder convolutional-neural-network part to form a second encoding vector; and generating the localization pose of the mobile platform by fusing the first encoding vector and the second encoding vector.

3. The method as recited in claim 2, wherein at least one layer of the first trained encoder convolutional-neural-network part and at least one corresponding layer of the second trained encoder convolutional-neural-network part have identical weights.

4. The method as recited in claim 2, wherein the first encoding vector and the second encoding vector are fused, in that the first encoding vector and the second encoding vector become joined together and are fully connected with at least one output layer of a fusion part of the convolutional neural network, an output signal of the output layer indicating the localization pose.

5. The method as recited in claim 1, wherein the aerial image of the area surrounding the mobile platform is generated using a satellite, or an aircraft, or a drone.

6. The method as recited in claim 1, wherein the aerial image is selected using a pose of the mobile platform, which is determined with a global navigation system and/or a navigation system based on a cellular network.

7. The method as recited in claim 1, wherein the ground image of the area surrounding the mobile platform is generated using a digital camera system.

8. A method for generating a trained convolutional neural network to determine a localization pose of an at least partially automated mobile platform using a ground image of an area surrounding the mobile platform and an aerial image of the area surrounding the mobile platform, the convolutional neural network having a first encoder convolutional-neural-network part, a second encoder convolutional-neural-network part, and a fusion part, and the trained convolutional neural network being generated utilizing a large number of training cycles, each of the training cycles including the following: providing a respective reference pose of the at least partially automated mobile platform; providing a ground image of the area surrounding the mobile platform in the respective reference pose; providing an aerial image of the area surrounding the mobile platform in the respective reference pose; using the ground image as an input signal of the first encoder convolutional-neural-network part to generate a first output signal; using the aerial image as an input signal of the second encoder convolutional-neural-network part to generate a second output signal; determining a respective localization pose using the fusion part, which fuses the first output signal and the second output signal; and adapting the convolutional neural network to minimize a deviation from the respective reference pose in determining the respective localization pose.

9. The method as recited in claim 8, wherein the fusing of the first output signal and the second output signal includes the following steps: forming a first encoding vector with the first output signal; forming a second encoding vector with the second output signal; and fusing the first encoding vector and the second encoding vector, by joining together the first encoding vector and the second encoding vector and full connection of the joined-together encoding vectors with an output layer of the fusion part of the convolutional neural network, the output layer indicating the respective localization pose.

10. The method as recited in claim 8, wherein in adapting the convolutional neural network, at least one layer of the first encoder convolutional-neural-network part and a corresponding layer of the second encoder convolutional-neural-network part mutually exchange corresponding weights of the at least one layer and the corresponding layer.

11. The method as recited in claim 1, further comprising: based on the localization pose, providing a control signal for controlling the at least partially automated mobile platform.

12. The method as recited in claim 1, further comprising: based on the localization pose, providing a warning signal to warn an occupant of the at least partially automated mobile platform.

13. A device configured to determine a localization pose of an at least partially automated mobile platform, the mobile platform being equipped to generate ground images of an area surrounding the mobile platform, and being equipped to receive aerial images of the area surrounding the mobile platform from an aerial-image system, the device configured to: provide a digital ground image of the area surrounding the mobile platform; receive an aerial image of the area surrounding the mobile platform; and generate the localization pose of the mobile platform using a trained convolutional neural network, which has a first trained encoder convolutional-neural-network part and a second trained encoder convolutional-neural-network part.

14. A device configured to generate a trained convolutional neural network to determine a localization pose of an at least partially automated mobile platform using a ground image of an area surrounding the mobile platform and an aerial image of the area surrounding the mobile platform, the convolutional neural network having a first encoder convolutional-neural-network part, a second encoder convolutional-neural-network part, and a fusion part, and the trained convolutional neural network being generated utilizing a large number of training cycles, in the device being configured to, in each of the training cycles: provide a respective reference pose of the at least partially automated mobile platform; provide a ground image of the area surrounding the mobile platform in the respective reference pose; provide an aerial image of the area surrounding the mobile platform in the respective reference pose; use the ground image as an input signal of the first encoder convolutional-neural-network part to generate a first output signal; use the aerial image as an input signal of the second encoder convolutional-neural-network part to generate a second output signal; determine a respective localization pose using the fusion part, which fuses the first output signal and the second output signal; and adapt the convolutional neural network to minimize a deviation from the respective reference pose in determining the respective localization pose

15. A non-transitory machine-readable storage medium on which is stored a computer program for determining a localization pose of an at least partially automated mobile platform, the mobile platform being equipped to generate ground images of an area surrounding the mobile platform, and being equipped to receive aerial images of the area surrounding the mobile platform from an aerial-image system, the computer program, when executed by a computer, causing the computer to perform the following steps: providing a digital ground image of the area surrounding the mobile platform; receiving an aerial image of the area surrounding the mobile platform; and generating the localization pose of the mobile platform using a trained convolutional neural network, which has a first trained encoder convolutional-neural-network part and a second trained encoder convolutional-neural-network part.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Exemplary embodiments of the present invention are represented with reference to FIGS. 1 and 2 and are explained in greater detail below.

[0058] FIG. 1 shows a flowchart of a method for determining a localization pose of an at least partially automated mobile platform, in accordance with an example embodiment of the present invention.

[0059] FIG. 2 shows a flowchart of a method for generating a trained convolutional neural network to determine a localization pose, in accordance with an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0060] FIG. 1, utilizing a data flowchart, schematically sketches method 100 for determining a localization pose 130 of an at least partially automated mobile platform. A digital ground image 110 of the area surrounding the mobile platform may be provided S1 by the mobile platform itself, for example, which is equipped to generate these ground images, e.g., via a digital camera system.

[0061] In addition, an aerial image of the area surrounding the mobile platform is received S2 by the mobile platform, e.g., from an aerial-image system. For example, such an aerial-image system is able to generate aerial image 120 of the area surrounding the mobile platform with the aid of a satellite, an aircraft or a drone. In order, for example, to generate or to select aerial image 120 of the area surrounding the mobile platform, a pose of the mobile platform may be determined with the aid of a global navigation system and/or a navigation system based on a cellular network. In this context, the mobile platform itself may be equipped with a system which determines such a pre-determination of the pose with the aid of a global navigation system and/or a navigation system based on a cellular network.

[0062] The localization pose of the mobile platform is then generated S3 with the aid of a trained convolutional neural network, the convolutional neural network having a first trained encoder convolutional-neural-network part 112, 114, 116 and a second trained encoder convolutional-neural-network part 122, 124, 126.

[0063] To generate S3 localization pose 130 of the mobile platform, ground image 110 is input S4 as input signal of first trained encoder convolutional-neural-network part 112, 114, 116, in order to form a first encoding vector 116. Parallel to that, aerial image 120 is input S5 as input signal of second trained encoder convolutional-neural-network part 122, 124, 126, in order to form a second encoding vector 126. Localization pose 130 of the mobile platform is generated S6 by fusing first encoding vector 116 and second encoding vector 126. To that end, at least one layer 112 of first trained encoder convolutional-neural-network part 112, 114, 116 and at least one corresponding layer 122 of second trained encoder convolutional-neural-network part 122, 124, 126 have identical weights. In that context, first encoding vector 116 and second encoding vector 126 are fused, in that first encoding vector 116 and second encoding vector 126 become joined together and are fully connected with at least one output layer of a fusion part 118 of the convolutional neural network, an output signal of the output layer of the fusion part of the convolutional neural network indicating localization pose 130.

[0064] FIG. 2, utilizing a data flowchart, schematically sketches method 200 for generating a trained convolutional neural network to determine a localization pose 130 of an at least partially automated mobile platform, with the aid of a ground image 110 of an area surrounding the mobile platform and an aerial image 120 of the area surrounding the mobile platform. The convolutional neural network has a first encoder convolutional-neural-network part 112, 114, 116 and a second encoder convolutional-neural-network part 122, 124, 126 and a fusion part 118.

[0065] Trained convolutional neural network 140 is generated utilizing a large number of training cycles, each training cycle having the following steps.

[0066] In a step S21, a reference pose 220 of the at least partially automated mobile platform is provided. In another step S22, a ground image 110 of the area surrounding the mobile platform in reference pose 220 is provided. In a further step S23, an aerial image of the area surrounding the mobile platform in reference pose 220 is provided. In a further step S24, ground image 110 is used as input signal of first encoder convolutional-neural-network part 112, 114, 116, in order to generate a first output signal. In a further step S25, aerial image 120 is used as input signal of second encoder convolutional-neural-network part 122, 124, 126, in order to generate a second output signal. In a further step, localization pose 130 is determined S26 with the aid of fusion part 118, which fuses the first output signal and the second output signal. And in a further step S27, the convolutional neural network is adapted in order to minimize a deviation from respective reference pose 220 in determining respective specific localization pose 130.

[0067] In adapting S27 the convolutional neural network, at least one layer 112 of first encoder convolutional-neural-network part 112, 114, 116 and a corresponding layer 122 of second encoder convolutional-neural-network part 122, 124, 126 mutually exchange corresponding weights of corresponding layers 112, 122, so that after the training, the corresponding layers have identical weights.

[0068] The fusing of the first output signal and the second output signal for generating the trained convolutional neural network has the following steps. In one step S28, a first encoding vector 116 is formed with the first output signal. In another step S29, a second encoding vector 126 is formed with the second output signal. In a further step S30, first encoding vector 116 and second encoding vector 126 are fused by joining together first encoding vector 116 and second encoding vector 126 and a full connection of joined-together encoding vectors 116, 126 with an output layer of fusion part 118 of the convolutional neural network, the output layer indicating localization pose 130.