METHOD OF VISION-BASED LONG-RANGE AND SHORT-RANGE GUIDANCE FOR AUTONOMOUS UAV LANDING

Abstract

Provided is a method for autonomous unmanned aerial vehicle (UAV) landing, including: collecting an unmanned vehicle image dataset in advance, training an unmanned vehicle detection model by using the unmanned vehicle image dataset combined with a YOLOv5 neural network; collecting, by the UAV during a landing process, images of an area below the UAV at specified time intervals, inputting the collected images into the unmanned vehicle detection model for recognition and detection; if an unmanned vehicle is recognized, further determining position information of the unmanned vehicle, and outputting, by a control module, a long-range guidance control instruction, to instruct the UAV to fly to a specified distance position above the unmanned vehicle; collecting, by the UAV, an image of a target and determining position information of the target, and outputting, by the control module, a short-range guidance control instruction to instruct the UAV to land on an unmanned vehicle platform.

Claims

1. A method of vision-based long-range and short-range guidance for autonomous unmanned aerial vehicle (UAV) landing, comprising the following steps: S1: collecting an unmanned vehicle image dataset in advance; S2: training an unmanned vehicle detection model by using the unmanned vehicle image dataset combined with a YOLOv5 neural network; S3: loading the unmanned vehicle detection model onto a control module of a UAV; S4: collecting, by the UAV during a landing process, images of an area below the UAV at specified time intervals, and inputting the collected images into the unmanned vehicle detection model for recognition and detection; S5: if an unmanned vehicle is recognized in the area below the UAV, determining position information of the unmanned vehicle through image analysis, and then proceeding to step S6; otherwise, returning to step S4; S6: outputting, by the control module, a long-range guidance control instruction based on the position information of the unmanned vehicle, to instruct the UAV to fly to a specified distance position above the unmanned vehicle; S7: capturing, by the UAV, an image of a target installed on the unmanned vehicle, and identifying position information of the target; and S8: outputting, by the control module, a short-range guidance control instruction based on the position information of the target, to instruct the UAV to land on an unmanned vehicle platform.

2. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 1, wherein step S2 comprises the following sub-steps: S21: preprocessing the unmanned vehicle image dataset and dividing the unmanned vehicle image dataset into a training set and a validation set according to a preset ratio; S22: annotating images in the training set with corresponding labels by using an annotation tool, and recording categories for cluster analysis, to build the YOLOv5 neural network; and S23: training and validating the YOLOv5 neural network based on the training set and the validation set, to obtain the UAV detection model.

3. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 2, wherein in step S21, the preprocessing specifically adopts Mosaic data augmentation, comprising but not limited to random horizontal or vertical flipping, cropping, and scale transformation operations.

4. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 2, wherein in step S23, before the training and validation of the YOLOv5 neural network, image sizes and resolutions are standardized: first, scaling down the images based on an input size required by the YOLOv5 neural network, and then adding black bars to shorter sides to form a square, thereby meeting input specifications of 608 pixels*608 pixels.

5. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 2, wherein during the training of the YOLOv5 neural network in step S23, a Generalized Intersection over Union (GIoU) loss function is specifically used to calculate a loss of a bounding box:
GIoU=|ABAB||C\(ABC|=IoU|C\(ABC| that is, for any two arbitrary boxes A and B, a smallest enclosed shape C is found, wherein C contains both A and B; then a ratio of an area of C outside A and B to a total area of C is calculated, and the ratio is subtracted from an Intersection over Union (IoU) of A and B to obtain a loss value of the bounding box.

6. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 1, wherein the target installed on the unmanned vehicle is a pattern comprising two AprilTags and an H-shaped geometric pattern, with the two AprilTags located in upper and lower grooves of the H-shaped geometric pattern.

7. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 6, wherein the target has a size of 0.5 m*0.5 m.

8. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 6, wherein step S7 comprises the following sub-steps: S71: capturing, by the UAV, an image of the target installed on the unmanned vehicle; S72: analyzing and processing the image of the target to obtain position information of the AprilTags in the image of the target; and S73: performing coordinate system transformation on the position information of the AprilTags in the image of the target, and then obtaining Euler angles through a homography matrix, to obtain real-time positions of the AprilTags.

9. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 1, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

10. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 9, wherein the driving unit is specifically configured to adjust a flight direction, a flight altitude, a flight speed, hovering, a landing speed, and a landing altitude of the UAV.

11. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 2, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

12. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 3, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

13. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 4, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

14. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 5, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

15. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 6, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

16. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 7, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

17. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 8, wherein the control module of the UAV comprises a recognition unit, a processing unit, and a driving unit; the recognition unit is equipped with the unmanned vehicle detection model; the processing unit is configured to calculate distance information between the UAV and the unmanned vehicle or the target, and output a corresponding control signal to the driving unit; and the driving unit is configured to adjust a flight status of the UAV according to the control signal.

18. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 11, wherein the driving unit is specifically configured to adjust a flight direction, a flight altitude, a flight speed, hovering, a landing speed, and a landing altitude of the UAV.

19. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 12, wherein the driving unit is specifically configured to adjust a flight direction, a flight altitude, a flight speed, hovering, a landing speed, and a landing altitude of the UAV.

20. The method of vision-based long-range and short-range guidance for autonomous UAV landing according to claim 13, wherein the driving unit is specifically configured to adjust a flight direction, a flight altitude, a flight speed, hovering, a landing speed, and a landing altitude of the UAV.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG. 1 is a schematic flowchart of a method according to the present disclosure;

[0039] FIG. 2 is a schematic diagram showing an application process of an embodiment;

[0040] FIG. 3 is a schematic flowchart of image processing during long-range and short-range guidance in the embodiment;

[0041] FIG. 4 is a schematic diagram of a construction process of an unmanned vehicle detection model;

[0042] FIG. 5 is a schematic principle diagram of a YOLOv5 algorithm;

[0043] FIG. 6 is a schematic diagram of a target pattern in the embodiment; and

[0044] FIG. 7 is a schematic flowchart of target recognition and positioning.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The present disclosure will be described in detail below with reference to the drawings and specific embodiments.

Embodiment

[0046] As shown in FIG. 1, a method of vision-based long-range and short-range guidance for autonomous UAV landing includes the following steps: [0047] S1: Collect an unmanned vehicle image dataset in advance. [0048] S2: Train an unmanned vehicle detection model by using the unmanned vehicle image dataset combined with a YOLOv5 neural network. [0049] S3: Load the unmanned vehicle detection model onto a control module of a UAV. [0050] S4: The UAV collects images of an area below the UAV at specified time intervals during a landing process, and inputs the collected images into the unmanned vehicle detection model for recognition and detection. [0051] S5: If an unmanned vehicle is recognized in the area below the UAV, determine position information of the unmanned vehicle through image analysis, and then proceed to step S6; otherwise, return to step S4. [0052] S6: The control module outputs a long-range guidance control instruction based on the position information of the unmanned vehicle, to instruct the UAV to fly to a specified distance position above the unmanned vehicle; [0053] S7: The UAV captures an image of a target installed on the unmanned vehicle, and identifies position information of the target. [0054] S8: The control module outputs a short-range guidance control instruction based on the position information of the target, to instruct the UAV to land on an unmanned vehicle platform.

[0055] In this embodiment, the above scheme is applied to guide autonomous landing of a UAV for collaborative operations with an unmanned vehicle in farmland, as shown in FIG. 2 and FIG. 3. The specific process mainly includes: [0056] 1. The UAV first captures photos of unmanned vehicle bodies in the farmland as training images. In this embodiment, the difference in the numbers of photos for different types of vehicle bodies is less than 100, and a total of approximately 2500 images related to vehicle bodies are captured. [0057] 2. The obtained images of unmanned vehicles in the farmland are preprocessed and divided into a training set and a validation set proportionally. [0058] 3. Images in the training set are annotated with labels by using an annotation tool, and categories are recorded for cluster analysis, to construct a YOLOv5 neural network. [0059] 4. The YOLOv5 neural network is trained and validated based on the training set and validation set to obtain a vehicle body identity detection model, namely, the unmanned vehicle detection model. [0060] 5. An unmanned vehicle image to be detected is inputted into the vehicle body identity detection model, and a corresponding vehicle identity detection result and a corresponding position detection result are outputted. [0061] 6. The UAV performs automatic calculation and is remotely guided, to control the UAV to fly to an area above the unmanned vehicle. The UAV hovers at a certain height (which set to 2 m in this embodiment) by rotating the direction and lowering the throttle for landing. The control module of the UAV includes a recognition unit, a processing unit, and a driving unit. An output end of the processing unit is connected to the recognition unit, and an output end of the processing unit is connected to the driving unit.

[0062] The processing unit calculates the distance between the UAV landing platform carried by the unmanned vehicle and the UAV. The driving unit is controlled according to the distance information, to drive the UAV to fly to a position above the unmanned vehicle at a height of 2 m.

[0063] The driving unit specifically adjusts the flight direction, flight altitude, flight speed, hovering, landing speed, and landing altitude of the UAV. [0064] 7. The UAV recognizes and locates the target, and when the recognition is successful, guides the UAV to descend at a short range. If the recognition fails, the UAV corrects hovering around an initial hover position and collects images until the target is successfully recognized. [0065] 8. The UAV corrects errors and aligns with the landing position by using a proportional-integral-derivative (PID) system.

[0066] In this embodiment, when constructing a sample dataset, it is necessary to classify and calibrate the sample dataset, and unifies sizes and resolutions of the images in the sample dataset. The sample dataset includes calibration data for various models of vehicle bodies. Mosaic data augmentation is employed in preprocessing, including random horizontal or vertical flipping, cropping, and scale transformation of images of vehicles in the farmland.

[0067] In this embodiment, as shown in FIG. 4 and FIG. 5, during the training of the YOLOv5 neural network, the final detection model is formed by identifying feature points of unmanned vehicles from the images captured by the UAV. The feature points of unmanned vehicles consist of vehicle names and vehicle bodies. The vehicle name and body shape are the main recognition criteria. YOLOv5 learns from the feature points marked in the training set and ultimately obtains the most suitable model data. After identifying the data corresponding to the above conditions, the vehicle name of the corresponding unmanned vehicle can be found.

[0068] In this embodiment, the image sizes and resolutions of the sample dataset are first unified: the images are scaled down according to the input size required by the YOLOv5 neural network, and then black bars are added to the shorter sides to form a square, thereby meeting the input specification of 608 pixels*608 pixels. In addition, the GIOU Loss is used to calculate the loss of the bounding box during the training process of the YOLOv5 neural network. The specific calculation method is as follows:

GIoU=|ABAB||C\(ABC|=IoU|C\(ABC|

[0069] That is, for any two arbitrary boxes A and B, a smallest enclosed shape C is found, where C contains both A and B; then a ratio of an area of C outside A and B to a total area of C is calculated, and the ratio is subtracted from an IoU of A and B.

[0070] In this embodiment, the target installed on the unmanned vehicle is shown in FIG. 6, which is configured as a physical landing target with a size of 0.5 m*0.5 m. The target pattern consists of two AprilTags and an H-shaped geometric pattern. Both AprilTags are composed of black and white, carrying various information.

[0071] As shown in FIG. 7, during short-range guidance, the target is detected using a preset AprilTag package, and the obtained serial number and position information of the obtained AprilTags in the target image are saved. The detection result and decoding result are saved to a temporary detection result set. Next, coordinate system transformation is performed on the position information of the obtained AprilTags in the image, and then Euler angles are obtained through a homography matrix to obtain real-time positions of the AprilTags, facilitating accurate and reliable directional positioning of the UAV.

[0072] In summary, this technical solution designs a method of long-range and short-range guidance for autonomous UAV landing, and proposes a new target pattern design, which can easily and efficiently achieve precise fixed-point landing of UAVs.