UAV real-time path planning method for urban scene reconstruction

Abstract

A method for urban scene reconstruction uses the top view of a scene as priori information to generate a UVA initial flight path, optimizes the initial path in real time, and realizes 3D reconstruction of the urban scene. There are four steps: (1): to analyze the top view of a scene, obtain the scene layout, and generate a UAV initial path; (2): to reconstruct the sparse point cloud of the building and estimate the building height according to the initial path, combine the scene layout to generate a rough scene model, and adjust the initial path height; (3): to use the rough scene model, sparse point cloud and the UAV flight trajectory to obtain the scene coverage confidence map and the details that need close-ups, optimize the flight path in real time; and (4): to obtain high resolution images, reconstruct them to obtain a 3D model of the scene.

Claims

1. A UAV real-time path planning method for urban scene reconstruction, comprising: analyzing a top view of a scene to obtain a layout of the scene, determining a safe flight area for the UAV according to the layout of the scene, generating, in the safe flight area, a UAV initial flight path with a fixed height and that traverses each side of a building, and at the same time giving each building the same initial height value, and adding an orientation of lens to the UAV initial flight path according to geometric constraints, so that the building photographed is in a center of a screen; reconstructing a sparse point cloud of the building in a flight process according to the initial flight path, using the sparse point cloud to estimate a height of the building, then combining the obtained layout of scene to generate a rough scene model, and at the same time optimizing the initial flight path according to the height of the building to generate a flight path with varying heights; by means of the generated rough scene model, and based on the sparse point cloud and the UAV flight path, predicting a completeness of scene collection information and judging details of the building to obtain a scene coverage confidence map and the details in need of close-up shots, and optimizing the flight path in real time to obtain the UAV flight path that can complete the scene collection information in real time and shoot close-ups of the details of the building; and in a fight process according to the optimized UAV flight path, obtaining high-resolution images with more than 19 million pixels, and using a multi-view stereo geometric technology to obtain a complete urban scene 3D model with the details of the building through reconstruction of the high-resolution images.

2. The UAV real-time path planning method for urban scene reconstruction according to claim 1, wherein the step 1 is implemented as: segmenting out the building in the top view of the scene, using Mask R-CNN instance segmentation neural network to segment out each building to obtain the layout of the scene, and according to the layout of scene, determining the UAV safe flight area that is an area other than that above the building, wherein buildings at the time are not connected, and there is a disconnected graph; to generate a coherent UAV flight path, needing to convert the disconnected graph to a connected graph by: viewing each building as a point, paths between the buildings as edges, and distances between the buildings as weights of the edges; obtaining a shortest path through each point by Dijkstra algorithm, this path being a shortest path through each building; and at the time representing each building in the scene by a geometric shape, wherein any two points of the building are connected by paths, and the connected graph are constructed; obtaining the connected graph; at the time, in order to save flight time, needing to obtain a shortest path that traverses each side of the building; adding repeated edges to make the connected graph become an Euler graph, and then using Fleury algorithm to find an optimal Euler tour, the optimal Euler tour obtained at the time being the UAV initial flight path with the fixed height in the safe flight area and that traverses each side of the building; and giving each building an initial height, calculating the orientation of lens according to geometric constraints, and adding the orientation of lens to the UAV initial flight path, so that top and sides of the building can be photographed at the same time, wherein the photographed building is in the center of the screen, and a path initialization is completed.

3. The UAV real-time path planning method for urban scene reconstruction according to claim 1, wherein the reconstruction is implemented as: in the UAV flight along the initial flight path generated in step 1, when photographing a current building, matching the photographed image with the top view of the scene in step 1, and extracting a Scale Invariant Feature Transform (SIFT) feature to determine an area of the building in a current shot; reconstructing the sparse point cloud of the building by Simultaneous Localization And Mapping (SLAM), searching points in the determined area, and determining heights of the points according to maximum and minimum values of the points in the area on z-axis; after determining the heights of the points, determine a scaling ratio according to a size of top area of the building and a size of the actual building top, multiplying the heights of the points and the scaling ratio to unify a scale of the reconstructed sparse point cloud, and restore a real height of the building; and on the basis of the obtained layout of the scene, combining the obtained real height of the building to generate the rough scene model, and optimizing the initial flight path according to the real height of the building to readjust the height of the initial flight path and calculate the orientation of lens.

4. The UAV real-time path planning method for urban scene reconstruction according to claim 1, wherein the step 3 is implemented as: combining the rough scene model to predict the completeness of the scene collection information, determining the area covered by the UAV based on the sparse point cloud recovered by SLAM, and determining the possible coverage area according to the recovered UAV flight path to generate the scene coverage confidence map; for the remaining uncovered area, calculating points to be added to the path and the orientation of onboard camera lens, optimizing the UAV flight path in real time, and enabling the UAV to complete the scene collection information in real time; and while reconstructing the sparse point cloud, judging the details of the building, calculating a density of the sparse point cloud, determining an area with the density of the sparse point cloud fairly great, that is, with more than 10 points per cubic meter, and optimizing the flight path so that the UAV can shoot close-ups of complex parts of the building, like balcony, water tank and fire ladder.

Description

DESCRIPTION OF FIGURES

(1) The accompanying drawings illustrate one or more embodiments of the present invention and, together with the written description, serve to explain the principles of the invention. Wherever possible, the same reference numbers are used throughout the drawings to refer to the same or like elements of an embodiment.

(2) FIG. 1 is a flowchart of the present disclosure;

(3) FIG. 2 is a schematic diagram of a path initialization flowchart of the present disclosure;

(4) FIG. 3 is a schematic diagram of the scene completeness prediction process of the present disclosure;

(5) FIG. 4 is an application sample diagram of the UAV real-time path planning of the present disclosure; and

(6) FIG. 5 is an application sample diagram of the urban scene reconstruction of the present disclosure.

DETAILED DESCRIPTION

(7) The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which exemplary embodiments of the present invention are shown. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure is thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like reference numerals refer to like elements throughout.

(8) In order to better understand the technical solution of the present disclosure, the specific mode of carry out the present disclosure is further described below with reference to the drawings.

(9) FIG. 1 is a flowchart of the present disclosure. The overall objective of the present disclosure is to propose a UAV real-time path planning method for urban scene reconstruction, which enables the UAV to optimize the flight path in one flight in real time, collect comprehensive information to complete the 3D reconstruction of the scene. The specific steps are to: first analyze the top view of a scene to obtain the layout of the scene and generate a UAV initial flight path with lens orientation; reconstruct the sparse point cloud of the building during the flight, estimate the height of the building from the point cloud, and combine the obtained scene layout to generate a rough scene model and adjust the height of the initial path; by means of the rough scene model and according to the sparse point cloud and the UAV flight path, obtain the confidence map of scene coverage and the details in need of close-up shots, and optimize the flight path in real time; and obtain high-resolution images, which are reconstructed to obtain a 3D model of the scene.

(10) As shown in FIG. 2, the method of the present disclosure first uses the instance segmentation model Mask R-CNN to obtain the top image (a) of the building in the map. Since each building is independent, and the flight path of the UAV is coherent, it is required to convert the image (a) to a connected graph. Each building is viewed as a point in the image, and the weight of the edge between points is the distance between buildings, as shown in image (b). The shortest path (c) through each building is obtained by Dijkstra's shortest path algorithm. At the time, in order to obtain the path traversing the side of each building, it is required to add repeated edges to make the connected graph being an Euler graph, and then use the Fleury algorithm to find the optimal Euler tour, wherein the finally obtained optimal Euler tour (d) is the shortest path that passes through each edge and may pass through some edges for more than once. Then an initial height is assigned to each building, and according to the following geometric constraint (e), the position and orientation of the camera are obtained:

(11) $\hspace{1em}\{\begin{array}{c}\frac{\sin \left[\pi -\left(\alpha +\beta \right)-\left(\frac{\pi -\mathrm{fov}}{2}\right)\right]}{c}=\frac{\sin \alpha }{w_p}\\ \frac{\sin \left(\frac{\pi -\mathrm{fov}}{2}\right)}{c}=\frac{\sin \beta }{h_p}\end{array}$
wherein, fov is the angle of view of the camera, α and β are the angle between the sides of the building and the camera in FIG. 1, c is the distance between the camera and the building, w.sub.p and h.sub.p are the length of the width w and height h of the building respectively imaged on the camera plane.

(12) As shown in FIG. 3, a schematic diagram of the scene completeness prediction process of the present disclosure is illustrated: First, the UAV flies according to the initial path, and uses the SLAM technology to reconstruct the sparse point cloud of the scene and estimate the camera pose during the flight; then estimate the height of the building currently photographed, match the image currently taken by the UAV with the image of the roof of the previously obtained building, and determine the area of the building in the current image by extracting the SIFT feature; the difference between the point z.sub.m with the largest value and the point z.sub.n with the smallest value on the z-axis in the area is taken as the height of the building; and at the same time, because the scale of the monocular camera is uncertain, the real height h of the building needs to be determined by the real scale s of the top view:

(13) $h=s.Math.\left(\frac{1}{M}\underset{m=1}{\overset{M}{.Math.}}\max \left(z_m\right)-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}\min \left(z_n\right)\right)$
wherein, M is the number of points with the largest value on the z-axis in the area, and N is the number of points with the smallest value on the z-axis in the area.

(14) After the height of the building is determined, the height of the UAV initial path and the orientation of lens are adjusted according to the height of the building, and the already obtained scene layout is combined to generate a rough model of the scene. By means of the rough scene model and sparse point cloud, the completeness of the scene is predicted to generate a confidence map of the scene coverage. Each face of the building is divided into several small grids, and for each small grid g its confidence E(g) can be expressed as:

(15) $E\left(g\right)=\left[I\left(n_p-n_{\mathrm{th}}\right)+\underset{i=m-\Delta m}{\overset{m+\Delta m}{.Math.}}I\left(t_i\right)\sin \left(t_i\right)\right].Math.G\left(g\right)$

(16) wherein, I(n.sub.p−n.sub.th) is that the number of points in the small grid exceeds the threshold, then it is considered that this small grid has been covered. Σ.sub.m−Δm.sup.m+ΔmI(t.sub.i)sin (t.sub.i) is based on the trajectory of the UAV to predict whether the grid is covered, t.sub.i is the angle between the lens orientation and the plane where the grid is located. G(g) is a function that measures the distance between the UAV and the observation plane:

(17) $G\left(g\right)=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{{\left(\frac{S_g}{\lambda S_f}-1\right)}^2}{2{\sigma }^2}\right)$

(18) This function is represented by a Gaussian distribution, where) represents the proportion that the small grid area S.sub.g expects to occupy in the screen S.sub.f, σ controls how fast the metric changes with the distance between the UAV and the observation plane.

(19) After the scene coverage confidence map is generated, the UAV trajectory needs to be optimized so that uncovered areas can be collected. For the uncovered area, first the center point is taken as the point of the path to be added, and then the time cost between points is calculated. For any two points p.sub.1, p.sub.2, the time cost C is:
C(p.sub.1,p.sub.2)=min[C.sub.s(p.sub.1,p.sub.2),C.sub.t(p.sub.1,p.sub.2)],

(20) wherein, C.sub.s represents the UAV from the side of the building to another point, and C.sub.t represents the UAV from the top of the building to the destination. Of the two, the one with less time is chosen as the optimal path. When points are added to the path, the times of the two strategies are calculated to complete C.sub.i immediately and complete C.sub.d later, and then the strategy with the least time is selected. The immediate completion requires the UAV from the current point to the destination point, and the round-trip time needs to be calculated, whereas the later completion is to calculate the time required from the point closest to the destination point to the destination point in the path:

(21) $\underset{P}{\arg \min }T=\underset{P}{\arg \min }\left\{\min \left[2C_i\left(p_c,p_a\right),C_d\left(p_{\mathrm{close}},p_a\right)\right]\right\}$

(22) wherein, P represents the path to be sought, T is the length of time that needs to be added after the point is added, p.sub.c is the current location of the UAV, p.sub.a is the destination point, and p.sub.close is the point closest to the destination point in the path.

(23) FIG. 4 is an application example diagram of the UAV real-time path planning of the present disclosure, which shows the benefits and advantages of the present disclosure in terms of time and collection efficiency. The advantages of the present disclosure are that: the UAV has a wide range of information collection angles, and because of the comprehensive information during reconstruction, the effect is better; the redundant information obtained by the collection is less, and the utilization rate is high, so the number of images collected in the same scene is less than the traditional collection method, and the reconstruction time is significantly reduced; the reconstructed rough model is not used as priori information, so there is no need to fly the UAV multiple times, thus saving time and effort; during the collection, the movement of the UAV is more in line with the manner of robot exploring environment, and the correlation between the images is high, so there are fewer errors during reconstruction. (a) is the traditional aerial mapping method, wherein the UAV is flying at a fixed height in the scene in the form of a polyline, and the lens orientation is also fixed, so the angle of the collected information is single, and many parts are not complete during reconstruction; (b) is presently more advanced method of using UAV to reconstruct the scene, which uses the rough model reconstructed in advance as priori information, calculates the best angle of view, and then uses the path to connect the angle of view, so the correlation between the images is not high, and it takes a lot of time to calculate the image correlation during reconstruction; and (c) is the result of the path planning of the present disclosure. Table 1 compares the reconstruction time (min) and reconstruction error (cm) under these three methods. It can be seen from Table 1 that the method of the present disclosure greatly reduces the time of the entire process where the reconstruction accuracy is similar to the Smith method.

(24) TABLE-US-00001 TABLE 1 Comparison of reconstruction time (min) and reconstruction error (cm) Flight Reconstruction Total Reconstruction Method Time Time Time Error Traditional 36.60 112.70 161.48 5.57 Surveying Smith 42.49 243.12 302.65 5.10 Present 23.73 146.84 185.77 5.11 disclosure

(25) In the method of the present disclosure, the initialization path traverses every side of the building to effectively collect the information of each building, the reconstruction accuracy is high, the flying distance of the UAV is short, and the time consumed is less. After combining the height, completeness and details etc. of the building, the initial path is optimized in real time, which further improves the accuracy and completeness of the reconstruction. The experimental results prove the feasibility, accuracy and universality of the method of the present disclosure. The experimental results are shown in FIG. 5. FIG. 5 shows the reconstruction result and reconstruction error under the traditional aerial mapping method, the path initialization method of the present disclosure, and the optimization method of the present disclosure. Among them, in (a), the traditional aerial mapping method, due to the fixed flying height of the UAV and the fixed orientation of the lens, has relatively large reconstruction errors of the building side. In (b), compared with the traditional aerial mapping method, the initialization method can reconstruct the side of the building more completely, but the accuracy is also not high either due to the fixed height and the lack of details. In (c), the optimization method of the present disclosure can adjust the flying height of the UAV, add close-ups of details, predict the completeness of the scene and make optimization, so the reconstructed scene model has higher accuracy and is more complete.

(26) The above are only some basic descriptions of the present disclosure, and any equivalent transformations made according to the technical solutions of the present disclosure shall fall within the protection scope of the present disclosure.

(27) The foregoing description of the exemplary embodiments of the present invention has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching.

(28) The embodiments were chosen and described in order to explain the principles of the invention and their practical application so as to activate others skilled in the art to utilize the invention and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present invention pertains without departing from its spirit and scope. Accordingly, the scope of the present invention is defined by the appended claims rather than the foregoing description and the exemplary embodiments described therein.