System and method for measuring a displacement of a mobile platform

Abstract

A method for detecting a displacement of a mobile platform includes acquiring a first frame and a second frame using an imaging device coupled to the mobile platform, obtaining an angle of the imaging device relative to a reference level while acquiring the first frame, and determining the displacement of the mobile platform based at least on the first frame, the second frame, and the angle of the imaging device relative to the reference level.

Claims

1. A method for detecting a displacement of a mobile platform, comprising: acquiring a first frame and a second frame using an imaging device coupled to the mobile platform; obtaining an angle of the imaging device relative to a reference level while acquiring the first frame; and determining the displacement of the mobile platform based at least on the first frame, the second frame, and the angle of the imaging device relative to the reference level.

2. The method of claim 1, wherein the angle of the imaging device relative to the reference level is obtained by a simultaneous localization and mapping (SLAM) device.

3. The method of claim 1, wherein obtaining the angle of the imaging device relative to the reference level comprises: obtaining a first angle of the mobile platform relative to the reference level; obtaining a second angle of the imaging device relative to a plane of the mobile platform; and combining the first angle and the second angle to obtain the angle of the imaging device relative to the reference level.

4. The method of claim 1, wherein the reference level is a ground level, a water level, or a reference plane associated with a structure.

5. The method of claim 1, further comprising: obtaining a height of the mobile platform relative to the reference level; and determining the displacement of the mobile platform further based on the height of the mobile platform.

6. The method of claim 5, wherein the height of the mobile platform is obtained using a barometer or an ultrasonic device.

7. The method of claim 1, wherein the first frame and the second frame are acquired within a time interval, wherein the time interval is no less than one-sixtieth of a second and no greater than one-twentieth of a second.

8. The method of claim 1, further comprising: obtaining rotation data of the mobile platform relative to the first frame using an inertial measurement unit (IMU) coupled to the mobile platform; and determining the displacement of the mobile platform further based on the rotation data.

9. The method of claim 8, further comprising: obtaining a stereoscopic point cloud based on the first frame, wherein the stereoscopic point cloud is an array of feature points {P.sub.1, P.sub.2, P.sub.3, . . . , P.sub.n}, n being a number of the feature points selected in the first frame.

10. The method of claim 9, further comprising: projecting the stereoscopic point cloud onto an x-y plane to obtain a first projective array {(x.sub.1.sup.1, y.sub.1.sup.1), (x.sub.2.sup.1, y.sub.2.sup.1), (x.sub.3.sup.1, y.sub.3.sup.1), . . . , (x.sub.n.sup.1, y.sub.n.sup.1)} comprising a plurality of first feature points of the first frame; and matching the plurality of first feature points to a plurality of second feature points of the second frame for generating a second projective array {(x.sub.1.sup.2, y.sub.1.sup.2), (x.sub.2.sup.2, y.sub.2.sup.2), (x.sub.3.sup.2, y.sub.3.sup.2), . . . , (x.sub.m.sup.2, y.sub.m.sup.2)} on the x-y plane, m being a number of matched feature points between the first frame and the second frame, wherein m is equal to or less than n.

11. The method of claim 10, further comprising: calculating a translation array based on a relation between the stereoscopic point cloud, the second projective array, and the rotation data.

12. The method of claim 11, wherein the relation is represented by ${\mathrm{RP}}_j+T=\mu \left(\begin{array}{c}{x}_j^2\\ y_j^2\\ 1\end{array}\right),$ wherein R is a rotation array associated the rotation data, Pj is a feature point of the stereoscopic point cloud, T is the translation array to be calculated, μ is a random number, and (x.sub.j.sup.2, y.sub.j.sup.2) is a matched feature point of the second projective array.

13. The method of claim 11, wherein calculating the translation array comprises: verifying the translation array T by introducing T into an equation ${\mathrm{RP}}_j+T=\mu \left(\begin{array}{c}{x}_j^2\\ y_j^2\\ 1\end{array}\right)$ with a plurality of points selected from the stereoscopic point cloud and their corresponding projection points of the second projective array to get a count of matching points; and selecting the translation array T with maximum count of matching points as a selected translation array T.

14. An apparatus for detecting a displacement of a mobile platform, comprising: an imaging device coupled to the mobile platform and configured to acquire a first frame and a second frame; and a processor coupled to the imaging device and configured to: obtain an angle of the imaging device relative to a reference level when the imaging device is acquiring the first frame; and determine the displacement of the mobile platform based at least on the first frame, the second frame, and the angle of the imaging device relative to the reference level.

15. The apparatus of claim 14, wherein the processor is further configured to: obtain a height of the mobile platform relative to the reference level; and determine the displacement of the mobile platform further based on the height of the mobile platform.

16. The apparatus of claim 14, further comprising: an inertial measurement unit (IMU) coupled to the mobile platform and configured to obtain rotation data; wherein the processor is further configured to determine the displacement of the mobile platform further based on the rotation data.

17. The apparatus of claim 16, wherein the processor is further configured to: obtain a stereoscopic point cloud based on the first frame, wherein the stereoscopic point cloud is an array of feature points {P.sub.1, P.sub.2, P.sub.3, . . . , P.sub.n}, n being a number of the feature points selected in the first frame; project the stereoscopic point cloud onto an x-y plane to obtain a first projective array {(x.sub.1.sup.1, y.sub.1.sup.1), (x.sub.2.sup.1, y.sub.2.sup.1), (x.sub.3.sup.1, y.sub.3.sup.1), . . . , (x.sub.n.sup.1, y.sub.n.sup.1)} comprising a plurality of first feature points of the first frame; and match the plurality of first feature points to a plurality of second feature points of the second frame for generating a second projective array {(x.sub.1.sup.2, y.sub.1.sup.2), (x.sub.2.sup.2, y.sub.2.sup.2), (x.sub.3.sup.2, y.sub.3.sup.2), . . . , (x.sub.m.sup.2, y.sub.m.sup.2)} on the x-y plane, m being a number of matched feature points between the first frame and the second frame, wherein m is equal to or less than n.

18. The apparatus of claim 17, wherein the processor is further configured to: calculate a translation array based on a relation between the stereoscopic point cloud, the second projective array, and the rotation data.

19. The apparatus of claim 18, wherein the relation is represented by ${\mathrm{RP}}_j+T=\mu \left(\begin{array}{c}{x}_j^2\\ y_j^2\\ 1\end{array}\right),$ wherein R is a rotation array associated the rotation data, Pj is a feature point of the stereoscopic point cloud, T is the translation array to be calculated, μ is a random number, and (x.sub.j.sup.2, y.sub.j.sup.2) is a matched feature point of the second projective array.

20. The apparatus of claim 19, wherein the processor is further configured to: verify the translation array T by introducing T into an equation ${\mathrm{RP}}_j+T=\mu \left(\begin{array}{c}{x}_j^2\\ y_j^2\\ 1\end{array}\right)$ with a plurality of points selected from the stereoscopic point cloud and their corresponding projection points of the second projective array to get a count of matching points; and select the translation array T with maximum count of matching points as a selected translation array T.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is an exemplary schematic diagram of a mobile platform.

(2) FIG. 2 is an exemplary top-level flowchart of a method for determining a displacement of the mobile platform of FIG. 1.

(3) FIG. 3 is an exemplary flowchart of an exemplary embodiment of the method of FIG. 2, wherein pairs of frames are matched.

(4) FIG. 4 is another exemplary flowchart of another embodiment of the method of FIG. 3, wherein projective arrays of the frames are calculated.

(5) FIG. 5 is another exemplary flowchart of another embodiment of the method of FIG. 3, wherein movements of the second frame are measured.

(6) FIG. 6 is an exemplary diagram of another embodiment of the method of FIG. 3, illustrating an exemplary method for matching two images with an overlapping area.

(7) FIG. 7 is an exemplary diagram illustrating another embodiment of the method of FIG. 6, wherein a first frame is matched with a second frame with a plurality of feature points.

(8) FIG. 8 is an exemplary diagram illustrating another embodiment of the method of FIG. 7, wherein each feature point is matched by calculating a similarity.

(9) It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are generally represented by like reference numerals for illustrative purposes throughout the figures. It also should be noted that the figures are only intended to facilitate the description of the embodiments. The figures do not illustrate every aspect of the described embodiments and do not limit the scope of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

(10) Since currently-available vision systems are restricted by conditions, a mobile platform and method that can meet the requirements of measuring a displacement of the mobile platform during a flight course at various heights under various conditions can prove desirable and provide a basis for accurate measurement of displacements, for systems such as UAV systems and other mobile platforms. This result can be achieved, according to one embodiment disclosed in FIG. 1.

(11) FIG. 1 illustrates an exemplary schematic diagram of a mobile platform 200. As shown in FIG. 1, the mobile platform 200 can detect a displacement d of the mobile platform 200. An imaging device 886 is shown as installed on the mobile platform 200, such as a UAV 250. When the mobile platform 200 is in the air, as shown in FIG. 1, the imaging device 886 can have a height H and an angle α relative to a ground level 880. The imaging device 886 can comprise a monocular imaging device or a multi-ocular imaging device. In other words, the imaging device 886 can include any suitable number of lenses. In some embodiments, one lens of the imaging device 886 can be used at selected time points for taking a first and second frames 811a, 811b (shown in FIG. 2) of an object of interest. The ground level 880 can be the actual ground, a water level or the ground with any structure. The imaging device 886 can be at a level 882 as illustrated in FIG. 1, which level 882 can have a height H. The mobile platform 200 can have a first angle relative to the ground level 880 and the imaging device 886 can have a second angle relative to a plane of the mobile platform 200. The first and second angles can be combined into an angle α of the imaging device 886 relative to the ground level 880.

(12) In some embodiments, the height H can be acquired with a barometer 251 and/or an ultrasonic device 252 (not shown). The angle α can be acquired with a Simultaneous Localization And Mapping (SLAM) device (not shown) in a manner shown and described below with reference to FIG. 5.

(13) FIG. 2 illustrates an exemplary embodiment of a method 100 for detecting the displacement d of the mobile platform 200 (collectively shown in FIG. 1). The method 100 of FIG. 2 is shown as including acquiring a first frame 811a and a second frame 811b, at 810. The displacement d of the mobile platform 200 can be determined, at 850, based upon the first and second stereoscopic frames 811a, 811b. In some embodiments, the method 100 can use a monocular imaging system 886 with a SLAM and certain height measurement device, such as a barometer 251 and/or an ultrasonic device 252.

(14) In an exemplary embodiment, the two frames 811a, 811b of images can be acquired at two different time points: a first time point and a second time point with a travel path of the imaging device 886. In some embodiments, the first frame 811a and the second frame 811b can be stereoscopic frames. In some embodiments, the two frames 811a, 811b can have at least an overlapping region with a predetermined percentage of the first frame 811a or the second frame 811b. The overlapping region of the two frames 811a, 811b is defined as points, on the two frames 811a, 811b, reflecting a same object. To ensure such overlapping region, a time span between the first time point and the second time point can be adjusted so that at least one object can be reflected on both frames 811a, 811b. In addition, the imaging device 886 stay still relative to the mobile platform 200 or move slowly in order to ensure the overlapping.

(15) When acquiring the two frames 811a, 811b, the time span between the first time point and the second time point can be less than one-sixtieth of a second and not greater than one-twentieth of a second. The time span can depend on requirements of actual applications or situations. As an exemplary example, when the mobile platform 200 is flying at a lower velocity, the time span can be set to a greater value because an overlapping between the two frames 811a, 811b can be ensured even under the greater time span. On the other hand, when the velocity of the mobile platform 200 is flying at a higher velocity, the time span can be set to a lower value to ensure the overlapping between the two frames 811a, 811b.

(16) At 850, the two frames 811a, 811b can be used to calculate a displacement of a mobile platform 200. Such displacement can be calculated in accordance with the manner shown and described below with reference to FIGS. 3-5.

(17) A system implementing the method 100 can be applicable to a wide height range of one meter to one hundred meters and can be less vulnerable to any ambient interference. In addition, the system does not rely on a Global Positioning System (“GPS”) signal, therefore, can be applicable in an environment lack of a GPS signal, such as an indoor setting. The barometer and the monocular imaging device can be easily installed on a mobile platform, for example, can be installed on a small size UAV.

(18) FIG. 3 illustrates another embodiment of the method 100. In the method 100 of FIG. 3, movement of the mobile platform 200 (shown in FIG. 1) can be calculated. As shown in FIG. 3, after two frames 811a, 811b of an object of interest are acquired, at 810, a height H of a mobile platform 200 can be calculated based on the two frames 811a, 811b, at 820. In some embodiments, the height of the mobile platform 200 can be determined with the barometer 251 and/or the ultrasonic device 252. Although shown and described as acquiring the height H after the two frames 811a, 811b are acquired, the height H of the mobile platform 200 can be acquired before, during or after the first frame 811a and/or the second frame 811b are acquired.

(19) At 830, the second frame 811b can be matched to the first frame 811a by matching feature points 355 (shown in FIG. 6) on the first frame 811a to certain points on second frames 811a, 811b. Under the present disclosure, the two frames 811a, 811b are matched when one or more points, reflecting a same object, on each of the two frames 811a, 811b are found. In other words, the points reflecting the same object define an overlapping region on the two frames 811a, 811b. The feature points 355 on the first frame 811a can be selected from those points reflecting an outstanding object. Such object can include, for example, a building, a tree, a road, a river or other type structures. In some embodiments, the object can be a stable object or a slowly moving object. Additional detail regarding matching the two frames 811a, 811b is shown and described below with reference to FIG. 4.

(20) At 840, six movements in three translation movements and three rotations of the mobile platform 200 can be measured with an Inertial Measurement Unit (“IMU”) and/or estimated with the two stereoscopic frames 811a, 811b. The three translation movements can include translation movements of the mobile platform 200 along each of an x-axis, a y-axis and a z-axis. The three rotations can include rotations of the mobile platform 200 around the x-axis, the y-axis and the z-axis, respectively. The rotations can be acquired in any conventional manner. One conventional manner for acquiring the rotations includes use of the IMU 150. In one embodiment, the translations can be calculated based on the rotation data, which is shown and described below with reference to FIG. 5. Detail regarding calculating the six movements will be provided below with reference to FIG. 5.

(21) The displacement of the mobile platform 200 is determined, at 850. Once acquired, the rotations and the translations, for example, can be applied to calculate the displacement of the mobile platform 200. In some embodiments, a velocity of the mobile platform 200 can be calculated by dividing the translations by the time span for acquiring the two frames 811a, 811b. In another exemplary embodiment, location information of the mobile platform at a time point can be acquired based upon the calculated translation array T and location information of the mobile platform at a time point for acquiring the first frame 811a. The time point can be a time for acquiring the second frame 811b.

(22) In another embodiment, after the displacement is determined, at 850, the next frame of image can be acquired and the system repeats the process in order to calculate the newly acquired frame or frames. The process continues over and over to realize uninterrupted displacement detection.

(23) FIG. 5 illustrates another embodiment of the method 100. Turning to FIG. 5, matching the two frames 811a, 811b is illustrated. As shown in FIG. 5, two frames 811a, 811b can be acquired via any of the methods shown and described with reference to FIG. 1, at 810. FIG. 4 illustrates one exemplary manner by which the method 100 can match, at 830, the second frame 811b with the first frame 811a. Turning to FIG. 4, the frames 811a, 811b consist of multiple points, and each of those points is represented with x, y and z coordinate values. The two frames 811a, 811b described herein can be acquired at different time points along with a travel path of the imaging device 100.

(24) Referring to FIG. 4, a stereoscopic point cloud can be acquired based on the first frame 811a {P.sub.1, P.sub.2, P.sub.3, . . . , P.sub.n} at 832. Each point P is a feature point 355 of the first frame 811a. In an x-y plane, the stereoscopic point cloud can be represented with {(x.sub.1.sup.1, y.sub.1.sup.1), (x.sub.2.sup.1, y.sub.2.sup.1), (x.sub.3.sup.1, y.sub.3.sup.1, . . . , (x.sub.n.sup.1, y.sub.n.sup.1)}. The number of the feature points 355 acquired various based on a processing speed, a size of the frame, a resolution of the frame as well as a calculation capacity of a processor (not shown) etc. In a typical embodiment disclosed herein, the number of points, or pixels, can be in an exemplarily range of one hundred to two hundred pixels.

(25) A second projective array can be calculated in the second frame 811b {(x.sub.1.sup.2, y.sub.1.sup.2), (x.sub.2.sup.2, y.sub.2.sup.2), (x.sub.3.sup.2, y.sub.3.sup.2), . . . , (x.sub.m.sup.2, y.sub.m.sup.2)}, at 834. Each of the points (x.sub.i.sup.2, y.sub.i.sup.2) of the second projective array represents a matched point, projected in an x-y plane, corresponding to a point P.sub.j or (x.sub.j′, y.sub.j.sup.1) of the stereoscopic point cloud of the first frame 811b. A size of the second projective array can be same as a size of the cloud array, in case all points in the cloud array {P.sub.1, P.sub.2, P.sub.3, . . . , P.sub.n} are matched onto the second frame 811b. However, in most cases, the size of the second projective array is less than the size of the cloud array {P.sub.1, P.sub.2, P.sub.3, . . . , P.sub.n} because not all points of the first frame 811a can be matched onto the second frame 811b. Matching of the stereoscopic point cloud of the first frame 811a to the second frame 811b can be accomplished by the method 100 described with reference to FIGS. 6-8, in which three-by-three pixels are compared to determine the similarity of two points.

(26) Now referring back to FIG. 5, an exemplary manner by which the method 100 can measure, at 840, movements of the second frame 811b is illustrated. In FIG. 5, at 842, an IMU 150 (not shown) can be configured to measure rotation measurements that can be passed to a processor (not shown). The rotation measurements can be represented by a three dimensional array R. With the rotation measurement array R, a relationship between the stereoscopic point cloud array of the first frame 811a and the projected point array of the second frame 811b can be represented as:

(27) $\begin{array}{cc}{\mathrm{RP}}_j+T=\mu \left(\begin{array}{c}{x}_j\\ y_j\\ 1\end{array}\right)& \mathrm{Equation}\left(6\right)\end{array}$
wherein R is a three-dimensional array representing the rotation measurements, Pj is a point of the first frame 811a, T represents a three-dimensional array of translation of the second frame 811b to be calculated and μ is a random number acting as a factor.

(28) To help ensure an accuracy of the relative rotation array measured by the IMU, the interval between the first time point when the first frame 811a can be taken and the second time point when the second frame 811b can be taken can be relatively short. The time interval between the first frame 811a and the second frame 811b usually can be within a range of twentieth to sixtieth seconds depending on the requirements of actual applications, as described with reference to FIG. 2.

(29) In Equation 6, three unknowns (Tx, Ty, Tz) exist; therefore, by mathematical principles, three equations can be needed to jointly solve the calculated translation array T with the three unknowns, at 844. However, each of the projected points has only two values in x.sub.i and y.sub.i. So, in order to resolve three unknowns in the calculated translation array T, three equations out of four equations available for two such points can be joined.

(30) In practical applications, because of errors and/or inaccuracies in matching the points between the first frame 811a and second frame 811b, the calculated translation array T can be inaccurate. At 846, the calculated translation array T can be introduced into the Equation 6, and calculated to determine the number of points that conform to the relationship defined in the equation. Then, another pair of points can be used in solving the calculated translation array T, at 844, which can be then used to calculate to determine the number of points that conform to the relationship of Equation 6 at 846. This process can iterate for a predetermined number of pairs of points, and the results can be a predetermined number of the calculated translation arrays T accompanied with a number of points of conformance to Equation 6 for each translation array.

(31) At 845, the numbers of conformed points can be compared among the calculated translation arrays T. A calculated translation array T with the greatest number of conformed points can be chosen. The process described with reference to FIG. 4 can be repeated again and again to perform continuous displacement detection.

(32) FIG. 6 illustrates an exemplary embodiment of a method 300 for matching a point of the second frame 811b with a corresponding point 355 of the first frame 811a. In FIG. 6, a three-by-three pixel block is taken with the compared point in the center from each of the frames 811a, 811b. When the first and second frames 811a and 811b are color images, values of color components can be compared for each pixel of the three-by-three pixel block. Conversely, when the frames 811a and 811b are black and white images, greyscale values for each pixel can be compared. Based on Equation 5, the point with smallest sum of value differences for all nine pixels can be selected as the matching point. This process can be repeated for all selected feature points on the first frame 811a.

(33) In some embodiments, a method of using Binary Robust Independent Elementary Features (“BRIEF”) descriptors can be used for matching the point of the second frame 811b with the corresponding point 355 of the first frame 811a. In an exemplary embodiment, a first binary string, representing a first region around the selected feature point of the first frame 811a, can be built by comparing intensities of each point pairs of the region. The first binary string can be the first BRIEF descriptor of the selected feature point of the first frame 811a.

(34) Similarly, a second binary string representing a second region around the point 355 of the second frame 811b can be built by comparing intensities of each point pairs of the second region. The second binary string can be a second BRIEF descriptor.

(35) A similarity between the selected feature point of the first frame 811a and the point 355 of the second frame 811b can be calculated by comparing a hamming distance between the first BRIEF descriptor and the second BRIEF descriptor. The point 355 of the second frame 811b can be determined as matching the selected feature point of the first frame 811a when a hamming distance between the first BRIEF descriptor and the second BRIEF descriptor is less than a first hamming threshold.

(36) Turning now to FIG. 7, an exemplary embodiment of the method 300 for acquiring the disparity d via feature points 355 of the object 198 of interest is illustrated. At 922, a plurality of feature points 355 on the object 198 of interest can be selected. The feature points 355 can be selected using one or more of a variety of different methods. In one exemplary embodiment, the feature points 355 can be identified as pre-defined shapes of the object 198 of interest. In another embodiment, the feature points 355 can be recognized as one or more portions of the object 198 of interest having a particular color or intensity. In another embodiment, the feature points 355 can be selected as random portions of the object 198 of interest. In another embodiment, the feature points 355 can be selected at regularly spaced intervals on the object 198 of interest, for example, every pixel, every other pixel, every third pixel, every fourth pixel, and so forth. The feature points 355 can take varying shapes and sizes, as desired. In some embodiments, a combination of methods described above can be used to select the feature points 355.

(37) At 924, the selected feature points 355 can be matched from the first frame 811a onto the second frame 811b. In some embodiments, matching of the feature points 355 consists of two procedures as shown in FIG. 8. In FIG. 8, at 924A, a feature point 355 of the first frame 811a can selected. A matching point can be scanned starting from a calculated point and along a line parallel to the centered line of a lens (not shown) being used to capture the two frames 811a, 811b. The matching starting point can be calculated based on the coordinates of the point on the first frame 811a. Although in some embodiments limited to only one direction along the selected line, the scanning can be performed in any of one or more predetermined directions.

(38) At 924B, while scanning for each point, a similarity is calculated between two points in the manner shown and described above in detail herein with reference to FIG. 6, and the point 355 of the second frame 811b with the minimum sum of differences with the feature point 355 of the first frame 811a can be selected as a matching point corresponding to the selected feature point 355.

(39) Returning to FIG. 7, a feature disparity d between each feature points 355 of the two frames 811a and 811b can be found, at 926. Any of a variety of methods can be used to determine the disparity d. In one embodiment, the disparity d can be found based on an average of the disparities d for each of the feature points 355. Exemplary types of averages can include an arithmetic mean, a geometric mean, a median, and/or a mode without limitation. In another embodiment, the disparity d can be found by selecting one or more of the feature points 355 and acquiring the disparity d based on the selected feature points 355.

(40) The described embodiments are susceptible to various modifications and alternative forms, and specific examples thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the described embodiments are not to be limited to the particular forms or methods disclosed, but to the contrary, the present disclosure is to cover all modifications, equivalents, and alternatives.