METHOD OF ESTIMATING THREE-DIMENSIONAL COORDINATE VALUE FOR EACH PIXEL OF TWO-DIMENSIONAL IMAGE, AND METHOD OF ESTIMATING AUTONOMOUS DRIVING INFORMATION USING THE SAME

Abstract

Proposed are a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, and a method of estimating autonomous driving information using the same, and more specifically, a method that can efficiently acquire information needed for autonomous driving using a mono camera. This method is able to acquire information having sufficient reliability in real-time without using expensive equipment such as a high-precision GPS receiver, a stereo camera or the like required for autonomous driving.

Claims

1. A method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, the method comprising: a camera height input step of receiving height of a mono camera installed in parallel to ground; a reference value setting step of setting at least one among a vertical viewing angle, an azimuth angle, and a resolution of the mono camera; and a pixel coordinate estimation step of estimating a three-dimensional coordinate value for at least some of pixels with respect to ground of the two-dimensional image captured by the mono camera, based on the inputted height of the mono camera and a set reference value.

2. The method according to claim 1, wherein the pixel coordinate estimation step includes a modeling process of estimating the three-dimensional coordinate value by generating a three-dimensional point using a pinhole camera model.

3. The method according to claim 2, wherein the pixel coordinate estimation step further includes, after the modeling process, a lens distortion correction process of correcting distortion generated by a lens of the mono camera.

4. The method according to claim 1, further comprising, after the pixel coordinate estimation step, a non-corresponding pixel coordinate estimation step of estimating a three-dimensional coordinate value of a pixel that is not corresponding to the three-dimensional coordinate value among the pixels of the two-dimensional image from a pixel corresponding to the three-dimensional coordinate value using a linear interpolation method.

5. A method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, the method comprising: a two-dimensional image acquisition step of acquiring the two-dimensional image captured by a mono camera; a coordinate system matching step of matching each pixel of the two-dimensional image and a three-dimensional coordinate system; and an object distance estimation step of estimating a distance to an object included in the two-dimensional image.

6. The method according to claim 5, wherein the coordinate system matching step includes the method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image of claim 4, and the object distance estimation step includes an object location calculation process of confirming the object included in the two-dimensional image, and estimating a direction and a distance to the object based on the three-dimensional coordinate value corresponding to each pixel.

7. The method according to claim 6, wherein at the object location calculation step, a distance to a corresponding object is estimated using a three-dimensional coordinate value corresponding to a pixel corresponding to the ground of the object included in the two-dimensional image.

8. A method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, the method comprising: a two-dimensional image acquisition step of acquiring the two-dimensional image captured by a mono camera; a coordinate system matching step of matching each pixel of the two-dimensional image and a three-dimensional coordinate system; and a semantic information location estimation step of estimating a three-dimensional coordinate value of semantic information for autonomous driving included in the ground of the two-dimensional image.

9. The method according to claim 8, wherein the coordinate system matching step includes the method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image of claim 4, and further includes, after the semantic information location estimation step, a localization step of confirming a location of a corresponding vehicle on a HD-map for autonomous driving based on the three-dimensional coordinate value of semantic information for autonomous driving.

10. The method according to claim 9, wherein the localization step includes: a semantic information confirmation process of confirming corresponding semantic information for autonomous driving on the HD-map for autonomous driving; and a vehicle location confirmation process of confirming a current location of the vehicle on the HD-map for autonomous driving by applying a relative location with respect to the semantic information for autonomous driving.

11. The method according to claim 2, further comprising, after the pixel coordinate estimation step, a non-corresponding pixel coordinate estimation step of estimating a three-dimensional coordinate value of a pixel that is not corresponding to the three-dimensional coordinate value among the pixels of the two-dimensional image from a pixel corresponding to the three-dimensional coordinate value using a linear interpolation method.

12. The method according to claim 3, further comprising, after the pixel coordinate estimation step, a non-corresponding pixel coordinate estimation step of estimating a three-dimensional coordinate value of a pixel that is not corresponding to the three-dimensional coordinate value among the pixels of the two-dimensional image from a pixel corresponding to the three-dimensional coordinate value using a linear interpolation method.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG. 1 is a flowchart illustrating an embodiment of a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention.

[0039] FIGS. 2 to 4 are views for describing each step of FIG. 1 in detail.

[0040] FIG. 5 is a flowchart illustrating another embodiment of FIG. 1.

[0041] FIG. 6 is a flowchart illustrating an embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention.

[0042] FIGS. 7 and 8 are views describing step S300 shown in FIG. 3.

[0043] FIGS. 9 to 12 are views describing step S400 shown in FIG. 3.

[0044] FIG. 13 is a flowchart illustrating another embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention.

[0045] FIGS. 14 and 15 are views describing FIG. 13.

[0046] FIG. 16 is a flowchart illustrating yet another embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention.

[0047] FIGS. 17 and 18 are views describing FIG. 16.

BEST MODE FOR CARRYING OUT THE INVENTION

[0048] Examples of a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, and a method of estimating autonomous driving information using the same according to the present invention may be diversely applied, and hereinafter, a most preferred embodiment will be described with reference to the accompanying drawings.

[0049] FIG. 1 is a flowchart illustrating an embodiment of a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention, and FIGS. 2 to 4 are views for describing each step of FIG. 1 in detail.

[0050] Referring to FIG. 1, a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image includes a camera height input step (S110), a reference value setting step (S120), and a pixel coordinate estimation step (S130).

[0051] The camera height input step (S110) is a process of receiving the height (h) of a mono camera installed in parallel to the ground as shown in FIG. 2, and a driver (user) of a vehicle equipped with the mono camera may input the height, or a distance measurement sensor may be configured on one side of the mono camera to automatically measure the distance to the ground, and in addition, the height of the mono camera may be measured and input in various ways in response to a request of those skilled in the art.

[0052] The reference value setting step (S120) is a process of setting at least one among the vertical viewing angle (θ), azimuth angle (φ), and resolution of the mono camera as shown in FIGS. 2 and 3, and it goes without saying that frequently used values may be set in advance or may be input and changed by a user.

[0053] The pixel coordinate estimation step (S130) is a process of estimating a three-dimensional coordinate value for at least some of the pixels with respect to the ground of the two-dimensional image captured by the mono camera, based on the inputted height of the mono camera and a previously set reference value, and it will be described below in detail.

[0054] First, referring to FIG. 2, the distance d to the ground according to the height h and the vertical viewing angle θ of the mono camera may be expressed as shown in Equation 1.

d=h/sin θ  (Equation 1)

[0055] In addition, as shown in FIG. 3, three-dimensional coordinates of a three-dimensional point generated on the ground may be determined by the azimuth φ and the resolution. Here, the three-dimensional point is a point displayed on the ground from the viewpoint of the mono camera, and may correspond to a pixel of a two-dimensional image in the present invention.

[0056] For example, a three-dimensional point X, Y, and Z with respect to the ground may be expressed as shown in Equation 2 in terms of distance d, height h, vertical viewing angle θ, and the azimuth angle φ of the mono camera.

X=d cos θ sin Ø

Y=d cos θ cos Ø

Z=−h  (Equation 2)

[0057] Thereafter, a three-dimensional coordinate value may be estimated by generating a three-dimensional point using a pinhole camera model.

[0058] FIG. 4 is a view showing a relation and a corresponding view between the pixel of a two-dimensional image with respect to the ground and a three-dimensional point using a pinhole camera model, and each of the rotation matrixes Rx, Ry and Rz for roll, pitch and yaw may be expressed as in Equation 3.

[00001]$\begin{array}{cc}{R}_x\left(\alpha \right)=\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \alpha & -\sin \alpha \\ 0& \sin \alpha & \cos \alpha \end{array}\right]R_y\left(\beta \right)=\left[\begin{array}{ccc}\cos \beta & 0& \sin \beta \\ 0& 1& 0\\ -\sin \beta & 0& \cos \beta \end{array}\right]R_z\left(\gamma \right)=\left[\begin{array}{ccc}\cos \gamma & -\sin \mathrm{\gamma 0}& 0\\ \sin \gamma & \cos \gamma & 0\\ 0& 0& 1\end{array}\right]& \left(\mathrm{Equation}3\right)\end{array}$

[0059] In addition, rotation matrix R for transforming the three-dimensional coordinate system of the mono camera's viewpoint into the coordinate system of a two-dimensional image may be expressed as shown in Equation 4.

R=R.sub.z(γ)R.sub.y(β)R.sub.x(α)  (Equation 4)

[0060] Finally, in order to transform a point X, Y and Z of the three-dimensional coordinate system to a point of a two-dimensional image of the camera's viewpoint, the point of the three-dimensional coordinate system is multiplied by rotation matrix R as shown in Equation 5.

[00002]$\begin{array}{cc}\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=R\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]& \left(\mathrm{Equation}5\right)\end{array}$

[0061] In this way, when the modeling process (S131) shown in FIG. 5 is performed, a lens distortion correction process (S132) of correcting distortion generated by the lens of the mono camera may be performed thereafter.

[0062] Generally, since a lens of a camera does not have a perfect curvature, distortion is generated in an image, and in order to estimate an accurate location, calibration for correcting the distortion is performed.

[0063] When external parameters of the mono camera are calculated through calibration of the mono camera, radial distortion coefficients k1, k2, k3, k4, k5 and k6 and tangential distortion coefficients p1 and p2 may be obtained.

[0064] The process as shown in Equation 6 is developed using the external parameters.

[00003]$\begin{array}{cc}\begin{array}{c}{x}^{\prime }=x/z\\ y^{\prime }=y/z\\ x^{″}=x^{\prime }\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+2p_1{x}^{\prime }{y}^{\prime }+p_2\left(r^2+2x^{\mathrm{\prime 2}}\right)\\ y^{″}=y^{\prime }\frac{1+k_1{r}^2+k_2{r}^4+k_3{r}^6}{1+k_4{r}^2+k_5{r}^4+k_6{r}^6}+p_1\left(r^2+2y^{\prime 2}\right)+2p_2{x}^{\prime }{y}^{\prime }\end{array}& \left(\mathrm{Equation}6\right)\end{array}$$\left(\mathrm{here},r^2=x^{\prime 2}+y^{\prime 2}\right)$

[0065] The relational equations of the image coordinate systems u and v obtained using the two points obtained before, focal lengths f.sub.x and f.sub.y, which are internal parameters of the mono camera, and principal points cx and cy are as shown in Equation 7.

u=f.sub.x*x″+c.sub.x

v=f.sub.y*y″+c.sub.y  (Equation 7)

[0066] In the process as described above, when the height of the mono camera and the pinhole camera model are used, pixels and three-dimensional points corresponding to the ground may be calculated.

[0067] Hereinafter, the process described above will be described using an image actually captured by a mono camera.

[0068] FIG. 6 is a flowchart illustrating an embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention, and FIGS. 7 and 12 are views describing the steps after step S130 shown in FIG. 3.

[0069] First, FIGS. 7 and 8 are views showing three-dimensional points at the pixels corresponding to the ground of a two-dimensional image through the process described above at the pixel coordinate estimation step (S130). As is understood from the enlarged portion, it can be seen that the spaces between the points are empty.

[0070] Referring to FIG. 6, when a three-dimensional coordinate value of a pixel that does not correspond to the coordinate value of a three-dimensional point among the pixels of the two-dimensional image is estimated after the pixel coordinate estimation step (S130) from a pixel corresponding to the coordinate value of the three-dimensional point using a linear interpolation method as shown in the enlarged portions of FIGS. 7 and 8 (S140), the three-dimensional point may be displayed as shown in FIGS. 9 to 12.

[0071] Here, FIGS. 9 and 10 show a view applying the linear interpolation method in the left and right directions, and FIGS. 11 and 12 show a view applying the linear interpolation method in the forward and backward directions after applying the linear interpolation method in the left and right directions.

[0072] The data passing through the process may be used at an object location calculation step S151, a localization step S152, and the like, and this will be described below in more detail.

[0073] FIG. 13 is a flowchart illustrating another embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention, and FIGS. 14 and 15 are views describing FIG. 13.

[0074] Referring to FIG. 13, the method of estimating autonomous driving information according to the present invention includes a two-dimensional image acquisition step (S210), a coordinate system matching step (S220), and an object distance estimation step (S230).

[0075] Describing in detail, a two-dimensional image captured by a mono camera is acquired at the two-dimensional image acquisition step (S210), and each pixel of the two-dimensional image and a three-dimensional coordinate system are matched at the coordinate system matching step (S220), and a distance to an object included in the two-dimensional image is estimated at the object distance estimation step (S230).

[0076] At this point, the coordinate system matching step (S220) may estimate a three-dimensional coordinate value for each pixel of the two-dimensional image through processes ‘S110’ to ‘S140’ of FIG. 6 described above.

[0077] Thereafter, at the object distance estimation step (S230), an object location calculation process of confirming an object (vehicle) included in the two-dimensional image as shown in FIG. 14, and estimating a direction and a distance to the object based on a three-dimensional coordinate value corresponding to each pixel may be performed.

[0078] Specifically, at the object location calculation process, a distance to a corresponding object may be estimated using a three-dimensional coordinate value corresponding to a pixel corresponding to the ground (the ground on which the vehicle is located) of the object included in the two-dimensional image.

[0079] FIG. 14 is a view showing a distance to a vehicle in front estimated according to the present invention, and as shown in FIG. 14, the distance to the vehicle estimated using the pixels at the lower ends of both sides of the bounding box recognizing the vehicle in front and the width and height of the bounding box is 7.35 m.

[0080] In addition, the distance measured using LiDAR in the same situation is about 7.24 m as shown in FIG. 15, and although an error of about 0.11 m with respect to FIG. 14 may occur, when the distance only to the ground on which the object is located is estimated, the accuracy may be further improved.

[0081] FIG. 16 is a flowchart illustrating another embodiment of a method of estimating autonomous driving information using a method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image according to the present invention, and FIGS. 17 and 18 are views describing FIG. 16.

[0082] Referring to FIG. 16, the method of estimating autonomous driving information according to the present invention includes a two-dimensional image acquisition step (S310), a coordinate system matching step (S320), and a semantic information location estimation step (S330).

[0083] Describing in detail, a two-dimensional image captured by a mono camera is acquired at the two-dimensional image acquisition step (S310), and each pixel of the two-dimensional image and a three-dimensional coordinate system are matched at the coordinate system matching step (S320), and a three-dimensional coordinate value of semantic information for autonomous driving included in the ground of the two-dimensional image is estimated at the semantic information location estimation step (S330).

[0084] At this point, the coordinate system matching step (S320) may estimate a three-dimensional coordinate value for each pixel of the two-dimensional image through processes ‘S110’ to ‘S140’ of FIG. 6 described above.

[0085] In addition, after the semantic information location estimation step (S330), a localization step (S340) of confirming the location of a corresponding vehicle (a vehicle equipped with a mono camera) on a high-definition map (HD-map) for autonomous driving based on the three-dimensional coordinate value of the semantic information for autonomous driving may be further included.

[0086] Particularly, the localization step (S340) may perform a semantic information confirmation process of confirming corresponding semantic information for autonomous driving on the HD-map for autonomous driving, and a vehicle location confirmation process of confirming the current location of a vehicle on the HD-map for autonomous driving by applying a relative location with respect to the semantic information for autonomous driving.

[0087] In other words, as shown in FIG. 17, when the three-dimensional coordinate value of the semantic information for autonomous driving (e.g., lanes) included in the ground of the two-dimensional image is estimated (S330), as shown in FIG. 18, corresponding semantic information may be confirmed on the HD-map, and the location of a corresponding vehicle (a vehicle equipped with a mono camera) may be grasped using a relative direction and distance with respect to the confirmed semantic information (S340).

[0088] A method of estimating a three-dimensional coordinate value for each pixel of a two-dimensional image, and a method of estimating autonomous driving information using the same according to the present invention have been described above. It will be appreciated that those skilled in the art may implement the technical configuration of the present invention in other specific forms without changing the technical spirit or essential features of the present invention.

[0089] Therefore, it should be understood that the embodiments described above are illustrative and not restrictive in all respects.