Vanishing point correction apparatus and method

Abstract

A vanishing point correction method may include the steps of: (a) extracting a contour candidate point of an object considered as the forward vehicle from the image taken by the camera; (b) tracking a movement of the contour candidate point over time; (c) determining whether the contour candidate point belongs to the vehicle; (d) setting the contour candidate point to a vehicle contour point of the forward vehicle, when it is determined at the step (c) that the contour candidate point belongs to the vehicle; (e) calculating a variation in width between vehicle contour points at the same height among the vehicle contour points; and (f) calculating a variation of the vanishing point.

Claims

1. A vanishing point correction method that corrects the position of a vanishing point in an image taken by a camera in an image recognition system mounted in a vehicle, the vanishing point correction method comprising the steps of: (a) extracting a contour candidate point of an object considered as the forward vehicle from the image taken by the camera; (b) tracking a movement of the contour candidate point over time; (c) determining whether the contour candidate point belongs to the vehicle; (d) setting the contour candidate point to a vehicle contour point of the forward vehicle, when it is determined at the step (c) that the contour candidate point belongs to the vehicle; (e) calculating a variation in width between vehicle contour points at the same height among the vehicle contour points; and (f) calculating a variation of the vanishing point through Equation 13 below: $\begin{array}{cc}{y}_{m2}=y_2-\frac{w_2}{w_1}\left(y_1-y_{m1}\right),& \mathrm{Equation}13\end{array}$ where y.sub.1 and y.sub.2 represent heights of the vehicle contour point in images taken by the camera at times t1 and t2, w.sub.1 and w.sub.2 represent the widths between the vehicle contour points at times t1 and t2, y.sub.m1 and y.sub.m2 represent the heights of the vanishing point at times t1 and t2, and the time t2 indicates a predetermined passage of time from the time t1.

2. The vanishing point correction method of claim 1, wherein the step (a) comprises extracting a point at which a horizontal edge and a vertical edges cross each other in the image taken by the camera, as the contour candidate point.

3. The vanishing point correction method of claim 1, wherein the step (b) comprises the steps of: (b-1) defining a first block having a plurality of pixels around the contour candidate point in a first image; (b-2) defining a second block having the same size as the first block in a second image after a predetermined passage of time from the first image; (b-3) determining whether a pixel value of the second block coincides with a pixel value of the first block, while moving the second block to around the contour candidate point based on the same position as the first block in the second image; and (b-4) determining that the contour candidate point has moved in the corresponding block, when it is determined at the step (b-3) that the pixel value of the second block coincides with the pixel value of the first block.

4. The vanishing point correction method of claim 1, wherein the step (c) comprises the steps of: (c-1) extracting the bottom of a rear wheel or the end of a lower shadow area of the forward vehicle, and recognizing the bottom of the rear wheel or the end of the lower shadow area as a distance reference point; (c-2) subtracting a relative distance to the distance reference point from a relative distance to the contour candidate point through Equation 6 below; (c-3) determining whether the subtraction result of the step (c-2) is constant over time within a predetermined error range; and (c-4) determining that the corresponding contour candidate point belongs to the vehicle, when it is determined at the step (c-2) that the subtraction result of the step (c-2) is not constant over time: $\begin{array}{cc}\mathrm{Zc}0-Z0=\frac{h}{\mathrm{yc}0}-\frac{h}{y0},& \mathrm{Equation}6\end{array}$ where Zc0 represents the relative distance to the contour candidate point, Z0 represents the relative distance to the distance reference point, yc0 represents the height of the contour candidate point in the image taken by the camera, and y0 represents the height of the distance reference point in the image taken by the camera.

5. The vanishing point correction method of claim 1, wherein the step (c) comprises the steps of: (c-1) extracting the bottom of a rear wheel or the end of a lower shadow area of the forward vehicle, and recognizing the bottom of the rear wheel or the end of the lower shadow area as a distance reference point; (c-2) calculating a height of the contour candidate point relative to the distance reference point through Equation 7 below; (c-3) determining whether the calculation result of the step (c-2) is constant over time within a predetermined error range; and (c-4) determining that the corresponding contour candidate point belongs to the vehicle, when it is determined at the step (c-3) that the calculation result of the step (c-2) is constant over time: $\begin{array}{cc}\mathrm{Yc}0=h\left(1-\frac{\mathrm{yc}0}{y0}\right),& \mathrm{Equation}7\end{array}$ where Yc0 represents the height of the contour candidate point, yc0 represents a height of the contour candidate point in the image taken by the camera, and y0 represents a height of the distance reference point in the image taken by the camera.

6. A vanishing point correction apparatus that is mounted in a vehicle and corrects the position of a vanishing point in an image taken by a camera, the vanishing point correction apparatus comprising: a contour candidate point extraction unit configured to extract a contour candidate point of an object considered as a forward vehicle from the image taken by the camera; a contour point determination unit configured to track a movement of the contour candidate point, determine whether the contour candidate point belongs to the vehicle, and set the contour candidate point to a vehicle contour point when the contour candidate point belongs to the vehicle; a contour point variation calculation unit configured to calculate a variation in width between vehicle contour points at the same height among the vehicle contour points; and a vanishing point correction unit configured to calculate a vanishing point variation from the width variation calculated through the contour point variation calculation unit, using Equation 13 below: $\begin{array}{cc}{y}_{m2}=y_2-\frac{w_2}{w_1}\left(y_1-y_{m1}\right),& \mathrm{Equation}13\end{array}$ where y.sub.1 and y.sub.2 represent heights of the vehicle contour point in images taken by the camera at times t1 and t2, w.sub.1 and w.sub.2 represent the widths between the vehicle contour points at times t1 and t2, y.sub.m1 and y.sub.m2 represent the heights of the vanishing point at times t1 and t2, and the time t2 indicates a predetermined passage of time from the time t1.

7. The vanishing point correction apparatus of claim 6, wherein the contour candidate point extraction unit extracts a point at which a horizontal edge and a vertical edge cross each other in the image taken by the camera, as the contour candidate point.

8. vanishing point correction apparatus 6, wherein the contour point determination unit comprises a relative distance tracking unit configured to extract the bottom of a rear wheel or the end of a lower shadow area of the forward vehicle, recognize the bottom of the rear wheel or the end of the lower shadow area as a distance reference point, determine whether a result obtained by subtracting a relative distance to the distance reference point from a relative distance to the contour candidate point through Equation 6 below is constant over time, and determine that the corresponding contour candidate point belongs to the vehicle, when the subtraction result is not constant over time: $\begin{array}{cc}\mathrm{Zc}0-Z0=\frac{h}{\mathrm{yc}0}-\frac{h}{y0},& \mathrm{Equation}6\end{array}$ where Zc0 represents the relative distance to the contour candidate point, Z0 represents the relative distance to the distance reference point, yc0 represents a height of the contour candidate point in the image taken by the camera, and y0 represents a height of the distance reference point in the image taken by the camera.

9. The vanishing point correction apparatus of claim 6, wherein the contour point determination unit comprises a height tracking unit configured to extract the bottom of a rear wheel or the end of a lower shadow area of the forward vehicle, recognize the bottom of the rear wheel or the end of the lower shadow area as the distance reference point, determine whether a height of the contour candidate point relative to the distance reference point through Equation 7 below is constant over time, and determine that the contour candidate point belongs to the vehicle, when the height of the contour candidate point is constant over time: $\begin{array}{cc}\mathrm{Yc}0=h\left(1-\frac{\mathrm{yc}0}{y0}\right),& \mathrm{Equation}7\end{array}$ where Yc0 represents the height of the contour candidate point, yc0 represents the height of the contour candidate point in the image taken by the camera, and y0 represents a height of the distance reference point in the image taken by the camera.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 illustrates that a vanishing point is varied in a forward collision warning system.

(2) FIG. 2 is a block diagram illustrating a vanishing point correction apparatus in accordance with an embodiment of the present invention.

(3) FIG. 3 is a flowchart illustrating a vanishing point correction method in accordance with an embodiment of the present invention.

(4) FIG. 4 is a flowchart illustrating a process of determining whether a contour candidate point belongs to a vehicle.

(5) FIG. 5 illustrates a process of extracting a contour candidate point.

(6) FIG. 6 illustrates an example of the process of determining whether a contour candidate point belongs to the vehicle.

MODE FOR INVENTION

(7) Hereafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to specific embodiments, and may include all variations, equivalents and substitutes within the scope of the present invention.

(8) Throughout the specification, the same components are represented by like reference numerals. The accompanying drawings of this specification are for convenience of descriptions, and the shapes and relative scales thereof may be exaggerated or omitted.

(9) When the embodiments are described in detail, the duplicated descriptions or the descriptions of techniques publicly known to those skilled in the art are omitted herein. In the following descriptions, when an element is referred to as including another element, the element may further include other elements unless referred to the contrary.

(10) The terms unit, er and module described in the specification may indicate a unit for processing one or more functions or operations, and the unit can be embodied by hardware, software or a combination of hardware and software. Furthermore, when an element is referred to as being electrically connected to another element, it may not only indicate that the former element is directly connected to the latter element, but also indicate that the former element is connected to the latter element with another element interposed therebetween.

(11) FIG. 2 is a block diagram illustrating a vanishing point correction apparatus in accordance with an embodiment of the present invention. Referring to FIG. 2, the vanishing point correction apparatus includes an image processor 200 for processing an image taken by a camera 100, and a vanishing point correction value outputted by the image processor 200 is transmitted to an ADAS (Advanced Driver Assistance system) such as an FCWS (Forward Collision Warning System) 300, LDWS (Lane Departure Warning System) 400 or LKAS (Lane Keeping Assistant System). For example, the vanishing point correction apparatus may be included in the listed ADAS or another ADAS which is not listed, or configured as a separate apparatus to transmit the vanishing point correction value to the ADAS.

(12) As illustrated in FIG. 2, the image processor 200 includes a contour candidate point extraction unit 210, a contour point determination unit 220, a relative distance tracking unit 230, a height tracking unit 240, a contour point variation calculation unit 250 and a vanishing point correction unit 260.

(13) The contour candidate point extraction unit 210 extracts contour candidate points of an object considered as a forward vehicle from an image taken by the camera 100. The contour point determination unit 220 determines whether the contour candidate points extracted by the contour candidate point extraction unit 210 belong to the vehicle. As illustrated in FIG. 4, the contour point determination unit 220 includes the relative distance tracking unit 230 and the height tracking unit 240.

(14) The contour point variation calculation unit 250 calculates a width variation between vehicle contour points at the same height among the vehicle contour points set by the contour point determination unit 220. The vanishing point correction unit 260 calculates a vanishing point variation using the width variation between the vehicle contour points.

(15) FIG. 3 is a flowchart illustrating a vanishing point correction method in accordance with an embodiment of the present invention, and FIG. 4 is a flowchart illustrating a process of determining whether a contour candidate point belongs to a vehicle. Referring to FIGS. 3 and 4, the operation relations of the respective units and the vanishing point correction method in accordance with the present embodiment will be described in more detail as follows.

(16) First, the contour candidate point extraction unit 210 of the image processor 200 receives image information from the camera 100, and extracts a contour candidate point of an object considered as a forward vehicle, at step ST110. For example, the contour candidate point extraction unit 210 extracts a point at which a horizontal edge and a vertical edge crosses each other in the image taken by the camera 100, as the contour candidate point.

(17) FIG. 5 illustrates the process of extracting a contour candidate point in accordance with the present embodiment. The example of FIG. 5 is based on the supposition that the position of a forward vehicle 510 relative to an ego vehicle 520 based on a coordinate axis can be represented by a coordinate (X, Z), and the width of the forward vehicle 510 is represented by W.

(18) As illustrated in the right side of FIG. 5, a left contour candidate point and a right contour candidate point between contour candidate points of the forward vehicle 510 based on a vanishing point P on an image 500 taken by the camera 100 may be represented by OL.sub.l and OL.sub.r, respectively. In this case, the coordinate of the contour candidate point O.sub.l is (x.sub.l, y.sub.c), and the coordinate of the contour candidate point OL.sub.r is (x.sub.r, y.sub.c). At this time, a width w between the two contour candidate points OL.sub.l and OL.sub.r on the image 500 may be expressed as Equation 5 below, according to a formula of a lens considering the vanishing point.

(19) $\begin{array}{cc}w=.Math.x_l-x_r.Math.=\frac{}{Z}.Math.X_l-X_r.Math.=\frac{W}{Z}& \left[\mathrm{Equation}5\right]\end{array}$

(20) In Equation 5, Z represents a relative distance between the ego vehicle and the forward vehicle, and represents the focal distance of the camera.

(21) That is, the width between the two contour candidate points OL.sub.l and OL.sub.r on the image 500 is proportional to the width of the forward vehicle 510 on the coordinate axis, and inversely proportional to the relative distance between the forward vehicle 510 and the ego vehicle 520.

(22) Then, the contour point determination unit 220 tracks a movement of the contour candidate point at step ST120. For example, the contour point determination unit 220 defines a first block including a plurality of pixels around the contour candidate point in a first image from which the contour candidate point was extracted. Then, the contour point determination unit 220 defines a second block having the same size as the first block in a second image acquired after a predetermined time. While moving the second block of the second image to around the contour candidate point based on the same position as the first block, the contour point determination unit 220 determines whether a pixel value of the second block coincides with a pixel value of the first block. For example, the contour point determination unit 220 may move the second block of the second image along a whirlwind shape. When the pixel value of the second block coincides with the pixel block of the first block, the contour point determination unit 220 determines that the contour candidate point has moved in the corresponding block.

(23) Then, the contour point determination unit 220 determines whether the contour candidate point belongs to the vehicle, at step ST130. The process of determining whether the contour candidate point belongs to the vehicle may be performed through the following two methods or any one of the following two methods.

(24) The first method is to track the relative distance of the contour candidate point through the relative distance tracking unit 230, and the second method is to track the height of the contour candidate point through the height tracking unit 240.

(25) FIG. 6 illustrates an example of the process of determining whether a contour candidate point belongs to the vehicle. Referring to FIG. 6, the process of determining whether a contour candidate point belongs to the vehicle will be described as follows. FIG. 6 illustrates a process of determining whether one contour candidate point belongs to the vehicle, and the determination process is equally applied to other contour candidate points.

(26) Referring to the left side of FIG. 6, the coordinate of a contour candidate point on the image 500 may be assumed to be (xc0, yc0). The y-axis coordinate value of the distance reference point corresponding to the bottom of a rear wheel or the end of a lower shadow area of the forward vehicle 510 is represented by y0.

(27) When the contour candidate point (xc0, yc0) of FIG. 6 does not belong to the vehicle, the contour candidate point is projected onto a point Zc0 on the road by the camera 100 as illustrated in the right top graph of FIG. 6. When the Z-axis coordinate value of the distance reference point on the actual coordinate axis is represented by Z0, an object on the road has a length corresponding to a subtraction of Z0 from Zc0, and the length may be expressed as Equation 6 below.

(28) $\begin{array}{cc}\mathrm{Zc}0-Z0=\frac{h}{\mathrm{yc}0}-\frac{h}{y0}& \left[\mathrm{Equation}6\right]\end{array}$

(29) In Equation 6, Zc0 represents a relative distance to the contour candidate point, Z0 represents a relative distance to the distance reference point, yc0 represents the height of the contour candidate point in the image taken by the camera, and y0 represents the height of the distance reference point in the image taken by the camera.

(30) When the contour candidate point (xc0, yc0) of FIG. 6 belongs to the vehicle, the contour candidate point is projected onto a point corresponding to the height Yc0 on the Z-axis distance Z0 of the distance reference point as illustrated in the right bottom graph of FIG. 6, and the height may be expressed as Equation 7.

(31) $\begin{array}{cc}\mathrm{Yc}0=h\left(1-\frac{\mathrm{yc}0}{y0}\right)& \left[\mathrm{Equation}7\right]\end{array}$

(32) In Equation 7, Yc0 represents the height of the contour candidate point, yc0 represents the height of the contour candidate point in the image taken by the camera, and y0 represents the height of the distance reference point in the image taken by the camera.

(33) When the subtraction result of Equation 6 over time is constant within a predetermined error range, the corresponding contour candidate point may be determined to be an object on the road. In this case, it may be considered that the extracted contour candidate point does not belong to the vehicle. Furthermore, when the height of Equation 7 over time is constant within a predetermined error range, the corresponding contour candidate point may be determined to be a portion of the vehicle. In this case, it may be considered that the extracted contour candidate point belongs to the vehicle.

(34) Referring back to FIG. 4, step ST130 of determining whether the contour candidate point belongs to the vehicle begins with step ST131 of extracting the distance reference point.

(35) The first relative distance tracking process is performed as follows. The contour point determination unit 220 subtracts the relative distance to the distance reference point from the relative distance to the contour candidate point through Equation 6 at t=0 at step ST132, and performs the same subtraction through Equation 6 at t=1 after a predetermined time at step ST134. Then, the contour point determination unit 220 determines whether the subtraction result at t=0 and the subtraction result at t=1 are equal to each other within a predetermined error range, at step ST136. When the subtraction result at t=0 and the subtraction result at t=1 are equal to each other, the procedure returns to the step ST110 to extract a new contour candidate point. On the other hand, when the subtraction result at t=0 and the subtraction result at t=1 are not equal to each other, the contour point determination unit 220 sets the corresponding contour candidate point to a vehicle contour point at step ST140.

(36) The second height tracking process is performed as follows. The contour point determination unit 220 calculates the height of the contour candidate point relative to the distance reference point through Equation 7 at t=0 at step ST133, and performs the same height calculation through Equation 7 at t=1 after a predetermined time at step ST135. Then, the contour point determination unit 220 determines whether the calculation result at t=0 and the calculation result at t=1 are equal to each other within a predetermined error range, at step ST136.

(37) When the calculation result at t=0 and the calculation result at t=1 are not equal to each other, the contour point determination unit 220 sets the corresponding contour candidate point to a vehicle contour point at step ST140. On the other hand, when the calculation result at t=0 and the calculation result at t=1 are not equal to each other, the procedure returns to the step ST110 to extract a new contour candidate point.

(38) Next, the contour point variation calculation unit 250 calculates a width variation between the vehicle contour points at the same height among the vehicle contour points as illustrated in FIG. 3, at step ST150. Then, the vanishing point correction unit 260 calculates a variation of the vanishing point using the width variation between the vehicle contour points at step ST160. Hereafter, times t1 and t2 in the following descriptions for the process of calculating the vanishing point variation indicate passages of time.

(39) When the position of the vanishing point at time t1 is represented by y.sub.m1, the relative distance to the forward vehicle 510 may be acquired through Equation 8 below, as described with reference to Equation 4 above. In Equation 4, since the distance reference point has been obtained based on the bottom of the forward vehicle 510, the height of the coordinate axis relative to the ground surface may be set to 0 (Y=0). In the present embodiment, the vehicle contour point is extracted from the contour line of the forward vehicle 510, unlike the conventional method for acquiring a distance reference point. Therefore, the height Y of the vehicle contour point on the coordinate axis needs to be considered.

(40) $\begin{array}{cc}{Z}_1=\frac{h-Y}{y_1-y_{m-1}}& \left[\mathrm{Equation}8\right]\end{array}$

(41) In Equation 8, Z.sub.1 represents the relative distance to the forward vehicle at time t1, Y represents the height of the vehicle contour point on the coordinate axis, y.sub.1 represents the height of the vehicle contour point in the image taken by the camera at time t1, and y.sub.m1 represents the height of the vanishing point at time t1.

(42) When the position of the vanishing point at time t2 after a predetermined time is varied to y.sub.m2, the relative distance to the forward vehicle may be expressed as Equation 9 below.

(43) $\begin{array}{cc}{Z}_2=\frac{h-Y}{y_2-y_{m2}}& \left[\mathrm{Equation}9\right]\end{array}$

(44) In Equation 9, Z.sub.2 represents the relative distance to the forward vehicle at time t2, Y represents the height of the vehicle contour point on the coordinate axis, y.sub.2 represents the height of the vehicle contour point in an image taken by the camera at time t2, and y.sub.m2 represents the height of the vanishing point at time t2.

(45) In the conventional method, however, the variation y.sub.m2 of the vanishing point is not recognized, and the vanishing point information y.sub.m1 at time t1 is used to calculate the relative distance Z.sub.2 at time t2. Therefore, a false warning may be caused. The present invention provides a method of calculating the variation y.sub.m2 of the vanishing point using the width variation between the vehicle vanishing points with reference to Equation 5 above.

(46) Referring to Equation 5, the denominator of the right side in Equation 8 may be expressed as Equation 10 below.

(47) $\begin{array}{cc}{y}_1-y_{m1}=\frac{}{Z_1}\left(h-Y\right)=\frac{w_1}{W}\left(h-Y\right)& \left[\mathrm{Equation}10\right]\end{array}$

(48) In Equation 10, w.sub.1 represents a width between the vehicle contour points at time t1.

(49) Furthermore, the denominator of the right side in Equation 9 may also be expressed as Equation 11 below.

(50) $\begin{array}{cc}{y}_2-y_{m2}=\frac{}{Z_2}\left(h-Y\right)=\frac{w_2}{W}\left(h-Y\right)& \left[\mathrm{Equation}11\right]\end{array}$

(51) In Equation 11, w2 represents a width between the vehicle contour points at time t2.

(52) In Equations 10 and 11, the width W of the forward vehicle on the coordinate axis, the height h of the camera, and the height Y of the vehicle contour point on the coordinate axis are all constants. Therefore, a proportional expression of Equations 10 and 11 may be expressed as Equation 12 below.

(53) $\begin{array}{cc}\frac{y_2-y_{m2}}{y_1-y_{m1}}=\frac{w_2}{w_1}& \left[\mathrm{Equation}12\right]\end{array}$

(54) Now, the width variation between the vehicle contour points on the camera may be used to obtain a y-axis variation of the vanishing point through Equation 13 below.

(55) $\begin{array}{cc}{y}_{m2}=y_2-\frac{w_2}{w_1}\left(y_1-y_{m1}\right)& \left[\mathrm{Equation}13\right]\end{array}$

(56) That is, the vanishing point correction apparatus and method can calculate the vanishing point variation by measuring the width variation between the vehicle contour points extracted from the image, and correct the position of the vanishing point using the vanishing point variation, thereby reducing a false warning which may occur when a vanishing point variation is not recognized by an ADAS such as FCWS.

(57) While various embodiments have been described above, it will be understood to those skilled in the art that the embodiments described are by way of example only. Accordingly, the disclosure described herein should not be limited based on the described embodiments.