Disparity computation method through stereo matching based on census transform with adaptive support weight and system thereof

Abstract

Provided is a method of computing precise disparity using a stereo matching method based on developed census transform with an adaptive support weight method in area based stereo matching. The method includes a step of setting an adaptive support weight window centered on a specific point of a left image and setting adaptive support weight windows with the same size with respect to one point positioned within a maximum disparity prediction value about a specific point of the left image in a right image.

Claims

1. A disparity calculation method through stereo matching based on census transform with an adaptive support weight, the method comprising: a first step of setting an adaptive support weight window centered on a specific point of a left image and setting adaptive support weight windows with the same size with respect to one point positioned within a maximum disparity prediction value about a specific point of the left image in a right image; a second step of computing an adaptive support weight for pixels in each of the weight windows of the left image and the right image; a third step of setting a matching window for pixels in the adaptive support weight windows set in the left image and the right image; a fourth step of computing a raw matching cost for pixels in the adaptive support weight windows using the matching window; a fifth step of computing a computed matching cost using the computed adaptive support weight and the raw matching cost; and a sixth step of computing a matching cost for other points positioned within a maximum disparity prediction value with respect to a specific point of the left image in the right image, and then computing a disparity value through comparison of the computed matching cost.

2. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 1, wherein the adaptive support weight is computed using a difference in a brightness value between a center pixel and other pixels of the adaptive support weight window.

3. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 2, wherein the adaptive support weight is computed by the following equation, $w (p, q) = \exp (- \frac{Δ p_{pq}}{γ_{p}})$ where, γ.sub.p is a constant coefficient, and Δp.sub.pq is a difference of brightness values of cells.

4. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 1, wherein the first and second steps and the third and fourth steps are simultaneously performed in parallel.

5. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 1, wherein the raw matching cost is computed through the stereo matching based on census transform.

6. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 5, wherein in the fifth step, the matching cost is computed by the following equation, $D_{ASW} (p, p_{d}) = \frac{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d}) e (q, q_{d})}{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d})}$ Where, N.sub.p and N.sub.pd are corresponding pixels in the left image window and the right image window, respectively, and e(q, q.sub.d) is a raw matching cost computed through the stereo matching based on census transform.

7. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 6, wherein the fifth step of computing the matching cost using the equation includes a fifth-1 step of summing numerators of the equation 3, a fifth-2 step of summing denominators of the equation 3, and a fifth-3 step of dividing a value of the summed numerators by a value of the summed denominators.

8. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 7, wherein the fifth-1 step and the fifth-2 step are simultaneously performed in parallel.

9. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 1, wherein the adaptive support weight window and the matching window are sparse-type windows using only selected pixels in a window area.

10. The disparity calculation method through stereo matching based on census transform with an adaptive support weight according to claim 9, wherein the adaptive support weight window and the matching window are different types of sparse-type windows.

11. A disparity computation system through stereo matching based on census transform with an adaptive support weight, the system comprising: a weight computation unit that computes an adaptive support weight for pixels in an adaptive support weight window set in each of a left image and a right image; a census transformation unit that sets a matching window for pixels in the adaptive support weight window, and computes a raw matching cost for pixels in the adaptive support weight window using the matching window; a matching cost computation unit that computes a matching cost using the computed adaptive support weight of pixels and the raw matching cost; and a disparity computation unit that computes matching costs for other points positioned within a maximum disparity prediction value with respect to a specific point of the left image in the right image, and then computes a disparity value through comparison of the computed matching costs.

12. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the weight computation unit includes a left adaptive support weight computation unit that computes an adaptive support weight for pixels in a left adaptive support weight window, and a right adaptive support weight computation unit that computes an adaptive support weight for pixels in a right adaptive support weight window.

13. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the census transformation unit includes a left hamming weight computation unit that computes a hamming weight using a brightness difference between a center pixel and other pixels in the left matching window, a right hamming weight computation unit that computes a hamming weight using a brightness difference between a center pixel and other pixels in the right matching window, and a hamming distance computation unit that computes a hamming distance using the computed left hamming weight and right hamming weight.

14. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the matching cost computation unit that computes the matching cost by the following equation, $D_{ASW} (p, p_{d}) = \frac{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d}) e (q, q_{d})}{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d})}$ where, N.sub.p and N.sub.pd are corresponding pixels in the left image window and the right image window,respectively, and e(q, q.sub.d) is a raw matching cost computed through the stereo matching based on census transform.

15. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 14, wherein the matching cost computation unit includes a numerator summation unit that performs sums the numerator values of the equation, a denominator summation unit that sums the denominator values of the equation, and a division unit that divides the numerator sum value by the denominator sum value.

16. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the weight computation unit and the census transformation unit are simultaneously driven in parallel.

17. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 14, wherein the numerator summation unit and the denominator summation unit are simultaneously driven in parallel.

18. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the adaptive support weight is computed using a difference in a brightness value between a center pixel and other pixels of the adaptive support weight window.

19. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the adaptive support weight window and the matching window are sparse-type windows using only selected pixels in a window area.

20. The disparity computation system through stereo matching based on census transform with an adaptive support weight according to claim 11, wherein the adaptive support weight window and the matching window are different types of sparse-type windows.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The above and other objects, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

(2) FIG. 1 illustrates a stereo matching method based on census transform using a disparity search range with a regular window size and a regular size in left and right images;

(3) FIG. 2 illustrates a method of computing a hamming weight window in a set window;

(4) FIG. 3 illustrates a method of generating a hamming bit window using left and right hamming weight windows;

(5) FIG. 4 is a diagram illustrating various examples of window shapes used in matching cost computation;

(6) FIG. 5 is a flowchart illustrating a process of performing a disparity computation method through stereo matching based on census transform with an adaptive support weight according to the invention;

(7) FIG. 6 is a diagram for explaining a method of acquiring disparity information with high reliability by fusing a stereo matching method and an adaptive support weight method according to an embodiment of the invention;

(8) FIG. 7 is a block diagram illustrating a configuration of a disparity computation system through stereo matching based on census transform with an adaptive support weight according to the invention;

(9) FIG. 8 illustrates a weight value based on a brightness difference according to an embodiment of the invention, and a pseudo code for acquiring the weight value;

(10) FIG. 9A and FIG. 9B are diagrams illustrating examples of various window shapes according to an embodiment of the invention;

(11) FIG. 10 is a diagram illustrating a use method of an adaptive support weight window and a matching window for stereo matching based on census transform according to an embodiment of the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

(12) Hereinafter, a preferred embodiment of the invention will be described in detail with reference to the accompanying drawings.

(13) FIG. 5 is a flowchart illustrating a process of performing a disparity computation method through stereo matching based on census transform with an adaptive support weight according to the invention, FIG. 6 is a diagram for explaining a method of acquiring disparity information with high reliability by fusing a stereo matching method based on census transform and an adaptive support weight method according to the embodiment of the invention, FIG. 7 is a block diagram illustrating a configuration of a disparity computation system through stereo matching based on census transform with an adaptive support weight according to the invention, and FIG. 8 illustrates a weight value based on a brightness difference and a pseudo code for acquiring the weight value according to the embodiment of the invention.

(14) Hereinafter, a method of acquiring disparity information with high reliability by fusing a stereo matching method based on census transform and an adaptive support weight method according to the invention will be described with reference to FIG. 5.

(15) First, a window for computing a matching cost is set in each of a left image and a right image (S100-1 and S100-2). As shown in FIG. 6, in order to find a point of the right image corresponding to a specific point P of the left image, an adaptive support weight window (W.sub.al, hereinafter, referred to as “weight window”) centered on the specific point P of the left image is set. In an embodiment of FIG. 6, the weight window W.sub.al with 5×5 size is exemplified. A weight window W.sub.ar with the same size is set for one point P.sub.d positioned within a maximum disparity prediction value with respect to the point P of the left image in the right image.

(16) When the weight windows W.sub.al and W.sub.ar are set in the left image and the right image, respectively, adaptive support weights (hereinafter, referred to as “weight”) are computed for all the pixels in each of the windows W.sub.al and W.sub.ar (S110-1 and S110-3).

(17) In the invention, only a brightness value difference is used to reduce an operation for weight computation, and the weight is computed by the following equation 2.

(18) $\begin{matrix} w (p, q) = \exp (- \frac{Δ p_{pq}}{γ_{p}}) & [Equation 2] \end{matrix}$

(19) γ.sub.p: constant coefficient

(20) Δp.sub.pq: difference of brightness values of cells

(21) That is, it can be known that the weight is exponential-functionally decreased as the brightness difference the center pixel P and adjacent pixels gets larger, and a weight computation function can be embodied by a simple pseudo code shown in FIG. 8. In the weight computation function shown in FIG. 8, the weight is exponentially decreased from the maximum 64 to 0.

(22) The weights are computed for all the pixels in the weight windows W.sub.al and W.sub.ar in the left image and the right image using the equation 2. Such computation can be embodied by hardware, and in order thereto, the disparity computation system according to the invention may include a weight computation unit 200 including a left adaptive support weight computation unit 210 and a right adaptive support weight computation unit 220.

(23) Simultaneously with the weight computation, census transform based on stereo is performed (S110-2). As described above, the weight computation and the census transform of the left image and the right image are simultaneously performed in parallel, and thus there is an advantage of shortening computation time.

(24) For all the pixels in the weight windows W.sub.al and W.sub.ar set in the left image and the right image for the census transform based on stereo, matching windows W.sub.cl and W.sub.cr for acquiring a raw matching cost are set.

(25) In FIG. 6, the matching window W.sub.cl with a regular size (3×3 in the embodiment) is set for the pixels in the left weight window W.sub.al, and the matching window W.sub.cr with the same size is also set for the corresponding pixels in the right weight window W.sub.ar. FIG. 6 illustrates that the matching windows W.sub.cl and W.sub.ar are set for the pixel q in the left weight window W.sub.al and the pixel q.sub.d which is the corresponding pixel in the right weight window W.sub.ar, respectively.

(26) The stereo matching method based on census transform is applied to the left matching window W.sub.cl and the right matching window W.sub.cr to compute a raw matching cost between the pixel q in the weight window W.sub.al and the pixel q.sub.d which is the corresponding pixel in the right weight window W.sub.ar. Such computation may be embodied by hardware, and in order thereto, the disparity computation system according to the invention may include a census transform unit 100 including a left hamming weight computation unit 110, a right hamming weight computation unit 120, and a hamming distance computation unit 130.

(27) In the description, when the weight and the raw matching cost are computed for the left and right images, D.sub.ASW(p, p.sub.d) is computed as a matching cost using the following equation 3.

(28) $\begin{matrix} D_{ASW} (p, p_{d}) = \frac{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d}) e (q, q_{d})}{\underset{q \in N_{p}, q_{d} \in N_{p_{d}}}{.Math.} w (p, q) w (p_{d}, q_{d})} & [Equation 3] \end{matrix}$

(29) Herein, N.sub.p and N.sub.pd are the corresponding pixels in the left image window and the right image window area, respectively, and e(q, q.sub.d) represents a raw matching cost computed through the stereo matching based on census transform differently from the equation 1 according to the related art.

(30) The process of computing the matching cost using the equation 3 is performed through the process (S120-1) of summing the numerators of the equation 3, the process of (S120-2) of summing the denominators of the equation 3, and then the process (S130) of dividing the summed numerator value by the denominator value. Such a process of summing and dividing the numerator and denominator value can be embodied by hardware, and in order thereto, the disparity computation system according to the invention may include a matching cost computation unit 300 including a numerator summation unit 310, a denominator summation unit 320, and a division unit 330.

(31) It is possible to improve an operation speed by also simultaneously performing the numerator computation process (S120-1) and the denominator computation process (S120-2).

(32) The matching costs described above are computed for the point p of the left image and all the points in the maximum disparity prediction range corresponding to the point p in the right image, and then a disparity value is computed through comparison of the computed matching costs (S140). Such disparity computation can also be embodied by hardware, and the disparity computation system according to the invention may include a disparity computation unit 400.

(33) FIG. 9A and FIG. 9B are diagrams illustrating examples of various window shapes according to a modified embodiment of the invention, and FIG. 10 is a diagram illustrating a method of using a weight window and a matching window according to the modified embodiment of the invention.

(34) FIG. 9A illustrates an example of a sparse weight window, FIG. 9B illustrates an example of a sparse matching window, and the modified embodiment of the invention is characterized in that a part of a window area is not used in stereo matching and weight computation.

(35) More specifically, FIG. 9A illustrates a circle weight window representing the use of eight points in a window with 7×7 size, and FIG. 9B illustrates a sparse matching window representing the use of eight points in a window with 5×5 size. The weight window determines whether or not to use the window area used in the weight computation processes (S110-1 and S110-2), and the matching window determines whether or not to use the window area used in the census transform process (S110-2).

(36) That is, in the equation 3, Np and Npd represent selected pixels in the left and right sparse window areas, and it is possible to reduce computation time by performing census transformation and the weight computation only for selected pixels.

(37) FIG. 10 illustrates a method of computing a matching cost using the weight window and the matching window exemplified in FIG. 9, the matching window is set only for selected pixels preset in the weight window, the stereo matching based on census transform is performed, and operation time is reduced through a process of performing census transformation only for selected pixels preset in the matching window which also applies to stereo matching based on census transform.

(38) According to the invention described above, it is possible to improve matching accuracy by combining the stereo matching based on census transform and the adaptive support weight method and at the same time it is also possible to prevent a rapid increase of the amount of computation by restricting the use pixels in the window area, and thus there is an advantage of achieving a high computation speed and a high matching degree.

(39) In addition, it is possible to raise the computation speed by simultaneously performing the weight computation and the process of the stereo matching based on census transform and by simultaneously performing the numerator computation and the denominator computation in the matching cost computation process.

(40) As described above, the preferred embodiment has been disclosed in the drawings and specification. Herein, the specific terms have been used, but have been used to merely describe the invention, and have not been used to restrict meaning or to restrict the scope of the invention described in the claim. Therefore, a person skilled in the art can understand that various modifications and other equivalent embodiments can be achieved from this. Accordingly, a real technical protective scope of the invention should be determined by a technical sprit of the accompanying claim.

Disparity computation method through stereo matching based on census transform with adaptive support weight and system thereof

Assignee

Inventors

Cpc classification

Classification Explorer

H01J2237/226

ELECTRICITY

Classification Explorer

H01J2237/2611

ELECTRICITY

Classification Explorer

H04N2013/0081

ELECTRICITY

Classification Explorer

G06T7/593

PHYSICS

Classification Explorer

H01J37/222

ELECTRICITY

International classification

Classification Explorer

H04N13/00

ELECTRICITY

Classification Explorer

G06T7/593

PHYSICS

Abstract

Claims

Description