DISTANCE IMAGE GENERATION DEVICE WHICH CORRECTS DISTANCE MEASUREMENT ABNORMALITIES

Abstract

A distance image generation device includes a light projection unit for projecting reference light onto a subject, a light reception unit having a plurality of two-dimensionally arrayed pixels, an optical system for guiding light from the subject to the light reception unit, an influence calculation means for calculating, based on an amount of light received by a target pixel and peripheral pixels thereof among the plurality of pixels, an influence of optical phenomena on the target pixel and the peripheral pixels, an impact calculation means for calculating the impact exerted by the peripheral pixels on the target pixel based on the influence, and a distance image generation means for generating a distance image of the subject based on the impact.

Claims

1. A distance image generation device, comprising: a light projection unit for projecting reference light onto a subject, a light reception unit having a plurality of two-dimensionally arrayed pixels, an optical system for guiding light from the subject to the light reception unit, an influence calculation means for calculating, based on an amount of light received by a target pixel and a peripheral pixel thereof among the plurality of pixels, an influence of optical phenomena on the target pixel and the peripheral pixel, an impact calculation means for calculating the impact exerted by the peripheral pixel on the target pixel based on the influence, and a distance image generation means for generating a distance image of the subject based on the impact.

2. The distance image generation device according to claim 1, wherein the impact calculation means generates impact images based on the impact for a plurality of phase images which have phases different than the reference light captured at a plurality of image timings, and the distance image generation means generates the distance image based on a difference image between the phase images and the impact images.

3. The distance image generation device according to claim 1, wherein the influence calculation means generates an influence image based on the influence, the impact calculation means generates an impact image based on the impact, and the distance image generation device further comprises a display means for displaying at least one of the influence image and the impact image.

4. The distance image generation device according to claim 1, wherein the influence calculation means calculates an influence coefficient for calculating the influence based on a distribution of the amount of received light at the target pixel and the peripheral pixel.

5. The distance image generation device according to claim 4, wherein the influence coefficient is calculated so as to be asymmetrical about the target pixel based on at least one of the distance from an image center to the target pixel or the peripheral pixel and position in a region constituted by the target pixel and the peripheral pixel.

6. The distance image generation device according to claim 1, wherein the influence calculation means calculates an influence coefficient for calculating the influence based on a distribution of the amount of received light at the target pixel and the peripheral pixel, and the impact calculation means calculates the impact based on the relationship between the influence coefficient and a measurement value of the peripheral pixel.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 is a block diagram of a distance image generation device according to an embodiment.

[0013] FIG. 2 is a view showing a distance measurement value calculation method according to the embodiment.

[0014] FIG. 3 is a view showing a distance measurement abnormality caused by stray light.

[0015] FIG. 4 is a view showing a distance image generation method according to an embodiment.

[0016] FIG. 5 is a view showing an optical phenomena influence calculation method according to an embodiment.

[0017] FIG. 6 is a view showing an impact image generation method according to an embodiment.

[0018] FIG. 7A is a view showing an example of an asymmetric filter.

[0019] FIG. 7B is a view showing an example of an asymmetric filter.

[0020] FIG. 8 is a flowchart showing the operations of the distance image generation device according to an embodiment.

[0021] FIG. 9 is an explanatory diagram detailing a distance measurement abnormality correction method according to another embodiment.

[0022] FIG. 10A is view showing an example of a distance measurement abnormality caused by a highly-reflective material.

[0023] FIG. 10B is a view showing an example of a distance measurement abnormality caused by a highly-reflectively material.

[0024] FIG. 11A is a view showing a distance measurement abnormality principle.

[0025] FIG. 11B is a view showing a distance measurement abnormality principle.

[0026] FIG. 11C is a view showing a distance measurement abnormality principle.

DETAILED DESCRIPTION

[0027] The embodiments of the present disclosure will be described in detail below referring to the attached drawings. In the drawings, identical or similar constituent elements are assigned the same or similar reference signs. Furthermore, the embodiments described below do not limit the technical scope of the invention and the definitions of the terms described in the claims.

[0028] FIG. 1 shows the structure of a distance image generation device 1 according to the present embodiment. The distance image generation device 1 comprises a light projection unit 10, a light reception unit 11, an optical system 12, an emission and imaging control unit 13, an A/D conversion unit 14, a memory 15, and a processor 16. The distance image generation device 1 may further comprise a display unit 17.

[0029] The light projection unit 10 comprises a light source (e.g., a light-emitting diode, semiconductor laser, etc.) which emits intensity-modulated reference light, and a diffuser which scatters the emitted reference light toward the subject O, and the light projection unit 10 projects the reference light onto the subject O. The light reception unit 11 comprises an image sensor (e.g., a CCD (charge-coupled device), CMOS (complementary metal-oxide semiconductor), etc.) having a plurality of two-dimensionally arrayed pixels.

[0030] The optical system 12 comprises a condenser lens which condenses the light from the subject O, and an optical filter which allows the wavelength band of the reference light to pass therethrough, and the optical system 12 guides the light from the subject O to the light reception unit 11. The emission and imaging control unit 13 comprises a control circuit which controls the light emission of the light projection unit 10 and the capturing of the light reception unit 11, and the emission and imaging control unit 13 is controlled by the processor 16. The A/D conversion unit 14 comprises an A/D converter which A/D converts the amount of light received by the light reception unit 11.

[0031] The memory 15 comprises a memory (e.g., a semiconductor memory, magnetic storage device, etc.) which stores various information. The processor 16 comprises a processor (e.g., a semiconductor integrated circuit such as a CPU (central processing unit) or FPGA (field-programmable gate array)) which controls the entirety of the distance image generation device 1. The display unit 17 comprises a display device (e.g., a liquid crystal display, organic EL display, etc.) which displays various information.

[0032] FIG. 2 shows a measurement value calculation method according to the present embodiment. The light reception unit 11 repeatedly captures images at a plurality of image timings, for example, four image timings (0, 90, 180, and 270), having phases different than the reference light, and acquires amounts of light received Q.sub.1, Q.sub.2, Q.sub.3, and Q.sub.4 for each pixel. The measurement value L of each pixel is determined from, for example, the formula below. In the formula below, c is the speed of light and f is the modulation frequency of the reference light.

[00001]$\begin{array}{cc}\left[\mathrm{Formula}.Math..Math.1\right]& \\ L=\frac{c}{4.Math..Math..Math.f}.Math.{\tan }^{-1}\left(\frac{Q_2-Q_4}{Q_1-Q_3}\right)& 1\end{array}$

[0033] FIG. 3 shows a distance measurement abnormality caused by stray light. As shown in FIG. 11A, though pixel a of the image sensor 34 should originally receive the light from path b, light from path a may be received inside the device (lens, image sensor, etc.). Since the amounts of light received Q1, Q2, Q3, and Q4 of path b and path a are different from each other, the measurement value L calculated with the above formula may become the distance between path a and path b. In order to eliminate the influence of path a, it is necessary to subtract the influence from path a on pixel a from the amount of light received by pixel a.

[0034] FIG. 4 shows a distance image generation method according to the present embodiment. Since the calculation of the measurement value is performed by generating phase images (phase 1 image to phase 4 image) captured at a plurality of image timings having phases different from the reference light, in each phase image, it is necessary to eliminate the influence of optical phenomena. Though it is considered that the influence of path a on each phase image cannot be determined unless the measurement value of path a is known, since the influence of optical phenomena such as stray light and aberrations appears as blur in an image, the influence of path a on each phase image can be determined by calculating the influence of optical phenomena based on the actual amounts of light received by a target pixel and peripheral pixels thereof, and calculating an impact exerted on the target pixel by the peripheral pixels based on the influence. In other words, impact images (phase 1 impact image to phase 4 impact image) are generated for each phase image, and by subtracting the impact images from each phase image, it is possible to specify only the amount of light received from path b. In other words, it is possible to simply and accurately correct distance measurement abnormalities caused by various optical phenomena.

[0035] The influence Fi of optical phenomena at each pixel can be described by, for example, the formula below. In the formula below, A is the amount of light received at the individual pixel, C.sub.1 is an influence coefficient, and C.sub.2 is a correction coefficient corresponding to individual characteristics (lens characteristics, sensor characteristics, etc.) inside the device.

[Formula 2]

Fi=C.sub.1A+C.sub.22

[0036] FIG. 5 shows a method for calculating the influence of optical phenomena according to the present embodiment. Since the influence Fi of the optical phenomena is equivalent to the degree of blur in the image, the influence coefficient C.sub.1 described above is determined from, for example, the formula below. In the formula below, w is the spread width of the shaded edge, and h is the height of the shaded edge.

[Formula 3]

C.sub.i=f(w,h)3

[0037] The influence coefficient C.sub.1 may be set in advance by, for example, the device provider by capturing an image of a subject having a predetermined pattern prepared in advance, or may be determined from the distribution of the amounts of light received by a target pixel and peripheral pixels thereof in phase images while using the device, i.e., from a spread width w and a height h of a shaded edge. Furthermore, the correction coefficient C.sub.2 can be experimentally determined in advance by the device provider in accordance with the individual characteristics within the device.

[0038] The influence Fi exerts a strong influence on pixels near the target pixel, and weakens as the distance increases. Thus, the influence coefficient C.sub.1 can be well described by a smoothing filter, in particular, a Gaussian filter. In another embodiment, another smoothing filter, for example, a moving average filter, a median filter, a low-pass filter, or the like, may be applied. Further, in some cases, a dedicated filter, for example, an asymmetric filter, which is described later, may be prepared.

[0039] A conventional Gaussian filter is described by the following formula. In the following formula, x and y are positions within the filter, with (0, 0) as a center, and a is the standard deviation of the Gaussian distribution. As the value of standard deviation a increases, the Gaussian distribution becomes flatter, while the effect of smoothing (blurring) increases, and the smaller the value of standard deviation a, the more the Gaussian distribution becomes convex, and the smaller the effect of smoothing (blurring). Thus, the standard deviation a may be obtained from formula 3 above, or the device provider may set a fixed value in advance.

[00002]$\begin{array}{cc}\left[\mathrm{Formula}.Math..Math.4\right]& \\ h_g\left(x,y\right)=\frac{1}{2.Math..Math.{}^2}.Math.\exp .Math.\left\{-\frac{x^2+y^2}{2.Math.{}^2}\right\}& 4\end{array}$

[0040] FIG. 6 shows an impact image generation method according to the present embodiment. Impact images are generated by applying, for example, a 55 Gaussian filter to each phase image.

[0041] Since the influence of optical phenomena such as stray light and aberrations differs depending on the distance from the image center and the position within the filter in some cases, in actual applications, an asymmetric filter in which at least one of the correction coefficient of distance (i, j) from the image center and the weight (a, b) of the position in the filter is applied to the Gaussian filter is more preferable. The asymmetric filter can be described by, for example, the following formula. In the following formula, the position of the pixel in the image is set as (i, j), the position of the image center pixel is set as (i.sub.c, j.sub.c), and the distance (i, j) from the image center is (ii.sub.c, jj.sub.c).

[00003]$\begin{array}{cc}\left[\mathrm{Formula}.Math..Math.5\right]& \\ h_g\left(i^{},j^{},x,y\right)=\frac{1}{2.Math..Math.{}^2}.Math.\frac{1}{a\left(i^{}\right).Math.b\left(j^{}\right)}.Math.\exp .Math.\left\{-\frac{a^2\left(i^{}\right).Math.x^2+b^2\left(j^{}\right).Math.y^2}{2.Math.{}^2}\right\}& 5\end{array}$

[0042] FIGS. 7A and 7B show examples of an asymmetric filter (corresponding to the influence coefficient) generated by the above formula. As shown in FIG. 7A, the asymmetric filter 20 has reduced asymmetry toward the image center (i.sub.c, j.sub.c) and increases in asymmetry toward the periphery of the image. Furthermore, it is also possible to change the Gaussian distribution, that is, the filter shape, in accordance with the weight (a, b) of the position in the filter. As a result, it is possible to represent the influence of various optical phenomena such as stray light and aberrations with high accuracy. FIG. 7B shows an example of an asymmetric filter applied to the periphery of an image.

[0043] FIG. 8 is a flowchart showing the operations of the distance image generation device 1 according to the present embodiment. When the distance image generation device 1 begins operations, in step S10, image capture is repeated at a plurality of image timings having phases different from the reference light to acquire each of the phase images. Each phase image may be stored in the memory.

[0044] In step S11, the influence of optical phenomena is calculated based on the amount of light received by the target pixel and peripheral pixels thereof in each phase image. In other words, in this step, a smoothing filter (corresponding to the influence coefficient) such as a Gaussian filter is generated from the distribution of the amounts of light received by the target pixel and the peripheral pixels, and the influence is calculated based on the relationship between the influence coefficients and the amounts of light received by the target pixel and the peripheral pixels. However, the smoothing filter (influence coefficient) may be set in advance by the device provider, and a fixed value may be loaded. Furthermore, influence images may be generated based on the influences calculated at this time, and the influence images may be stored in the memory.

[0045] In step S12, the impact exerted on the target pixel by the peripheral pixel is calculated based on the calculated influence. In other words, in this step, the smoothing filter (corresponding to the influence coefficient) is applied to each phase image, and impact images are generated. At this time, the impact image may be stored in the memory.

[0046] In step S13, the impact images are subtracted from each phase image to generate difference images. The difference images correspond to the phase images of the light which should be originally received. In step S14, a distance image is generated from each difference image based on the distance calculation formula of formula 1 above. At this time, the difference images may be stored in the memory. Note that the generated distance image may be applied to a control system which controls machines such as robots, machine tools, and automobiles, an object monitoring system that monitors objects in a monitoring region, etc.

[0047] Though not indispensable, in step S15, at least one of the phase images, the influence images, the impact images, and the difference images are displayed on the display unit. In particular, by displaying the influence images and impact images, investigation of the cause of the distance measurement abnormality is facilitated. Since the distance measurement abnormality becomes great in locations at which the impact is large, it is possible to take measures such as moving, to the location, the subject causing the distance measurement abnormality by specifying the location where the influence is presumed to be exerted.

[0048] A distance measurement abnormality correction method according to another embodiment will be described with reference to FIG. 9. Note that this embodiment differs from the embodiment described above in that phase images, impact images, difference images, etc., are not used. FIG. 9 shows a target pixel E and peripheral pixels thereof A, B, C, D, F, G, H, and I.

[0049] The influence of the distance measurement abnormality is determined in accordance with the actual distance value of the target pixel E, which receives the influence, the actual distance values of the peripheral pixels A, B, C, D, F, G, H, and I, which exert the influence, and the influence coefficients of the target pixel and the peripheral pixels. Thus, the measurement value l of the target pixel E can be described by the formula below. In the formula below, l.sub.a, l.sub.b, l.sub.d, l.sub.d, l.sub.e, l.sub.f, l.sub.g, l.sub.h, and l.sub.i are the actual distance values of the target pixel and the peripheral pixels, and a, b, c, d, e, f, g, h, and i are influence coefficients.

[00004]$\begin{array}{cc}\left[\mathrm{Formula}.Math..Math.6\right]& \\ l=\frac{{\mathrm{al}}_a+b.Math.l_b+c.Math.l_c+d.Math.l_d+e.Math.l_e+f.Math.l_f+g.Math.l_g+h.Math.l_h+{\mathrm{il}}_i}{a+b+c+d+e+f+g+h+i}& 6\end{array}$

[0050] By solving the formula above for the actual distance value l.sub.e, the formula below is obtained. Since the actual distance values l.sub.a, l.sub.b, l.sub.c, l.sub.d, l.sub.f, l.sub.g, l.sub.h, and l.sub.i of the peripheral pixels are unknown in the formula below, in actual applications, the measurement values of the peripheral pixels may be used in place of the actual distance values. Furthermore, the influence coefficients a, b, c, d, e, f, g, h, and i may be determined from the distribution (e.g., formula 3 above) of the amounts of light received by the target pixel and the peripheral pixels.

[00005]$\begin{array}{cc}\left[\mathrm{Formula}.Math..Math.7\right]& \\ l_e=\frac{\begin{array}{c}l\left(a+b+c+d+e+f+g+h+i\right)-\\ {\mathrm{al}}_a-b.Math.l_b-c.Math.l_c-d.Math.l_d-f.Math.l_f-g.Math.l_g-h.Math.l_h-i.Math.l_i\end{array}}{e}& 7\end{array}$

[0051] As shown in the formula above, the actual distance value l.sub.e (i.e., the distance image) of the target pixel E can be determined by calculating the influence coefficients a, b, c, d, e, f, g, h, and i, for calculating the influence, based on the distribution of the amounts of light received at the target pixel and the peripheral pixels, and calculating the impacts al.sub.a, bl.sub.b, cl.sub.d, dl.sub.d, fl.sub.f, gl.sub.g, hl.sub.h, and il.sub.i based on the relationship between the influence coefficients and the measurement values of the peripheral pixels.

[0052] According to the embodiments described above, it is possible to easily specify the amount of light which should be originally received by the target pixel, and correct distance measurement abnormalities caused by various optical phenomena with high accuracy.

[0053] Furthermore, the program carried out by the distance image generation device 1 may be stored on and supplied from computer-readable non-temporary recording medium, for example, a CD-ROM or the like.

[0054] Though the various embodiments have been described in the present description, the present invention is not limited to the embodiments described above, and it should be appreciated that various modifications can be made within the scope described in the claims below.