Image processing apparatus, image processing method, and imaging apparatus

Abstract

An imaging device that includes a first color filter, a second color filer and an image sensor. The first color filter includes a first plurality of color filter regions that transmit a color of a first set of colors. The second color filter includes that transmit a color of a second set of colors. The second set of colors is different from the first set of colors. The image sensor detects light that passes through both the first color filter and the second color filter and generates an image signal.

Claims

1. An imaging device comprising: a first color filter configured to receive light and transmit a first portion of the light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors; a second color filter configured to receive the first portion of the light transmitted through the first color filter and transmit a second portion of the light, the second color filter comprising a second plurality of color filter regions, wherein each color filter region of the second plurality of color filter regions transmits a color of a second set of colors, wherein all the colors of the second set of colors are different from all the colors of the first set of colors; and an image sensor configured to detect the second portion of the light transmitted through the second color filter and to generate an image signal, wherein the first color filter comprises cyan, magenta, yellow and white color filter regions of equal areas, and wherein the second color filter comprises red, blue, green and infrared color filter regions.

2. The imaging device of claim 1, wherein the first set of colors is complementary to the second set of colors.

3. The imaging device of claim 1, wherein the second color filter is formed integrally with the image sensor.

4. The imaging device of claim 1, further comprising: a lens, wherein the lens receives and transmits the first portion of the light from the first color filter.

5. The imaging device of claim 1, wherein the second plurality of color filter regions are arranged randomly within the second color filter.

6. The imaging device of claim 1, further comprising: an image processor configured to generate depth information from the image signal.

7. An imaging device comprising: a first color filter configured to receive light and transmit a first portion of the light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors; a second color filter configured to receive the first portion of the light transmitted through the first color filter and transmit a second portion of the light, the second color filter comprising a second plurality of color filter regions, wherein each of the second plurality of color filter regions transmits a color of a second set of colors, wherein the second plurality of color filter regions are arranged randomly within the second color filter and wherein all the colors of the first set of colors are different from all the colors of the second set of colors; and an image sensor configured to detect the second portion of the light transmitted through the second color filter and to generate an image signal, wherein the first color filter comprises cyan, magenta, yellow and white color filter regions of equal areas, and wherein the second color filter comprises red, blue, green and infrared color filter regions.

8. The imaging device of claim 7, wherein the first set of colors is complementary to the second set of colors.

9. The imaging device of claim 7, further comprising: an image processor configured to generate depth information from the image signal.

10. The imaging device of claim 7, wherein the second color filter is formed integrally with the image sensor.

11. The imaging device of claim 7, further comprising: a lens, wherein the lens receives and transmits the first portion of the light from the first color filter.

12. A method of determining depth information, the method comprising: irradiating a first color filter with light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors; irradiating a second color filter with a first portion of the light transmitted through the first color filter, the second color filter comprising a second plurality of color filter regions, wherein each color filter region of the second plurality of color filter regions transmits a color of a second set of colors, wherein all the colors of the second set of colors are different from all the colors of the first set of colors; detecting, using an image sensor, a second portion of the light transmitted through the second color filter and generating a corresponding image signal comprising a plurality of pixels, wherein the first color filter comprises cyan, magenta, yellow and white color filter regions of equal areas, and wherein the second color filter comprises red, blue, green and infrared color filter regions; and determining depth information from the image signal.

13. The method of claim 12, further comprising: determining a color displacement amount for each pixel of the plurality of pixels, wherein the act of determining the depth information is based on the color displacement amount.

14. The method of claim 13, wherein the act of determining a color displacement amount comprises: creating a plurality of restored images from the image signal, wherein each restored image of the plurality of restored images is associated with a respective color displacement; calculating an error for each pixel of each restored image of the plurality of restored images; and selecting a color displacement for each pixel based at least in part on the error associated with the respective pixel.

15. The method of claim 14, wherein selecting the color displacement for each pixel further comprises using a graph cut method.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a diagram showing the structure of an embodiment of an image processing apparatus to which the present technology is applied.

(2) FIG. 2 is a diagram showing an example structure of the primary color filter placed at the aperture of the lens unit.

(3) FIG. 3 is a diagram showing an example structure of the secondary color filter placed on the front surface of the image sensor.

(4) FIG. 4 is a diagram schematically showing observational images obtained from the light that has passed the respective paths.

(5) FIG. 5 is a diagram showing the amounts of light that passes through the respective filters of CMY in the respective wavelength regions.

(6) FIG. 6 is a diagram showing the amounts of light that passes through the respective filters of RGB in the respective wavelength regions.

(7) FIG. 7 is a diagram for explaining observational images to be obtained from color filter arrangement to which the present technology is applied.

(8) FIG. 8 is a diagram for explaining observational images to be obtained from conventional color filter arrangement.

(9) FIG. 9 is a diagram for explaining the principles of a process of restoring an object image, taking advantage of compressed sensing.

(10) FIG. 10 is a diagram for explaining the principles of a process of restoring an object image, taking advantage of compressed sensing.

(11) FIG. 11 is a diagram schematically showing the color component that passes through the primary color filter in a case where attention is paid to the R channels.

(12) FIG. 12 is a diagram schematically showing the color component that passes through the secondary color filter in a case where attention is paid to the R channels.

(13) FIG. 13 is a flowchart for explaining the flow of image processing to be performed by the image processing apparatus.

(14) FIG. 14 is a diagram showing the structure of an embodiment of an imaging apparatus to which the present technology is applied.

DESCRIPTION OF EMBODIMENTS

(15) The following is a description of embodiments of the present technology, with reference to the drawings. Explanation will be made in the following order.

(16) 1. Structure of an image processing apparatus

(17) 2. Principles of ensuring of incident light

(18) 3. Restoration process using compressed sensing

(19) 4. Flow of image processing

(20) 5. Structure of an imaging apparatus

(21) <1. Structure of an Image Processing Apparatus>

(22) FIG. 1 is a diagram showing the structure of an embodiment of an image processing apparatus to which the present technology is applied.

(23) The image processing apparatus 10 shown in FIG. 1 has a structure in which a color filter is placed on the aperture of a lens unit to take advantage of the principles of Color-Filtered Aperture. In FIG. 1, the image processing apparatus 10 includes a lens unit 111, a primary color filter 112, an image sensor 113, a secondary color filter 114, and an image processing unit 115. In FIG. 1, the lens unit 111 and the primary color filter 112 are shown separately from each other, for ease of explanation. In practice, however, the primary color filter 112 is placed inside the lens unit 111. The secondary color filter 114 is formed integrally with the image sensor 113.

(24) The lens unit 111 includes one or more lenses, and guides light (image light) from an object 1 located in front of a background 2 onto the light receiving surface of the image sensor 113. The primary color filter 112 is placed inside the lens unit 111.

(25) FIG. 2 shows the external appearance of the lens unit 111. In FIG. 2, the primary color filter 112 is placed at the aperture (the diaphragm portion) of the lens unit 111, and light from the object 1 enters the image sensor 113 of the later stage via the primary color filter 112. In FIG. 2, the rhomboid-shaped primary color filter 112 is divided into four regions, and the respective regions are formed with filters of the following colors: cyan (Cy), magenta (Mg), yellow (Ye), and white (W).

(26) Referring back to FIG. 1, the image sensor 113 is a solid-state imaging device such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. The image sensor 113 includes a pixel array unit in which pixels having photoelectric conversion elements (photodiodes) are two-dimensionally arranged in a matrix fashion, and a peripheral circuit unit that drives the pixels and performs AD (Analog-to-Digital) conversion and the like.

(27) The image sensor 113 photoelectrically converts light from the lens unit 111, and outputs the light as an image signal to the image processing unit 115. The secondary color filter 114 is placed on the front surface of the image sensor 113.

(28) FIG. 3 shows the external appearance of the secondary color filter 114. In FIG. 3, the secondary color filter 114 includes filters of the following colors: red (R), green (G), and blue (B). The secondary color filter 114 also includes IR (Infrared Ray) pass filters (IR) that can pass infrared rays. In this secondary color filter 114, the filters of red (R), green (G), and blue (B), and the IR pass filters (IR) are randomly arranged.

(29) That is, the primary color filter 112 placed at the lens aperture and the secondary color filter 114 placed on the sensor pixels have different characteristics from each other, and the colors that pass through the primary color filter 112 (cyan (Cy), magenta (Mg), and yellow (Ye), which are the three primary colors in the case of subtractive color mixing) are complementary to the colors that pass through the secondary color filter 114 (red (R), green (G), and blue (B), which are the three primary colors in the case of additive color mixing).

(30) In the description below, the respective filters of cyan (Cy), magenta (Mg), yellow (Ye), and white (W) in the primary color filter 112 will be referred to as the Cy filter, the Mg filter, the Ye filter, and the W filter. Also, the respective filters of red (R), green (G), and blue (B) in the secondary color filter 114 will be referred to as the R filters, the G filters, and the B filters.

(31) In the description below, the color filters of cyan (Cy), magenta (Mg), and yellow (Ye) will be also referred to as the CMY filters. The color filters of red (R), green (G), and blue (B) will be also referred to as the RGB filters.

(32) Referring back to FIG. 1, the image processing unit 115 performs predetermined image processing on the image signal that is output from the image sensor 113. This image processing may be a process of restoring an image of the object 1 (an object image) by applying compressed sensing to the image signal that is output from the image sensor 113, for example. The process of restoring the image of the object 1 (the object image) using compressed sensing will be described later in detail.

(33) The image processing apparatus 10 shown in FIG. 1 has the above described structure. In this image processing apparatus 10, light from the object 1 is gathered by the lens unit 111 having the primary color filter 112 placed at its aperture, the light is then photoelectrically converted by the image sensor 113 having the secondary color filter 114 placed on its front surface, and the image signal obtained through the photoelectric conversion is processed by the image processing unit 115.

(34) <2. Principles of Ensuring of Incident Light>

(35) Next, the principles of ensuring of sufficient incident light by the light receiving surface of the image sensor 113 in the image processing apparatus 10 shown in FIG. 1 are described.

(36) As shown in FIG. 4, in the image processing apparatus 10 shown in FIG. 1, the image sensor 113 outputs an image signal that is observational images in accordance with the paths of light that passes through the filters of the respective colors of the primary color filter 112 placed at the aperture of the lens unit 111.

(37) In this case, four observational images, which are an observational image of a path Ye obtained from the light that has passed through the Ye filter, an observational image of a path Cy obtained from the light that has passed through the Cy filter, an observational image of a path Mg obtained from the light that has passed through the Mg filter, and an observational image of a path W obtained from the light that has passed through the W filter, are output based on the respective paths of the light that passes through the filters of the respective colors of the primary color filter 112. In practice, an image obtained by superimposing (combining) these four observational images on one another is observed at the image processing unit 115. In the description below, this observation result will be referred to as y.

(38) Specifically, in the primary color filter 112, the observational image obtained from the light that has passed through the Ye filter and been received by the image sensor 113 via the secondary color filter 114 (the observational image of the path Ye in FIG. 4) includes red (R) and green (G) pixels obtained from the light that has passed through the R filters and the G filters randomly arranged in the secondary color filter 114, and yellow (Ye) pixels obtained from the light that has passed through the IR pass filters.

(39) FIG. 5 shows the characteristics of the respective filters of cyan (Cy), magenta (Mg), and yellow (Ye), which are the three primary colors in the case of subtractive color mixing. FIG. 6 shows the characteristics of the respective filters of red (R), green (G), and blue (B), which are the three primary colors in the case of additive color mixing. In FIGS. 5 and 6, the abscissa axis indicates the wavelengths of the respective colors, and the ordinate axis indicates the amounts of light that passes through the filters of the respective colors.

(40) In accordance with the characteristics of these filters, the wavelength region of yellow (Ye) shown in FIG. 5 corresponds to the wavelength regions of red (R) and green (G) shown in FIG. 6. Accordingly, the Ye filter of the primary color filter 112 also functions as a filter that passes the red (R) component and the green (G) component, and red (R) pixels and green (G) pixels are obtained from the light that has passed through the R filters and the G filters randomly arranged in the secondary color filter 114. However, the Ye filter of the primary color filter 112 does not function as a filter that passes the blue (B) component. Therefore, blue (B) pixels are not obtained as indicated by the black squares in FIG. 4.

(41) Also, in accordance with the characteristics of these filters, the wavelength region of cyan (Cy) shown in FIG. 5 corresponds to the wavelength regions of green (G) and blue (B) shown in FIG. 6. Therefore, the Cy filter of the primary color filter 112 also functions as a filter that passes the green (G) component and the blue (B) component, but does not function as a filter that passes the red (R) component.

(42) Because of such characteristics of the Cy filter, the observational image of the path Cy (FIG. 4) includes green (G) pixels and blue (B) pixels obtained from the light that has passed through the G filters and the B filters randomly arranged in the secondary color filter 114, and cyan (Cy) pixels obtained from the light that has passed through the IR pass filters. In the case of the observational image of the path Cy (FIG. 4), the Cy filter does not pass the red (R) component, and therefore, red (R) pixels are not obtained as indicated by the black squares in the drawing.

(43) Also, in accordance with the characteristics of these filters, the wavelength region of magenta (Mg) shown in FIG. 5 corresponds to the wavelength regions of ref (R) and blue (B) shown in FIG. 6. Therefore, the Mg filter of the primary color filter 112 also functions as a filter that passes the red (R) component and the blue (B) component, but does not function as a filter that passes the green (G) component.

(44) Because of such characteristics of the Mg filter, the observational image of the path Mg (FIG. 4) includes red (R) pixels and blue (B) pixels obtained from the light that has passed through the R filters and the B filters randomly arranged in the secondary color filter 114, and magenta (Mg) pixels obtained from the light that has passed through the IR pass filters. In the case of the observational image of the path Mg (FIG. 4), the Mg filter does not pass the green (G) component, and therefore, green (G) pixels are not obtained as indicated by the black squares in the drawing.

(45) Further, the filter of white (W) of the primary color filter 112 functions as a filter that passes all the components of red (R), green (G), and blue (B). Accordingly, the observational image of the path W (FIG. 4) includes red (R) pixels, green (G) pixels, and blue (B) pixels obtained from the light that has passed through the R filters, the G, filters, and the B filters randomly arranged in the secondary color filter 114, and pixels obtained from the light that has passed through the IR pass filters.

(46) The structure in which the primary color filter 112 formed with the respective colors of the three primary colors (cyan (Cy), magenta (Mg), and yellow (Ye)) of the subtractive color mixing is placed at the aperture of the lens unit 111, and the secondary color filter 114 formed with the respective colors of the three primary colors (red (R), green (G), and blue (B)) of the additive color mixing is placed on the front surface of the image sensor 113 is employed as described above, so that the four observational images (the observation result y) shown in FIG. 4 are obtained.

(47) To sum up, the Ye filter of the primary color filter 112 passes the red (R) component and the green (G) component of light that comes from the object 1 and enters the light receiving surface of the image sensor 113 from the lens unit 111. Further, the R filters of the secondary color filter 114 pass the red (R) component, and the G filters pass the green (G) component, as shown in the upper part of FIG. 7. As a result, an image including red (R) pixels and green (G) pixels is obtained as the observational image of the path Ye (FIG. 4).

(48) Also, the Cy filter of the primary color filter 112 passes the green (G) component and the blue (B) component of the light from the object 1. Further, the G filters of the secondary color filter 114 pass the green (G) component, and the B filters pass the blue (B) component, as shown in the middle part of FIG. 7. As a result, an image including green (G) pixels and blue (B) pixels is obtained as the observational image of the path Cy (FIG. 4).

(49) Also, the Mg filter of the primary color filter 112 passes the red (R) component and the blue (B) component of the light from the object 1. Further, the R filters of the secondary color filter 114 pass the red (R) component, and the B filters pass the blue (B) component, as shown in the lower part of FIG. 7. As a result, an image including red (R) pixels and blue (B) pixels is obtained as the observational image of the path Mg (FIG. 4).

(50) For comparison, FIG. 8 shows observational images to be obtained in a case where a structure in which color filters each formed with the respective colors of the three primary colors (red (R), green (G), and blue (B)) of the additive color mixing are provided at the aperture of the lens unit 111 and on the front surface of the image sensor 113 is employed.

(51) Specifically, the R filters of the primary color filter 112 placed at the aperture of the lens unit 111 pass only the red (R) component of light that comes from the object 1 and enters the light receiving surface of the image sensor 113 from the lens unit 111. Further, the R filters of the secondary color filter 114 placed on the front surface of the image sensor 113 pass only the red (R) component, as shown in the upper part of FIG. 8. As a result, an image including only red (R) pixels is obtained as an observational image of the path R.

(52) Also, the G filters of the primary color filter 112 pass only the green (G) component of the light from the object 1, and the G filters of the secondary color filter 114 further pass only the green (G) component, as shown in the middle part of FIG. 8. As a result, an image including only green (G) pixels is obtained as an observational image of the path G.

(53) Also, the B filters of the primary color filter 112 pass only the blue (B) component of the light from the object 1, and the B filters of the secondary color filter 114 further pass only the blue (B) component, as shown in the lower part of FIG. 8. As a result, an image including only blue (B) pixels is obtained as an observational image of the path B.

(54) In a case where RGB filters with the same characteristics are used as the primary color filter 112 and the secondary color filter 114 as described above, each of the color filters passes only a single color component. Therefore, the amount of light that is received by the light receiving surface of the image sensor 113 becomes smaller and the S/N ratio becomes lower than in a case where CMY filters that are complementary to RGB filters are used to have different characteristics.

(55) In the image processing apparatus 10 shown in FIG. 1, on the other hand, RGB filters are used as the secondary color filter 114 placed on the front surface of the image sensor 113 while CMY filters that are complementary to the RGB filters are used as the primary color filter 112 placed at the aperture of the lens unit 111. In this manner, the primary color filter 112 and the secondary color filter 114 are made to have different characteristics from each other. In a case where such a structure is employed, the CMY filters can pass two or more color components on to the RGB filters. Accordingly, light loss due to a filter combination can be made smaller, a larger amount of light can be ensured to be received by the light receiving surface of the image sensor 113, and the S/N ratio can be made higher than in a case where only RGB filters are used.

(56) <3. Restoration Process Using Compressed Sensing>

(57) Next, a restoration process to be performed by the image processing apparatus 10 shown in FIG. 1 using compressed sensing is described.

(58) As described above, in the image processing apparatus 10 shown in FIG. 1, the image processing unit 115 applies compressed sensing to an image signal (observational images) that is output from the image sensor 113, so that an image of the object 1 (an object image (a natural image)) is restored.

(59) Compressed sensing (Compressive Sampling) is a framework to restore a high-dimensional signal having sparsity (characteristics with a large number of zero elements) from a small number of observations. Here, the high-dimensional signal to be restored is the image of the object 1 (the object image), and the number of observations is the observation result y obtained by superimposing observational images on one another.

(60) Specifically, as shown in FIG. 9, where the image signal (the image obtained by superimposing the four observational images shown in FIG. 4 on one another) that is output from the image sensor 113 is the observation result y, the image of the object 1 (the object image) is an original signal x, and the process of passing through the primary color filter 112 placed at the aperture of the lens unit 111 and the secondary color filter 114 placed on the front surface of the image sensor 113 is an observational matrix A, the following Expression (1) is true.
[Math.1]
y=Ax(1)

(61) As the simultaneous equation shown in Expression (1) is solved, the original signal x can be determined from the observation result y and the observational matrix A.

(62) As shown in FIG. 10, the observation number M of the observation result y has lower dimensions than the N dimensions of the original signal x, and therefore, a solution is not uniquely determined. However, the facts that the N-dimensional original signal x is k-sparse and the number of non-zero elements is k are taken advantage of in this case. That is, the known characteristic of a natural image is that it contains a large number of zero elements. For example, the original signal x is restored with a certain probability by employing an L1 restoration method (an L1 reconstruction method) using the observational matrix A having random properties when the number of non-zero elements in the original signal x is smaller than the number of dimensions of the observation result y. In other words, a natural image is redundant in the spatial frequency direction, and therefore, can be restored even if the amount of sampling is reduced. This aspect is taken advantage of in this case.

(63) In a case where attention is paid to the R channels, as shown in FIG. 11, the Mg filter, the Ye filter, and the W filter in the primary color filter 112 placed at the aperture of the lens unit 111 pass the red (R) component, but the Cy filter does not pass the red (R) component as indicated by the black square in the drawing. In this case, the process in which light passes through the primary color filter 112 is defined as B.sub.R. However, this B.sub.R depends on the focal position.

(64) In a case where attention is paid to the R channels in the secondary color filter 114 placed on the front surface of the image sensor 113, as shown in FIG. 12, the R filters pass the red (R) component, but the G filters and the B filters do not pass the red (R) component as indicated by the black squares in the drawing. In this case, the process in which light passes through the secondary color filter 114 is defined as C.sub.R.

(65) Where B.sub.R and C.sub.R are defined in the above manner, the observational matrix A representing the process of passing through the primary color filter 112 and the secondary color filter 114 can be expressed with B.sub.R and C.sub.R. Accordingly, the process of restoring the object image in the R channels is realized by solving the simultaneous equation shown in the following Expression (2).
[Math.2]
y.sub.R=C.sub.RB.sub.Rx(2)

(66) As is apparent from the arrangement in the secondary color filter 114 in FIG. 12, where attention is paid to the R channels, sampling can have random properties, since the R filters are randomly arranged. Also, this restoration process is performed with the information about the entire image, instead of local information. Accordingly, the amount of moire can be made smaller than in a case where conventional demosaic processing is performed.

(67) Although a case where attention is paid to the R channels has been described above, the observational matrix A can be expressed with B.sub.G and C.sub.G or with B.sub.B and C.sub.B in the same manner as above in a case where attention is paid to the G channels or the B channels.

(68) That is, the process of restoring the object image in the G channels is expressed by solving the simultaneous equation shown in the following Expression (3).
[Math.3]
y.sub.G=C.sub.GB.sub.Gx(3)

(69) Also, the process of restoring the object image in the B channels is expressed by solving the simultaneous equation shown in the following Expression (4).
[Math.4]
y.sub.B=C.sub.BB.sub.Bx(4)

(70) As described above, the image processing apparatus 10 shown in FIG. 1 has a structure that utilizes the principles of Color-Filtered Aperture by placing the primary color filter 112 at the aperture of the lens unit 111. However, RGB filters are used as the secondary color filter 114 placed on the front surface of the image sensor 113 while CMY filters that are complementary to the RGB filters are used as the primary color filter 112. In this manner, the primary color filter 112 and the secondary color filter 114 are made to have different characteristics from each other. As such a combination of CMY filters and RGB filters is employed, light loss due to a filter combination can be made smaller, a larger amount of light can be ensured to be received by the light receiving surface of the image sensor 113, and the S/N ratio can be made higher than in a case where only RGB filters are used.

(71) In the image processing apparatus 10, the filters of the respective colors of red (R), green (G), and blue (B), and the IR pass filters (IR) are randomly arranged in the secondary color filter 114 placed on the front surface of the image sensor 113. Accordingly, the amount of moire in the object image to be restored through the restoration process can be reduced. As a result, the image processing apparatus 10 can reduce the amount of moire while improving the S/N ratio by ensuring a sufficient amount of light entering the light receiving surface of the image sensor 113.

(72) Although the combination of the primary color filter 112 and the secondary color filter 114 is a combination of CMY filters and RGB filters in the above described example, the combination of color filters is not limited to that. For example, some other combination of color filters that have different characteristics from each other such as different wavelength regions for the same color may be employed.

(73) Since the secondary color filter 114 placed on the front surface of the image sensor 113 includes the IR pass filters that can pass infrared rays, switching may be performed so that an infrared image is generated in a dark place or the like. However, it is arbitrary to incorporate the IR pass filters as well as the RGB filters into the secondary color filter 114.

(74) <4. Flow of Image Processing>

(75) Referring now to the flowchart shown in FIG. 13, the flow of image processing to be performed by the image processing apparatus 10 shown in FIG. 1 is described.

(76) In step S101, the image processing unit 115 acquires an image signal that is output from the image sensor 113.

(77) In step S102, the image processing unit 115 performs predetermined image processing on the image signal from the image sensor 113. As described above, this image processing may be a process of restoring the image of the object 1 (the object image) by applying compressed sensing to the image signal (observational images) from the image sensor 113.

(78) The image processing may involve not only the process of restoring the object image, but also determination of depth information using disparity information (shift amounts of the respective colors) obtained based on the principles of Color-Filtered Aperture, for example. The depth information may be calculated by using the above described compressed sensing. For example, due to the arrangement of color filters in the primary color filter 112, the red image, blue image and green image recorded by the image sensor are displaced with respect to one another. The amount of displacement is a function of the depth of the objects in the recorded images. For example, using a thin lens approximation, the displacement amount (d) is related to the depth (D) by the following equation.
[Math.5]
1/D=1/f(1+d/b)/a,(5)

(79) Where f is the focal distance of lens unit 111, b is the displacement of the center of each of the color filters of the primary color filter 112 with respect to the center of the lens unit 111, and a is the distance between the lens unit 111 and the image sensor 113. This relationship between displacement amount (d) and depth (D) can be used to determine the depth (D) of portions of the image using techniques similar to those described in US Patent Application Publication Number 2009/0284627, filed Mar. 9, 2009, titled Image Processing Method, which is herein incorporated by reference in its entirety.

(80) To determine the depth of different portions of the image, first a plurality of restored images are created, each restored image compensating for a different amount of displacement for all the pixels of the image.

(81) Second, an error is calculated for each of the pixels in each of the plurality of restored images. The error calculation may, for example, be based on a linear color model where it is assumed that natural mages are free from color displacement in a three-dimensional color space.

(82) Third, a color displacement amount is selected for each pixel based on the calculated error associated with each of the plurality of restored images. For example, the displacement associated with the restored image with the smallest error at for a particular pixel may be selected. To increase the likelihood that the selected displacement for a particular pixel is smoothly selected relative to its surrounding pixels, an estimation method, such as a graph cut method, may be used when associating a pixel with a color displacement amount.

(83) Finally, using formula 5 from above, the color displacement associated with each pixel is used to determine a depth associated with each pixel of the image. In this way, the depth associated with different portions of the image may be determined.

(84) In step S103, the image processing unit 115 outputs a result of the image processing performed in step S102 to a circuit or the like in a later stage. When the processing in step S103 is completed, the image processing shown in FIG. 13 comes to an end.

(85) The image processing to be performed by the image processing apparatus 10 shown in FIG. 1 has been described. In this image processing, light from an object is gathered by the lens unit 111 having the primary color filter 112 placed at its aperture, the light is then photoelectrically converted by the image sensor 113 having the secondary color filter 114 placed on its front surface, and the image signal obtained through the photoelectric conversion is processed by the image processing unit 115.

(86) <5. Structure of an Imaging Apparatus>

(87) FIG. 14 is a diagram showing an embodiment of an imaging apparatus to which the present technology is applied.

(88) In FIG. 14, an imaging apparatus 300 includes a lens unit 301, an image sensor 302, an image processing unit 303, a frame memory 304, a display unit 305, a recording unit 306, an operating unit 307, and a power supply unit 308. In the imaging apparatus 300, the image processing unit 303, the frame memory 304, the display unit 305, the recording unit 306, the operating unit 307, and the power supply unit 308 are connected to one another via a bus line 309.

(89) The lens unit 301 is equivalent to the above described lens unit 111 (FIG. 1). The lens unit 301 includes one or more lenses, and guides light (image light) from an object onto the light receiving surface of the image sensor 302. Although not shown in the drawing, a primary color filter that is divided into four regions formed with filters of the respective colors of cyan (Cy), magenta (Mg), yellow (Ye), and white (W) (the primary color filter 112 shown in FIG. 1) is placed at the aperture of the lens unit 301.

(90) The image sensor 302 is equivalent to the above described image sensor 113 (FIG. 1) formed with a CMOS image sensor or the like. The image sensor 302 includes a pixel array unit in which pixels each including a photoelectric conversion element are two-dimensionally arranged in a matrix fashion, and a peripheral circuit unit. The image sensor 302 converts the amount of incident light forming an image on the light receiving surface through the lens unit 301, into an electrical signal pixel by pixel, and outputs the electrical signal as an image signal to the image processing unit 303. Although not shown in the drawing, a secondary color filter including filters of the respective colors of red (R), green (G), and blue (B) (the secondary color filter 114 shown in FIG. 1) is placed on the front surface of the image sensor 302.

(91) The image processing unit 303 is equivalent to the above described image processing unit 115 (FIG. 1). The image processing unit 303 performs predetermined image processing on the image signal that is output from the image sensor 302. This image processing may be a process of restoring an object image by applying compressed sensing to the image signal that is output from the image sensor 302, for example.

(92) The image processing unit 303 also performs camera signal processing on the image signal that is output from the image sensor 302. The image data obtained through this signal processing is temporarily stored into the frame memory 304, and is then supplied to the display unit 305 or the recording unit 306.

(93) The display unit 305 is formed with a liquid crystal panel or an organic EL (Electro-Luminescence) panel, for example, and displays a moving image or a still image sensed by the image sensor 302. The recording unit 306 records the image data of the moving image or the still image sensed by the image sensor 302 into a recording medium such as a semiconductor memory, a video tape, or the like.

(94) The operating unit 307 issues an operation instruction with respect to a function of the imaging apparatus 300, in accordance with an operation from a user. The power supply unit 308 supplies the image processing unit 303, the frame memory 304, the display unit 305, the recording unit 306, and the operating unit 307 with the power necessary for operations thereof.

(95) An imaging apparatus to which the present technology is applied does not necessarily have the above described structure, and may have some other structure. For example, such an imaging apparatus may not be a digital still camera or a digital video camera, but may be an information processing apparatus having an imaging function, such as a portable telephone, a smartphone, a tablet device, or a personal computer. Alternatively, an imaging apparatus to which the present technology is applied may be a camera module that is attached to and used in some other information processing apparatus (or is mounted as an incorporated device).

(96) The above described series of processes (the image processing shown in FIG. 13, for example) may be performed by hardware or may be performed by software. In a case where the above described series of processes are performed by software, the program that forms the software may be installed in a computer incorporated into special-purpose hardware, or may be installed from a recording medium into a general-purpose personal computer that can execute various kinds of functions by installing various kinds of programs, for example.

(97) This recording medium may not be formed with a removable medium that is distributed so as to provide users with programs separately from a computer, such as a magnetic disk, an optical disk, a magnetooptical disk, or a semiconductor memory, but may be formed with a ROM or a recording unit that is incorporated into a computer before provided to a user and has programs recorded therein.

(98) The program for performing the above described series of processes may be installed into a computer via a wired or wireless communication medium such as a local area network, the Internet, digital satellite broadcasting, using an interface such as a router or a modem as necessary.

(99) It should be noted that embodiments of the present technology are not limited to the above described embodiments, and various modifications may be made to them without departing from the scope of the present technology.

(100) For example, aspects of some configurations of various embodiments include:

(101) (1) An imaging device comprising:

(102) a first color filter configured to receive light and transmit a first portion of the light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors;

(103) a second color filter configured to receive the first portion of the light transmitted through the first color filter and transmit a second portion of the light, the second color filter comprising a second plurality of color filter regions, wherein each color filter region of the second plurality of color filter regions transmits a color of a second set of colors, wherein the second set of colors is different from the first set of colors; and

(104) an image sensor configured to detect the second portion of the light transmitted through the second color filter and generate an image signal.

(105) (2) The imaging device of configuration (1), wherein the first set of colors is complementary to the second set of colors.

(106) (3) The imaging device of configuration (2), wherein:

(107) the first set of colors comprises cyan, magenta and yellow; and

(108) the second set of colors comprises red, blue and green.

(109) (4) The imaging device of configuration (3), wherein:

(110) the first set of colors further comprises white; and

(111) the second set of colors further infrared.

(112) (5) The imaging device of configuration (1), wherein:

(113) the first set of colors comprises red, blue and green; and

(114) the second set of colors comprises cyan, magenta and yellow.

(115) (6) The imaging device of configuration (1), wherein the second color filter is formed integrally with the image sensor.

(116) (7) The imaging device of configuration (1), further comprising: a lens, wherein the lens receives and transmits the first portion of the light from the first color filter.

(117) (8) The imaging device of configuration (1), wherein the second plurality of color filter regions are arranged randomly within the second color filter.

(118) (9) The imaging device of configuration (1), further comprising:

(119) an image processor configured to generate depth information from the image signal.

(120) (10) An imaging device comprising:

(121) a first color filter configured to receive light and transmit a first portion of the light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors;

(122) a second color filter configured to receive the first portion of the light transmitted through the first color filter and transmit a second portion of the light, the second color filter comprising a second plurality of color filter regions, wherein each of the second plurality of color filter regions transmits a color of a second set of colors, wherein the second plurality of color filter regions are arranged randomly within the second color filter; and

(123) an image sensor configured to detect the second portion of the light transmitted through the second color filter and generate an image signal.

(124) (11) The imaging device of configuration (10), wherein the first set of colors is the same as the second set of colors.

(125) (12) The imaging device of configuration (10), wherein the first set of colors is different from the second set of colors.

(126) (13) The imaging device of configuration (12), wherein the first set of colors is complementary to the second set of colors.

(127) (14) The imaging device of configuration (10), further comprising:

(128) an image processor configured to generate depth information from the image signal.

(129) (15) The imaging device of configuration (10), wherein the second color filter is formed integrally with the image sensor.

(130) (16) The imaging device of configuration (10), further comprising:

(131) a lens, wherein the lens receives and transmits the first portion of the light from the first color filter.

(132) (17) A method of determining depth information, the method comprising:

(133) irradiating a first color filter with light, the first color filter comprising a first plurality of color filter regions, wherein each color filter region of the first plurality of color filter regions transmits a color of a first set of colors;

(134) irradiating a second color filter with a first portion of the light transmitted through the first color filter, the second color filter comprising a second plurality of color filter regions, wherein each color filter region of the second plurality of color filter regions transmits a color of a second set of colors, wherein the second set of colors is different from the first set of colors;

(135) detecting, using an image sensor, a second portion of the light transmitted through the second color filter and generating a corresponding image signal comprising a plurality of pixels; and

(136) determining depth information from the image signal.

(137) (18) The method of configuration (17), further comprising:

(138) determining a color displacement amount for each pixel of the plurality of pixels,

(139) wherein the act of determining the depth information is based on the color displacement amount.

(140) (19) The method of configuration (18), wherein the act of determining a color displacement amount comprises: creating a plurality of restored images from the image signal, wherein each restored image of the plurality of restored images is associated with a respective color displacement; calculating an error for each pixel of each restored image of the plurality of restored images; and

(141) selecting a color displacement for each pixel based at least in part on the error associated with the respective pixel.

(142) (20) The method of configuration (19), wherein selecting the color displacement for each pixel further comprises using a graph cut method.

REFERENCE SIGNS LIST

(143) 10 Image processing apparatus 111 Lens unit 112 Primary color filter 113 Image sensor 114 Secondary color filter 115 Image processing unit 300 Imaging apparatus 301 Lens unit 302 Image sensor 303 Image processing unit