Optimal color filter array and a demosaicing method thereof

Abstract

A method for capturing raw digital color image data by an image sensor array that includes a color filter array (CFA), at least one type of color filter elements of which transmits an additive (or subtractive) mixture of plural color-components of an additive (or subtractive) color-space, unlike conventional CFAs (akin to the color filter array) wherein single color-component of the color-space passes through each color filter element. An image demosaicing method is disclosed to convert the raw digital color image data wherein each pixel has a raw grayscale value into a full-color image wherein each pixel has values in all color-channels of the color-space. Further, a method of optimizing relationship between low-light sensitivity of the image sensor array, effective resolutions of color-component images, of color-components of the color-space, that are associated with the raw digital color image data and color depth of the full-color image, by means of adjusting spectral transmission characteristics of different types of color filter elements forming the CFA is disclosed.

Claims

1. A method for designing color filter arrays (CFAs), each color filter array comprising: a plurality of tiled side-by-side identical minimal repeating units, wherein each of said minimal repeating units comprises: wavelength selective color filter elements (optical filters), each of which is permanently superposed in one-to-one registry on a physically separate light-sensitive element (photodetector) in an array of solid state broad spectrum photodetectors so as to form a photosite group of a color image sensor of a camera module; wherein said color filter elements are of types so that each of said color filter elements transmits at least one of color-components of an additive (or subtractive) color-space, each of said color-components is transmitted by at least one of said color filter elements and said types of color filter elements are at least equal to number of said color-components in total; wherein at least one type of color filter elements is adapted to transmit an additive (or subtractive) mixture of plural color-components of said color-space, and wherein spectral transmission characteristic of said type of color filter elements is a superimposition of (perhaps overlapping) light transmittances of said plural color-components for said type of color filter elements so as to store sum of (perhaps fractional) values of said plural color-components in pixel location(s) of a raw image pattern, wherein said pixel location(s) is/are associated with said type of color filter elements and said photosite group is configured to electronically capture said raw image pattern; and wherein at least one of type(s) of color filter elements, that is/are adapted to transmit an additive (or subtractive) mixture of plural color-components of said color-space, has at least two of (non-zero) maximum values of light transmittances of said plural color-components that are distinct so as to transmit unequal amounts of at least two of said plural color-components.

2. A color conversion method, for reconstructing raw image data that is associated with a color filter array according to claim 1, wherein each pixel of said raw image data has a raw grayscale value of a color-channel associated with a color filter element that corresponds to said pixel, to generate a mosaiced-image that is separated into color-component images of color-components of a color-space as defined in claim 1 so as to demosaic said mosaiced-image into a full-color image wherein pixels have values in all color-channels of said color-space, by using any known demosaicing technique (for example, linear interpolation, bilinear, etc.).

3. An optimization method to obtain spectral transmission characteristics of different types of color filter elements forming a color filter array according to claim 1, wherein said spectral transmission characteristics are associated with optimal tradeoff among low-light sensitivity of a color image sensor, effective resolutions of color-component images of color-components of a color-space as defined in claim 1 and color depth of a demosaiced full-color image, that are associated with said color filter array.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0018] The invention is described with reference to the accompanying drawings, wherein:

[0019] FIG. 1 depicts a two-dimensional plot of spectral transmittance curves of different (types of) color filter elements forming ideal Bayer CFA pattern, wherein transmittance of the color filter elements is plotted along y-axis against wavelength in nanometres λ on x-axis, and wherein spectral transmittance curves R, G and B of three different band-pass optical filters of the RGB color filter element group are assumed to be sin.sup.3((0.0125(x-450))°):[450,700].fwdarw.[0,1], sin.sup.2((0.0125(x-400))°):[400,650].fwdarw.[0,1] and sin.sup.2((0.02(x-380))°):[380,535].fwdarw.[0,1] respectively, in order to ease calculations and are merely illustrative;

[0020] FIG. 2 depicts a two-dimensional plot of tentative and actual spectral transmission characteristic of ideal color filter element F.sub.1,2 according to the present invention, wherein transmittance of the color filter element is plotted along y-axis against wavelength in nanometres λ on x-axis.

DETAILED DESCRIPTION OF THE INVENTION

[0021] In general, a photosite is mathematically representable by matrix equation

[00001]$\text{ζ}\left(C_{1\times N}\right)\cdot F_{N\times 1}=G_{1\times 1}$

, wherein matrices

[00002]$C_{1\times N}\in \left\{\left[\begin{array}{ccc}{v}_{c_1}& .Math.& v_{c_N}\end{array}\right]:v_{c_{k\le N}}\in {ℝ}_{\ge 0},k\in \mathbb{N}\right\},$

[00003]$\begin{array}{l}{F}_{N\times 1}\in \left\{\left[\begin{array}{c}{\text{τ}}^{c_1}\\ .Math.\\ {\text{τ}}^{c_N}\end{array}\right]:{\text{τ}}^{c_{k\le N}}\in {ℝ}_{\le 1}^{+}\cup \left\{0\right\},k\in \mathbb{N}\right\}\text{and}\\ G_{1\times 1}\in \left\{\left[s\right]:s\in {\mathbb{Z}}_{\ge 0}\right\}\end{array}$

store incident light as intensities v.sub.c1,..., v.sub.cN of different color-components C.sub.1,..., C.sub.N of an output color-space, maximum values τ.sup.C1,..., τ.sup.CN of (perhaps overlapping) light transmittances of the color-components for the color filter element and a raw grayscale value s that is measured by a photodetector (light-sensitive) element, underlying the color filter element, that is associated with the photosite, respectively; ζ() is a quantization function for converting input original image data to quantized image data. In other words, the color filter element selectively transmits towards the photodetector element, light of a color-component, that is

[00004]$\left[\begin{array}{ccc}{c}_1& .Math.& c_N\end{array}\right]\cdot \left[\begin{array}{c}{\text{τ}}^{c_1}\\ .Math.\\ {\text{τ}}^{c_N}\end{array}\right],$

of a raw color-space derived through conversion from the output color-space. Since each photodetector element produces an electrical charge that is directly proportional to light intensity it receives with no wavelength specificity, v.sub.c1, ..., v.sub.cN are independent variables.

[0022] Inductively, a photosite group that is associated with a minimal repeating unit comprising an a × b set of individual color filter elements, under assumption that light intensities, that is C.sub.1×N, of c.sub.1, ...,c.sub.N that are incident on each of the color filter elements are uniform, may be expressed by matrix equation

[00005]${\text{ξ}}_{a\times a}\cdot P_{a\times b}=R_{a\times b}$

, wherein ξ.sub.a×a = diag(ξ(C.sub.1×N),...,ζ(C.sub.1×N)), and matrices P.sub.a×b and R.sub.a×b store maximum values of (perhaps overlapping) light transmittances of c.sub.1,...,c.sub.N for the color filter elements and raw grayscale values of color-components of the raw color-space that are measured by a photodetector group, underlying the minimal repeating unit, that is associated with the photosite group, respectively, and wherein P.sub.a×b[i,j]=F.sub.i,j (defined as

[00006]$\left(F_{i,j}=\left[\begin{array}{c}{\text{τ}}_{i,j}^{c_1}\\ .Math.\\ {\text{τ}}_{i,j}^{c_N}\end{array}\right]\right)$

) wherein F.sub.i,j ∈ {F.sub.N×1} store maximum values of the light transmittances of c.sub.1,...,c.sub.N for a color filter element, that corresponds to a color-component

[00007]$\left[\begin{array}{ccc}{c}_1& .Math.& c_N\end{array}\right]\cdot \left[\begin{array}{c}{\text{τ}}_{i,j}^{c_1}\\ .Math.\\ {\text{τ}}_{i,j}^{c_N}\end{array}\right]$

of the raw color-space, at pixel location (i, j) in the minimal repeating unit, and R.sub.a×b[i,j] = G.sub.i,j wherein G.sub.i,j ∈ {G.sub.1×1} stores raw grayscale value that is measured by a photodetector element that is associated with the color filter element at pixel location (i,j) in the minimal repeating unit. diag(e,...,e) denotes diagonal matrix wherein all diagonal entries are equal to e, and M[r,c] denotes the entry in r-th row and c-th column of a matrix M. Clearly,

[00008]${\text{ξ}}_{a\times a}=R_{a\times b}\cdot P_{a\times b}^{-1}$

represents solution of the matrix Equation (1), wherein matrix R.sub.a×b represents an input raw image pattern, that is associated with the raw color-space, before transformation and matrix ξ.sub.a×a represents output color pixel data, that is associated with the output color-space, after transformation, and therefore, precomputing the color-space transform matrix

[00009]$P_{a\times b}^{-1}$

greatly reduces computational time complexity of the above color conversion method.

[0023] Note that C.sub.1×n may also be deemed as variable matrix representing color information about the raw image pattern R.sub.a×b, that can be closely determined as ζ(C.sub.1×N) by using a color conversion method that requires solving of matrix Equation (1) for ξ.sub.a×a under condition that total number of unique color filter elements F.sub.i,j (each representing an independent linear equation) constituting P.sub.a×b is equal to or greater than N (that is total number of unknown variables v.sub.Ck in C.sub.1×N), and then diagonalizing ξ.sub.a×a, if necessary, to obtain a diagonal matrix Ω.sub.a×a which best approximates ξ.sub.a×a. Virtually, the assumption that incident light intensity over each of color filter elements in P.sub.a×b is uniform is not always true, thus all the diagonal elements of Ω.sub.a×a are not always equal. Therefore, more practically, ζ(C.sub.1×N) can be approximated as average of the diagonal elements of Ω.sub.a×a, that is

[00010]$\text{ζ}\left(C_{1\times N}\right)\approx \frac{{\displaystyle {.Math.}_{n=1}^a{\text{Ω}}_{a\times a}\left[n,n\right]}}{a}.$

Thereafter, a full-color image can be obtained by applying any known demosaicing method to R.sub.a×b, after performing image transform that is defined by conditional expression:

[00011]$\left({\text{τ}}_{i,j}^{c_k}=\max \left({\text{τ}}_{1,1}^{c_k},.Math.,{\text{τ}}_{a,b}^{c_k}\right)\right)\iff \left(R_{a\times b}\left[i,j\right]=\left[v_{c_k}\right]\right)\forall i,j,k$

, wherein max(N.sub.1,...,N.sub.n) denotes the largest value in the set of values N.sub.1, ..., N.sub.n. Hereinafter, the latter is illustrated in more detail, by way of example, in paragraph [0025].

[0024] Optimal ideal color filter pattern P.sub.a×b with respect to a conventional CFA pattern can be formulated via computational optimization methods by substituting spectral transmittances of color filter elements of the conventional CFA pattern into matrix C.sub.1×N (C.sub.1×3 = [R G B] for example, wherein R, G and B are spectral transmittances of color filter elements forming the ideal Bayer color filter array); then solving equation

[00012]${\displaystyle \underset{i=1}{\overset{a}{.Math.}}{\displaystyle \underset{j=1}{\overset{b}{.Math.}}{F}_{i,j}^{\text{T}}\cdot \text{diag}\left(C_{1\times N}\right)=\mu \cdot D_{1\times N}}}$

for maximum values

[00013]${\text{τ}}_{1,1}^{c_1},.Math.,{\text{τ}}_{a,b}^{c_N}$

of (perhaps overlapping) light transmittances of color-components c.sub.1,...,c.sub.N of a color-space that is associated with the conventional CFA pattern

[00014]$\left({\text{τ}}_{i,j}^R,{\text{τ}}_{i,j}^G\text{and}{\text{τ}}_{i,j}^B\right)$

for example, since the conventional ideal Bayer CFA pattern is associated with a RGB color-space) for ideal color filter elements F.sub.1,1,...,F.sub.a,b constituting P.sub.a×b to maximize .Math.∈ℝ.sup.+, wherein

[00015]$F_{i,j}^{\text{T}}$

is transpose of matrix F.sub.i,j, diagonal matrix diag(C.sub.1×N) is defined as

[00016]$\text{diag}\left(C_{1\times N}\right)\left[r,c\right]=\left\{\begin{array}{cc}{C}_{1\times N}\left[1,c\right],& \text{if}r=c\\ 0,& \text{if}r\ne c\end{array}\right)$

, and matrix D.sub.1×N store intensity of each of the color-components that are transmitted by the conventional CFA pattern, which is intended to be optimized (D.sub.1×3 = [R 2G B] for example, in case of ideal 2×2 Bayer CFA pattern). Note that in a case where wavelength ranges of plural color-components are overlapping, actual maximum values of the light transmittances for an ideal color filter element that transmit the plural color-components are obtained after min-max normalization of sum of initially assumed light transmittances for the ideal color filter element, which is illustrated in more detail, by way of example, in paragraph [0024]. Depending on particular desired purpose,

[00017]${\text{τ}}_{1,1}^{c_1},.Math.,{\text{τ}}_{a,b}^{c_N}$

may be configured to achieve optimal tradeoff between image sensor low-light sensitivity (which is proportional to .Math.) and color depth of a demosaiced full-color image that is associated with raw image data captured by a color image sensor that is associated with P.sub.a×b, under constraint condition that

[00018]${\text{τ}}_{1,1}^{c_1},.Math.,{\text{τ}}_{a,b}^{c_N}$

associated with maximum color depth are chosen for every value of the variable .Math., wherein the color depth is positively correlated with

[00019]${\displaystyle {.Math.}_{k=1}^N{\text{σ}}_k}$

wherein σ.sub.k is standard deviation of maximum values of light transmittances of color-component c.sub.k, for each of the ideal color filter elements, that is

[00020]$\left\{{\text{τ}}_{i,j}^{c_k}:1\le i\le a,1\le j\le b\right\}.$

Fortunately, any loss in color depth due to the optimization can be compensated by increasing sensor depth (or bit depth). A remarkable feature of above method is that each of the color-components c.sub.1,...,c.sub.N is partly sampled at plural pixel locations, thereby reducing color information loss in color-component images of c.sub.1,...,c.sub.N in the raw image data, that is inevitable in case of an undersampled raw digital color image data captured by a digital image sensor which includes a CFA that is associated with the conventional CFA pattern, wherein each of the color-components is sampled at single pixel location.

[0025] For simplicity and clarity, consider an explanatory illustration in which a 2 × 2 photosite group that is associated with a conventional RGB CFA pattern is given by

[00021]$\left[\begin{array}{cc}\text{ζ}\left(C_{1\times 3}\right)& 0\\ 0& \text{ζ}\left(C_{1\times 3}\right)\end{array}\right]\cdot \left[\begin{array}{cc}{F}_{1,1}& F_{1,2}\\ F_{2,1}& F_{2,2}\end{array}\right]=\left[\begin{array}{cc}{G}_{1,1}& G_{1,2}\\ G_{2,1}& G_{2,2}\end{array}\right]$

, wherein

[00022]$C_{1\times 3}=\left[\begin{array}{ccc}{v}_R& v_G& v_B\end{array}\right]\text{and}{F}_{i,j}=\left[\begin{array}{c}{\text{τ}}_{i,j}^R\\ {\text{τ}}_{i,j}^G\\ {\text{τ}}_{i,j}^B\end{array}\right].$

[0026] In (Bayer US3971065A), ideal 2×2 Bayer CFA pattern is configured as

[00023]$F_{1,2}=\left[\begin{array}{c}1\\ 0\\ 0\end{array}\right],F_{1,1}=F_{2,2}=\left[\begin{array}{c}0\\ 1\\ 0\end{array}\right],F_{2,1}=\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$

, or equivalently s.sub.1,2 = v.sub.R, s.sub.1,1 = s.sub.2,2 = v.sub.G, s.sub.2,1 = v.sub.B wherein s.sub.i,j = G.sub.i,j[1,1]. Therefore, amount of light intensity transmitted by the CFA pattern on underlying photodetector group is

[00024]${\displaystyle \underset{i=1}{\overset{2}{.Math.}}{\displaystyle \underset{j=1}{\overset{2}{.Math.}}{s}_{i,j}=v_R+2v_G+v_B}}.$

[0027] Consider next an alternative version P.sub.2×2 of the ideal 2×2 Bayer CFA pattern wherein ideal optical filter elements F.sub.1,1,...,F.sub.2,2 have ratios of maximum values of (overlapping) light transmittances of Red, Green and Blue (RGB) color-components (values are rounded off for simplicity) as follows:

[00025]$\begin{array}{l}{\text{τ}}_{1,2}^R:{\text{τ}}_{1,2}^G:{\text{τ}}_{1,2}^B=4/5:1:3/5,\\ {\text{τ}}_{1,1}^R:{\text{τ}}_{1,1}^G:{\text{τ}}_{1,1}^B={\text{τ}}_{2,2}^R:{\text{τ}}_{2,2}^G:{\text{τ}}_{2,2}^B=1/3:1:1/3,\\ {\text{τ}}_{2,1}^R:{\text{τ}}_{2,1}^G:{\text{τ}}_{2,1}^B=3/5:1:4/5.\end{array}$

Let ℸ= [R G B].Math.(a.Math.F.sub.1,2), as shown in FIG. 2, be a tentative spectral transmittance curve of F.sub.1,2 which is a superimposition of light transmittances of the RGB color-components for F.sub.1,2 wherein initially assumed maximum values, that is

[00026]$a\cdot F_{1,2}=\left[\begin{array}{c}4/5\\ 1\\ 3/5\end{array}\right]$

wherein a∈R.sup.+, of the light transmittances are the same as their ratio that is specified in Equation (4), and wherein R, G and B are spectral transmittance curves of color filter elements forming ideal Bayer-pattern as shown in FIG. 1. Since maximum value of transmission characteristic of a color filter is equal to or lesser than 1, actual spectral transmission characteristic of F.sub.1,2, defined as J = [R G B] .Math.F.sub.1,2, is obtained by applying min-max normalization method to perform [0,1] normalization on ℸ, as shown in FIG. 2. Thus, actual maximum values of the light transmittances of F.sub.1,2 is given by formula:

[00027]$F_{1,2}=\frac{1}{a}\left[\begin{array}{c}4/5\\ 1\\ 3/5\end{array}\right]$

, wherein a = max ℸ is maximum value of ℸ. Therefore,

[00028]$F_{1,2}=\left[\begin{array}{c}0.561\\ 0.702\\ 0.421\end{array}\right].$

Finally, the ideal absorptive/subtractive color filter F.sub.1,2 is manufactured so as to have a maximum value of 56.1% of light transmittance of Red color-component R, a maximum value of 70.2% of light transmittance of Green color-component G and a maximum value of 42.1% of light transmittance of Blue color-component B at a nearly vertical angle of incident light. The above operations may be performed iteratively for each F.sub.i,j constituting P.sub.2×2 to finally obtain

[00029]$F_{1,2}=\left[\begin{array}{c}0.561\\ 0.702\\ 0.421\end{array}\right],F_{1,1}=F_{2,2}=\left[\begin{array}{c}0.272\\ 0.816\\ 0.272\end{array}\right],F_{2,1}=\left[\begin{array}{c}0.375\\ 0.625\\ 0.5\end{array}\right]$

In this case, light intensity transmitted by the CFA pattern in Equation (5) on underlying photodetector group is

[00030]${\displaystyle \underset{i=1}{\overset{2}{.Math.}}{\displaystyle \underset{j=1}{\overset{2}{.Math.}}{s}_{i,j}=1.48v_R+2.959v_G+1.465v_B}}$

that is equivalent to about 148% as much image sensor sensitivity as available in case of the conventional ideal Bayer filter pattern.

[0028] Let ζ(C.sub.1×3) = [v.sub.R v.sub.G v.sub.B] be solution to a raw image pattern R.sub.2×2 captured by an image sensor array that is associated with P.sub.2×2 as given in Equation (5), which is obtained by solving Equation (3) according to paragraph [0020]. Since

[00031]$\begin{array}{cc}{\text{τ}}_{1,1}^G=\max \left({\text{τ}}_{1,1}^G,{\text{τ}}_{1,2}^G,{\text{τ}}_{2,1}^G,{\text{τ}}_{2,2}^G\right),& {\text{τ}}_{1,2}^R=\max \left({\text{τ}}_{1,1}^R,{\text{τ}}_{1,2}^R,{\text{τ}}_{2,1}^R,{\text{τ}}_{2,2}^R\right),\\ {\text{τ}}_{2,1}^B=\max \left({\text{τ}}_{1,1}^B,{\text{τ}}_{1,2}^B,{\text{τ}}_{2,1}^B,{\text{τ}}_{2,2}^B\right),& {\text{τ}}_{2,2}^G=\max \left({\text{τ}}_{1,1}^G,{\text{τ}}_{1,2}^G,{\text{τ}}_{2,1}^G,{\text{τ}}_{2,2}^G\right)\end{array}$

in .sub.2×2, it follows from Equation (2) that one applies to each raw image pattern, that is in the form of R.sub.2×2, in raw color image data captured by the image sensor array, the image transform

[00032]$R_{2\times 2}=\left[\begin{array}{cc}\left[v_G\right]& \left[v_R\right]\\ \left[v_B\right]& \left[v_G\right]\end{array}\right]$

to transform the raw color image data into raw Bayer-type image data so as to make it compatible with known color demosaicing algorithms that are used to demosaic raw Bayer-type image data (as discussed in Gunturk, B. K., et al.). The obtained raw Bayer-type image data is then demosaiced into a final full-color image by using any one of the color demosaicing algorithms. Note further that final RGB color values in ζ(C.sub.1×3) are weighted averages of all pixel values in R.sub.2×2 prior to its transformation, and thus color accuracy in final full-color image is considerably better than when measured by Bayer image sensor array, wherein each color-component value is measured at a single image sensor pixel location, and then interpolated for neighboring pixels.

[0029] The invention has been described in detail with particular respect to implementations thereof, but it will be appreciated that variations and modifications can be effected within the spirit and scope of the invention. For example, a variety of sensors might be employed, including the sensors of CMOS or CCD imaging arrays. Moreover, color-sensitive elements for use in the invention may have inherent selective sensitivity or may incorporate filters either adjacent to or removed from a broad-wavelength-range sensor, which filters selectively limit the range of sensitivity for individual sensors. Also, while the invention is cast in the environment of camera utilizations, it has other uses, for example, in connection with color printing devices and color display devices.