Method for processing signals from a matrix for taking colour images, and corresponding sensor

Abstract

The invention relates to the processing operation of interpolating the colours of a Bayer mosaic image sensor. A first elementary matrix filter, which is a bilinear interpolation filter, of size mm, m being an odd number larger than or equal to 3, a low-pass matrix filter of size nn, n being an odd number larger than or equal to 3, and a high-pass matrix filter, complementary to the low-pass filter, of size nn, are defined. The first matrix filter is convoluted with the low-pass filter, resulting in a low-frequency interpolation filter of size (m+n1)(m+n1), and the first matrix filter is convoluted with the high-pass filter, resulting in a high-frequency interpolation filter of size (m+n1)(m+n1). The matrix of digital signals arising from the pixels is filtered separately, using the pixels of each colour, by the low-frequency interpolation filter. The complete matrix of signals is filtered using the high-frequency interpolation filter. The result of the low-frequency filtering operation and the result of the high-frequency filtering operation are added together, for each pixel, in order to obtain a numerical value of a given colour of that pixel.

Claims

1. A method for processing the image delivered by a colour image sensor comprising a mosaic of N coloured pixels arranged in a regular geographic distribution of three colours in which each pixel is associated with one point of the image to be established, each pixel delivering a digital signal whose value represents the luminance component of an image point in the colour corresponding to that pixel, the signals arising from the N pixels being arranged in a matrix and the image processing operation comprising a calculation of interpolating signals of the matrix, allowing each image point to be given three numerical values of luminance components corresponding to the three colours, respectively, while the pixel corresponding to that image point delivers a signal corresponding to a single colour, the method comprising the following steps: defining a first elementary matrix filter, which is a bilinear interpolation filter, of size mm, m being an odd number larger than or equal to 3; defining a low-pass matrix filter of size nn, n being an odd number larger than or equal to 3; defining a high-pass matrix filter, complementary to the low-pass filter, of size nn; convoluting the first elementary matrix filter with the low-pass filter, resulting in a low-frequency interpolation filter of size (m+n1)(m+n1); convoluting the first elementary matrix filter with the high-pass filter, resulting in a high-frequency interpolation filter of size (m+n1)(m+n1); filtering the matrix of digital signals, using the pixels of the first colour only, with the low-frequency interpolation filter, filtering the matrix of pixels, using the pixels of the second colour only, with the low-frequency interpolation filter, and filtering the matrix of pixels, using the pixels of the third colour only, with the low-frequency interpolation filter, these three filtering operations resulting in a first intermediate numerical value being given to each point of the image for each of the colours; filtering the entire matrix of pixels with the high-frequency interpolation filter, without taking the colours into account, in order to establish a second intermediate numerical value for each point of the image; adding, for each pixel, in order to obtain a numerical value of a given colour of that pixel, the first intermediate numerical value corresponding to that pixel and to that colour and the second numerical value corresponding to that pixel.

2. The method of claim 1, wherein the size of the first elementary matrix filter is 33 and it is represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccc}1/4& 1/2& 1/4\\ 1/2& 1& 1/2\\ 1/4& 1/2& 1/4\end{array}\right].& \end{array}$

3. The method of claim 2, wherein the low-pass matrix filter is a simple mean filter, of size 33, and is represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccc}1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\end{array}\right]& \end{array}$ and the high-pass filter is represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccc}-1/9& -1/9& -1/9\\ -1/9& 8/9& -1/9\\ -1/9& -1/9& -1/9\end{array}\right].& \end{array}$

4. The method of claim 2, wherein in the case in which the pixels of one determined colour have a pitch of geographic distribution that is twice smaller than the pitch of the pixels of the other colours, the filtering of the matrix of pixels using the pixels of this determined colour with the low-frequency interpolation filter is followed by a division by two in order to deliver the first intermediate value for that colour.

5. The method of claim 1, wherein the low-pass matrix filter is a simple mean filter, of size 33, and is represented by the following matrix: $\left[\begin{array}{ccc}1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\end{array}\right]$ and the high-pass filter is represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccc}-1/9& -1/9& -1/9\\ -1/9& 8/9& -1/9\\ -1/9& -1/9& -1/9\end{array}\right].& \end{array}$

6. The method of claim 5, wherein in the case in which the pixels of one determined colour have a pitch of geographic distribution that is twice smaller than the pitch of the pixels of the other colours, the filtering of the matrix of pixels using the pixels of this determined colour with the low-frequency interpolation filter is followed by a division by two in order to deliver the first intermediate value for that colour.

7. The method of claim 1, wherein in the case in which the pixels of one determined colour have a pitch of geographic distribution that is twice smaller than the pitch of the pixels of the other colours, the filtering of the matrix of pixels using the pixels of this determined colour with the low-frequency interpolation filter is followed by a division by two in order to deliver the first intermediate value for that colour.

8. A method for processing the image delivered by a colour image sensor comprising a mosaic of N coloured pixels arranged in a regular geographic distribution of three colours in which each pixel is associated with one point of the image to be established, each pixel delivering a digital signal whose value represents the luminance component of an image point in the colour corresponding to that pixel, the digital signals arising from the N pixels being arranged in a matrix and the image processing operation comprising an interpolation calculation allowing each image point to be given three numerical values of luminance components corresponding to the three colours, respectively, while the pixel corresponding to this image point delivers a signal corresponding to a single colour, the method comprising the following steps: defining a low-frequency interpolation filter of size 55, represented by the following matrix: $\left[\begin{array}{ccccc}1/36& 3/36& 4/36& 3/36& 1/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 4/36& 12/36& 16/36& 12/36& 4/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 1/36& 3/36& 4/36& 3/36& 1/36\end{array}\right]$ defining a high-frequency interpolation filter of size 55 represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccccc}-1/36& -3/36& -4/36& -3/36& -1/36\\ -3/36& 0& 6/36& 0& -3/36\\ -4/36& 6/36& 20/36& 6/36& -4/36\\ -3/36& 0& 6/36& 0& -3/36\\ -1/36& -3/36& -4/36& -3/36& -1/36\end{array}\right]& \end{array}$ filtering the matrix of signals, using the pixels of the first colour only, with the low-frequency interpolation filter, filtering the matrix of signals, using the pixels of the second colour only, with the low-frequency interpolation filter, and filtering the matrix of signals, using the pixels of the third colour only, with the low-frequency interpolation filter, these three filtering operations resulting in a first intermediate numerical value being given to each point of the image for each of the colours; filtering the entire matrix of signals with the high-frequency interpolation filter, in order to establish a second intermediate numerical value for each point of the image; for each pixel, adding, in order to obtain a numerical value of a given colour of that pixel, the first intermediate numerical value corresponding to that pixel and to that colour and the second numerical value corresponding to that pixel.

9. A colour image sensor comprising a mosaic of N coloured pixels arranged in a regular geographic distribution of three colours in which each pixel is associated with one point of the image to be established, each pixel delivering a digital signal whose value represents the luminance component of an image point in the colour corresponding to that pixel, the digital signals arising from the N pixels being arranged in a matrix and the image processing operation comprising an interpolation calculation allowing each image point to be given three numerical values of luminance components corresponding to the three colours, respectively, while the pixel corresponding to that image point delivers a signal corresponding to a single colour, the sensor comprising: means for filtering the matrix of signals, using the pixels of the first colour only, with a low-frequency interpolation filter, filtering the matrix of signals, using the pixels of the second colour only, with the low-frequency interpolation filter, and filtering the matrix of signals, using the pixels of the third colour only, with the low-frequency interpolation filter, these three filtering operations resulting in a first intermediate numerical value being given to each point of the image for each of the colours; means for filtering the entire matrix of signals with a high-frequency interpolation filter, in order to establish a second intermediate numerical value for each point of the image; means for adding in order to obtain, for each pixel, a numerical value of a given colour of that pixel which is the sum of the first intermediate numerical value corresponding to that pixel and to that colour and the second numerical value corresponding to that pixel; the low-frequency interpolation filter of size 55, being represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccccc}1/36& 3/36& 4/36& 3/36& 1/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 4/36& 12/36& 16/36& 12/36& 4/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 1/36& 3/36& 4/36& 3/36& 1/36\end{array}\right]& \end{array}$ and the high-frequency interpolation filter of size 55 being represented by the following matrix: $\begin{array}{cc}\left[\begin{array}{ccccc}-1/36& -3/36& -4/36& -3/36& -1/36\\ -3/36& 0& 6/36& 0& -3/36\\ -4/36& 6/36& 20/36& 6/36& -4/36\\ -3/36& 0& 6/36& 0& -3/36\\ -1/36& -3/36& -4/36& -3/36& -1/36\end{array}\right].& \end{array}$

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Other characteristics and advantages of the invention will become apparent upon reading the detailed description which follows, given with reference to the appended drawings in which:

(2) FIG. 1 shows a typical matrix of colour pixels, referred to as a Bayer matrix;

(3) FIG. 2 shows the step of low-frequency interpolation filtering;

(4) FIG. 3 shows the step of high-frequency interpolation filtering and the step of adding the results of the two interpolation filtering operations.

DETAILED DESCRIPTION

(5) FIG. 1 shows a mosaic of pixels of a colour image sensor comprising colour pixels arranged according to the principle of a Bayer mosaic. The colour for a given pixel is represented by a letter on that pixel. The letter R denotes a red-coloured pixel, the letter G a green pixel, and the letter B a blue pixel. A pixel of a given colour collects a signal representing the component of that colour in the light received by the pixel. There are two green pixels, one red pixel and one blue pixel in an elementary pattern repeated in the mosaic.

(6) In the prior art, an interpolation filter would typically be used to assign, to any pixel of any colour, three numerical values corresponding to the three colours. The filter is represented by a matrix of weighting coefficients. Typically, a 33 bilinear filter with nine coefficients is represented by the following matrix:

(7) $\begin{array}{cc}\left[\begin{array}{ccc}1/4& 1/2& 1/4\\ 1/2& 1& 1/2\\ 1/4& 1/2& 1/4\end{array}\right]& \end{array}$

(8) The position of a coefficient in the filter represents the position of a pixel in a block of 33 pixels. The coefficient associated with a pixel position gets smaller as the position of the pixel gets further away from the centre pixel.

(9) Any pixel is taken. The centre position in a block of 33 pixels is assigned thereto, i.e. the block is formed around this chosen pixel. The nine pixels of this block all deliver signals, each in its specific colour.

(10) The sum of the values of signals delivered by the green pixels of the block is then assigned, as the green value Vg, to the centre pixel, each value being assigned the weighting coefficient which, in the matrix filter, corresponds to the position of the green pixel in the block.

(11) Likewise, the sum of the values of signals delivered by the blue pixels of the block is assigned, as the blue value Vb, to the centre pixel, each value being assigned the weighting coefficient which, in the matrix filter, corresponds to the position of the blue pixel in the block.

(12) Lastly, the sum of the values of the red pixels of the block is then assigned, as the red value Vr, to the centre pixel, each value being assigned the coefficient corresponding, in the matrix filter, to the position of the red pixel in the block.

(13) Thus, any pixel of the mosaic, regardless of the colour to which it is sensitive, is given three levels of received light, corresponding to the three colours.

(14) In the method according to the invention, interpolations from blocks of pixels centred on any pixel, and from a matrix of coefficients representing the weighting coefficients that must be assigned to each pixel of the block, are also carried out. However, the particularity of the invention lies in the manner in which the interpolation filter, in this instance two interpolation filters, is chosen, and in the manner in which the results of the two filtering operations carried out are combined.

(15) The first interpolation filter, represented by a first matrix of coefficients, is obtained by convoluting a bilinear filter of size mm (for example, m=3) and a low-pass filter of size nn (for example, n=3). The low-pass filter is preferably a simple mean value filter represented by a matrix of coefficients all equal to 1/9. The coefficient 1/9 arises from the fact that there are nine pixels and that the sum of the signals must be divided by nine if it is desired to reduce it to a normalized value corresponding to one pixel.

(16) $\left[\begin{array}{ccc}1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\\ 1/9& 1/9& 1/9\end{array}\right]$

(17) The second interpolation filter, represented by a second matrix of coefficients, is obtained by convoluting the same bilinear filter of size mm and a high-pass filter, complementary to the low-pass filter, and of the same size nn. The low-pass and high-pass filters are complementary in the sense that they have the same cutoff frequency. The high-pass filter is represented by the matrix of coefficients:

(18) $\left[\begin{array}{ccc}-1/9& -1/9& -1/9\\ -1/9& 8/9& -1/9\\ -1/9& -1/9& -1/9\end{array}\right]$

(19) The bilinear filter is preferably the simplest possible bilinear filter that is represented by the matrix:

(20) $\left[\begin{array}{ccc}1/4& 1/2& 1/4\\ 1/2& 1& 1/2\\ 1/4& 1/2& 1/4\end{array}\right]$

(21) The principle of convoluting two matrices is recalled.

(22) The convolution of a matrix mm by a kernel matrix nn (where n is odd) gives a matrix of (m+n1)(m+n1). It is established in the following manner: the matrix mm is extended by supplementing the mm values with n1 rows of zeros above, n1 rows of zeros below, n1 columns of zeros on the left, and n1 columns of zeros on the right; next, for each of the points of a resulting matrix of (m+n1)(m+n1) points, a block of nn points, centred on a corresponding point of the extended matrix, is formed in the extended matrix, and the sum of the values of the nn points of this block, which are weighted by the corresponding coefficients of the kernel matrix nn, is calculated; this sum constitutes the value of the point in question of the matrix of (m+n1)(m+n1) values.

Example

(23) TABLE-US-00001 Matrix m m (3 3): Kernel matrix n n (3 3): A B C j k p D E F q r s G H J t u v

(24) Matrix mm extended by rows and columns of zeros:

(25) ##STR00001##

(26) Result of the convolution: 55 matrix

(27) 0$\begin{array}{ccccc}{A}^{}& B^{}& C^{}& D^{}& E^{}\\ F^{}& G^{}& H^{}& I^{}& J^{}\\ K^{}& L^{}& M^{}& N^{}& P^{}\\ Q^{}& R^{}& S^{}& T^{}& U^{}\\ V^{}& W^{}& X^{}& Y^{}& Z^{}\end{array}$

(28) where:

(29) A=the sum of the coefficients of the first block mm on the left of the extended matrix (surrounded by a solid line) weighted by the coefficients of the kernel matrix mm; A=Av

(30) B=the sum of the coefficients of the next block to the right of the first (surrounded by a dotted line) weighted by the coefficients of the kernel matrix;
B=Au+Bv
C=At+Bu+Cv
etc. up to Z=k*J, via
M=Aj+Bk+Cp+Dq+Er+Fs+Gt+Hu+Jv

(31) This is a conventional convolution operation.

(32) According to the invention, the convolution of a bilinear filter and of a low-pass filter is therefore carried out, resulting in an interpolation filter referred to as a low-frequency interpolation filter.

(33) Likewise, the convolution of a bilinear filter and of a high-pass filter, complementary to the low-pass filter, is carried out, resulting in an interpolation filter referred to as a high-frequency interpolation filter.

(34) The result of the conventions is then the following:

(35) a) Low-Frequency Interpolation Filter:

(36) $\begin{array}{cc}\left[\begin{array}{ccccc}1/36& 3/36& 4/36& 3/36& 1/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 4/36& 12/36& 16/36& 12/36& 4/36\\ 3/36& 9/36& 12/36& 9/36& 3/36\\ 1/36& 3/36& 4/36& 3/36& 1/36\end{array}\right]& \end{array}$

(37) b) High-Frequency Interpolation Filter:

(38) $\left[\begin{array}{ccccc}-1/36& -3/36& -4/36& -3/36& -1/36\\ -3/36& 0& 6/36& 0& -3/36\\ -4/36& 6/36& 20/36& 6/36& -4/36\\ -3/36& 0& 6/36& 0& -3/36\\ -1/36& -3/36& -4/36& -3/36& -1/36\end{array}\right]$

(39) These are 55 filters if starting from a 33 bilinear filter and a 33 low-pass filter. The interpolation of all of the pixels of the mosaic is carried out by the low-frequency interpolation filter (FIG. 2), for each of the respective colours, in order to give, to each of the pixels and regardless of the colour to which it is sensitive, a first red numerical value R1, a first green numerical value G1, and a first blue numerical value B1. These numerical values result from the signals (R for red, G for green, B for blue) detected by the 25 pixels of the block and the corresponding coefficients of the matrix. The interpolation is therefore carried out colour by colour. In practice, and this is an advantage of the invention, this interpolation by the low-frequency interpolation filter is calculated in a single pass, since, in each position of the filter, the colour being processed is known, which is the colour of the pixel on which the filter is centred.

(40) Furthermore, the interpolation (FIG. 3) of all of the pixels of the mosaic is carried out by the high-frequency interpolation filter, without taking the colour into account, in order to give, to each of the pixels and regardless of the colour to which they are sensitive, a second numerical value W. This second numerical value, assigned to any pixel, results from the weighted sum of the signals detected by the 25 pixels of the 55 block formed around the pixel in question. The weighting is that of the matrix above, representing the high-frequency interpolation filter, in which the weighting coefficient used for one pixel of the block is that which, in the matrix, corresponds to the position of that pixel in the block.

(41) Then, in order to give a final red numerical value to any pixel, the second numerical value of that pixel is added to the first red numerical value of that pixel. Likewise, the second numerical value of the pixel is added to the first green numerical value of that pixel in order to obtain a final green value. Likewise again, the second numerical value of the pixel is added to the first blue numerical value of that pixel in order to obtain a final blue value.

(42) It should be noted that if there are more green pixels than red or blue pixels, as is generally the case with Bayer matrices, provision will be made for the low-frequency filter for the colour green to be assigned a coefficient of so that it has the same weight as the other colours.

(43) The final numerical values VR, VG, VB assigned to a pixel for the three colours are therefore, respectively:
VR=R1+W
VG=(G1)/2+W
VB=B1+W