IMAGE DEMOSAICING TAKING ACCOUNT OF CHROMATIC ABBERATION

Abstract

A method of chromatically-corrected image demosaicing for a sensor with a colour filter array (CFA) such as a Bayer filter array. The demosaicing makes use of local chromatic aberration correction data to determine chromatically-corrected pixel positions for spatial interpolation. In implementations a chromatically-corrected quadratic adjustment is also applied.

Claims

1. A method of chromatically-corrected image demosaicing, the method comprising: inputting image data from an image sensor of an imaging system with chromatic aberration, wherein the image sensor has a plurality of pixels at respective pixel positions and a colour filter array defining a pattern of colour filters, one colour filter over each pixel, and wherein the image data comprises a pixel value for each pixel of the image sensor; and demosaicing the image data to determine a colour value for each of one or more image colours at each pixel position, wherein the demosaicing comprises determining a first colour value for a target pixel covered by a filter of a second colour by: determining a neighbouring set of pixels, each covered by a filter of the first colour, which have pixel positions neighbouring the target pixel, wherein determining the neighbouring set of pixels includes compensating for chromatic aberration by i) determining a chromatically-corrected position of the target pixel, or ii) determining chromatically-corrected positions of each of the neighbouring set of pixels, when determining pixel positions neighbouring the target pixel, spatially interpolating between pixel values of the neighbouring set of pixels to determine the first colour value, and adjusting the first colour value dependent upon an image brightness variation in a region around the target pixel.

2. The method of claim 1 wherein the region around the target pixel comprises a local region of the colour filter array.

3. The method of claim 1 wherein the image brightness variation is defined by a set of third colour values of a set of pixels in the region each covered by a filter of a third colour.

4. The method of claim 3 wherein determining the chromatically-corrected position of a pixel comprises adjusting a position of the pixel by a chromatic position correction dependent upon the chromatic aberration, and wherein the adjusting comprises: determining an adjustment from the third colour values, modifying the adjustment using the chromatic position correction for the or each chromatically-corrected pixel position to determine a modified adjustment, and applying the modified adjustment to the first colour value.

5. The method of claim 4 wherein the adjustment is dependent upon or substantially equal to a linear combination of one or more functions of the or each chromatic position correction, the method further comprising determining one or more coefficients of the linear combination from the third colour values.

6. The method of claim 5, wherein the one or more functions of the or each chromatic position correction are quadratic functions of the or each chromatic position correction.

7. The method of claim 5, wherein the one or more coefficients comprise linear combinations of the third colour values.

8. The method of claim 1 wherein adjusting the first colour value dependent upon an image brightness variation in a region around the target pixel comprises performing a convolution over pixel values in the region around the target pixel.

9. The method of claim 1 wherein adjusting the first colour value comprises determining an adjustment to the first colour value and scaling the adjustment dependent upon a distance between the target pixel and one or more of the neighbouring pixels compensated for chromatic aberration.

10. The method of claim 4 wherein the colour filter array is a 2D array defining a 2D pattern of colour filters, wherein the neighbouring set of pixels comprises a set of pixels in two dimensions, and wherein determining the adjustment comprises determining first and second partial adjustments representing the image brightness variation in each of two orthogonal directions in the array, and combining the first and second partial adjustments to determine the adjustment.

11. The method of claim 1 wherein the first colour is red or blue and the second colour is green.

12. The method of claim 1 wherein the first colour is red or blue and the second colour is red or blue.

13. The method of claim 3, wherein the colour filter array comprises a Bayer filter and wherein the third colour is green.

14. The method of claim 1 wherein determining the chromatically-corrected position of a pixel comprises applying x- and y-offset values to a position of the pixel.

15. The method of claim 1 wherein the neighbouring set of pixels comprises a set of four pixels, and wherein the spatial interpolating comprises bilinear interpolation.

16. The method of claim 1 wherein adjusting the first colour value dependent upon the image brightness variation in the region around the target pixel comprises adjusting the first colour value dependent upon an estimated second or higher partial derivative, with respect to position, of a set of third colour values of a set of pixels in the region each covered by a filter of a third colour.

17. The method of claim 1 wherein adjusting the first colour value dependent upon the image brightness variation in the region around the target pixel comprises adjusting the first colour value dependent upon a linear combination of quadratic functions of two spatial coordinates, with coefficients given by estimated second partial derivatives of the third colour values with respect to the spatial coordinates.

18. A non-transitory data carrier carrying processor control code to implement the method of claim 1.

19. An image processing system for chromatically-corrected image demosaicing, the system comprising: an input to receive image data from an image sensor of an imaging system with chromatic aberration, wherein the image sensor has a plurality of pixels at respective pixel positions and a colour filter array defining a pattern of colour filters, one colour filter over each pixel, and wherein the image data comprises a pixel value for each pixel of the image sensor; and a demosaicing system to demosaic the image data to determine a colour value for each of one or more image colours at each pixel position, wherein the demosaicing system is configured to determine a first colour value for a pixel covered by a filter of a second colour, wherein the demosaicing system is further configured to: determine a neighbouring set of pixels, each covered by a filter of the first colour, which have pixel positions neighbouring the target pixel, compensate for chromatic aberration by i) determining a chromatically-corrected position of the target pixel, or ii) determining chromatically-corrected positions of each of the neighbouring set of pixels, when determining the neighbouring set of pixels, spatially interpolate between pixel values of the neighbouring set of pixels to determine the first colour value, and adjust the first colour value dependent upon an image brightness variation in a region around the target pixel.

Description

DRAWINGS

[0040] These and other aspects of the system will now be further described by way of example only, with reference to the accompanying figures, in which:

[0041] FIG. 1 a schematic diagram of an image sensor with a Bayer filter;

[0042] FIG. 2 illustrates chromatic aberration in an image sensor;

[0043] FIG. 3 is a flow diagram of chromatically-corrected image demosaicing procedure;

[0044] FIG. 4 illustrates a geometry for the procedure of FIG. 3; and

[0045] FIGS. 5a and 5b show, respectively, a block diagram of an imaging system configured to implement chromatically-corrected image demosaicing, and a data capture and sharing system incorporating the imaging system.

[0046] In the figures like elements are indicated by like reference numerals.

DESCRIPTION

[0047] Referring to FIG. 1, this shows an image sensor 100 comprising a plurality of light sensitive pixels 102 at respective pixel positions in this example in a rectangular grid. The sensor is provided with a colour filter array 104 which covers each pixel with a colour filter having, in this example, one of three colours, red, blue, and green. Light is imaged onto the sensor by a lens or other focussing element (not shown in FIG. 1).

[0048] FIG. 1 shows a colour filter array which has a Bayer filter pattern. Thus green filters cover half the sensor pixels, and red and blue filters cover the remaining sensor pixels, all in a regular pattern.

[0049] FIG. 2 illustrates chromatic aberration in the image sensor 100. In FIG. 2 one of the colours, here green, is taken as a reference. Light at the red pixel locations, R, absent chromatic aberration, would have been imaged at the locations indicated by the (solid) arrows from the red pixels. That is the red light at a red pixel location should be considered to have come from the locations at the tips of the red pixel arrows. In a similar way blue light at a blue pixel location should be considered to have come from the locations at the tips of the (dashed) blue pixel arrows.

[0050] FIG. 2 also shows how light from an object such as a cross 200 is split into R, G and B images 202a,b,c by chromatic aberration. The red image 202a, for example, is shifted down and to the left from its true position. The shift corresponds to the correction indicated by the arrows.

[0051] In implementations of the later described system the aberration is modelled by assuming that, locally, all the red pixels are displaced by a fixed amount with respect to the green pixels; blue pixels are treated similarly That is, arbitrarily, the locations of the green pixels are used as a local reference. This can model typical chromatic aberrations such as red/blue or green/purple fringing effects in the corners of an image, and red, green and blue images of slightly different sizes.

[0052] Red and blue colour values at, e.g. green pixel locations are effectively determined by interpolation between locations where the pixels should have been (with a perfect lens). The interpolation may be performed using bilinear or higher order interpolation or a more complex technique. Since there is an assumption of local invariance when determining the colour value at a location the nearest neighbor pixels are used for interpolation rather than more distant pixels, and chromatically corrected pixel positions are used when determining the nearest neighbours.

[0053] The raw image data from sensor 100 comprises pixel values for each of the pixels 102. The task of the image demosaicing procedure is to determine red, green and blue colour values for each pixel position.

[0054] As described above the red and blue pixel values are displaced from their correct positions by chromatic aberration, and thus the procedure should determine corrected red and blue values at the red and blue pixel positions as well as the missing colour values i.e. red and blue values for pixels covered by a green filter, and so forth.

[0055] Since the green pixel positions are taken as a reference grid, green colour values at these positions may be taken as the pixel values from the green pixels. Alternatively an interpolation and/or correction technique as described below may be applied to the green pixel values to determine green colour values at these pixels. For example a green colour value at a green pixel position may be determined by computing a local convolution over green pixel values only (i.e. disregarding red and blue pixels). For example the convolution kernel below may be applied to the green pixel values, to interpolate between green pixel values (the kernel below is centred at a green pixel position):

[00001]$\frac{1}{9}\left(\begin{array}{ccccc}& & -1& & \\ & 2& & 2& \\ -1& & 5& & -1\\ & 2& & 2& \\ & & -1& & \end{array}\right)$

[0056] This can, for example, help to smooth the image and reduce artefacts.

[0057] The image demosaicing procedure of FIG. 3 may be used to determine red and blue colour values at the location of a green pixel; this procedure compensates for chromatically aberration in an imaging system focussing light onto sensor 100. The procedure for determining a red colour value at the location of a green target pixel will be described; the same procedure may be used for determining a blue colour value (i.e. for “red” read “blue”).

[0058] Thus at step S300 the procedure inputs raw image data from sensor 100. Chromatic position correction data is provided, S301, for example by retrieving this from non-volatile memory associated with the sensor and its imaging system. As described later the chromatic position correction data may comprise x- and y-offset values (δx,δy) to be applied to the target pixel position. Thus at step S302 the target pixel position is adjusted according to the chromatic position correction data (offset values), to compensate for the chromatic aberration. This may be performed as an initial step as illustrated in FIG. 3, or online during the procedure. In an alternative implementation the red or blue pixel positions may be chromatically corrected rather than the target pixel position.

[0059] Then for each chromatically-corrected target pixel position a set of nearest neighbour red (or blue) pixel positions is identified (S304). For example in some implementations four nearest neighbour pixel positions may be identified; in other implementations more or fewer nearest neighbours may be used.

[0060] The procedure then interpolates between the red (or blue) pixel values at the nearest neighbour pixel positions to determine the red (or blue) colour value, later, L, at the target pixel position (S306). The interpolation may be a bilinear interpolation or a higher order, or a simpler, interpolation. Details of an example bilinear interpolation are described later.

[0061] In some implementations the procedure then determines an adjustment (correction) to be applied to the red (or blue) colour value at the target, e.g. green, pixel position. For example the procedure may determine a curvature of green colour values in the x- and y-directions, and then determine the error that would be incurred by performing bilinear interpolation on green pixel values as it has been performed on red (or blue) pixel values. This provides an adjustment value (correction) which compensates for the crudity of bilinear interpolation. It is assumed that the change in red (or blue) colour values approximates the same curvature, and thus this correction may be applied to the red (or blue) value derived from the interpolation to provide a corrected red (or blue) value. This assumes that the correction relates more to image brightness than hue i.e. that brightness changes faster than hue. However the changes in red (or blue) pixel values and the changes in green pixel values are misaligned due to chromatic aberration. Thus, when making this correction, the chromatic position correction data is used to compensate for the misalignment. Further, the error incurred in bilinear interpolation depends upon the position at which the interpolant is evaluated: in particular, the error is zero if the position of interest after chromatic aberration correction happens to lie exactly under a pixel of the first colour. Hence the adjustment computed above should be scaled dependent upon the relation between the chromatically corrected location of the target pixel and the actual locations of pixels of the first colour, as explained later.

[0062] The correction has been described with reference to determining a curvature of the pixel value data in e.g. x- and y-directions, i.e. a second spatial derivative. However the correction may also be determined using some other spatial gradient, for example, a derivative of higher order. When using curvature the correction may be approximated by determining discrete approximations to the second derivatives using a set of local pixels i.e. pixels near to the chromatically-corrected position of the pixel whose colour value is to be determined. This may be implemented as a convolution over the local pixels.

[0063] Referring again to FIG. 3, the procedure may determine Laplacian-like corrections, based on curvature in the x- and y-directions, over green pixels neighbouring the target green pixel position (S308). These corrections may then be scaled using the chromatic position correction data (S310), and may be combined to determine an overall quadratic correction, later, Q. This quadratic correction may then be applied to the bilinear interpolated colour value, L, to determine an improved red (or blue) colour value for the green target pixel position (S312).

[0064] Essentially the same procedure may be used to determine red and blue colour values for red and blue pixel positions, as explained further later.

[0065] As an example implementation of the procedure of FIG. 3, suppose the local red chromatic aberration is (δx,δy). This means that a red pixel value near (x,y) comes from the same place as a green pixel value near (x+δx,y+δy). This implies that to obtain a red value near the same location as green pixel (x, y) the procedure needs to look near (x−δx,y−δy).

[0066] This is illustrated in FIG. 4, where for convenience of illustration the green pixel is shown as displaced (though in implementations the green pixels define the reference grid).

[0067] Let the four nominally-red pixel locations surrounding (x−δx,y−δy) be (x.sub.0,y.sub.0), (x.sub.0,y.sub.1), (x.sub.1,y.sub.0), and (x.sub.1,y.sub.1) where 0.1=0.0+2 (recalling that red pixels are spaced two apart on each axis) and (δx,δy) are in units of pixel spacing (e.g δx=1 corresponds to one pixel spacing in the x-direction). Let the red pixel values corresponding to these locations be denoted r.sub.00,r.sub.01,r.sub.10,r.sub.11 respectively. Thus the differences between the x-positions and y-positions for the locations between which the interpolation is performed are δx.sub.i=x−δx−x.sub.i and δy.sub.i=x−δy−y.sub.i. Thus −2≤δx.sub.1,δy.sub.1≤0≤δx.sub.0,δy.sub.0≤+2 and δx.sub.0=δx.sub.1+2, and likewise for y.

[0068] Thus a bilinear interpolation, L, between the red pixel locations to obtain a red value at the chromatically-corrected target pixel position is given by:

¼[(−δx.sub.1)(−δy.sub.1)r.sub.00+(−δx.sub.1)δy.sub.0r.sub.01+δx.sub.0(−δy.sub.1)r.sub.10+δx.sub.0δy.sub.0r.sub.11]

[0069] Discrete approximations to the second derivative may be used to estimate the curvature of the local image brightness. These are applied to the green pixel values only, and may be implemented as convolutions. For example a value α for determining a correction based on curvature in the x-direction, i.e. a correction with quadratic dependence on x, may be given by e.g. a convolution over green pixel values with a kernel (centred at a green pixel position):

[00002]$\frac{1}{48}\left(\begin{array}{ccccc}& & 1& & \\ & -2& & -2& \\ 7& & -8& & 7\\ & -2& & -2& \\ & & 1& & \end{array}\right)$

[0070] Other kernels may be used, e.g. adjusting the coefficients for more efficient computation or using more or fewer points.

[0071] For example a value β for determining a correction based on curvature in the y-direction may be determined similarly, for example by convolution with the transpose of the above kernel.

[0072] A quadratic correction, Q, to be applied to the red colour value, corrected for chromatic aberration (local chromatic displacement), is then given by δx.sub.0δx.sub.1α+δy.sub.0δy.sub.1β.

[0073] In one approach the red colour value used is then given by L+Q. In another approach the correction can be scaled by a ratio of local red and green values, R, G, e.g. from one or more nearby pixels, optionally averaged so that the red colour value used is given by

[00003]$L+\frac{R}{G}Q.$

In a further approach the correction may be scaled by a ratio of differences between the local red and green values and respective white values. For example if R.sub.w,G.sub.w denote red and green values when a white object is imaged then the red colour value used may be given by

[00004]$L+\frac{R_w-R}{G_w-G}Q$

[0074] The same process as that described above may be used to determine a blue colour value at a green pixel position.

[0075] Determination of a green colour value at a green pixel position has already been described. In implementations this only uses pixel values from green pixels.

[0076] There is now described determination of green colour values for red (and blue) pixels; and determination of red and blue colour values for red and blue pixels.

[0077] Implementations of the above described method may also be used to determine chromatically corrected green values at red and blue pixel locations; as described previously the first colour may be green and the second colour may be red or blue.

[0078] To determine green colour values at red (or blue) pixel locations the procedure simply interpolates green pixel values; no correction Q is required. For example the interpolated pixel value may be determined by convolution with the kernel below (centred at a red or blue pixel position):

[00005]$\frac{1}{32}\left(\begin{array}{ccccc}& -1& & -1& \\ -1& & 10& & -1\\ & 10& & 10& \\ -1& & 10& & -1\\ & -1& & -1& \end{array}\right).$

[0079] This convolution is performed for the red and blue pixel positions i.e. for the non-green pixels (i.e. it is not a convolution which is applied to every pixel of the image). As can be seen, the convolution is applied to the green pixel values—that is in implementations all green pixel values are obtained only from green pixels and red and blue pixel values are not used in computing green (output) pixel values.

[0080] The procedure for determining red and blue colour values at red and blue pixel positions is very similar to that for determining red and blue colour values at green pixel positions. The red and blue pixel values are each offset from their true positions by chromatic aberration. Thus in implementations a correction is made for chromatic aberration when determining red colour values for both red and blue pixels, and when determining blue colour values for both red and blue pixels. The only difference is in the calculation of the correction, since it is derived from green pixel values and we are now considering pixel positions that do not lie on the green-pixel lattice.

[0081] Thus the procedure may comprise determining the local chromatic position correction data (δx,δy), for example by retrieving this from storage. Four red/blue pixel positions surrounding the chromatically corrected target position (x−δx,y−δy) are then identified and bilinear interpolation is performed as previously described, e.g. using the same formula for L.

[0082] The quadratic correction, Q, is likewise determined in the same way as previously described, except that a different kernel is used for determining α and β (describing the quadratic dependence on x and y respectively).

[0083] For example to determine α a 12-point estimate of the second derivative with respect to x may be determined by convolution over non-green pixel values with the kernel below. The kernel is centred at a red or blue pixel position; the non-zero values in the kernel are defined at green pixel positions:

[00006]$\frac{1}{608}\left(\begin{array}{ccccc}& -5& & -5& \\ 43& & -46& & 43\\ & -30& & -30& \\ 43& & -46& & 43\\ & -5& & -5& \end{array}\right)$

[0084] Convolution with the transpose of this kernel may be used for determining β. The correction Q may then be determined as previously described, and L and Q may be combined, for example by summation, to determine a red or blue colour value for the target pixel.

[0085] As previously described, approximations to the example kernels given above may be employed, e.g. to speed up computation. For example the factor of 1/608 may be converted to a factor of 1/128, which is easily implemented by bit-shifting, by approximating the kernel as:

[00007]$\frac{1}{128}\left(\begin{array}{ccccc}& -1& & -1& \\ 9& & -10& & 9\\ & -6& & -6& \\ 9& & -10& & 9\\ & -1& & -1& \end{array}\right).$

[0086] FIG. 5a shows a block diagram of an imaging system 400 configured to implement the above described chromatically-corrected image demosaicing.

[0087] The imaging system comprises one or more optical elements 402, such as one or more lenses, configured to form an image on an image sensor 404 bearing a colour filter array, for example an image sensor as shown in FIG. 1. Image data from the image sensor, which may be raw or pre-processed image data, comprising pixel value data for the sensor pixels, is provided as an input to a processor 406.

[0088] In some implementations processor 406 operates under control of stored program code (as shown) to implement the described image demosaicing and provide a demosaiced image output 408. In other implementations processor 406 implements some or all of the image demosaicing functions in hardware, for example in an ASIC or FPGA.

[0089] As illustrated, processor 406 is coupled to non-volatile memory 410 which stores chromatic position correction data, as well as pixel position corrector code, bilinear interpolator code and quadratic corrector code to determine chromatically-corrected pixel positions and L and Q values as described.

[0090] FIG. 5b shows an example application of the imaging system 400 in a data capture and sharing system 450. Thus a camera 452 comprising the imaging elements 402 and image sensor 404 is mounted by a support 454 on a wall 456 so that it may be directed an acute angle towards a surface 458 such as a whiteboard to capture a colour image of writing/drawing on the surface. The acute angle image capture and sharp edges and fine detail in the image make chromatic aberration a particular problem. Thus the above described image demosaicing techniques are particularly helpful for such an application. The image processing may be implemented locally, for example in or in association with the camera, or remotely.

[0091] As the resolution of camera image sensors continues to rise the effects of chromatic aberration are also becoming more important in general purpose cameras. Thus other applications include (but are not limited to) the image sensors of colour cameras in consumer electronic devices such as mobile phones and mobile computing devices.

[0092] Many alternatives will occur to the skilled person. The invention is not limited to the described embodiments and encompasses modifications apparent to those skilled in the art lying within the spirit and scope of the claims appended hereto.