METHOD AND DEVICE FOR SUPPRESSING CHROMATIC ABERRATION IN A DIGITAL IMAGE

Abstract

A method for suppressing chromatic aberration, especially blue or red fringing, in a digital image with multiple color channels is disclosed. The method comprises negatively correcting a first color channel by subtracting an overshoot component of the first color channel. The subtraction is subject to a lower threshold, which is dependent on a local value of at least one further color channel.

Claims

1. A method for suppressing chromatic aberration in a digital image with multiple color channels, the method comprising negatively correcting a first color channel by subtracting an overshoot component of the first color channel, wherein the subtraction is subject to a lower threshold, which is dependent on a local value of at least one further color channel.

2. The method of claim 1, wherein the lower threshold value is a local value of a second color channel.

3. The method of claim 1, wherein the lower threshold value is a combination of local values of a second and a third color channel.

4. The method of claim 1, wherein the lower threshold value has a granularity of one pixel or a group of multiple pixels.

5. The method of claim 1, wherein the first color channel represents a blue, bluish, red or reddish primary color.

6. The method of claim 1, wherein the lower threshold is enforced uniformly throughout the digital image.

7. The method of claim 1, wherein the lower threshold is enforced only in subareas of the digital image.

8. The method of claim 1, further comprising extracting the overshoot component by computing a difference between the first color channel of the digital image and a low-pass filtered copy of the same color channel.

9. The method of claim 8, wherein the extraction of the overshoot component further includes composing the difference with one or more of the following: a ramp function; a constant scaling factor; a scaling function with superlinear asymptotics at zero.

10. The method of claim 1, further comprising applying said negative correction to a second color channel.

11. The method of claim 1, further comprising positively correcting the first or a different one of the color channels by adding an undershoot component of the same color channel subject to an upper threshold, which is dependent on a local value of at least one other color channel.

12. The method of claim 1, wherein the negative correction and any positive correction are executed prior to any dynamic-range compression, tone mapping or sharpening of the digital image.

13. A non-transitory computer-readable storage medium having stored thereon instructions for implementing a method for suppressing chromatic aberration in a digital image with multiple color channels, when executed on a device having processing capabilities, the method comprising: negatively correcting a first color channel by subtracting an overshoot component of the first color channel, wherein the subtraction is subject to a lower threshold, which is dependent on a local value of at least one further color channel.

14. An image processing device for suppressing chromatic aberration in a digital image with multiple color channels, the image processing device comprising a memory and processing circuitry configured to perform a method for suppressing chromatic aberration in a digital image with multiple color channels, the method comprising: negatively correcting a first color channel by subtracting an overshoot component of the first color channel, wherein the subtraction is subject to a lower threshold, which is dependent on a local value of at least one further color channel.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, on which:

[0026] FIG. 1 is a schematic view illustrating effects of axial chromatic aberration;

[0027] FIG. 2 is a schematic view illustrating effects of lateral chromatic aberration;

[0028] FIG. 3 is a flowchart of a method for suppressing chromatic aberration in a digital image, according to embodiments of the present disclosure;

[0029] FIG. 4 is a block diagram of an imaging system;

[0030] FIGS. 5, 6, and 7 show different ways in which an image processing device according to the present disclosure can be arranged in or at an imaging system; and

[0031] FIG. 8 is a schematic signal processing diagram illustrating some aspects of chromatic aberration suppression according to embodiments herein.

DETAILED DESCRIPTION

[0032] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the disclosure are shown. These aspects may, however, be embodied in many different forms and should not be construed as limiting; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of the disclosure to those skilled in the art. Like numbers refer to like elements throughout the description.

[0033] FIGS. 1 and 2, which aim to elucidate some aspects of chromatic aberration, have been described above.

[0034] FIG. 4 is a schematic view of an imaging system, in this case a digital video camera. The imaging system has a lens system 120 imaging a scene on an image sensor 122. The image sensor 122 is provided with a color filter array, e.g., a Bayer filter, such that different pixels will receive radiation of a particular wavelength region, in accordance with a known, usually mosaic-like pattern. After having read the signal of individual sensor pixels of the image sensor 122, demosaicing is carried out 124. At this point, the color information in the scene has been extracted, i.e., the detected signal from the scene has been separated into the components of the color system for which the color filter array is designed. The color channels may be wavelengths constituting primary colors, which can be combined into desired points in a subset (gamut) of the visible color space. The color channels may for example correspond to specified red R, green G, and blue B colors. To correct inadequacies in the color reproduction, a color correction matrix is applied 126 on the demosaiced sensor output. The color correction matrix could be described as a calibration matrix which, for each pixel of the image sensor 122, takes a wavelength-dependent response into account. In a straightforward process, the color correction matrix is deduced by imaging a known target with the imaging system, the target typically being a color rendition chart. Consequently, there is a target of known color properties, a resulting image sensor response, and a color correction matrix that when applied should bring the sensor response closer to the known target, effectively resulting in an optimization task that tends to produce an optimal color correction matrix. An error in color may be described in different ways, and there are many algorithms for deducing a minimum total error. The present disclosure does not advocate any specific color error measure or optimization algorithm. When applying the color correction matrix 126, the response from the image sensor (following demosaicing 124) is transformed to, or at least closer to, the value it should have according to the color rendition chart. In state-of-the-art imaging systems, the actual step of performing imaging of the color rendition chart may be left out, and by using spectral response data for the particular sensor and imaging system, a color correction matrix may be deduced.

[0035] Following application of the color correction matrix 126, the response from the image sensor, pixel by pixel, should have been corrected to a true, or at least more accurate, color representation in which a total color error is minimized. The resulting image is forwarded to an image processing pipeline 128. It should be noted that the application of the color correction matrix 126 could be seen as an integral part of the image processing pipeline 128. The image processing pipeline 128 may include noise filtering (for eliminating spatial and/or temporal noise), distortion correction (for eliminating effects of, e.g., barrel distortion), global and/or local tone mapping (e.g., enabling imaging of scenes containing a wide range of intensities), transformation (e.g., rotation), flat-field correction (e.g., for removal of the effects of vignetting), application of overlays (e.g., privacy masks, explanatory text), dynamic-range correction (including dynamic range compression), sharpening etc. These example operations may be arranged in the order the implementer sees fit. The image processing pipeline 128 may also be associated with an analytics engine performing object detection, object recognition, alarm triggering, etc. Following the image processing pipeline 128, the digital image may be forwarded to an encoder 130, which is responsible for coding the image data according to an encoding protocol and forward to a recipient, a server, a cloud storage, etc. in a desired image format.

[0036] FIG. 3 is a flowchart of a method 300 for suppressing chromatic aberration in a digital image, according to embodiments of the present disclosure. The method 300 may be implemented in an image processing device 510 of the type illustrated in FIGS. 5-7. In relation to the successive stages of the imaging system of FIG. 4, it is advantageous to perform the method 300 prior to the image processing pipeline 128. Preferably, the method 300 is performed after the color correction matrix 126 has been applied. Alternatively, the method 300 may be integrated into the image processing pipeline 128, where it is preferably performed after any noise filtering and prior to any dynamic-range compression, tone mapping or sharpening operation therein. In dual-exposure and similar systems, where two exposures of the same scene are merged, it is preferable to perform the method 300 after the exposure merging, i.e., on the merged image.

[0037] On an overall level, the method 300 is structured into a negative correction block 310 and an optional positive correction block 320. Each of the blocks 310, 320 can be applied to one color channel of the digital image or to multiple color channels. The positive correction 320, if any, can be applied to a different color channel than that or those which undergo negative correction 310. Preferably, the corrections 310, 320 are applied to color channels representing non-central wavelengths in the imaging system's design spectrum, such as blue (bluish) or red (reddish) color channels. The order of the blocks 310, 320 may be as shown in FIG. 3 or inverted. Also possible are staggered sequences, where one or more negative corrections alternate with one or more positive corrections. For example, a negative correction of a first color channel may be followed by a positive correction of the first component, which is followed, in turn, by a negative correction of a second color channel, with optional further positive and/or negative corrections.

[0038] When the negative correction 310 is applied to a first color channel of the digital image, it includes a step of extracting 312 an overshoot component from image data relating to the first color channel of the digital image. Since blue fringing is known to exaggerate sharp edges, the overshoot component may be chosen to represent a high-frequency component of the first color channel, where frequency refers to the spatial variation of the image data. One way of obtaining such a high-frequency component is to first compute a low-pass (LP) filtered copy of the first color channel and then subtract the LP-filtered copy from the original first color channel.

[0039] It is assumed that the digital image is provided in the form of a spatial representation with three color channels {B(u, v), R(u, v), G(u, v)}, each of which expresses the corresponding values (e.g., intensity, luminance) as functions of an index pair (u, v) which designates a pixel. (As announced above, a coarser granularity can be used, wherein the index pair (u, v) designates a predefined group of pixels.) The value functions may be represented as tables, in particular lookup tables, where the entries may be 8-bit (4+4) floating-point numbers. The three color channels are expressed in common or comparable units. Preferably, the three color channels are in common linear units, whereby cross-channel processing is simplified, e.g., linear scaling of a value triplet essentially preserves the hue. The color channels may have an original or near-original dynamic range, or they may have a pre-compressed dynamic range. Because the method 300 does not rely on the occurrence of pixel saturation to detect image areas potentially affected by aberration, the method 300 can be applied also to high dynamic range (HDR) images. The method 300 may in fact perform better when applied to HDR images, since local values of the further color channel(s), from which the lower threshold depends, have not been altered by saturation.

[0040] A spatial representation of one of these types would be a straightforward format in which to exchange an image among components of a digital imaging system. If the method 300 is utilized for postprocessing outside the imaging system which acquired the digital image, the digital image may be provided in a compressed format suitable for storage or data transfer. In this case, it may be necessary to decompress (or unpack) the image data into a spatial representation where all color channels are in common or comparable units. A similar procedure may be a useful preparation to the method 300 if the digital image is a frame of a prediction-coded video sequence, optionally with recourse to earlier or later frames of the sequence. It is noted furthermore that the method 300 cannot readily be applied to image data in YCbCr format, where neither of the luma Y and chroma Cb, Cr components has a direct relationship with a color. A preliminary conversion into a representation in terms of color channels, such as RGB, is therefore necessary.

[0041] Returning to step 312, the literature discloses several suitable algorithms for providing 312.1 a LP-filtered copy of a channel of a digital image encoded according to one of the above types. Examples include the Fourier transform. One option is to Fourier transform (e.g., using a fast Fourier transform (FFT) or direct Fourier transform (DFT)) the spatial representation of a color channel

{B(u, v):u=1,2, . . . , U, v=1,2, . . . V},

where U×V is the resolution of the digital image, into a frequency representation:

{{circumflex over (B)}(k): k=(k.sub.u, k.sub.v), where k.sub.u=1,2, . . . , k.sub.v=1,2, . . . },

such that {circumflex over (B)}(k)=0 for |k|>K.sub.max, and then inversely transforming this into a spatial representation including only frequency content below K.sub.max:

{B.sup.LP(u, v): u=1,2, . . . , U, v=1,2, . . . V}.

To ensure that {circumflex over (B)}(k)=0 for |k|>K.sub.max, one may either design the Fourier-type transformation so that it forgoes the computation of the higher frequency components, or one may eliminate those frequency components subsequently, before the inverse transformation. The upper bound K.sub.max may be determined empirically by executing the method 300 for a tentative K.sub.max value on images of similar scenes and/or with comparable brightness and coloring, and make adjustment until a visually pleasing result is achieved. Alternatively, one may utilize a non-perceptual evaluation and instead compare the frequency response and/or sharpness of the compensated first color channel with the same quantity of a color channel corresponding to a centrally located wavelength, such as green or greenish.

[0042] Another option is to compute 312.1 the low-pass filtered copy by convolving the spatial representation B (u, v) with a kernel κ:

B.sup.LP(u, v)=(B*κ)(u, v)=∫∫B(u′, v′)κ(u′, v′; u, v)du′dv′

The convolution may correspond to uniform blurring, e.g., Gaussian blurring, in which case the kernel is symmetric with respect to the origin:

κ(u′, v′; u, v)=κ(|u′−u|, |v′−v|)

Alternatively, or additionally, the kernel K may be designed to compensate a pointspread function (PSF) of optics in the imaging system by which the digital image was acquired. If the PSF is different for on-axis and off-axis image points, the kernel may not have the above symmetry property; rather, the kernel depends on u, v. It is noted that the convolution operation can be designed to absorb one or more further steps of the method 300, including the different computation 312.2 to be described next, for a suitably chosen kernel κ.

[0043] In implementations, to reduce the requirement for runtime (e.g., on-chip) memory, a kernel κ which is supported on a large interval can be conveniently split into a sum multiple sub-kernels κ=κ.sub.1+κ.sub.2+ . . . , where each sub-kernel is supported on a relatively smaller interval. Optionally, the implementer may define variable offsets for the sub-kernels, wherein the offsets (rather than the sub-kernels themselves) can be made dependent on the distance from the optical axis.

[0044] Next follows a step of computing 312.2 a difference of the color channel and the LP-filtered copy:

B(u, v)−B.sup.LP(u, v).

This difference can optionally be processed further before it is used in the subsequent steps, e.g., by composing 312.3 the difference function with at least one other function g, so that the overshoot component is obtained as:

g(B(u, v)−B.sup.LP(u, v)).

Preferably, the function g is non-decreasing almost everywhere or increasing almost everywhere. In a computer implementation of the method 300, the function g can be represented as a piecewise linear (PWL) function, for which efficient software code is publicly available.

[0045] A first option is to use a ramp function, such as a positive-part or negative-part function as g, respectively defined as:

[00001]$$\begin{gathered} x_{+}=\{\begin{array}{c}x,\mathrm{if}x>0,\\ 0,\mathrm{else},\end{array} \\ {x}_{-}=\{\begin{array}{c}-x,\mathrm{if}x<0,\\ 0,\mathrm{else},\end{array} \end{gathered}$$

or locally smoothed versions of these.

[0046] A second option is to use a constant scaling factor g(x)=α.sub.1x, where the coefficient α.sub.1 can be used to adjust the magnitude of the negative compensation 310. Preferably, the coefficient α.sub.1 is constant for the first color channel throughout the digital image. A corresponding coefficient α.sub.2 for a second color channel can be independent of the first coefficient α.sub.1. Similar to the constant K.sub.max discussed above, the coefficient α.sub.1 may be determined empirically by trying different tentative values and choosing the visually most pleasing one. A non-perceptual evaluation is possible as well, wherein frequency responses and/or sharpness indicator values of the compensated first color channel and a reference color channel corresponding to a centrally located wavelength are compared.

[0047] A third option is to use a nonlinear scaling function. For example, one may use a non-decreasing scaling function with superlinear asymptotics at zero, such as

[00002]$\begin{array}{cc}g\left(x\right)=\{\begin{array}{c}x{.Math.x.Math.}^{p-1},.Math.x.Math.\le 1,\\ x,\mathrm{else},\end{array}& \end{array}$

which for p>1 goes to zero faster than any multiple of x when |x|.fwdarw.0. If the difference B(u, v)−B.sup.LP(u, v) is composed with a scaling function with superlinear asymptotics, the negative compensation .sub.310 will be applied to pixels (or pixel groups) where the difference is significant but not elsewhere. Indeed, those image areas where blue fringing is negligible can be left intact, so as not to run the risk of introducing artefacts unnecessarily. The scaling function may be replaced by a dilated version

[00003]$\begin{array}{cc}{g}_a\left(x\right)=g\left(\frac{x}{a}\right),a>0,& \end{array}$

if the scaling needs to be adapted to the perception of ‘significant’ differences and ‘negligible’ fringing in the use case at hand.

[0048] In different embodiments, the first, second and third options can be advantageously combined by composing the corresponding functions into said g.

[0049] Next, there follows a step of subtracting 314 the overshoot component from the first color channel. According to the method 300, the subtraction is not unrestricted but is subject to (or conditional upon) a lower threshold T.sub.B(u, v) , which is dependent on a local value of at least one further color channel:

B(u, v)max{B(u, v)−g(B(u, v)−B.sup.LP(u, v)), T.sub.B(u, v)},   (1)

where g is the function introduced in the optional step 312.3. In embodiments where the method 300 does not include the optional step 312.3, one may conceptually set g(x)=x in equation (1). The maximum operation ensures that the pixel value B(u, v) will be replaced by a value which is equal to at least the threshold T.sub.B(u, v).

[0050] In some embodiments of the method 300, the threshold can be set equal to the local value of another color channel, e.g., T.sub.B(u, v)=G(u, v) or T.sub.B(u, v)=R(u, v). In other embodiments, the threshold is a combination of the local values of two other color channels:

T.sub.B(u, v)=β.sub.1G(u, v)+β.sub.2R(u, v), β.sub.1, β.sub.2>0.

The combination may be a plain mean value (β.sub.1=β.sub.2=½) or a weighted average (β.sub.1+β.sub.2=1). A further option is to use β.sub.1+β.sub.2=⅔, which in some conditions (when G(u, v) and R(u, v) are approximately equal) will cause the three color channels to have approximately equal intensity, whereby the compensated pixel will have an unobtrusive greyish hue. The combination may as well be a maximum or a minimum of the two other color channels:

T.sub.B(u, v)=max{G(u, v), R(u, v)},

T.sub.B(u, v)=min{G(u, v), R(u, v)}.

[0051] As explained above, the method 300 may be performed with different granularities, that is, the index pair (u, v) may refer to a pixel or to a group of multiple pixels. Specifically, a rule may be applied by which the subtraction step 314 assigns the threshold value to all pixels in a pixel group as soon as any pixel in the group would end up below the threshold value if the overshoot component was subtracted.

[0052] A further option is to perform the subtraction step 314 with a one-pixel granularity but compute the threshold T.sub.B with a coarser granularity. One way to achieve this is to partition the pixel space [1, U]×[1, V] into pixel groups {W.sub.i⊂.sup.2:i ∈ I}, where I is an index set, such that:

[00004]$\underset{i\in I}{.Math.}{W}_i=\left[1,U\right]\times \left[1,V\right]$

and W.sub.i∩W.sub.j=ø for any i≠j. Starting from a threshold with pixel-group granularity, T.sub.B=T.sub.B(W.sub.i), the subtraction operation 314 may still be formulated with one-pixel granularity, though with a modified threshold {tilde over (T)}.sub.B:

B(u, v)max{B(u, v)−g(B(u, v)−B.sup.LP(u, v)), {tilde over (T)}.sub.B(u, v)},   (2)

where {tilde over (T)}.sub.B (u, v) is piecewise constant, i.e., {tilde over (T)}.sub.B(u, v)=T.sub.B(W.sub.i) for i such that (u, v) ∈ W.sub.i. Alternatively, the modified threshold {tilde over (T)}.sub.B(u, V) in (2) may be replaced by an interpolation of {tilde over (T)}.sub.B(u, V). For example, if each W.sub.i is a group of 9×9 pixels, then the interpolation may have a finer granularity of 5×5 pixels. A threshold function with pixel-group granularity may be defined as T.sub.B(W.sub.i)=R(u.sub.i, v.sub.i), where (u.sub.i, v.sub.i) is a central pixel in W.sub.i. Alternatively, an average of the local values may be formed:

[00005]$T_B\left(W_i\right)=\frac{1}{.Math.W_i.Math.}\underset{\left(u,v\right)\in W_i}{.Math.}R\left(u,v\right),$

where |W.sub.i| is the number of pixels in W.sub.i.

[0053] Along the same lines, moreover, it is possible to modify the method 300 such that the threshold T.sub.B has one-pixel granularity but the subtraction 314 is performed with a coarser granularity.

[0054] For a summary of the processing steps described so far, reference is made to the diagram in FIG. 8. Starting from a digital image with three color channels 890, one of the color channels (here: B) is fed to a low-pass filter 810, and a signed summation component 820 forms a difference, which is composed with a ramp function 830 (alternatively: a scaling function 830′ with superlinear asymptotics at zero) and a scaling factor 840, whereby the overshoot component is obtained. The overshoot component is subtracted at position 850 from the original color channel, upon which a maximum operator 860 ensures that the resulting difference is nowhere less than a threshold computed from one or both of the two further color channels (here: G, R). The original color channel is replaced by its value after subtraction by the overshoot component or, in pixels where this is necessary, the threshold value. Save for components 830, 840 and 860, the processing diagram in FIG. 8 may be described as a type of unsharp masking applied to the B color channel.

[0055] In some embodiments, the subtraction 314 subject to the lower threshold is carried out by applying the rule (1) uniformly throughout the first color channel of the digital image.

[0056] In other embodiments, the rule (1) is applied only in some subareas of the digital image. In the remainder of the image, the overshoot component may not be subtracted at all, or it may be subtracted in an unrestricted way without regard to the lower threshold. The subareas may be those with the greatest likelihood of being affected by blue fringing. In one example, an edge filter may be used to localize sharp dark/bright edges and other image features where blue fringing is known to occur frequently, and the subareas may be defined as an environment thereof. In another example, saturated pixels and/or pixels with hues typical of blue fringes are localized, and the subareas are defined as an environment of these pixels. Alternatively, a trained machine-learning model may be utilized to find the dark/bright edges. Optionally, to avoid visible boundaries between the subareas where the substitution rule (1) is applied and a remainder of the image where it is not applied, one or more transition zones may be defined, where a modified rule is used:

B(u, v)max{B(u, v)−γ.sub.1g(B(u, v)−B.sup.LP(u, v)), T.sub.B(u, v)}.   (1′)

The constant γ.sub.1, suitably chosen in the interval (0,1), will have the effect of moderating the subtraction in such manner that the substituted pixel value is closer to the original pixel value B(u, v). A transition zone may be a neighborhood of each subarea with a predefined thickness ϵ>0.

[0057] Returning to FIG. 3, if there is no more color channel due to be processed by the negative correction 310 in the current round (N branch from step 318), the execution flow proceeds to the optional second block, positive correction 320. If the negative correction 310 can be said to address the blue fringes on the dark side of an edge in the image, the role of the positive correction 320 is to restore the ‘missing’ amount of blue on the bright side of the edge. An area with a deficit in blue color may have an artificially yellowish appearance.

[0058] To carry out the positive correction 320, an undershoot component is extracted 322 in an analogous manner as the overshoot component, though with opposite sign inside the g function:

g(B.sup.LP(u, v)−B(u, v)).

The conditional subtraction operation 314 described above corresponds, in the positive correction 320, to a conditional addition 324 subject to an upper threshold. The upper threshold is dependent on a local value of at least one other color channel. The function T.sub.B according to any of the descriptions above may be used as upper threshold as well. This can be summarized as the following substitution rule:

B(u, v)min{B(u, v)+g(B.sup.LP(u, v)−B(u, v)), T.sub.B(u, v)}.   (3)

The positive correction 320 can be modified by inclusion of the same optional features (with a sign change, where necessary) as have been outlined in relation to the negative correction 310. This said, the specifics of the positive correction 320 (including the tuning and adaptation of numerical parameters, the choice of the function g, if any) remain independent of the negative correction .sub.310 and need not resemble or match the negative correction 310.

[0059] The execution of the method 300 may end when there are no more color channels to which the positive correction 320 is to be applied (N branch from step 328). The digital image may then be processed by operations such as dynamic-range compression, tone mapping and sharpening if desired.

[0060] An image processing device 510 with a memory 512 and suitably configured processing circuitry 514 may be used to carry out the method 300. As illustrated in FIG. 5, the image processing device 510 may be an integral part of a digital video camera 500. The video camera 500 may be one suitable for a monitoring use case. It may be for example be fixedly mounted and configured to deliver its outputs as a live stream.

[0061] Alternatively, as shown in FIG. 6, the image processing device 510 may be configured to process outputs of a video camera 500. The image processing device 510 can be co-localized with the video camera 500 or may receive its outputs remotely over a communication network. The image processing device 510 preferably operates in real time, so that a user can inspect a video stream captured by the video camera 500 and processed by the image processing device 510 with a controlled (and typically small) delay.

[0062] FIG. 7 shows a hybrid implementation, where the image processing device 510 is included in the video camera 500 and has the option of delegating some of its computational tasks to a networked (or cloud) processing resource 520.

[0063] The aspects of the present disclosure have mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the disclosure, as defined by the appended patent claims.