SIMULTANEOUS AND CONSISTENT HANDLING OF IMAGE DATA AND ASSOCIATED NOISE MODEL IN IMAGE PROCESSING AND IMAGE SYNTHESIS

Abstract

A method for processing image data having noise and information, including: acquiring input raw image data having pixels of an image sensor used to take the image data, processing the input data, and outputting processed image-output data. The step of acquiring input data includes acquiring an input-noise model from the input data, and the step of processing the input raw image data includes a preprocessing operation and determining an output-noise model adapted to reflect noise in the output data, and producing output raw-image data consistent with the output-noise model, and the step of outputting the processed image data includes storing and/or transmitting the output raw image data and the output-noise model, which together form the output data, in a manner linking the output raw image data to the output-noise model, thereby allowing processing of the output data, as input data, such that the processing is adapted for pipeline processing.

Claims

1. A method for processing of image data, wherein said image data comprises noise and information, the method comprising the steps of: acquiring input data comprising input raw image data to be processed for storage and/or transmission, said input raw image data comprising values y of pixels of an image sensor used to take the image data, processing said input raw image data, outputting the processed image data by providing output data, wherein the step of acquiring input data comprises the step of acquiring an input noise model from said input data, said input noise model being adapted to reflect noise present in said input raw image data, said input raw image data and said input noise model together forming said input data in a manner linking the input raw image data to the input noise model, wherein the step of processing said input raw image data comprises at least one of the processing operations of correcting pixel values of said input raw image data, applying flat-field correction to said input raw image data, applying correction of photo-response-non-uniformity to said input raw image data, applying a binning and/or averaging operation to said input raw image data, performing normalization of said input raw image data, applying a rounding and/or truncation operation and/or integer quantization to said input raw image data, performing image synthesis, in particular by creating and/or adding synthetic images based upon said input raw image data and/or by modulating said input raw image data, said processing step further comprising the steps of determining, based upon said input noise model and depending on said processing operation(s) applied to the input raw image data, an output noise model adapted to reflect noise present in said output data, and producing, based upon said processing operation(s) applied to the input raw image data as well as on said input noise model and/or said output noise model, output raw image data which is statistically consistent with said output noise model, wherein the step of outputting the processed image data comprises storing and/or transmitting said output raw image data and said output noise model, which together form said output data, in a manner linking the output raw image data to the output noise model, such as to allow for processing of said output data, as input data, by any one of said processing operations, alone or in any combination thereof, such that said processing is adapted for pipeline processing.

2. The method according to claim 1, wherein said input noise model and/or said output noise model is/are represented by a mapping σ.sub.y(μ.sub.y) of mean pixel values μ.sub.y of the values y of a given pixel of the image sensor used to take the image data to pixel value standard deviations σ.sub.y, a mapping σ.sub.y.sup.2(μ.sub.y) of mean pixel values μ.sub.y to pixel value variances σ.sub.y.sup.2, or a mapping SNR.sub.y(μ.sub.y) of mean pixel values μ.sub.y to signal-to-noise ratios SNR.sub.y.

3. The method according to claim 1, wherein said input noise model and/or said output noise model is/are represented by: a finite set of points which is adapted for obtaining a continuous model σ.sub.y(μ.sub.y) by linear or non-linear interpolation a set of coefficients adapted for constructing a polynomial function approximating a continuous model σ.sub.y(μ.sub.y), or a function with free parameters for a continuous model σ.sub.y(μ.sub.y), combined with specific values for said parameters, where σ.sub.y is the standard deviation of the values y of a given pixel of an image sensor used to take the image data to its mean value μ.sub.y.

4. The method according to claim 1, wherein said input noise model and/or said output noise model is/are adapted to represent noise in said input raw image data and/or said output raw image data which follows a Poisson-Gaussian distribution described by ${\sigma }_y\left({\mu }_y\right)=\{\begin{array}{cc}{\sigma }_{y.\mathrm{dark}}& \mathrm{for}{\mu }_y<{\mu }_{y.\mathrm{dark}}\\ \sqrt{K\left({\mu }_y-{\mu }_{y.\mathrm{dark}}\right)+{\sigma }_{y.\mathrm{dark}}^2}& \mathrm{otherwise}.\end{array},$ where σ.sub.y is the standard deviation of the values y of a given pixel of the image sensor used to take the image data to its mean value μ.sub.y, σ.sub.y.dark is the standard deviation of the pixel values in the absence of light, μ.sub.y.dark is the mean value of the pixel values in the absence of light, and K is a gain factor.

5. The method according to claim 1, wherein said processing step comprises applying a transformation S(y, q) representing a noise-consistent operation to the values y of a given pixel of an image sensor used to take the image data, where q is a pixel value reduction factor, with 0<q<1 , said processing step using an output noise model which is identical to the input noise model.

6. The method according to claim 5, wherein said transformation S(y, q) representing a noise-consistent scaling operation is described by y′=q.Math.y+δ(y, q), where y′ is a corrected pixel value after applying said transformation, δ is a (pseudo-)random number sampled from a normal distribution with mean value 0 and variance σ.sub.δ.sup.2=σ.sub.y.sup.2.Math.(q′.sup.2−q.sup.2) and where $q^{\prime }=\frac{{\sigma }_y\left(\mathrm{qy}\right)}{{\sigma }_y\left(y\right)}$ is a factor by which the standard deviation σ.sub.y(qy) of the noise of the output raw image data (150) is reduced for pixel value reduction factor q , with q<q′<1.

7. The method according to claim 4, wherein the variance of the distribution of said (pseudo-)random number δ of said transformation S(y, q) representing a noise-consistent scaling operation is described by ${\sigma }_{\delta }^2=\frac{y-{\mu }_{y.\mathrm{dark}}}{K}q\left(1-q\right)+{\sigma }_{y.\mathrm{dark}}^2\left(1-q^2\right).$

8. The method according to claim 1, wherein said processing step comprises applying a noise-inconsistent operation to the values y of a given pixel of an image sensor used to take the image data, said processing step using an output noise model which is different as compared to the input noise model and consistent with the noise of the output raw image data of the output data.

9. The method according to claim 5, wherein said at least one processing operation applied during the step of processing said input raw image data is chosen from the group of processing operations comprising correcting pixel values of said input raw image data, applying flat-field correction to said input raw image data, applying correction of photo-response-non-uniformity to said input raw image data, each of said processing operations being applied to the input raw image data in a manner that the output noise model is identical to the input noise model, by ensuring that the input noise model remains valid for all pixel values of the output raw image data.

10. The method according to claim 9, wherein correcting pixel values of said input raw image data is performed by replacing bad pixel values y.sub.C by using neighboring pixel values y.sub.L and y.sub.R for assigning corrected pixel values y′.sub.C=S(Y.sub.L+Y.sub.R−μ.sub.y.dark, q=1/2), where q is the pixel value reduction factor and μ.sub.y.dark is the mean value of the pixel values in the absence of light, and/or applying flat-field correction to said input raw image data is performed by assigning corrected pixel values y′=q.Math.y+δ(y, q) by using a pixel-dependent reduction factor q(i, j) according to $q\left(i,j\right)=\frac{\underset{i,j}{\min }N\left(i,j\right)}{N\left(i,j\right)},$ with N(i,j) representing a reference flat-field image and with i and j denoting pixel coordinates, and/or applying correction of photo-response-non-uniformity to said input raw image data is performed by assigning corrected pixel values y′=q.Math.y+δ(y, q) by using a pixel-dependent reduction factor q(i, j) according to $q\left(i,j\right)=\frac{\underset{i,j}{\min }N\left(i,j\right)}{N\left(i,j\right)},$ with N(i, j) representing a spatially varying relative detection efficiency, determined by calibration, of an image sensor used to take the image data and with i and j denoting pixel coordinates.

11. The method according to claim 8, wherein at least one processing operation applied during the step of processing said input raw image data is chosen from the group of processing operations comprising applying a binning and/or averaging operation to said input raw image data, performing normalization of said input raw image data, applying a rounding and/or truncation operation and/or integer quantization to said input raw image data, performing image synthesis by creating and/or adding synthetic images based upon said input raw image data and/or by modulating said input raw image data, each of said processing operations being applied to the input raw image data in a manner that the output noise model is different as compared to the input noise model, by ensuring that the output noise model determined during the step of processing is valid for all pixel values of the output raw image data.

12. The method according to claim 11, wherein: applying a binning operation to said input raw image data is performed by forming groups of N=n.Math.m pixels or of N exposures, assigning to each group an output pixel value Y=Σ.sub.i=1.sup.Ny.sub.i, with i denoting pixel coordinates, and determining the corresponding output noise model σ.sub.Y(μ.sub.Y) by increasing the black level to μ.sub.Y.dark=Nμ.sub.y.dark and the readout noise to σ.sub.Y.dark=√{square root over (N)}.sub.y.dark, and/or applying a binning operation to said input raw image data is performed by forming groups of N=n.Math.m pixels, assigning to each group an output pixel value Y=(Σ.sub.i=1.sup.Ny.sub.i)−(N−1)μ.sub.y.dark, with i denoting pixel coordinates, and determining the corresponding output noise model σ.sub.Y(μ.sub.Y) by maintaining the black level μ.sub.y.dark and increasing the readout noise to σ.sub.Y.dark=√{square root over (N)}σ.sub.y.dark, and/or applying an averaging operation to said input raw image data is performed by forming groups of N=n.Math.m pixels, assigning to each group an output pixel value Ý=1/NΣ.sub.i=1.sup.Ny.sub.i and determining the corresponding output noise model σ.sub.Y(μ.sub.Y) by maintaining the black level μ.sub.Y.dark and reducing the readout noise to σ.sub.Ý.dark=σ.sub.y.dark/√{square root over (N)}, and/or performing normalization of said input raw image data is done by determining, for a sequence i=1,2 . . . , M} of image input data coming from a number M of different devices with Poisson-Gaussian noise-models having different parameters {(K.sub.i, μ.sub.y.dark.sup.(i), σ.sub.y.dark.sup.(i))∨i=1,2 . . . , M}, the input noise model having the highest normalized readout noise ${\sigma }_{ϵ}={\max }_i\frac{{\sigma }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i},$ assigning to each pixel value y.sub.i of image input data (100) with index number i an output pixel value ${ϵ}_i=\frac{y_i-{\mu }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i}+{\delta }_i,$ with δ.sub.i being a (pseudo-)random number sampled from a normal distribution with mean value 0 and variance ${\sigma }_{ϵ}^2-{\left(\frac{{\sigma }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i}\right)}^2,$ and with the output noise model, having the parameters {K=1, μ.sub.∈.dark=0, σ.sub.∈.dark=σ.sub.∈}, being common to all output raw image data (150) of the sequence i=1,2 . . . , M} of image output data, and/or applying a rounding and/or truncation operation and/or integer quantization to said input raw image data is done by addition of a random number Δ, which is different for each pixel, before said rounding/truncation/quantization operation according to y′=round(y+Δ) by simultaneously adapting the readout noise of the output noise model according to σ′.sub.y.dark.sup.2=σ.sub.y.dark.sup.2+σ.sub.66.sup.2, where σΔ.sup.2 is the variance of the distribution for the random number Δ, and/or performing image synthesis, in particular by creating and/or adding synthetic images based upon said input raw image data and/or by modulating said input raw image data, is done by simulating images having a reduced signal-to-noise ratio as compared to the input raw image data and having the same output noise model, this output noise model being realized by the noise model of a single set of input raw image data or by of a set of normalized input raw image data or is done by simulating images having an increased signal-to-noise ratio as compared to the input raw image data.

13. The method according to claim 1, wherein storing and/or transmitting said output raw image data and said output noise model, which together form said output data, in a manner linking the output raw image data to the output noise model is performed by: storing the output raw image data together with the output noise model in a container file, or storing the output raw image data in a first file and the output noise model in a second, separate sidecar file which is connected to the first file comprising the output raw image data, said sidecar file being adapted to store additional (meta-)data that is not supported by said first file, or storing the output noise model in a database adapted to link specific output raw image data to its associated output noise model.

14. The method according to claim 13, wherein storing and/or transmitting the output noise model is performed by storing and/or transmitting a digital representation of the parameters of said model, said parameters being passed to a function with fixed definition or to a catalog of functions, the specific function to be used in the latter case being selected by an additional identifying parameter.

15. A computer program stored in a tangible computer-readable medium adapted to implement the method according to claim 1.

16. A device equipped with the computer program according to claim 15, wherein the device is chosen from a group comprising a microprocessor, a field-programmable gate array, an image sensor, a mobile phone, in particular a smart phone equipped with a digital camera, a digital photo apparatus, a digital video camera, a scanning device, a tablet, a personal computer, a server, a microscope, a telescope, and a satellite.

17. The method according to claim 6, wherein the variance of the distribution of said (pseudo-)random number δ of said transformation S(y, q) representing a noise-consistent scaling operation is described by ${\sigma }_{\delta }^2=\frac{y-{\mu }_{y.\mathrm{dark}}}{K}q\left(1-q\right)+{\sigma }_{y.\mathrm{dark}}^2\left(1-q^2\right).$

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The attached figures exemplarily and schematically illustrate the principles as well as several embodiments of the present invention.

[0039] FIG. 1 is a schematic illustration of a noise-consistent image processing according to the present invention.

[0040] FIG. 2 is an illustration of an embodiment of a method of noise-consistent image processing according to the present invention combining several features to optimize the performance of the image processing pipeline which starts from initial input data that consist of input raw image data and input noise model.

[0041] FIG. 3 schematically illustrates that, for some embodiments of a method of noise-consistent image processing according to the present invention, two or more processing operations operating on input data may be combined into a single processing operation that performs the combined function of each of said processing operations.

DETAILED DESCRIPTION OF THE INVENTION

[0042] In the following, the invention shall be described in detail with reference to the above mentioned figures.

[0043] The concept of this invention is illustrated in FIG. 1. The invention is concerned with the processing of raw image data 110, but in contrast to established methods, the raw image data 110 only is one part of the input data 100 to the processing algorithm 130. The processing algorithm 130 requires a noise model 120 as a second input. Raw image data 110 and noise model 120, forming together the input data 100, are then processed together to produce output data 140, consisting of output raw image data 150 and an output noise model 160. The processing 130 produces output raw image data 150 where every pixel of the output is consistent with the output noise model 160. Examples of noise-consistent processing 130 will be given below. For some processing algorithms, the output noise model 160 will be identical to the input noise model 120.

[0044] The reason that the output image data 150 is labelled as raw is that it is statistically equivalent to actual raw image data coming from a sensor for which the output noise model 160 applies. Hence, the output data 140 is adapted to serve as input data 100 for another step of noise-consistent processing 130, such that a whole processing pipeline can be constructed based thereupon.

Noise Models

[0045] Noise models are an indispensable building block of the current invention. According to, e.g., the EMVA1288 standard, the following considers as a noise model a mapping σ.sub.y(μ.sub.y) that relates the standard deviation σ.sub.y of the values y of a given pixel to its mean value μ.sub.y. In general, the mean valueμ.sub.y depends on the flux and wavelength of the photons hitting the pixel, but the standard deviationσ.sub.y only depends on the mean valueμ.sub.y. While the following will focus on the above introduced mapping σ.sub.y(σ.sub.y), equivalent mappings exist which can be used to extract σ.sub.y(μ.sub.y), such as the dependence of the pixel variance σ.sub.y.sup.2 on the mean valueμ.sub.y or the signal-to-noise ratio

[00001]${\mathrm{SNR}}_y=\frac{{\mu }_y-{\mu }_{y.\mathrm{dark}}}{{\sigma }_y},$

where μ.sub.y.dark is the black level, i.e. the mean pixel value in absence of illumination.

[0046] The above mentioned mapping/model is related to so-called temporal noise which quantifies the pixel value fluctuations for repeated acquisitions under identical conditions. In principle, each pixel of a given image sensor can have its own characteristic mapping σ.sub.y(μ.sub.y), but modern image sensors have reached an extremely good uniformity, such that in many cases using the same model for all pixels of the sensor will form a good approximation. Nevertheless, an image sensor can exhibit spatial noise that quantifies the differences of the mean values of different pixels exposed to identical illumination. Spatial noise can often be reduced or even removed with the help of preprocessing. Bad pixels can be considered as an extreme form of spatial noise.

[0047] Depending on the type of image data to be processed, a suitable parametrization of the mappingσ.sub.y(μ.sub.y)may serve as digital representation for the noise model. Examples hereof, usable as input noise model 120 or output noise model 160, are [0048] a finite set of points {(μ.sub.i, σ.sub.i} from which a continuous model σ.sub.y(μ.sub.y) is obtained by linear or non-linear interpolation or by any other appropriate approximation for said set of points, or [0049] a set of coefficients used to construct a polynomial that approximates σ.sub.y(μ.sub.y), or [0050] any kind of function with free parameters combined with specific values for those parameters.

[0051] For the purpose of noise-consistent processing 130, the input raw image data 110 is combined with the digital representation of the input noise model 120 to form the input data 100. In various embodiments of the invention which will be described in the following, this combination can be achieved in different ways or any of its combination, such as [0052] storing the raw image data together with the noise model in a suitable (single) container file, for example a file in the Tagged Image File Format (TIFF), or [0053] storing the noise model in a so-called sidecar file, that is, a (second) file that is connected to the (first) file wherein are stored the raw image data, said sidecar file being adapted to store additional (meta-)data that is not supported by the raw image data file, or [0054] storing the noise model in a database that links specific raw image data, stored a corresponding (single) raw image data file, to its associated noise model.

[0055] In any case, according to the present invention, storing the noise model means storing a digital representation of the parameters for the model. In some embodiments of the invention, these parameters are passed to a function with fixed definition. In other embodiments, the implementation consists of a catalog of functions, and the specific function to be used is selected by an additional identifying parameter. While some embodiments of the invention make use of noise models in the very general sense, as discussed so far, other embodiments will specialize on the Poisson-Gaussian noise model typically used for linear image sensors, where the sensor output is directly proportional to the amount of incoming light. The Poisson-Gaussian noise model is given by

[00002]${\sigma }_y\left({\mu }_y\right)=\{\begin{array}{cc}{\sigma }_{y.\mathrm{dark}}& \mathrm{for}{\mu }_y<{\mu }_{y.\mathrm{dark}}\\ \sqrt{K\left({\mu }_y-{\mu }_{y.\mathrm{dark}}\right)+{\sigma }_{y.\mathrm{dark}}^2}& \mathrm{otherwise}.\end{array},$

[0056] where σ.sub.y.dark represents the standard deviation of the pixel values in the absence of light and is often termed readout noise, μ.sub.y.dark is the mean of the pixel values in the absence of light, and K is a gain factor. Hence, the Poisson-Gaussian noise model is described by the 3 parameters {K, μ.sub.y.dark, σ.sub.y.dark}.

[0057] It should be noted that the Poisson-Gaussian noise model is applicable to raw image data output by the vast majority of image sensors. It is a formal expression of the statistics of the fluctuations of pixel values that naturally occur in image acquisition with linear sensors. Despite its ubiquity, its importance is generally ignored in prior art concerning image data correction and processing.

Noise-Consistent Processing

[0058] The image data processing that is the subject of this invention can be divided into two categories. The first category is image preprocessing, mainly with the purpose of correcting imperfections in acquisition hardware or conditions, or of standardizing the image data in a specific way. The second category is concerned with the synthesis of image data, which can then be used to test and improve the reliability of image processing algorithms, or to enrich the training data set in a machine-learning context.

[0059] While both categories of processing exist in the prior art, no attention is paid in prior art to the underlying noise model, and no effort is made to produce output data that is consistent with the noise model or to consider how the processing affects the image statistics, and hence the noise model. In prior art, only processing algorithms specifically concerned with noise, like denoising and deconvolution, occasionally use noise models in a specific and limited manner, such as mentioned in the introduction. Even in these cases, the focus of prior art has always been on the single operation concerned, without attempting to generalize the methods to a whole processing pipeline and without producing output data that remains consistent with some noise model.

[0060] Nevertheless, images with a consistent and accurate noise model across the entire image allow for much more reliable processing, in particular if the noise model is taken into account in the processing algorithm. Conversely, images that do not follow any consistent noise model have the tendency to confuse algorithms with their unexpected statistics. This is in particular true for algorithms based on deep learning and neural networks which consider all aspects of the image data in their analysis, including noise and noise correlations.

[0061] Two important remarks in this context are that a) if the input is only a single pixel value and its noise model, no operation can reliably increase the signal-to-noise ratio for that pixel, but a reduction is possible; and b) the signal-to-noise ratio typically increases for higher illumination (and hence larger pixel values), but the improvement is not linear or proportional. On this basis, reducing the illumination corresponding to a multiplication of the initial mean pixel values by a factor q, where 0<q<1, may be simulated, respectively achieved. To do so, the initial pixel value y is replaced by a corrected pixel value y′=q.Math.y+δ(y, q), where δ is a (pseudo-)random number sampled from a normal distribution with mean value 0 and variance

σ.sub.δ.sup.2=σ.sub.y.sup.2.Math.(q′.sup.2−q.sup.2),

where

[00003]$q^{\prime }=\frac{{\sigma }_y\left(\mathrm{qy}\right)}{{\sigma }_y\left(y\right)}$

is the factor by which the standard deviation σ.sub.y(q, y) of the noise of the output raw image data 150 is reduced for a real pixel value reduction of a factor q. Note that q<q′<1. This transformation represents a noise-consistent scaling operation, which will be denoted by the symbol S(y, q), that is, S(y, q)=q.Math.y+δ(y, q), and the use of which in the context of the present invention will become more clear in the following. In fact, when applied correctly, this transformation can be used to perform more sophisticated operations in the context of image correction.

[0062] In the specific case of a Poisson-Gaussian noise model, the variance to be used for the pseudorandom number δ(y, q) is given by

[00004]${\sigma }_{\delta }^2=\frac{y-{\mu }_{y.\mathrm{dark}}}{K}q\left(1-q\right)+{\sigma }_{y.\mathrm{dark}}^2\left(1-q^2\right).$

Bad Pixel Correction

[0063] In prior art bad pixels are typically corrected in the hardware of the imaging device or in software as part of the processing pipeline, by interpolation with neighboring pixels which are working correctly. As a result of the interpolation, the resulting pixel value fluctuates less than it would for a working pixel, and interpolated pixels can be clearly identified by a different photo-response curve.

[0064] Since the noise of the bad pixels is below the expectation, their statistics can be made consistent with the noise model by adding noise. This is achieved with the help of noise-consistent scaling. Consider, for example, a bad pixel whose value y.sub.C should be replaced by using the left and right neighbors with values y.sub.L and y.sub.R. One may assign to the value y.sub.C the equation

y′.sub.C=S(y.sub.L+y.sub.R−μ.sub.y.dark, q=1/2)

which reproduces on average the same value as by using interpolation and which in addition reproduces the standard deviation of y.sub.C that is consistent with the noise model.

Flat-Field Correction

[0065] Apertures, lenses and filters often result in a spatially inhomogeneous “capturing” of incoming photons, as for example in vignetting: A uniformly illuminated scene appears darker towards the edges when imaged. In the prior art, a standard way to correct for this is by normalizing to a reference flat-field image N(i, j), where i and j denote pixel coordinates, through

y′(i, j)=y(i, j)/N(i, j).

[0066] Here, the flat-field image should be as noise-free as possible, which can be achieved by fitting a mathematically smooth function to the reference image, or by averaging a large number of reference images. In addition one usually wants to preserve the brighter parts of the original image (usually the center), such that N(i, j) should be normalized to the range 0<N(i, j)≤1.

[0067] While this method can be used to normalize the brightness of the image, it will no longer be possible to use the same noise model for the entire image: The previously darker parts now have the same brightness, but they have larger relative noise than the previously brighter parts of the image.

[0068] In the present invention, a noise-consistent flat-field correction is achieved by using noise-dependent scaling with a pixel-dependent factor q(i, j) to darken the brighter parts of the image. All the considerations of noise-consistent scaling still apply, namely with respect to use of the equation y′=q.Math.y+δ(y, q) for determining corrected pixel values, and the value for q(i, j) such as to ensure noise-consistent flat-field correction is

[00005]$q\left(i,j\right)=\frac{\underset{i,j}{\min }N\left(i,j\right)}{N\left(i,j\right)}.$

Correction of Photo-Response-Non-Uniformity

[0069] Photo-Response-Non-Uniformity is a special kind of inhomogeneous illumination caused by the varying detection efficiency of the individual pixels. The differences are usually on the order of 1% to 2% and can, among other things, be caused by differences of absorption in the active area of the pixel.

[0070] The varying detection efficiency is equivalent to inhomogeneous illumination or exposure and can be corrected in the same way as flat-field correction by using as N(i,j) the spatially varying relative detection efficiency, determined through careful calibration, of an image sensor used to take the image data.

Binning and Averaging

[0071] The methods of noise-consistent processing presented above ensure that the output raw image data 150 remain consistent with the input noise model 120. In many use cases, it is required or of interest to perform processing that leads to a modified output noise model 160, which, however, remains applicable to the entirety of the output raw image data 150. One may, for example, sacrifice spatial resolution with a binning operation, and in turn gain in signal-to-ratio. It is here important to update the noise model, such that the next processing step can distinguish binned data with a consistently modified noise model from cropped data where the noise model remains the same and thus is inconsistent with the binned data.

[0072] One realization of a binning operation consists of forming a sum of groups of N=n.Math.m pixels, such that for each group there is an output pixel with value

Y=Σ.sub.i=1.sup.Ny.sub.i,

where the index i labels the spatial position in the block to be binned.

[0073] The output noise model σ.sub.Y(μ.sub.Y) must be updated to take into account the new dark conditions. Specifically, Y has N times the contribution of the black level, i.e. μ.sub.Y.dark=Nμ.sub.y.dark, and the readout noise in the output model is σ.sub.Y.dark=√{square root over (N)}σ.sub.y.dark.

[0074] In a different realization of the binning operation, the output pixel value is calculated as

Y=(Σ.sub.i=1.sup.Ny.sub.i)−(N−1)μ.sub.y.dark.

[0075] This has the advantage that the black level of the output noise model is identical to that of the input noise model, such that the range of scarcely used pixel values between 0 and μ.sub.y.dark is not unnecessarily enlarged.

[0076] In contrast to the binning operation based on average described below, the above two binning operations map integers to integers, and there is no risk for loss of precision associated to rounding or discarding fractional parts.

[0077] In another version of the binning operation, the output pixel value is obtained by forming the average of the input pixel values,

[00006]$\stackrel{\prime }{Y}=\frac{1}{N}{.Math.}_{i=1}^N{y}_i.$

[0078] This is particularly useful when the digital representation of the output raw image data 150 is a bounded data type, such as an integer value with fixed bit depth, because it avoids clipping of sums of large values. In the case of averaging, the black level of the data remains unchanged, but the output noise is smaller than the input noise, σ.sub.Ý(μ)≤σ.sub.y(μ). In particular, σ.sub.Ý.darkσ.sub.y.dark√{square root over (N)}, and in the case of the Poisson-Gaussian noise model, the gain parameter of the output model is equal to K/N.

[0079] One can similarly sum or average pixels at the same position but of different exposures. In this case, the output raw image data 150 will have the same dimension as the input raw image data 110, but the output noise model 160 must be modified in the same way as for spatial sums or averages. For example, N exposures can be averaged to obtain an output image with pixel values

[00007]$\stackrel{\prime }{Y}=\frac{1}{N}{.Math.}_{i=1}^N{y}_i,$

where the index i now labels the exposure number, and the values y.sub.i represent the same position in the image data as the output pixel value Ý. In the output noise model of the exposure average, σ.sub.Ý.dark=σ.sub.y.dark/√{square root over (N)}, and in the case of a Poisson-Gaussion noise model, the output gain parameter is equal to K/N.

Normalization

[0080] Some embodiments for noise-consistent processing will have as input raw image data 110 a sequence of images coming from a number M of different devices with Poisson-Gaussian noise-models that have different parameters {(K.sub.i, μ.sub.y.dark.sup.(i), σ.sub.y.dark.sup.(i))∨i=1,2 . . . , M}, and the goal of the processing is to produce output raw image data 150 formed by a sequence of output images following a single output noise model 160. This can be achieved by determining which noise model has the highest normalized readout noise

[00008]${\sigma }_{ϵ}={\max }_i\frac{{\sigma }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i},$

and assigning for each pixel value y.sub.i of input image i an output pixel value ∈.sub.i as

[00009]${ϵ}_i=\frac{y_i-{\mu }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i}+{\delta }_i,$

[0081] where δ.sub.i is a (pseudo-)random number sampled from a normal distribution with mean value 0 and variance

[00010]${\sigma }_{ϵ}^2-{\left(\frac{{\sigma }_{y.\mathrm{dark}}^{\left(i\right)}}{K_i}\right)}^2.$

The parameters for the common output noise model 160 will then be {K=1, μ.sub.∈.dark=0, σ.sub.∈.dark=σ.sub.∈}.

Integer Quantization

[0082] In several of the above-mentioned operations, corrections use floating point operations or floating point random numbers, and changes with respect to the input pixel value may be small. The overall correction may in fact be smaller than unity, for example, if a reduction of 2% should be applied to a pixel value of 10, the effective corrected pixel value may be 9.8. In general, such small corrections may be applied by saving the corrected value in a non-integer representation (e.g.

[0083] fixed-point, floating-point or rational number). However, in many use-cases, having an output in integer representation is desirable. Naïve implementations, such as truncating or rounding, may result in a difference between the desired correction coefficient and the effective applied correction. For example, truncating or rounding may introduce an unintended shift of the statistical mean of the image data, also known as bias. To avoid this, a preferred embodiment of a method according to the present invention provides a noise-consistent rounding operation that involves dithering, i.e the addition of a suitably chosen random number Δ, which is different for each pixel, before the actual rounding operation, i.e.

y′=round(y+Δ).

[0084] Of particular interest are random numbers that follow a uniform distribution between −½ and +½, and random numbers that follow a triangular distribution between −1 and +1, i.e. the distribution that is formed by adding two independent samples of said uniform distribution, because both distributions introduce zero bias. Moreover, the latter gives quantization noise that is uncorrelated with Δ. Some applications may benefit from the use of other random distributions for Δ.

[0085] The quantization operation introduces a slight amount of extra noise, which requires to adapt the readout noise of the output noise model according to

σ′.sub.y.dark.sup.2=σ.sub.y.dark.sup.2+σ.sub.Δ.sup.2,

where σ.sub.Δ.sup.2 is the variance of the distribution for the random number Δ. Because of this additional noise and to minimize the introduction of processing errors, preferred implementations of processing pipelines according to the present invention perform quantization only once and as the last operation of the pipeline.

Image Synthesis by Exposure and Gain Scaling

[0086] The normalization procedure presented above is well-suited for machine learning and algorithm testing, because it allows to gather data from a multitude of sources without increasing the number of noise models that the algorithm must be able to handle or be tolerant against. To further increase the amount of training or testing data, one can simulate images that have a reduced signal-to-noise ratio within the same noise model. This noise model can either be the model of a single set of original input data 100, or of a set of normalized input data generated by following the procedure set out above. Synthetic images with properties equivalent to a reduction of exposure time or of detection efficiency by a factor q can then be generated by applying to each pixel with value y of the input raw image data 110 the noise-consistent scaling

[0087] S(y, q) such as defined above in more detail.

[0088] In other situations, it is desirable to increase the noise level while keeping the signal constant. For a Poisson-Gaussian input noise model, this can be achieved in two ways. The trivial way is to add (pseudo-)random values δ with 0 mean and desired standard deviation σ.sub.δ to all pixel values of the input image data 110. This effectively corresponds to increasing the read-out noise parameter, which in the output noise model should be changed to √{square root over (σ.sub.y.dark+σ.sub.δ.sup.2)}. Another approach consists of the combination of normalization and exposure scaling by factor q, followed by an application of a new gain factor K′=K/q and adding back the input black level μ.sub.y.dark,

[00011]$y^{\prime }=\frac{K}{q}S\left(\frac{y-{\mu }_{y.\mathrm{dark}}}{K},q\right)+{\mu }_{y.\mathrm{dark}}.$

With this transformation, the output raw image data 150 will statistically have the same mean values as the input raw image data 110, and the output noise model has the new parameters {K′=K/q, μ.sub.y.dark, σ.sub.y.dark′=σ.sub.y.dark/q}.

[0089] It is clear from the above description that said processing step 130 of a method according to the present invention in general, i.e. in all cases, comprises, on the one hand, determining, based upon said input noise model 120 and depending on said processing operation(s) applied to the input raw image data 110, an output noise model 160 adapted to reflect noise present in said output data 140, as well as, on the other hand, producing, based upon said processing operation(s) applied to the input raw image data 110 as well as on said input noise model 120 and/or said output noise model 160, output raw image data 150 which is statistically consistent with said output noise model 160, these two steps of course depending on one another.

[0090] In practice, either the processing consists in an operation (or series of operations) which is noise-consistent and thus happens within the input noise model, i.e. by using the input noise model as the output noise model, or the processing consists in an operation (or series of operations) which isn't noise-consistent and thus simultaneously modifies the input noise model such as to produce an output noise model that is consistent with the processed image data.

[0091] In case of the first solution mentioned here above, said processing step 130 preferably comprises applying a transformation S(y, q) representing a noise-consistent operation to the values y of a given pixel of an image sensor used to take the image data, where q is a pixel value reduction factor, with 0<q<1, said processing step 130 using an output noise model 160 which is identical to the input noise model 120.

[0092] In case of the second solution mentioned here above, said processing step 130 preferably comprises applying a noise-inconsistent operation to the values y of a given pixel of an image sensor used to take the image data, said processing step 130 using an output noise model 160 which is different as compared to the input noise model 120 and statistically consistent with the noise of the output raw image data 150 of the output data 140.

[0093] A way to verify that acquired image data is consistent with an associated noise model is to acquire a set of images under identical conditions such that all differences between pairs of individual images can be attributed to noise. Preferably, the pixel values of a single image should cover a large fraction of all possible pixel values. For each pixel position i, the mean μ.sub.i and standard deviation σ.sub.i is then calculated using all images in the set. The larger the number of images in the set, the better the estimates of mean μ.sub.i and standard deviation σ.sub.i. Finally, it is checked if the points (μ.sub.i, σ.sub.i) are well approximated by the function σ.sub.y(μ.sub.y) of the noise model. Similarly, it can be verified that the output noise model of a processing operation is consistent with the output noise model by passing each individual image of said set through the processing operation and extracting means and standard deviations (μ′.sub.i, σ′.sub.i) for each pixel position in the output image data. The output noise model is considered to be consistent with the output image data if all points (μ′.sub.i, σ′.sub.i) are well approximated by the function σ′.sub.y(μ′.sub.y) of the output noise model.

[0094] FIG. 2 is an illustration of an embodiment of a method of noise-consistent image processing according to the present invention combining several features to optimize the overall performance of the image processing pipeline which starts from initial input data 100 that consists of input raw image data 110 and input noise model 120. Each two arrows 311, 312 connecting subsequent blocks 310, 320, 330, 340, 350, 360, 370, which each schematically represent a processing operation, symbolize the fact that both raw image data and noise model information is propagated through each processing operation, given that in the context of the present invention the input data 100 need to consist of both input raw image data 110 and noise model 120. This input data is processed through one or more operations 310, 320, 330, 340, 350, 360, 370. The operations treat input data (image data and noise model) and deliver processed output raw image data 150 and output noise model 160 associated with the processed output raw image data, these two parts together forming output data 140. The availability of the noise model at operations 310, 320, 330, 340, 350, 360, 370 enhances the performance of the operations. Optionally, a specific processing operation 385, i.e. a specific element of the processing pipeline, is able to make use of both image and noise data, however it is only able to output image data, such as schematically illustrated in FIG. 2 by the fact that only one arrow 311 exits the block corresponding to this specific processing operation 385, which might be added at the end of the processing pipeline and which might for example consist in deconvolution. Before this specific processing operation 385, however, at each intermediate step, both raw image and noise model data are available, so that the output data 365 from a previous processing operation is the input data 365 for the next processing operation. Optionally, certain additional processing operations 390, which might also be added at the end of the processing pipeline and which might consist in classification or segmentation, may not require and/or output noise model data, but may still benefit from the higher quality of the processing pipeline up to that point in a method of noise-consistent image processing according to the present invention, so that the end result 395 is more precise, better qualified and more reliable.

[0095] FIG. 3 schematically illustrates that for some embodiments of a method of noise-consistent image processing according to the present invention, two or more processing operation, symbolized in FIG. 3 by blocks 520, 521, 522 and operating on input data 100, may be combined into a single processing operation 510 that performs the combined function of each of the individual processing operations 520, 521, 522 such as to produce output data 140 consisting of both output raw image data 150 and output noise model 160. As an example, pixel linearization and vignetting correction may be combined into one single processing operation 510. Each two arrows connecting in FIG. 3 subsequent blocks 100, 510, 140 again symbolize the fact that both raw image data and noise model information is propagated through each processing operation combined into a single processing operation.

[0096] Finally, the present invention is also related to computer program means stored in a computer readable medium adapted to implement the method set out above as well as to a device equipped with such computer program means. For example, such device may consist in a microprocessor, a field-programmable gate array, an image sensor, a mobile phone, in particular a smart phone equipped with a digital camera, a digital photo apparatus, a digital video camera, a scanning device, a tablet, a personal computer, a server, a microscope, a telescope, or a satellite.

[0097] In light of the above description of various embodiments of a method according to the present invention, its advantages are clear.

[0098] First, access to a consistent noise model according to the present invention provides strong benefits to a variety of image processing applications. In fact, as the noise model describes the statistical uncertainty of the image data, it provides valuable insights in terms of tolerance limits and reproducibility in particular for scientific and metrological applications. This is enabled by a method according to the present invention. Conversely, the absence of a consistent noise model reduces the reliability of image processing and analysis, where the outcome may depend on the degree of correction or preprocessing applied to the image data. This may be avoided by use of a method according to the present invention.

[0099] Secondly, access to a consistent noise model according to the present invention also provides strong benefits to applications relying on machine learning techniques for the processing and/or interpretation of image data, because this allows for the availability of large amounts of high-quality training data due to the fact noise that models may be used to enhance the efficiency and eliminate a possible point of confusion of the algorithm. In the context of machine learning, this is even more important, because machine learning algorithms make much stronger use of (statistical) properties of the data than human observers who ignore statistical properties, such as noise and noise correlations.