Method and arrangement for estimating at least one cross-channel colour mapping model from an set of tuples of corresponding colours relative to at least two images

Abstract

The present invention generally relates to a method and arrangement for estimating said at least one cross-channel color mapping model from a set of tuples of corresponding colors relative to at least two images. The method is characterized in that it uses a processor for selecting at least one intermediate tuple of corresponding colors from the set of tuples of corresponding colors by using at least one channel-wise color mapping model, and for estimating the at least one cross-channel color mapping model from the at least one intermediate selected tuple of corresponding colors.

Claims

1. A method for compensation of colour differences between at least two images imaging a same scene, the colours of which are represented according to m colour channels, comprising: extracting from said images a set of tuples of corresponding colours; estimating from said set of tuples of corresponding colours a channel-wise colour mapping model for each of said m colour channels; selecting within said set of tuples of corresponding colours at least one intermediate tuple having colours with a difference to said estimated channel-wise colour mapping model that are smaller than a determined threshold; estimating from said at least one selected intermediate tuple of corresponding colours at least one cross-channel coulour mapping model for at least one of said m colour channels; generating a final set of final tuples of corresponding colours from said at least one selected intermediate tuple of corresponding colours such that said final tuples have colors with a difference to said estimated cross-channel colour mapping model that are smaller than a determined threshold; and compensating colour difference between said images based on said final set of final tuples of corresponding colours.

2. The method for compensation of colour differences according to claim 1, wherein said extracting is based on matching feature points between said images.

3. The method for compensation of colour differences according to claim 1, wherein estimating the channel-wise colour mapping model for each of said m colour channels is performed such as to model, in said colour channel, colour differences between colours of each tuple.

4. The method for compensation of colour differences according to claim 1, wherein said at least one cross-channel colour mapping model is based on thin plate splines.

5. An arrangement for compensation of colour differences between at least two images imaging a same scene, the colours of which are represented according to m colour channels, comprising: a processor coupled to a memory, the processor being configured to: extract from said images a set of tuples of corresponding colours; estimate from said set of tuples of corresponding colours a channel-wise colour mapping model for each of said m colour channels; select within said set of tuples of corresponding colours at least one intermediate tuple having colours with a difference to said estimated channel-wise colour mapping model that are smaller than a determined threshold; estimate from said at least one selected intermediate tuple of corresponding colours at least one cross-channel coulour mapping model for at least one of said m colour channels; generate a final set of final tuples of corresponding colours from said at least one selected intermediate tuple of corresponding colours such that said final tuples have colors with a difference to said estimated cross-channel colour mapping model that are smaller than a determined threshold; and compensate colour difference between said images based on said final set of final tuples of corresponding colours.

6. The arrangement for compensation of colour differences according to claim 5, wherein said extracting is based on matching feature points between said images.

7. The arrangement for compensation of colour differences according to claim 5, wherein estimating the channel-wise colour mapping model for each of said m colour channels is performed such as to model, in said colour channel, colour differences between colours of each tuple.

8. The arrangement for compensation of colour differences according to claim 5, wherein said at least one cross-channel colour mapping model is based on thin plate splines.

Description

4. LIST OF FIGURES

(1) The embodiments will be described with reference to the following figures:

(2) FIG. 1 shows a diagram of the main steps of the method according to the present invention,

(3) FIG. 2 shows an arrangement comprising processing means configured to implement the step of the method according to an embodiment of the invention.

5. DETAILED DESCRIPTION OF THE INVENTION

(4) Generally speaking, the invention concerns a method for estimating a colour mapping model from a set of tuples of corresponding colours relative to at least two images.

(5) Multiple approaches may be used to define such a set of tuples of corresponding colours from images (or views). The invention does not request any specific approach. In the literature, several type of detectors exist that are to extract tuples of image features. For example, the well-known MSER (J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing, 22(10):761-767, 2004.) and DOG (D. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004) detectors may be used.

(6) Resulting from the detector are multiple patches which belongs to a same image and/or to multiple images. Generally speaking, these patches belong to n different images (usually n=2). Each patch may be described, for example by a SIFT descriptor (D. Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.). Any other descriptor may be used without limiting the scope of the invention.

(7) A feature matching provides a match between patches of those n images and a tuple of corresponding colours is built for each set of matched patches. For example the well-known method of Hasan Sheik-Faridul entitled Optimization of Sparse Color Correspondences for Color Mapping published in the Colour and Imaging Conference 2012 proposes to select a tuple of corresponding colours in a spatial neighborhood of the matched patches. A tuple of corresponding colours thus comprises n elements, each one being a colour of a patch of an image.

(8) Colour spaces in images include RGB, YCbCr and XYZ colour spaces. Other colour spaces do exist, also with more or less three coordinates (m>3 or m<3). For example, the RGB colour space can be defined by the standard ITU-R BT.709, the YCbCr colour space according to the standard SMPTE S274M or the XYZ by the CIE in 1931.

(9) A tuple of corresponding colours is then a tuple with n colours defined in a m-dimensional colour space and the set of tuples of corresponding colours is then formed by bringing together these tuples of corresponding colours.

(10) According to the invention, the method comprises steps for selecting at least one intermediate tuple of corresponding colours from the set of tuples of corresponding colours by using at least one channel-wise colour mapping model, and for estimating at least one cross-channel colour mapping model from said at least one intermediate selected tuple of corresponding colours.

(11) According to an embodiment, shown in FIG. 1, the step for selecting said at least one intermediate tuple of corresponding colours (step 1 on FIG. 1) comprises estimating from the set of tuples of corresponding colours a channel-wise colour mapping model for each channel of a m-dimensional colour space and comparing a criterion obtained from said at least one estimated channel-wise colour mapping model to a determined threshold.

(12) A channel-wise colour mapping model is a 1-dimensional colour mapping model which is an estimate of a single colour channel without taking into account cross channel influence.

(13) The number of channel-wise colour mapping models equals the dimension of the colour space.

(14) For example, three channel-wise colour mapping models are required for a RGB colour space (m=3), one for the red channel, one for the green channel and one for the blue channel.

(15) The definition of each channel-wise colour mapping model depends on the number of input images. For example, when a two input images I.sub.1 and I.sub.2 in a RGB color space are considered (n=2, m=3), three channel-wise colour mapping models f.sub.R,f.sub.G,f.sub.B are required that provides estimates , , of the colour channels R.sub.I1,G.sub.I1,B.sub.I1 of the image I.sub.1 from the colour channels R.sub.I2,G.sub.I2,B.sub.I2 of the image I.sub.2 respectively according to:
=.sub.R(R.sub.I2)
=.sub.G(G.sub.I2)
=.sub.B(B.sub.I2)

(16) According to an embodiment, the channel-wise colour mapping model is a colour transform, preferably a Gamma-Offset-Gain (GOG) model given by:
=G.sup.C{C.sub.I2}.sup..sup.C+b.sup.C

(17) where C.sub.I1 is a color channel of an input image I.sub.1 (C.sub.I1 {R.sub.I1, G.sub.I1, B.sub.I1}) and C.sub.I2 is a color channel of another input image I.sub.2 (C.sub.I2 {R.sub.I2, G.sub.I2, B.sub.I2})), and parameters G.sup.C defines a gain for a color channel C (C{R, G, B}), .sup.C a gamma correction for the color channel C and b.sup.C an offset for the color channel C. The GOG model was designed originally only for cases such as color change due to an uncalibrated camera. In this invention, the GOG-based model helps cleaning the corresponding colours by a robust fitting and makes it ready for further modeling. Success of the GOG modeling is decisive, if later data driven models are used that can be heavily influenced by outliers.

(18) Variants of this embodiment consists in considering only some of these three parameters in the model.

(19) The use of a GOG model is advantegeous because this is a simple, powerful and stable color mapping model which removes noisy tuple of corresponding colors from the set of tuples of corresponding colouers when their parameters are optimized.

(20) According to an embodiment, the parameters .sup.C={G.sup.C, .sup.C, b.sup.C} of a GOG model are estimated by solving a linear regression problem.

(21) According to an embodiment, the parameters of the GOG model are iteratively estimated by minimizing a L.sub.p norm.

(22) According to an embodiment, the parameters of a GOG model are iteratively estimated by minimizing a reweighted least-squares criteria.

(23) According to this embodiment, at step t+1, the parameters .sub.t+1.sup.C are then given from the parameters .sub.t.sup.C estimated at the previous step t by:

(24) ${}_{t+1}^C=\arg \underset{{}^C}{\min }\stackrel{n}{\underset{i=1}{.Math.}}{w}_i^t{.Math.C_{I1}-.Math.}^2$
where w.sub.i.sup.t is a Huber function given preferably by

(25) $w_i^t=\frac{1}{\max \left(1,.Math.r^t.Math.\right)}$
with r.sup.t=C.sub.I1 is the residual error at step t between the colour component of a channel of an image and a version of this channel estimated by the corresponding channel-wise colour model.

(26) Once a channel-wise colour mapping model is estimated for each channel of the colour space, at least one tuple of corresponding colours, said intermediate, is selected from the set of tuples of corresponding colours by comparing a criterion obtained from at least one estimated channel-wise colour mapping model to a predefined threshold T.

(27) According to an embodiment, the criterion is the residual error r.sup.t.

(28) According to another embodiment, the criteria is a distance measure in the colour space of the residual errors calculed for each channel of this colour space.

(29) For example for a RGB color space between two images I.sub.1 and I.sub.2, the criterion is an Euclidean distance given by:

(30) $\underset{C\left\{R,G,B\right\}}{.Math.}{.Math.C_{I1}-.Math.}^2$

(31) According to another embodiment, the criteria is a multiple of the robust standard deviation of residuals (RSDR) as defined by Motulsky and Brown (H. Motulsky and R. Brown. Detecting outliers when fitting data with nonlinear regressiona new method based on robust nonlinear regression and the false discovery rate. BMC bioinformatics, 7(1):123, 2006). The RSDR is a method to quantify points scattered around a curve generated by robust nonlinear regression. Motulsky quantifies the scatter by calculating the 68.27 percentile of the absolute values of the residuals because 68.27% of values in a Gaussian distribution lie within one standard deviation of the mean.

(32) The selection of at least one intermediate tuple of corresponding colours may be e.g. varied by making the threshold for remaining colour differences adapted to the colour signal, for example setting it to a multiple of the colour difference before colour mapping or after colour mapping, or using colour ratios instead of colour differences without to depart from the scope of the present invention.

(33) According to the invention, the method comprises steps for estimating at least one cross-channel colour mapping model from said at least one intermediate selected tuple of corresponding colours (step 2 on FIG. 2).

(34) A cross-channel colour model is a colour mapping model which is an estimate of the color coordinate of a single colour channel but taking into account cross channel influence. A cross-channel colour model is then relative to a colour channel of a colour space and is a function of at least one other colour channel of this colour space.

(35) The number of cross-channel colour mapping models equals the dimension of the colour space.

(36) For example, three cross-channel colour mapping models are required for a RGB colour space (m=3), one for the red channel, one for the green channel and one for the blue channel.

(37) The definition of each cross-channel colour mapping model depends on the number of input images but also of the number of adjacent channels which are taken into account.

(38) Preferably, when the RGB colour space is used, a function .sub.tps.sup.R for the estimate of the red channel and a function .sub.tps.sup.G for the estimate of the green channel of an image I.sub.1 use the red and green channels of an image I.sub.2 and a function .sub.tps.sup.B for the estimates of the blue channel of the image I.sub.1 is estimated from the red and blue channels of the image I.sub.2. Instead of taking all three channels (red, green and blue), only two are taken at a time.

(39) These assumptions are motivated from red, green and blue channel sampling in a typical camera. Moreover, the resulting model will be smooth with respect to the thirdmissingcolor channel and therefore can be generalized more easily for all colors of the color space. Finally, not using always all color channels reduces the computational load and increases the robustness of model estimation.

(40) Based on these assumptions, the three cross-channel models are given by:
=.sub.tps.sup.R(R.sub.I2, G.sub.I2)
=.sub.tps.sup.G(R.sub.I2, G.sub.I2)
=.sub.tps.sup.B(T.sub.I2, B.sub.I2)

(41) These assumptions do not limit the scope of the invention because the function of each (or at least one) cross-channel colour mapping model may also takes into account the influence of more than two other colour channels.

(42) According to an embodiment, a cross-channel colour mapping model relative to a colour channel C (C {R, G, B} in case of RGB color space) is based on thin-plate splines. More precisely, the function .sub.tps.sup.C of a cross-channel colour mapping model relative to a colour channel C (C {R, G, B} in case of RGB color space) uses thin-plate splines that estimates the channel C.sub.I1 of an image I.sub.1 from at least one channel of an image I.sub.2.

(43) A complete definition of thin-plate spline may be found in Bookstein (F. Bookstein. Principal warps: Thin-plate splines and the decomposition of deformations. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 11(6):567-585, 1989).

(44) In brief, a thin-plate spline fits smoothly clattered data using pieces offor examplecubic spline functions. Since the set of intermediate tuples of corresponding colours is cleaned from the worse outliers, such a model allows to precisely model cross-channel changes.

(45) According to an embodiment, for the case of a 3-dimensional colour space having the color channels C, K and L, the thin-plate spline .sub.tps.sup.C for the color channel having the colour coordinate C is regularized using a data driven model which is an unique minimizer of a weighted sum given by:
.sub.tps.sup.C=argmin[E(.sub.tps.sup.C)+(1)RO(.sub.tps.sup.C)]
with E(.sub.tps.sup.C) an error measure given by

(46) $E\left(f_{\mathrm{tps}}^C\right)=\underset{C\left\{R,G,B\right\}}{.Math.}{.Math.C_{I1}-f_{\mathrm{tps}}^C\left(C_{I1},K_{I1}\right).Math.}^2$
and RO(.sub.tps.sup.C) a roughness measure is a sum of first and second derivatives of the thin-plate spline function. In this case, the cross-channel influence of coordinates C and K are considered for modeling C while the influence of L is not considered.

(47) For example, to estimate the red channel R of the Image I.sub.1 from the red and green channels of I.sub.2, the roughness measure is given by

(48) $\mathrm{RO}\left(f_{\mathrm{tps}}^R\right)=\left[{\left(\frac{{}^2{f}_R}{R^2}\right)}^2+{\left(\frac{{}^2{f}_R}{RG}\right)}^2+{\left(\frac{{}^2{f}_R}{G^2}\right)}^2\right]\mathrm{dRdG}$
where is a smoothing parameter that controls the smoothness in the colour space. If is near to 1 it's a least square approximation and the model will try to go through all input color data and therefore might overfit. On the other hand, if is near to zero, it is a linear polynomial approximation.

(49) According to an embodiment, a cross-channel colour mapping model has a smoothness parameter fixed with a predefined value. In the previous embodiment such a parameter is the parameter .

(50) Such a parameter is common in estimation and model fitting when the colour mapping model has a high degree of freedom. The smoothness parameter is then used to avoid over-fitting of the model to the colour correspondences.

(51) For some applications, it is useful to have a final set of tuples of corresponding colours that are well in coherence with an estimated cross-channel colour model. In block 3 of FIG. 1, this final set of tuples of corresponding colours can be either calculated from the estimated cross-channel model or by other methods allowing to generate such a final set using the cross-channel colour model. Such a final set is useful to create a look-up-table, or representation of image colours or for other applications.

(52) According to a particular application of the invention, the invention relates also to a method for generating a final set of tuples of corresponding colours corresponding from a set of tuples of corresponding colors. The method is characterized in that at least one cross-channel colour mapping channel model is estimated from the set (initial) of tuples of corresponding colors using a method as above-explained in relation with FIG. 1 and the final set of tuples of corresponding colours has colours with a difference to said at least one cross-channel colour mapping model that is smaller than a determined threshold.

(53) FIG. 2 shows an arrangement 200 that can be used to implement the method of the invention. The arrangement comprises the following components, interconnected by a digital data and address bus 20: a processing unit 23 (or CPU for Central Processing Unit); a memory 25; a network interface 24, for interconnection of arrangement 200 to other arrangement or devices connected in a network via connection 21.

(54) Processing unit 23 can be implemented as a microprocessor, a custom chip, a dedicated (micro-) controller, and so on. Memory 25 can be implemented in any form of volatile and/or non-volatile memory, such as a RAM (Random Access Memory), hard disk drive, non-volatile random-access memory, EPROM (Erasable Programmable ROM), and so on. Arrangement 200 is suited for implementing a data processing device according to the method of the invention.

(55) A set of tuples of corresponding colours and/or a channel-wise colour mapping model may be obtained from the network interface 24 via the connection 21 or from the memory 25 or resulting of a cooperation between the processing unit 23 and the memory 25.

(56) The processing unit 23 and the memory 25 are configured to cooperate with each other to select at least one intermediate tuple of corresponding colours from the set of tuples of corresponding colours by using at least one channel-wise colour mapping model, and to estimate at least one cross-channel colour mapping model from said at least one intermediate selected tuple of corresponding colours. Such a selection and estimation conform to the method described in relation with the FIG. 1.

(57) According to an embodiment, the processing unit 23 and the memory 25 are configured to cooperate with each other to generating a final set of tuples of corresponding colours corresponding from a set of tuples of corresponding colors. For that, the processing unit 23 and the memory 25 are configured to estimate a cross-channel colour mapping channel model from said set of tuples of corresponding colors according to a method described in relation with the FIG. 1, said final set of tuples of corresponding colours having colours with a difference to said at least one cross-channel colour mapping model that is smaller than a predefined threshold.

(58) According to a particular embodiment, the invention is entirely implemented in hardware, for example as a dedicated component (for example as an ASIC, FPGA or VLSI) (respectively Application Specific Integrated Circuit, Field-Programmable Gate Array and Very Large Scale Integration) or according to another variant embodiment, as distinct electronic components integrated in a device or, according to yet another embodiment, in a form of a mix of hardware and software.