IMAGE PROCESSING DEVICE AND METHOD FOR COLOR BALANCING

Abstract

An image processing device performing color balancing of a first image and at least a second image is provided. The image processing device comprises a color balancing determination unit and a color balancing calculation unit.

The color balancing determination unit determines a global gain vector (t) comprising at least two gain factors (â.sup.n, â.sup.n+1) of the first and second images by minimizing a cost function based upon reference pixel values of the first and second images. The first and second image reference pixels depict a shared color scene of the two images.

The color balancing calculation unit performs color balancing of the first image based upon the gain factor (â.sup.n) of the first image and to perform a color balancing of the second image based upon the gain factor (â.sup.n+1) of the second image.

Claims

1. An image processing device for performing a color balancing of at least a first image recorded by a first camera and a second image recorded by a second camera, comprising a color balancing determination unit and a color balancing calculation unit, wherein the color balancing determination unit is adapted to determine a global gain vector (t) comprising at least one gain factor (â.sup.n) of the first image and at least one gain factor (â.sup.n+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images, and wherein the color balancing calculation unit is adapted to perform a color balancing of the first image based upon the at least one gain factor (â.sup.n) of the first image, and perform a color balancing of the second image based upon the at least one gain factor (â.sup.n+1) of the second image.

2. The image processing device according to claim 1, wherein the first image and the second image each comprise at least three color channels, wherein the color balancing determination unit is adapted to minimize separate cost functions for each color channel of the images, wherein the color balancing calculation unit is adapted to perform the color balancing of the first image separately for each color channel, and perform the color balancing of the second image separately for each color channel.

3. The image processing device according to claim 1, wherein the shared color scene of the first image and the second image is a scene that captures at least one same object or area, and wherein the at least one same object or area is within a shared field of view of the first camera and the second camera.

4. The image processing device according to claim 1, wherein the shared color scene of the first image and the second image is a scene that captures at least one same color, and wherein the at least one same color is not within a shared field of view of the first camera and the second camera.

5. The image processing device according to claim 1, wherein the image processing device is adapted to perform a color balancing of at least the first image recorded by the first camera, the second image recorded by the second camera, and a third image recorded by a third camera, wherein the color balancing determination unit is adapted to, after the color balancing calculation unit has performed the color balancing of the first image and the second image, resulting in a color balanced first image and a color balanced second image: determine a further global gain vector comprising at least one further gain factor of the second image and at least one gain factor of the third image by minimizing a pre-determined cost function based upon reference pixel values of the second color balanced image and reference pixel values of the third image, wherein the reference pixels of the color balanced second image and the third image depict a shared color scene of the color balanced second image and the third image, and wherein the color balancing calculation unit is adapted to perform a color balancing of the second color balanced image based upon the at least one further gain factor of the second image, and perform a color balancing of the third image based upon the at least one gain factor of the third image.

6. The image processing device according to claim 1, wherein minimizing the pre-determined cost function comprises minimizing
J′=[y.sub.j.sup.(1) y.sub.j.sup.(2)][a.sup.(2)−a.sup.(1)].sup.T, wherein j is a pixel number, y.sub.j.sup.(1) is a value of pixel number j measured by the first camera, y.sub.1.sup.(2) is a value of pixel number j measured by the second camera, a.sup.(1) is a gain factor by which a reference color value of the pixel j has to be multiplied to equal y.sub.j.sup.(1), and a.sup.(2) is a gain factor by which the reference color value of the pixel j has to be multiplied to equal y.sub.j.sup.(2).

7. The image processing device according to claim 6, wherein a global gain vector t is assumed to achieve
J=Y.Math.t, where
Y=[y.sup.(1) y.sup.(2)],
y.sup.(n)=[y.sub.1.sup.(n). . . y.sub.i.sup.(n). . . y.sub.1.sup.(n)]T, and
t=[a.sup.(2)−a.sup.(1)].sup.T, wherein t is the global gain vector, Y is a vector of values y measured for a single pixel by first camera and the second camera, n is a pixel number, i is an intermediary pixel number, and 1 is a last pixel number, and minimizing the pre-determined cost function comprises determining a non-trivial nullspace of a matrix R.sub.yy=Y.sup.T.Math.Y by preferably performing an eigenvalue decomposition of R.sub.yy Ryy=Â.Math.Ryy.Math.Â.sup.T wherein Ryy is a diagonal matrix with the eigenvalues at its main diagonal, and  is a unitary matrix, and using the eigenvectors with the smallest eigenvalues {circumflex over (t)} as the first color balancing information â.sup.(1) and the second color balancing information â.sup.(2), wherein
t=[a.sup.(2)−a.sup.(1)].sup.T.

8. The image processing device according to claim 7, wherein performing of the color balancing comprises: dividing pixel values y.sub.j.sup.(1) of the first camera by the gain factor â.sup.(1), and dividing pixel values y.sub.j.sup.(1) of the second camera by the gain factor â.sup.(2).

9. An imaging system comprising an image processing device according to claim 1 and at least a first camera and a second camera.

10. The imaging system according to claim 9, wherein the first camera and the second camera have an at least partially shared field of view.

11. The imaging system according to claim 9, comprising a third camera.

12. The imaging system according to claim 11, wherein the each of the cameras has at least a partially shared field of view with at least one further of the cameras.

13. Image processing method for performing a color balancing of at least a first image recorded by a first camera and a second image recorded by a second camera, wherein the following steps are performed: determining a global gain vector (t) comprising at least one gain factor (â.sup.n) of the first image and at least one gain factor (â.sup.n+1) of the second image by minimizing a pre-determined cost function based upon reference pixel values of the first image and reference pixel values of the second image, wherein the first and second image reference pixels depict a shared color scene of the first and second images, and performing a color balancing of the first image based upon the at least one gain factor (â.sup.n) of the first image, and performing a color balancing of the second image based upon the at least one gain factor (â.sup.n+1) of the second image.

14. A non-transitory computer readable medium comprising a program code for performing the method according to claim 13 when the computer program runs on a computer.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0074] The present disclosure is in the following explained in detail in relation to embodiments of the disclosure in reference to the enclosed drawings, in which

[0075] FIG. 1 shows an overview of an embodiment of the imaging system according to the second aspect of the disclosure in a block diagram;

[0076] FIG. 2 shows an embodiment of the image processing device according to the first aspect of the present disclosure in a block diagram;

[0077] FIG. 3 shows a further embodiment of the first aspect of the disclosure in a block diagram;

[0078] FIG. 4 shows another embodiment of the first aspect of the disclosure in a block diagram, and

[0079] FIG. 5 shows an embodiment of the third aspect of the disclosure in a flow chart.

DESCRIPTION OF EMBODIMENTS

[0080] First we demonstrate the general construction and function of the imaging system and image processing device according to the first and second aspects of the disclosure along FIG. 1. Along FIGS. 2 - 4 we further elaborate on the construction and function of the respective aspects of the disclosure. Finally, with regard to FIG. 5, the function of an embodiment of the inventive method according to the third aspect is described. Similar entities and reference numbers in different figures have been partially omitted.

[0081] In FIG. 1, an embodiment of the inventive imaging system 1 according to the second aspect of the disclosure is shown. The imaging system 1 comprises a plurality of cameras 11, 12, 13 and 14 and an image processing device 10 according to the first aspect of the disclosure. The cameras 11-14 are connected to the image processing device 10. Images 11a, 12a, 13a and 14a generated by the cameras 11, 12, 13 and 14 are handed to the image processing device 10 by the cameras 11-14. The image processing device 10 performs a color balancing of the individual images 11a-14a and generates color balanced images 11b, 12b, 13b and 14b. For performing the color balancing, pairs of images sharing at least a color scene depicting reference pixels are formed between the images 11a-14a. A color balancing of the respective pairs is performed. After the color balancing of the pairs has been performed, further color balancing between the already color balanced images of the pairs are performed. Regarding the color balancing of more than two images, it is referred to the elaborations regarding FIG. 3 and FIG. 4. Further details regarding the color balancing of the pairs of images, it is referred to the elaborations regarding FIG. 2.

[0082] In FIG. 2, an embodiment of the image processing device 10 of the first aspect of the disclosure is shown. The image processing device 10 comprises a balancing determination unit 20 and connected to it a color balancing calculation unit 21.

[0083] A first image 11a generated by a first camera 11 and a second image 12a generated by a second camera 12 are provided to the color balancing determination unit 20. The first image 11a and the second image 12a each comprise reference pixels depicting a shared color scene of the first and second images. A color scene is the spectral radiances of a view of the natural world as measured from one or more vantage points in space. A shared color scene is a scene or scene portion that captures at least one same object or area within a shared view of the first camera 11 and the second camera 12 or captures at least one same color that may not be within a shared field of view (e.g., different areas of a uniformly blue sky).

[0084] The color balancing determination unit 20 determines a global gain vector t comprising at least one gain factor ân of the first image 11a and at least one gain factor ân+1 of the second image 12a by minimizing a pre-determined cost function based upon the reference pixel values. These gain factors are handed on to the color balancing calculation unit 21, which performs a color balancing of the first image 11a and the second image 12a by dividing the pixel values of the first image 11a by the gain factor ân and by dividing the pixel values of the second image 12a by the gain factor ân+1. Resulting from this calculation by the color balancing calculation unit 21 is a first color balanced image 11b and a second color balanced image 12b.

[0085] In the following, the detailed mathematics performed within the color balancing determination unit 20 and color balancing calculation unit 21 are shown.

[0086] Assuming a number of J=2 cameras, the output of each camera in each channel is given as a set of pixels y.sub.i.sup.(n) with

y.sub.i.sup.(1)=a.sup.(1).Math.x.sub.i,   (1)

y.sub.i.sup.(2)=a.sup.(2).Math.x.sub.i,   (2)

where x.sub.i is denoting the i-th pixel of a reference image (e.g., a “true” color value of an object) and x.sub.i ∈ {x.sub.i,r, x.sub.i,g, x.sub.i,b} where the indices r, g, and b corresponds to the camera color channels, red, green, and blue respectively or also some other color basis such as {HSV}, {YUV}, etc.

[0087] The superscript denotes the camera index. Accordingly, a.sup.(j) ∈ {a.sub.r.sup.(j), a.sub.g.sup.(j), a.sub.b.sup.(j)}, with j ∈ {1, . . . , J}.

[0088] To calibrate the cameras to the common reference multichannel image which is represented by the pixels x.sub.i, identify the channels gain factors a.sub.i.

[0089] Multiplying the output of each camera by the gain of the other one leads to an identical quantity. Hence, in our approach, we minimize quantity

[00001]$\begin{array}{cc}{J}^{\prime }:=\left[\begin{array}{cc}{y}_i^{\left(1\right)}& y_i^{\left(2\right)}\end{array}\right]\left[\begin{array}{c}{a}^{\left(2\right)}\\ -a^{\left(1\right)}\end{array}\right].& \left(3\right)\end{array}$

[0090] In the assumed model, a global channel gain factor has to be determined for all pixels of a reference image. Let us define

J:=Y.Math.t,   (4)

Where

[0091]
Y:=[y.sup.(1) y.sup.(2)],   (5)

y.sup.(n):=[y.sub.1.sup.(n) . . . y.sub.i.sup.(n) . . . y.sub.l.sup.(n)].sup.T,   (6)

a vector built up with the pixel values, and {.Math.}.sup.T is denoting the transponation operator, and

[00002]$\begin{array}{cc}t:=\left[\begin{array}{c}{a}^{\left(2\right)}\\ -a^{\left(1\right)}\end{array}\right].& \left(7\right)\end{array}$

[0092] Minimizing the energy of the vector is equivalent to determine the non-trivial nullspace of the matrix R.sub.YY:=Y.sup.TY. We determine this, preferably, by an eigenvalue decomposition of the R.sub.YY.

R.sub.YY=ÂR.sub.YYÂ.sup.T,   (8)

where R.sub.YY is a diagonal matrix with the eigenvalues at its main diagonal,
and  is a unitary matrix.

[0093] The nullspace is spanned by the eigenvectors corresponding to the zero-valued eigenvalues

[00003]$\hat{t}.Math..Math.\text{=:}.Math.\left[\begin{array}{c}{\hat{a}}^{\left(2\right)}\\ -{\hat{a}}^{\left(1\right)}\end{array}\right].$

[0094] In practice, due to the camera noise, the eigenvalues are in general all different from noise; therefore, we select the eigenvetors with the smallest eigenvalues. This eigenvector is then interpreted as an estimation of t. To calibrate both cameras. The output is divided by the estimated gain factors in {circumflex over (t)}.

[0095] The resulting gain factors are estimated such that [0096] i) the camera corrected colors have minimal difference, and [0097] ii) the corrected colors tend to the unknown but common reference colors.

[0098] It is important to note that the shared color scene does not mean that the first image and the second image necessarily share a field of view. The shared color scene merely means that either the images depicted an identical object or area and thereby share a field of view or the images merely depict objects of closely matching color without the images sharing a field of view.

[0099] Moreover, it is important to note, that in case of color images, the above-described process is separately performed for each different color channel of the images. This means that a separate cost function is minimized for each color channel of the images and a separate color balancing is performed for each individual color channel of the images.

[0100] In FIG. 3, an extension of the above-shown process to more than two images is shown. Especially, FIG. 3 shows a different embodiment of the image processing device 10. The image processing device 10 here also comprises a color balancing determination unit 20 and a color balancing calculation unit 21. For example a color balancing of a first image 11 a and a second image 12a as shown in FIG. 2 has already been performed. A gain factor ân−1, which corresponds to the gain factor of the second image 12a is already available. The color balancing determination unit 20 is supplied with the second image 12a, a third image 13a generated by a third camera 13 and the gain factor ân−1 and therefrom computes a further gain factor ân for the third image 13a. This gain factor is handed on to the color balancing calculation unit 21 along with the third image 13a. Therefrom, a color balanced third image 13b is calculated. In the following, the detailed mathematics underlying the briefly-specified process is shown:

[0101] Let us assume, we have already estimations of â.sup.(1) and â.sup.(2) for all channels of the two cameras. To obtain an estimation for the channels gain â.sup.(3) by constraining â.sup.(1) we consider again the nullspace determination problem

[00004]$\begin{array}{cc}{R}_{\mathrm{YY}}^{\prime }.Math.\left[\begin{array}{c}{a}^{\left(3\right)}\\ -a^{\left(1\right)}\end{array}\right]=\left[\begin{array}{c}0\\ 0\end{array}\right],& \left(9\right)\end{array}$

with

[00005]$\begin{array}{cc}{R}_{\mathrm{YY}}^{\prime }:={\left[\begin{array}{cc}{y}^{\left(1\right)}& y^{\left(3\right)}\end{array}\right]}^T\left[\begin{array}{cc}{y}^{\left(1\right)}& y^{\left(3\right)}\end{array}\right].& \left(10\right)\\ R_{\mathrm{YY}}^{\prime }.Math..Math.\left(\left[\begin{array}{c}{a}^{\left(3\right)}\\ -a^{\left(1\right)}\end{array}\right]-.Math.\left[\begin{array}{c}0\\ -a^{\left(1\right)}\end{array}\right]\right)=R_{\mathrm{YY}}^{\prime }.Math.\left[\begin{array}{c}{a}^{\left(3\right)}\\ 0\end{array}\right],& \left(11\right)\\ R_{\mathrm{YY}}^{\prime }.Math.\left[\begin{array}{c}{a}^{\left(3\right)}\\ -a^{\left(1\right)}\end{array}\right]-\underset{\underset{:=b}{}}{R_{\mathrm{YY}}^{\prime }.Math.\left[\begin{array}{c}0\\ -a^{\left(1\right)}\end{array}\right]}=R_{\mathrm{YY}}^{\prime }.Math.\left[\begin{array}{c}{a}^{\left(3\right)}\\ 0\end{array}\right].& \left(12\right)\end{array}$

With (9), we obtain

[00006]$\begin{array}{cc}{R}_{\mathrm{YY}}^{\prime }\left[\begin{array}{c}{a}^{\left(3\right)}\\ 0\end{array}\right]=-b.& \left(13\right)\end{array}$

[0102] For an inverse of R.sub.YY′ that insures, the second coefficient of the unknown vector is 0, we use

[00007]$\begin{array}{cc}{R}_{\mathrm{YY}}^{\prime -1}=\left[\begin{array}{cc}\frac{1}{r_{\mathrm{yy},11}}& 0\\ 0& 0\end{array}\right],& \left(14\right)\end{array}$

With r.sub.yy,11 the element at the first row and first column of the matrix R.sub.YY′. Hence,

[00008]$\begin{array}{cc}{\hat{a}}^{\left(3\right)}=\frac{{\hat{a}}^{\left(1\right)}.Math.r_{\mathrm{yy},21}}{r_{\mathrm{yy},11}},& \left(15\right)\end{array}$

where we employed for â.sup.(1) in the vector b the estimated value from a previous step.

[0103] Given 2 images from paired cameras, the channel gain factors are estimated such that

[0104] i) the camera corrected colors have minimal difference, and

[0105] ii) the corrected colors tend to the reference colors.

[0106] The calibration itself is done finally by dividing the pixels by the estimating channel gain parameters.

[0107] Moreover, in FIG. 4, again the overall structure of the proposed imaging system 1, which allows a calibration of more than two cameras is shown. Here, images 11a, 12a, 13a, 40a, 41a are provided to the image processing device 10, which performs a pairwise color balancing. In this example, the images 11a and 12a are used for generating color balanced images 11b, 12b. A resulting gain factor al is used for performing an incremental color balancing of a third image 13a resulting in a color balanced third image 13b. Accordingly it is continued with further images 40a, 41.

[0108] In FIG. 5, an embodiment of the third aspect of the disclosure is shown. In an optional first step 100, at least first and second images 11a, 12a are recorded. The images comprise a shared color scene, as described earlier.

[0109] In a second step 101, a global gain vector t comprising a gain factor ân of the first image 11a and a gain factor ân+1 of the second image 12a is determined by minimizing a cost function based upon reference pixel values from the first and second images 11a, 12a.

[0110] In a third step 102, a color balancing of the first and second images 11a, 12a is performed using the respective gain factors ân, ân+1.

[0111] Regarding the details of the individual steps, it is also referred to the elaborations regarding FIG. 1-4.

[0112] The disclosure is not limited to the examples and especially not to a specific number of images. Also the disclosure is not limited to images with or without overlapping fields of view. The characteristics of the exemplary embodiments can be used in any combination.

[0113] The disclosure has been described in conjunction with various embodiments herein. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed disclosure, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in usually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0114] Embodiments of the disclosure may be implemented in a computer program for running on a computer system, at least including code portions for performing steps of a method according to the disclosure when run on a programmable apparatus, such as a computer system or enabling a programmable apparatus to perform functions of a device or system according to the disclosure.

[0115] A computer program is a list of instructions such as a particular application program and/or an operating system. The computer program may for instance include one or more of: a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, a source code, an object code, a shared library/dynamic load library and/or other sequence of instructions designed for execution on a computer system.

[0116] The computer program may be stored internally on computer readable storage medium or transmitted to the computer system via a computer readable transmission medium. All or some of the computer program may be provided on transitory or non-transitory computer readable media permanently, removably or remotely coupled to an information processing system. The computer readable media may include, for example and without limitation, any number of the following: magnetic storage media including disk and tape storage media; optical storage media such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; nonvolatile memory storage media including semiconductor-based memory units such as FLASH memory, EEPROM, EPROM, ROM; ferromagnetic digital memories; MRAM; volatile storage media including registers, buffers or caches, main memory, RAM, etc.; and data transmission media including computer networks, point-to-point telecommunication equipment, and carrier wave transmission media, just to name a few.

[0117] A computer process typically includes an executing (running) program or portion of a program, current program values and state information, and the resources used by the operating system to manage the execution of the process. An operating system (OS) is the software that manages the sharing of the resources of a computer and provides programmers with an interface used to access those resources. An operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as a service to users and programs of the system.

[0118] The computer system may for instance include at least one processing unit, associated memory and a number of input/output (I/O) devices. When executing the computer program, the computer system processes information according to the computer program and produces resultant output information via PO devices.

[0119] The connections as discussed herein may be any type of connection suitable to transfer signals from or to the respective nodes, units or devices, for example via intermediate devices. Accordingly, unless implied or stated otherwise, the connections may for example be direct connections or indirect connections. The connections may be illustrated or described in reference to being a single connection, a plurality of connections, unidirectional connections, or bidirectional connections. However, different embodiments may vary the implementation of the connections. For example, separate unidirectional connections may be used rather than bidirectional connections and vice versa. Also, plurality of connections may be replaced with a single connection that transfers multiple signals serially or in a time multiplexed manner. Likewise, single connections carrying multiple signals may be separated out into various different connections carrying subsets of these signals. Therefore, many options exist for transferring signals.

[0120] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative and that alternative embodiments may merge logic blocks or circuit elements or impose an alternate decomposition of functionality upon various logic blocks or circuit elements. Thus, it is to be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures can be implemented which achieve the same functionality.

[0121] Thus, any arrangement of components to achieve the same functionality is effectively “associated” such that the desired functionality is achieved. Hence, any two components herein combined to achieve a particular functionality can be seen as “associated with” each other such that the desired functionality is achieved, irrespective of architectures or intermedial components. Likewise, any two components so associated can also be viewed as being “operably connected,” or “operably coupled,” to each other to achieve the desired functionality.

[0122] Furthermore, those skilled in the art will recognize that boundaries between the above described operations merely illustrative. The multiple operations may be combined into a single operation, a single operation may be distributed in additional operations and operations may be executed at least partially overlapping in time. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

[0123] Also for example, the examples, or portions thereof, may implemented as soft or code representations of physical circuitry or of logical representations convertible into physical circuitry, such as in a hardware description language of any appropriate type.

[0124] Also, the disclosure is not limited to physical devices or units implemented in nonprogrammable hardware but can also be applied in programmable devices or units able to perform the desired device functions by operating in accordance with suitable program code, such as mainframes, minicomputers, servers, workstations, personal computers, notepads, personal digital assistants, electronic games, automotive and other embedded systems, cell phones and various other wireless devices, commonly denoted in this application as ‘computer systems’.