Method for processing images

Abstract

An image processing method using a first image sensor covering the visible range and a second sensor covering the infra-red range. The method includes a) acquiring a first image I.sub.1 of said given zone of space by means of the first image sensor, b) acquiring a second image I.sub.2 of said given zone of space by means of the second image sensor, c) performing a decomposition of the first image I.sub.1 so as to obtain at least one luminance image I.sub.1L, d) obtaining an image I.sub.f resulting from digital fusion of the luminance image I.sub.1L and of the second image I.sub.2, e) adding colour information to the fusion image I.sub.f or during the fusion stage d).

Claims

1. An image processing method using a device comprising at least one casing comprising at least one image sensor capable of covering a first spectral range and at least one second image sensor capable of covering a second spectral range different from the first spectral range, wherein the first and second sensors are capable of forming images of a same zone of space, with the first sensor being capable of covering a spectral range belonging to the range of wavelengths of the visible spectrum, wherein the method comprises: a) acquiring a first image I.sub.1 of said given zone of space by means of the first image sensor, b) acquiring a second image I.sub.2 of said given zone of space by means of the second image sensor, c) performing a decomposition of the first image I.sub.1 so as to obtain at least one luminance image .sub.1L, d) obtaining an image I.sub.f resulting from digital fusion of the luminance image I.sub.1L and of the second image I.sub.2, and e) adding colour information to the fusion image I.sub.f, wherein the digital fusion involves: i. carrying out decompositions, noted respectively F.sub.n.sup.1L and F.sub.n.sup.2, into successive spatial frequencies n of the luminance image I.sub.1L and of the second image I.sub.2, ii. performing an energy calculation in at least some of the zones of the images F.sub.n.sup.1L, F.sub.n.sup.2 each associated with a frequency component n, iii. calculating a weighting image P.sub.n.sup.1L, P.sub.n.sup.2, for each of the images F.sub.n.sup.1L, F.sub.n.sup.2 associated with a frequency component n, based on the local analysis performed at the preceding stage, iv. for each image F.sub.n.sup.1L, F.sub.n.sup.2 associated with a frequency component, performing the following calculation:
F.sub.n(x,y)=P.sub.n.sup.1L(x,y).Math.F.sub.n.sup.1L(x,y)+P.sub.n.sup.2(x,y).Math.F.sub.n.sup.2(x,y), and v. performing a recombination of all the images F.sub.n each associated with a frequency component n, so as to obtain a fused image I.sub.f of all the spatial frequencies.

2. A method according to claim 1, wherein prior to step c), adapting the dimensions of that of the first image I.sub.1 and of the second image I.sub.2 having the lowest resolution to the dimensions of the other of the first image and the second image.

3. A method according to claim 1, wherein prior to step c), matching the dynamics of the first image I.sub.1 to the dynamics of the second image I.sub.2.

4. A method according to claim 1, wherein the digital fusion includes a step of carrying out successive spatial frequency decompositions n of the luminance image I.sub.1L, and of the second image I.sub.2.

5. A method according to claim 4, wherein the decomposition into spatial frequencies involves successively applying an averaging or low-pass filter to each luminance image I.sub.1L, and second image I.sub.2, according to the following equation:
F.sub.n.sup.1Lou2=G.sub.n=G.sub.n1*S where G.sub.1=I.sub.e*S S denotes a low-pass filter I.sub.e denotes the input image I.sub.1L or I.sub.2.

6. A method according to claim 1, wherein step ii) involves calculating the energy according to the following formulation: $E_n^{1\mathrm{Lou}2}\left(x,y\right)=\mathrm{Var}\left(F_n^{1\mathrm{Lou}2}\left(\left[x-\frac{k_x}{2},x+\frac{k_x}{2}\right],\left[y-\frac{k_y}{2},y+\frac{k_y}{2}\right]\right)\right)$ where k.sub.x and k.sub.y represent the dimensions of an analysis window.

7. A method according to claim 6, wherein the weighting images P.sub.n.sup.1, P.sub.n.sup.2 are obtained as follows: $P_n^{1L}\left(x,y\right)=\frac{E_n^{\max }\left(x,y\right)+E_n\left(x,y\right)}{2.Math.E_n^{\max }}$ $P_n^2\left(x,y\right)=\frac{E_n^{\max }\left(x,y\right)-E_n\left(x,y\right)}{2.Math.E_n^{\max }}$ Where: E.sub.n(x,y)=E.sub.n.sup.1L(x,y)E.sub.n.sup.2(x,y) and E.sub.n.sup.max=max(E.sub.n(x,y)).

8. A method according to claim 1, wherein the recombination stage involves performing the following calculation: $I_f=\underset{n=1}{\overset{h}{.Math.}}{F}_n^{}$ where h represents the number of frequency components.

9. A method according to claim 1, wherein step c) involves carrying out a decomposition of the first image I.sub.1 into a luminance image I.sub.1L, and two chrominance images I.sub.1C.sub.b and I.sub.1C.sub.r.

10. A method according to claim 9, wherein step e) involves summing the fusion image I.sub.f with the chrominance images I.sub.1C.sub.b and I.sub.1C.sub.r.

11. A method according to claim 1, wherein the first sensor is capable of covering a spectral range belonging to the interval included between 0.4 and 1.1 m.

12. A method according to claim 1, wherein the second sensor is capable of covering a second spectral range belonging to the range of infra-red wavelengths.

13. A method according to claim 12, wherein the second sensor is capable of covering a spectral range belonging to the interval included between 7.5 and 14 m.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a flow chart of the main processing stages performed using the method according to the invention;

(2) FIG. 2 is a flow chart of the main processing stages performed during the digital fusion stage; and

(3) FIG. 3 is a table illustrating standardised levels of darkness.

DETAILED DESCRIPTION OF THE INVENTION

(4) Reference is first made to FIG. 1 which shows the principle of image acquisition and processing according to the invention. The image processing method according to the invention is intended to be implemented in a device comprising a casing housing at least two image sensors capable of covering different spectral ranges.

(5) By way of an example, the casing thus comprises a first image sensor covering a spectral range included between 0.4 and 1.1 m located in the visible range and a second image sensor capable of covering a spectral range included between 7.5 and 14 m located in the infra-red range. Both sensors are arranged in the casing so as to be capable of forming images of a same zone of space.

(6) The method 10 according to the invention involves acquiring a first image I.sub.1 in 12 of a given zone of space using the first sensor and acquiring a second image I.sub.2 in 14 of the same zone of space using the second sensor.

(7) Firstly, the second image I.sub.2 undergoes recalibration on the first image I.sub.1 in 16. This choice essentially results from the dimensions of the sensors used. Generally, infra-red sensors, such as the second sensor, have a lower resolution than sensors in the visible range such as the first sensor. For example, the first sensor may have an image resolution of 1280 by 1024 pixels for 10 bits and the second sensor may have an image resolution of 640 by 480 pixels for 14 bits. Therefore, recalibration essentially involves adapting the resolution of the second image obtained with the second sensor so that it corresponds to the resolution of the first image obtained with the first sensor in order to keep the resolution of the first best defined sensor intact.

(8) This recalibration stage also involves performing complete distortion of the image in order to take account of the enlargement to be made, the rotation, offset and geometric aberrations. In order to perform the recalibration, elements of information can be used a priori, such as the virtually parallel position of the optical axes of the optical systems of the first and second sensors. It is furthermore possible to make the assumption that the corrections to be made to the second image I.sub.2 are minimal.

(9) During another stage 18, which can be performed immediately after acquisition of the first image I.sub.1, a decomposition of the first image I.sub.1 into a luminance image I.sub.1L and two chrominance images I.sub.1C.sub.b and I.sub.1C.sub.r is performed. Since the image I.sub.1 is a model image (RGB), this model image (RGB) needs to be converted into an image of a model defining a colorimetric space with three components, namely a luminance component and the two chrominance components. The values of each pixel vary between 0 and 255 for each R, G, and B channel.

(10) An example of a model that can be used is the YCbCr model. Other models can be used, such as the YPbPr model, the YUV model, the TSL model or the Lab model. Generally speaking, any model with which one luminance component and two chrominance components can be obtained may be used.

(11) Calculations with which conversion of a model image (RGB) into one luminance component I.sub.1 and two chrominance components I.sub.1C.sub.b and I.sub.1C.sub.r can be performed according to a YCbCr model are provided here by way of non-limiting examples.

(12) Thus, the image I.sub.1 is broken down into: a luminance image noted I.sub.1L calculated according to the equation:
I.sub.1L=0.299R+0.587G+0.114B, a chrominance image I.sub.1C.sub.b calculated according to the equation:
I.sub.1C.sub.b=0.1687R0.3313G+0.5B+128, and a chrominance image I.sub.1C.sub.r calculated according to the equation:
I.sub.1C.sub.r=0.5R0.4187G0.0813B+128.

(13) For both models (RGB) and YCbCr, the pixel values for each channel vary in this case between 0 and 255.

(14) During a subsequent stage 20, the method involves adapting the dynamics of the image I.sub.1L with the second image I.sub.2. As mentioned earlier, the first and second images are not acquired with the same dynamics, 10 bits for example for the first image and 14 bits for the second image. It is therefore important to bring the first and second images to a same scale that makes it possible to obtain information concerning each of the first and second images that can be compared. In practice, adaptation of the dynamics allows adjustment of the dynamics of the luminance images I.sub.1L and the two chrominance images I.sub.1C.sub.b, I.sub.1C.sub.r with the second image I.sub.2.

(15) During a subsequent stage 22, fusion of the luminance image I.sub.1C.sub.b with the second image I.sub.2 is performed, thereby obtaining the image I.sub.f.

(16) Finally, in one or two final stages 24, colour information is added to the fusion image I.sub.f so as to obtain in 30 a fused final image including colour information.

(17) The actual fusion stage 22 will now be described in detail with reference to FIG. 2.

(18) In order to be able to interpret the contents of each of the luminance images I.sub.1L and of the second image I.sub.2 locally, i.e. at the level of each pixel analysed, decomposition into spatial frequencies is performed. Thus, decomposition into spatial frequencies is performed for the luminance image I.sub.1L and the second image I.sub.2. Reference is made here to the images I.sub.1L and I.sub.2 which are images obtained following recalibration and adaptation of the dynamics.

(19) Generally speaking, in order to obtain the first spatial frequency F1 (the highest frequency), the image is initially filtered by convolution with an averaging or low-pass filter K, which can adopt the form of a simple local average in a window of m by m pixels, i.e. the form of a Gaussian shape that subsequently weights the pixels of this same calculation window by coefficients following a Gaussian shape. A filtered implying low-pass filtered image G.sub.1 is obtained in this manner:
G.sub.1=I.sub.e*S

(20) This filtered image is subtracted from the original image. An image F1 that only contains high frequencies is obtained in this case:
F.sub.1=I.sub.eG.sub.1=I.sub.e(I.sub.e*S)

(21) The second spatial frequency F.sub.2 is obtained by reproducing the previous pattern, but using the previous filtered image F.sub.1 as the input image. This filtered image is therefore filtered again using the same low-pass filter as before and an image F.sub.2 is obtained by difference:
F.sub.2=G.sub.1G.sub.2=G.sub.1(G.sub.1*S)

(22) By recurrence, the following is obtained
G.sub.n+1=(G.sub.n*S) and F.sub.n+1=G.sub.n(G.sub.n*S)

(23) The last frequency h will be represented by the last convolution and will also represent the local average level of the image:
F.sub.h=G.sub.h=G.sub.h1*S

(24) Thus, one will note F.sub.n.sup.1L=G.sub.n1*S with G.sub.1=I.sub.1L*S and F.sub.n.sup.2=G.sub.n1*S with G.sub.1=I.sub.2*S.

(25) During a second processing stage, an energy calculation in at least some of the zones of the images of each of the frequency components F.sub.n.sup.1L and F.sub.n.sup.2 is performed. It is chosen here to carry out an analysis compatible with real-time processing allowing instantaneous restitution of data to an observer. To do this, an analysis by comparison of the energies of each of the images F.sub.n.sup.1L and F.sub.n.sup.2 is performed. This analysis involves calculating the local variance in a window of dimensions kxky centered on the pixel analysed, which yields:

(26) $E_n^{1L}\left(x,y\right)=\mathrm{Var}\left(F_n^{1L}\left(\left[x-\frac{k_x}{2},x+\frac{k_x}{2}\right],\left[y-\frac{k_y}{2},y+\frac{k_y}{2}\right]\right)\right)$$\mathrm{and},E_n^2\left(x,y\right)=\mathrm{Var}\left(F_n^2\left(\left[x-\frac{k_x}{2},x+\frac{k_x}{2}\right],\left[y-\frac{k_y}{2},y+\frac{k_y}{2}\right]\right)\right)$

(27) During a subsequent stage, the method involves calculating a weighting image P.sub.n.sup.1L, P.sub.n.sup.2, for each of the images F.sub.n.sup.1L, F.sub.n.sup.2 associated with a frequency component n, based on the local analysis performed at the preceding stage. For this purpose, the difference of these energies is calculated:
E.sub.n(x,y)=E.sub.n.sup.BNL(x,y)E.sub.n.sup.1R(x,y)

(28) The maximum is subsequently calculated:
E.sub.n.sup.max=max(E.sub.n(x,y))

(29) Next, a weighting image is calculated for each frequency component, with these weightings being such that:

(30) $P_n^{1L}\left(x,y\right)=\frac{E_n^{\max }\left(x,y\right)+E_n\left(x,y\right)}{2.Math.E_n^{\max }}$$P_n^2\left(x,y\right)=\frac{E_n^{\max }\left(x,y\right)-E_n\left(x,y\right)}{2.Math.E_n^{\max }}$$\mathrm{with}{P}_n\left(x,y\right)\left[0,1\right]$

(31) The weighting images for each spatial frequency P.sub.n.sup.1L(x,y) and P.sub.n.sup.2(x,y) are subsequently applied to the respective images F.sub.n.sup.1L and F.sub.n.sup.2 and are finally summed according to the following equation:
F.sub.n(x,y)=P.sub.n.sup.1L(x,y).Math.F.sub.n.sup.1L(x,y)+P.sub.n.sup.2(x,y).Math.F.sub.n.sup.2(x,y)

(32) The synthetic image F.sub.n(x,y) for the spatial frequency n thus closer, owing to the weighting applied, to the image F.sub.n.sup.1L and F.sub.n.sup.2 which contains the most information.

(33) Recomposition of the spatial frequencies is the reverse process of stage 1. One therefore begins with the lowest frequency h, i.e. the last obtained from the decomposition described above, resulting in:
G.sub.h=F.sub.h

(34) Recomposition is subsequently performed using the previous spatial frequencies:
G.sub.h1=F.sub.h1+G.sub.h

(35) If I.sub.f is the result of recombination, then:

(36) $I_f=G_0=\underset{n=1}{\overset{h}{.Math.}}{F}_n^{}$ where h represents the number of frequency components.

(37) After obtaining the fusion image I.sub.f, the method involves adding colour information to the fusion image I.sub.f.

(38) Hence, the method involves using the colour of the first sensor. For this purpose, addition of the colour information is performed by summing or resynchronising the fusion image I.sub.f with the chrominance images I.sub.1C.sub.b and I.sub.1C.sub.r, the dynamics of which have been adapted as mentioned above.

(39) In practice, it will be understood that this type of addition of colour information will prove less and less relevant as the level of luminosity of the observed area decreases. Indeed, the colour information derived from the chrominance images will gradually decrease.

(40) It should be remembered that levels of darkness have long been standardised according to the table below, which is repeated for information purposes and is well known from the state of the art.

(41) The units indicated in the table above are in lux, which is the unit derived from the international system, such that 1 lux=1 cd.Math.sr.Math.m.sup.2. This table shows the darkness level values depending on the illuminance, a type of sky being associated with each darkness level.