BI-LEVEL OPTIMIZATION-BASED INFRARED AND VISIBLE LIGHT FUSION METHOD

Abstract

The present invention proposes a bi-level optimization-based infrared and visible light fusion method, adopts a pair of infrared camera and visible light camera to acquire images, and relates to the construction of a bi-level paradigm infrared and visible light image fusion algorithm, which is an infrared and visible light fusion algorithm using mathematical modeling. Binocular cameras and NVIDIA TX2 are used to construct a high-performance computing platform and to construct a high-performance solving algorithm to obtain a high-quality infrared and visible light fusion image. The system is easy to construct, and the input data can be acquired by using stereo binocular infrared and visible light cameras respectively; the program is simple and easy to implement; and the fusion image is divided into an image domain and a gradient domain for fusion by means of mathematical modeling according to different imaging principles of infrared and visible light cameras.

Claims

1. A bi-level optimization-based infrared and visible light fusion method, wherein the method comprises the following steps: 1) obtaining registered infrared and visible light images, and respectively calibrating each lens and jointly calibrating the respective systems of the visible light binocular camera and the infrared binocular camera; 1-1) respectively calibrating the infrared camera and the visible light camera by the Zhangzhengyou calibration method to obtain internal parameters including focal length and principal point position and external parameters including rotation and translation of each camera; 1-2) calculating the positional relationship of the same plane in the visible light image and the infrared image by using RT obtained by joint calibration and the detected checker corners, and registering the visible light image to the infrared image by using a homography matrix; 2) converting the color space of the visible light image from an RGB image to an HSV image, extracting the value information of the color image as the input of image fusion, and retaining the original hue and saturation; 3) carrying out bi-level paradigm-based mathematical modeling on the input infrared image and the visible light image with the color space converted; establishing two separate models, namely the upper-level sub-problem Leader and the lower-level sub-problem Follower to solve the same problem. $\mathrm{Leader}\left(l\right):\underset{F}{\min }{.Math.F-I.Math.}_2^2+\gamma {.Math.F-V.Math.}_2^2$ $\mathrm{Follower}\left(f\right):s.t.F\in \underset{F}{\mathrm{argmin}}{.Math.\nabla F-\nabla I.Math.}_1+\beta {.Math.\nabla F-\nabla V.Math.}_1$ wherein F represents the fused image, the infrared image and the visible light image are respectively represented by I, V, ∇ represents the operator for obtaining the gradient, and γ, β represent parameters of Leader and Follower respectively; 4) solving the upper-level sub-problem to obtain the fusion result of the target in the image domain; the target result is obtained by solving the following formula: $F_1^{k+1}=\underset{F}{\mathrm{argmin}}{.Math.F-I.Math.}_2^2+\gamma {.Math.F-V.Math.}_2^2$ wherein F.sub.1.sup.k+1 represents the result of the upper-level problem; and the goal has a simple closed-form solution and thus is directly obtained by the closed-form solution shown as the following formula: $F_1^{k+1}=\frac{1+\gamma V}{1+\alpha }$ wherein F represents the fused image, I represents the infrared image, V represents the visible light image, γ represents the weight parameter of the visible light and infrared image, and α represents the weight variable parameter; 5) solving the lower-level sub-problem to obtain the fusion result of the target in the gradient domain; and using the alternating direction multiplier method to transform an unconstrained problem into a constrained problem by introducing auxiliary variables, and then solving the problem under the framework; 6) obtaining two estimates F.sub.1.sup.k+1, F.sub.f.sup.k+1 of the fusion result under different characteristics by solving the upper-level and lower-lever sub-problems, and to fuse the two components into an image F, linearly combining the two components, which is expressed as the following form:
F.sup.k+1=α.sup.kF.sub.l.sup.k+1+(1−α.sup.k)F.sub.f.sup.k+1 wherein F.sup.k+1 is the final result of each iteration and is manually selected based on experience; 7) converting the color space: converting the fusion image back to the RGB image, and adding the hue and saturation previously retained; restoring to the RGB color space from the HSV color space by updating the V information saved into the fusion image in combination with the previously retained H and S; 8) enhancing the color: enhancing the color of the fusion image to generate a fusion image with higher resolution and contrast; and performing pixel-level image enhancement for the contrast of each pixel; performing color correction and enhancement on the restored image to generate a three-channel image that is consistent with observation and detection; and performing color enhancement on the R channel, G channel and B channel respectively to obtain the final fusion image.

2. The bi-level optimization-based infrared and visible light fusion method according to claim 1, wherein the color space conversion of the visible light image in step 2) comprises: 2-1) converting the RGB color space to the HSV color space, wherein V is value, H is hue and S is saturation; and extracting the value information of the visible light image to be fused with the infrared image, and retaining the hue and saturation, wherein the specific conversion is shown as follows:
R′=R/255 G′=G/255 B′=B/255
Cmax=max(R′,G′,B′)
Cmin=min(R′,G′,B′)
Δ=Cmax−Cmin
V=Cmax 2-2) extracting the V channel as the input of visible light, retaining H and S to the corresponding matrix, and retaining the color information for the subsequent color restoration after fusion.

3. The bi-level optimization-based infrared and visible light fusion method according to claim 1, wherein the solving method in step 5): first introducing two auxiliary variables u, w, wherein u=∇F−∇I, w=∇F−∇V; and through variable substitution, transforming to minimizing the following problem: $\underset{F,u,w}{\min }{.Math.u.Math.}_1+\beta {.Math.w.Math.}_1+\frac{{\rho }_1}{2}{.Math.\nabla F-\nabla I-u-\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2+\frac{{\rho }_2}{2}{.Math.\nabla F-\nabla V-w-\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2+\frac{\tau }{2}{.Math.\nabla F-\nabla {\hat{F}}^k.Math.}_2^2$ wherein ∇ represents the gradient operator, λ.sub.1, λ.sub.2 are two multipliers, ρ.sub.1, ρ.sub.2 are parameters of a penalty term, and three sub-problems respectively about u, w, F are obtained through variable separation: $\underset{u}{\min }{.Math.u.Math.}_1+\frac{{\rho }_1}{2}{.Math.u-\nabla F+\nabla I+\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2$ $\underset{w}{\min }\beta {.Math.w.Math.}_1+\frac{{\rho }_2}{2}{.Math.w-\nabla F+\nabla V+\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2$ $\underset{F}{\min }\frac{{\rho }_1}{2}{.Math.\nabla F-\nabla I-u-\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2+\frac{{\rho }_2}{2}{.Math.\nabla F-\nabla V-w-\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2$ 5-1) for the update of F.sub.f.sup.k+1 of the lower-level problem, the closed-form solution of the formula is used to obtain: $F_f^{k+1}=\frac{{\rho }_1{\nabla }^T\left(\nabla I+u+\frac{{\lambda }_1}{{\rho }_1}\right)+{\rho }_2{\nabla }^T\left(\nabla V+w+\frac{{\lambda }_2}{{\rho }_2}\right)+I*\nabla {\hat{F}}^k}{{\rho }_1{\nabla }^T\nabla +{\rho }_2{\nabla }^T\nabla +I*{\nabla }^T\nabla }$ 5-2) the multipliers λ.sub.1, λ.sub.2 need updating after each iteration, and the specific update mode is as follows:
λ.sub.1.sup.k+1=λ.sub.1.sup.k−ρ.sub.1(∇F−∇V−w)
λ.sub.2.sup.k+1=λ.sub.2.sup.k−ρ.sub.2(∇F−∇I−u).

4. The bi-level optimization-based infrared and visible light fusion method according to claim 1, wherein the specific formulas for color space conversion in step 7) are shown as follows: $C=V\times S$ $X=C\times \left(1-.Math.\left(H/60°\right)\mathrm{mod}2-1.Math.\right)$ $m=V-C\left(R^{\prime },G^{\prime },B^{\prime }\right)=\{\begin{array}{cc}\left(C,X,0\right)& 0°\le H<60°\\ \left(X,C,0\right)& 60°\le H<120°\\ \left(0,C,X\right)& 120°\le H<180°\\ \left(0,X,C\right)& 180°\le H<240°\\ \left(X,0,C\right)& 240°\le H<300°\\ \left(C,0,X\right)& 300°\le H<360°\end{array}$
R′,G′,B′=((R′+m)×255,(G′+m)×255,(B′+m)×255) wherein C is the product of the value and the saturation; and in is the difference of the value and C.

5. The bi-level optimization-based infrared and visible light fusion method according to claim 1, wherein the specific formulas for color enhancement in step 8) are shown as follows:
R.sub.out=(R.sub.in).sup.1/gamma
R.sub.display=(R.sub.in.sup.(1/gamma)).sup.gamma
G.sub.out=(G.sub.in).sup.1/gamma
G=(G.sub.in.sup.(1/gamma))gamma
B.sub.out=(B.sub.in).sup.1/gamma
B.sub.display=(B.sub.in.sup.(1/gamma)).sup.gamma wherein gamma is the correction parameter, R.sub.in, G.sub.in and B.sub.in are the values of the three input channels R, G, and B respectively, R.sub.out, G.sub.out and B.sub.out are the intermediate parameters, and R.sub.display, G.sub.display and B.sub.display are the values of the three channels after enhancement.

Description

DESCRIPTION OF DRAWINGS

[0037] FIG. 1 is a flow chart of a visible light and infrared fusion algorithm.

[0038] FIG. 2 is a final fusion image.

DETAILED DESCRIPTION

[0039] The present invention proposes a method for real-time image fusion by an infrared camera and α visible light camera, and will be described in detail below in combination with drawings and embodiments.

[0040] The binocular stereo cameras are placed on a fixed platform, the image resolution of the experiment cameras is 1280×720, and the field of view is 45.4°. To ensure real-time performance, NVIDIA TX2 is used for calculation. On this basis, a real-time infrared and visible light fusion method is designed, and the method comprises the following steps:

[0041] 1) Obtaining registered infrared and visible light images;

[0042] 1-1) Respectively calibrating each lens and jointly calibrating the respective systems of the visible light binocular camera and the infrared binocular camera;

[0043] 1-2) Respectively calibrating the infrared camera and the visible light camera by the Zhangzhengyou calibration method to obtain internal parameters such as focal length and principal point position and external parameters such as rotation and translation of each camera.

[0044] 1-3) Calculating the positional relationship of the same plane in the visible light image and the infrared image by using RT obtained by joint calibration and the detected checker corners, and registering the visible light image to the infrared image by using a homography matrix.

[0045] 2) Converting the color space of the image

[0046] 2-1) In view of the problem that the visible light image has RGB three channels, converting the RGB color space to the HSV color space, extracting the V (value) information of the visible light image to be fused with the infrared image, and retaining H (hue) and S (saturation), wherein the specific conversion is shown as follows:

R′=R/255 G′=G/255 B′=B/255

Cmax=max(R′,G′,B′)

Cmin=min(R′,G′,B′)

Δ=Cmax−Cmin

V=Cmax

[0047] 2-2) Extracting the V (value) channel as the input of visible light, retaining H (hue) and S (saturation) to the corresponding matrix, and retaining the color information for the subsequent color restoration after fusion.

[0048] 3) Carrying out bi-level paradigm-based mathematical modeling for the input infrared image and the visible light image with the color space converted. The core of the idea is to establish two separate models, namely the upper-level sub-problem Leader and the lower-level sub-problem Follower to solve the same problem. First, the objective function and the problem constraint in optimization are assumed to be two participants in the game, wherein the objective function is regarded as the Leader and the problem constraint is the Follower. In this kind of competition, the next optimization of the Leader needs to consider the result of the Follower. The two competitors are defined as two composite minimization sub-problems:

[00008]$\mathrm{Leader}\left(l\right):\underset{F}{\min }{.Math.F-I.Math.}_2^2+\gamma {.Math.F-V.Math.}_2^2$$\mathrm{Follower}\left(f\right):s.t.F\in \underset{F}{\mathrm{argmin}}{.Math.\nabla F-\nabla I.Math.}_1+\beta {.Math.\nabla F-\nabla V.Math.}_1$

[0049] wherein F represents the fused image, and the infrared image and the visible light image are respectively represented by I, V, ∇ represents the operator for obtaining the gradient, and γ, β represent parameters of Leader and Follower respectively.

[0050] 4) Solving the upper-level sub-problem; the target result can be obtained by solving the following formula:

[00009]$F_1^{k+1}=\underset{F}{\mathrm{argmin}}{.Math.F-I.Math.}_2^2+\gamma {.Math.F-V.Math.}_2^2$

[0051] wherein F.sub.1.sup.k−1 represents the result of the upper-level problem. The goal has a simple closed-form solution and thus can be directly obtained by the closed-form solution shown as the following formula:

[00010]$F_1^{k+1}=\frac{1+\gamma V}{1+\alpha }$

[0052] 5) Solving the lower-level sub-problem; it can be found that the problem is non-convex and non-smooth and difficult to solve directly and is solved by the alternating direction multiplier method widely used. Therefore, it is necessary to transform an unconstrained problem into a constrained problem by introducing auxiliary variables, and then the problem is solved under the framework. Specifically, first introducing two auxiliary variables u, w, wherein u=∇F−∇I, w=∇F−∇V. Through variable substitution, transforming to minimizing the following problem:

[00011]$\underset{F,u,w}{\min }{.Math.u.Math.}_1+\beta {.Math.w.Math.}_1+\frac{{\rho }_1}{2}{.Math.\nabla F-\nabla I-u-\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2+\frac{{\rho }_2}{2}{.Math.\nabla F-\nabla V-w-\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2+\frac{\tau }{2}{.Math.\nabla F-\nabla {\hat{F}}^k.Math.}_2^2$

[0053] wherein ∇ represents the gradient operator, λ.sub.1, λ.sub.2 are two multipliers, ρ.sub.1, ρ.sub.2 are parameters of a penalty term, and three sub-problems respectively about u, w, F are obtained through variable separation:

[00012]$\underset{u}{\min }{.Math.u.Math.}_1+\frac{{\rho }_1}{2}{.Math.u-\nabla F+\nabla I+\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2$$\underset{w}{\min }\beta {.Math.w.Math.}_1+\frac{{\rho }_2}{2}{.Math.w-\nabla F+\nabla V+\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2$$\underset{F}{\min }\frac{{\rho }_1}{2}{.Math.\nabla F-\nabla I-u-\frac{{\lambda }_1}{{\rho }_1}.Math.}_2^2+\frac{{\rho }_2}{2}{.Math.\nabla F-\nabla V-w-\frac{{\lambda }_2}{{\rho }_2}.Math.}_2^2$

[0054] 5-1) For the update of F.sub.f.sup.k+1 of the lower-level problem, the closed-form solution of the formula is used to obtain:

[00013]$F_f^{k+1}=\frac{{\rho }_1{\nabla }^T\left(\nabla I+u+\frac{{\lambda }_1}{{\rho }_1}\right)+{\rho }_2{\nabla }^T\left(\nabla V+w+\frac{{\lambda }_2}{{\rho }_2}\right)+I*\nabla {\hat{F}}^k}{{\rho }_1{\nabla }^T\nabla +{\rho }_2{\nabla }^T\nabla +I*{\nabla }^T\nabla }$

[0055] 5-2) The multipliers λ.sub.1, λ.sub.2 need updating after each iteration, and the specific update mode is as follows:

λ.sub.1.sup.k+1=λ.sub.1.sup.k−ρ.sub.1(∇F−∇V−w)

λ.sub.2.sup.k+1=λ.sub.2.sup.k−ρ.sub.2(∇F−∇I−u)

[0056] 6) Obtaining two estimates F.sub.1.sup.k+1, F.sub.f.sup.k+1 of the fusion result under different characteristics by solving the upper-level and lower-lever sub-problems, and to fuse the two components into an image F, linearly combining the two components, which is expressed as the following form:

F.sup.k+1=α.sup.kF.sub.l.sup.k+1+(1−α.sup.k)F.sub.f.sup.k+1

[0057] wherein F is the final result of each iteration, and α is a parameter weighing the two components. The parameter needs to be manually selected based on experience, and is selected as 0.5 herein.

[0058] 7-1) Restoring to the RGB color space from the HSV color space by updating the V (value) information saved into the fusion image in combination with the previously retained H (hue) and S (saturation), wherein the specific formulas are shown as follows:

[00014]$C=V\times S$$X=C\times \left(1-.Math.\left(H/60°\right)\mathrm{mod}2-1.Math.\right)$$m=V-C\left(R^{\prime },G^{\prime },B^{\prime }\right)=\{\begin{array}{cc}\left(C,X,0\right)& 0°\le H<60°\\ \left(X,C,0\right)& 60°\le H<120°\\ \left(0,C,X\right)& 120°\le H<180°\\ \left(0,X,C\right)& 180°\le H<240°\\ \left(X,0,C\right)& 240°\le H<300°\\ \left(C,0,X\right)& 300°\le H<360°\end{array}$
R′,G′,B′=((R′+m)×255,(G′+m)×255,(B′+m)×255)

[0059] wherein C is the product of the value and the saturation; and in is the difference of the value and C.

[0060] 7-2) performing color correction and enhancement on the image restored in step 7-1 to generate a three-channel image that is consistent with observation and detection; and performing color enhancement on the R channel, G channel and B channel respectively, wherein the specific formulas are shown as follows:

R.sub.out=(R.sub.in).sup.1/gamma

R.sub.display=(R.sub.in.sup.(1/gamma)).sup.gamma

G.sub.out=(G.sub.in).sup.1/gamma

G=(G.sub.in.sup.(1/gamma)).sup.gamma

B.sub.out=(B.sub.in).sup.1/gamma

B.sub.display=(B.sub.in.sup.(1/gamma)).sup.gamma

[0061] wherein gamma is the correction parameter, R.sub.in, G.sub.in and B.sub.in are the values of the three input channels R, G, and B respectively, R.sub.out, G.sub.out and B.sub.out are the intermediate parameters, and R.sub.display, G.sub.display and B.sub.display are the values of the three channels after enhancement.