MULTI-EXPOSURE IMAGE FUSION METHOD BASED ON FEATURE DISTRIBUTION WEIGHT OF MULTI-EXPOSURE IMAGE

Abstract

The present disclosure provides a multi-exposure image fusion (MEF) method based on a feature distribution weight of a multi-exposure image, including: performing color space transformation (CST) on an image, determining a luminance distribution weight of the image, determining an exposure distribution weight of the image, determining a local gradient weight of the image, determining a final weight, and determining a fused image. The present disclosure combines the luminance distribution weight of the image, the exposure distribution weight of the image and the local gradient weight of the image to obtain the final weight, and fuses the input image and the weight with the existing pyramid-based multi-resolution fusion method to obtain the fused image, thereby solving the technical problem that an existing MEF method does not consider the overall feature distribution of the multi-exposure image.

Claims

1. A multi-exposure image fusion (MEF) method based on a feature distribution weight of a multi-exposure image, comprising the following steps: (1) performing color space transformation (CST) on an image transforming red, green, and blue (RGB) color components of an input multi-exposure image into a luminance component Y.sub.n(x,y), a blue chrominance component Cb.sub.n(x,y), and a red chrominance component Cr.sub.n(x,y):
Y.sub.n(x,y)=0.257R.sub.n(x,y)+0.564G.sub.n(x,y)+0.098B.sub.n(x,y)+16
Cb.sub.n(x,y)=−0.148R.sub.n(x,y)−0.291G(x,y)+0.439B.sub.n(x,y)+128
Cr.sub.n(x,y)=0.439R.sub.n(x,y)−0.368G.sub.n(x,y)−0.071B.sub.n(x,y)+128   (1) wherein, R.sub.n(x,y) is a red component at a pixel (x,y) in an nth image, G.sub.n(x,y) is a green component at the pixel (x,y) in the nth image, B.sub.n(x,y) is a blue component at the pixel (x,y) in the nth image, n is a finite positive integer, nϵ[1,N], (x,y) is a position coordinate of the pixel, and N is a number of exposure images and is a finite positive integer; (2) determining a luminance distribution weight of the image determining the luminance distribution weight w.sub.1,n(x,y) of the image according to eq. (2): $\begin{array}{cc}{w}_{1,n}\left(x,y\right)=\exp \left(-\frac{{\left(Y_n\left(x,y\right)-\left(1-m_n\right)\right)}^2}{2S_n^2\left(x,y\right)}\right)& \left(2\right)\end{array}$ $S_n\left(x,y\right)=\frac{Y_n\left(x,y\right)}{{.Math.}_{n=1}^N{Y}_n\left(x,y\right)}$ wherein, m.sub.n is a mean of the nth image; (3) determining an exposure distribution weight of the image determining the exposure distribution weight w.sub.2,n(x,y) of the image according to eq. (3): $\begin{array}{cc}{w}_{2,n}\left(x,y\right)=1-D_n\left(x,y\right)& \left(3\right)\end{array}$ $D_n\left(x,y\right)=.Math.Y_n\left(x,y\right)-m\left(x,y\right).Math.$ $m\left(x,y\right)=\frac{1}{N}{.Math.}_{n=1}^N{Y}_n\left(x,y\right)$ (4) determining a local gradient weight of the image determining the local gradient weight w.sub.3,n(x,y) of the image according to eq. (4): $\begin{array}{cc}{w}_{3,n}\left(x,y\right)=\frac{{\mathrm{grad}}_n\left(x,y\right)}{{.Math.}_{n=1}^N{\mathrm{grad}}_n\left(x,y\right)+\epsilon }& \left(4\right)\end{array}$ ${\mathrm{grad}}_n\left(x,y\right)=\sqrt{d_x^2\left(x,y\right)+d_y^2\left(x,y\right)}$ $d_x\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial x}$ $d_y\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial y}$ $G_1\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_1^2}\exp \left(-\frac{{\left(x-\left(k_1+1\right)\right)}^2+{\left(y-\left(k_1+1\right)\right)}^2}{2{\sigma }_1^2}\right)$ wherein, ε is in a range of [10.sup.−14,10.sup.−10], .Math. is a filtering operation, G.sub.1(x,y) is a two-dimensional (2D) Gaussian kernel function, k.sub.1 is a parameter on a window size of a filter, k.sub.1ϵ[2,16], and σ.sub.1 is a standard deviation of the filter, σ.sub.1∈(0,5]; (5) determining a final weight determining the final weight W.sub.n(x,y) according to eq. (5): $\begin{array}{cc}{W}_n\left(x,y\right)=\frac{g_n\left(x,y\right)}{{.Math.}_{n=1}^N{g}_n\left(x,y\right)+\epsilon }& \left(5\right)\end{array}$ $g_n\left(x,y\right)=w_n\left(x,y\right).Math.G_2\left(x,y\right)$ $G_2\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_2^2}\exp \left(-\frac{{\left(x-\left(k_2+1\right)\right)}^2+{\left(y-\left(k_2+1\right)\right)}^2}{2{\sigma }_2^2}\right)$ $w_n\left(x,y\right)=w_{1,n}\left(x,y\right)\times w_{2,n}\left(x,y\right)\times w_{3,n}\left(x,y\right)$ wherein, G.sub.2(x,y) is a 2D Gaussian kernel function, k.sub.2 is a parameter on a window size of a filter, k.sub.2∈[2,20], and σ.sub.2 is a standard deviation of the filter, σ.sub.2ϵ(0,10]; and (6) determining a fused image fusing the input image and the weight with an existing pyramid-based multi-resolution fusion method to determine the fused image; and completing the MEF method based on a feature distribution weight of a multi-exposure image

2. The MEF method based on a feature distribution weight of a multi-exposure image according to claim 1, wherein in step (4), the k.sub.1 is 8, and the σ.sub.1 is 2.

3. The MEF method based on a feature distribution weight of a multi-exposure image according to claim 1, wherein in step (5), the k.sub.2 is 10, and the σ.sub.2 is 5.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] FIG. 1 is a flow chart according to Embodiment 1 of the present disclosure;

[0028] FIG. 2 illustrates four input images of a multi-exposure cave image of Embodiment 1; and

[0029] FIG. 3 illustrates a fused image of Embodiment 1.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] The present disclosure will be further described below in detail with reference to accompanying drawings and embodiments, but the present disclosure is not limited to the following embodiments.

Embodiment 1

[0031] Taking four input images of the multi-exposure cave image as an example, the MEF method based on a feature distribution weight of a multi-exposure image in the embodiment includes the following steps (refer to FIG. 1):

[0032] (1) CST is performed on an image

[0033] RGB color components of an input multi-exposure image are transformed into a luminance component Y.sub.n(x,y), a blue chrominance component Cb.sub.n(x,y), and a red chrominance component Cr.sub.n(x,y):

Y.sub.n(x,y)=0.257R.sub.n(x,y)+0.564G.sub.n(x,y)+0.098B.sub.n(x,y)+16

Cb.sub.n(x,y)=−0.148R.sub.n(x,y)−0.291G.sub.n(x,y)+0.439B.sub.n(x,y)+128

Cr.sub.n(x,y)=0.439R.sub.n(x,y)−0.368G.sub.n(x,y)−0.071B.sub.n(x,y)+128   (1)

[0034] where, R.sub.n(x,y) is a red component at a pixel (x,y) in an nth image, G.sub.n(x,y) is a green component at the pixel (x,y) in the nth image, B.sub.n(x,y) is a blue component at the pixel (x,y) in the nth image, n is a finite positive integer, nϵ[1,N], (x,y) is a position coordinate of the pixel, and N is a number of exposure images. In the embodiment, the input images are as shown in FIG. 2, and the N is 4. For different input images, the N and the number of input images are the same.

[0035] (2) A luminance distribution weight of the image is determined

[0036] The luminance distribution weight w.sub.1,n(x,y) of the image is determined according to Eq. (2):

[00005]$\begin{array}{cc}{w}_{1,n}\left(x,y\right)=\exp \left(-\frac{{\left(Y_n\left(x,y\right)-\left(1-m_n\right)\right)}^2}{2S_n^2\left(x,y\right)}\right)& \left(2\right)\end{array}$$S_n\left(x,y\right)=\frac{Y_n\left(x,y\right)}{{.Math.}_{n=1}^N{Y}_n\left(x,y\right)}$

[0037] where, m.sub.n is a mean of the nth image.

[0038] (3) An exposure distribution weight of the image is determined

[0039] The exposure distribution weight w.sub.2,n(x,y) of the image is determined according to Eq. (3):

[00006]$\begin{array}{cc}{w}_{2,n}\left(x,y\right)=1-D_n\left(x,y\right)& \left(3\right)\end{array}$$D_n\left(x,y\right)=.Math.Y_n\left(x,y\right)-m\left(x,y\right).Math.$$m\left(x,y\right)=\frac{1}{N}{.Math.}_{n=1}^N{Y}_n\left(x,y\right)$

[0040] (4) A local gradient weight of the image is determined

[0041] The local gradient weight w.sub.3,n(x,y) of the image is determined according to Eq. (4):

[00007]$\begin{array}{cc}{w}_{3,n}\left(x,y\right)=\frac{{\mathrm{grad}}_n\left(x,y\right)}{{.Math.}_{n=1}^N{\mathrm{grad}}_n\left(x,y\right)+\epsilon }& \left(4\right)\end{array}$${\mathrm{grad}}_n\left(x,y\right)=\sqrt{d_x^2\left(x,y\right)+d_y^2\left(x,y\right)}$$d_x\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial x}$$d_y\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial y}$$G_1\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_1^2}\exp \left(-\frac{{\left(x-\left(k_1+1\right)\right)}^2+{\left(y-\left(k_1+1\right)\right)}^2}{2{\sigma }_1^2}\right)$

[0042] where, ε is in a range of [10.sup.−14,10.sup.−10] and the ε is 10.sup.−12 in the embodiment, .Math. is a filtering operation, G.sub.1(x,y) is a 2D Gaussian kernel function, k.sub.1 is a parameter on a window size of a filter, k.sub.1ϵ[2,16] and the k.sub.1 is 8 in the embodiment, and σ.sub.1 is a standard deviation of the filter, σ.sub.1∈(0,5] and the σ.sub.1 is 2 in the embodiment.

[0043] (5) A final weight is determined

[0044] The final weight W.sub.n(x,y) is determined according to Eq. (5):

[00008]$\begin{array}{cc}{W}_n\left(x,y\right)=\frac{g_n\left(x,y\right)}{{.Math.}_{n=1}^N{g}_n\left(x,y\right)+\epsilon }& \left(5\right)\end{array}$$g_n\left(x,y\right)=w_n\left(x,y\right).Math.G_2\left(x,y\right)$$G_2\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_2^2}\exp \left(-\frac{{\left(x-\left(k_2+1\right)\right)}^2+{\left(y-\left(k_2+1\right)\right)}^2}{2{\sigma }_2^2}\right)$$w_n\left(x,y\right)=w_{1,n}\left(x,y\right)\times w_{2,n}\left(x,y\right)\times w_{3,n}\left(x,y\right)$

[0045] where, G.sub.2 (x,y) is a 2D Gaussian kernel function, k.sub.2 is a parameter on a window size of a filter, k.sub.2∈[2,20] and the k.sub.2 is 10 in the embodiment, and σ.sub.2 is a standard deviation of the filter, σ.sub.2ϵ(0,10] and the σ.sub.2 is 5 in the embodiment.

[0046] By determining the luminance distribution weight of the image, the exposure distribution weight of the image and the local gradient weight of the image, the embodiment combines the luminance distribution weight of the image, the exposure distribution weight of the image and the local gradient weight of the image to obtain the final weight. When the weight of the input image is determined, considerations are given to the feature distribution of the multi-exposure image. The weight function can adaptively compute the weight for each pixel in the images, and can obtain the weights of the images simply and quickly, to solve the technical problem that the existing MEF method does not consider the overall feature distribution of the multi-exposure image. The fused image has the advantages of high fusion quality, simple computation, abundant information and clear details, and so on.

[0047] (6) A fused image is determined

[0048] The input image and the weight are fused with an existing pyramid-based multi-resolution fusion method to determine the fused image. The pyramid-based multi-resolution fusion method has been disclosed on the following website:

[0049] https://github.com/tkd1088/multi-exposure-image-fusion/blob/master/code/functions/fus ion_pyramid.m.

[0050] The MEF method based on a feature distribution weight of multi-exposure image is completed. The fused image is as shown in FIG. 3. It can be seen from FIG. 3 that the image processed with the method in the embodiment includes abundant information and clear details.

Embodiment 2

[0051] Taking four input images of the multi-exposure cave image as an example, the MEF method based on a feature distribution weight of a multi-exposure image in the embodiment includes the following steps:

[0052] (1) CST is performed on an image

[0053] The step is the same as that of Embodiment 1.

[0054] (2) A luminance distribution weight of the image is determined

[0055] The step is the same as that of Embodiment 1.

[0056] (3) An exposure distribution weight of the image is determined

[0057] The step is the same as that of Embodiment 1.

[0058] (4) A local gradient weight of the image is determined

[0059] The local gradient weight w.sub.3,n(x,y) of the image is determined according to Eq. (4):

[00009]$\begin{array}{cc}{w}_{3,n}\left(x,y\right)=\frac{{\mathrm{grad}}_n\left(x,y\right)}{{.Math.}_{n=1}^N{\mathrm{grad}}_n\left(x,y\right)+\epsilon }& \left(11\right)\end{array}$${\mathrm{grad}}_n\left(x,y\right)=\sqrt{d_x^2\left(x,y\right)+d_y^2\left(x,y\right)}$$d_x\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial x}$$d_y\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial y}$$G_1\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_1^2}\exp \left(-\frac{{\left(x-\left(k_1+1\right)\right)}^2+{\left(y-\left(k_1+1\right)\right)}^2}{2{\sigma }_1^2}\right)$

[0060] where, ε is in a range of [10.sup.−14,10.sup.−10] and the ε is 10.sup.−14 in the embodiment, .Math. is a filtering operation, G.sub.1(x,y) is a 2D Gaussian kernel function, k.sub.1 is a parameter on a window size of a filter, k.sub.1ϵ[2,16] and the k.sub.1 is 2 in the embodiment, and σ.sub.1 is a standard deviation of the filter, σ.sub.1∈(0,5] and the σ.sub.1 is 0.5 in the embodiment.

[0061] (5) A final weight is determined

[0062] The final weight W.sub.n(x,y) is determined according to Eq. (5):

[00010]$\begin{array}{cc}{W}_n\left(x,y\right)=\frac{g_n\left(x,y\right)}{{.Math.}_{n=1}^N{g}_n\left(x,y\right)+\epsilon }& \left(12\right)\end{array}$$g_n\left(x,y\right)=w_n\left(x,y\right).Math.G_2\left(x,y\right)$$G_2\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_2^2}\exp \left(-\frac{{\left(x-\left(k_2+1\right)\right)}^2+{\left(y-\left(k_2+1\right)\right)}^2}{2{\sigma }_2^2}\right)$$w_n\left(x,y\right)=w_{1,n}\left(x,y\right)\times w_{2,n}\left(x,y\right)\times w_{3,n}\left(x,y\right)$

[0063] where, G.sub.2(x,y) is a 2D Gaussian kernel function, k.sub.2 is a parameter on a window size of a filter, k.sub.2∈[2,20] and the k.sub.2 is 2 in the embodiment, and σ.sup.2 is a standard deviation of the filter, σ.sub.2ϵ(0,10] and the σ.sub.2 is 0.5 in the embodiment.

[0064] Other steps are the same as those of Embodiment 1.

[0065] The MEF method based on a feature distribution weight of a multi-exposure image is completed.

Embodiment 3

[0066] Taking four input images of the multi-exposure cave image as an example, the MEF method based on a feature distribution weight of a multi-exposure image in the embodiment includes the following steps:

[0067] (1) CST is performed on an image

[0068] The step is the same as that of Embodiment 1.

[0069] (2) A luminance distribution weight of the image is determined

[0070] The step is the same as that of Embodiment 1.

[0071] (3) An exposure distribution weight of the image is determined

[0072] The step is the same as that of Embodiment 1.

[0073] (4) A local gradient weight of the image is determined

[0074] The local gradient weight w.sub.3,n(x,y) of the image is determined according to Eq. (4):

[00011]$\begin{array}{cc}{w}_{3,n}\left(x,y\right)=\frac{{\mathrm{grad}}_n\left(x,y\right)}{{.Math.}_{n=1}^N{\mathrm{grad}}_n\left(x,y\right)+\epsilon }& \left(13\right)\end{array}$${\mathrm{grad}}_n\left(x,y\right)=\sqrt{d_x^2\left(x,y\right)+d_y^2\left(x,y\right)}$$d_x\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial x}$$d_y\left(x,y\right)=\frac{\partial \left(Y_n\left(x,y\right).Math.G_1\left(x,y\right)\right)}{\partial y}$$G_1\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_1^2}\exp \left(-\frac{{\left(x-\left(k_1+1\right)\right)}^2+{\left(y-\left(k_1+1\right)\right)}^2}{2{\sigma }_1^2}\right)$

[0075] where, ε is in a range of [10.sup.−14,10.sup.−10] and the ε is 10.sup.−10 in the embodiment, .Math. is a filtering operation, G.sub.1(x,y) is a 2D Gaussian kernel function, k.sub.1 is a parameter on a window size of a filter, k.sub.1ϵ[2,16] and the k.sub.1 is 16 in the embodiment, and σ.sub.1 is a standard deviation of the filter, σ.sub.1∈(0,5] and the σ.sub.1 is 5 in the embodiment.

[0076] (5) A final weight is determined

[0077] The final weight W.sub.n(x,y) is determined according to Eq. (5):

[00012]$\begin{array}{cc}{W}_n\left(x,y\right)=\frac{g_n\left(x,y\right)}{{.Math.}_{n=1}^N{g}_n\left(x,y\right)+\epsilon }& \left(14\right)\end{array}$$g_n\left(x,y\right)=w_n\left(x,y\right).Math.G_2\left(x,y\right)$$G_2\left(x,y\right)=\frac{1}{2{\mathrm{\pi \sigma }}_2^2}\exp \left(-\frac{{\left(x-\left(k_2+1\right)\right)}^2+{\left(y-\left(k_2+1\right)\right)}^2}{2{\sigma }_2^2}\right)$$w_n\left(x,y\right)=w_{1,n}\left(x,y\right)\times w_{2,n}\left(x,y\right)\times w_{3,n}\left(x,y\right)$

[0078] where, G.sub.2(x,y) is a 2D Gaussian kernel function, k.sub.2 is a parameter on a window size of a filter, k.sub.2∈[2,20] and the k.sub.2 is 20 in the embodiment, and σ.sub.2 is a standard deviation of the filter, σ.sub.2ϵ (0,10] and the σ.sub.2 is 10 in the embodiment.

[0079] Other steps are the same as those of Embodiment 1.

[0080] The MEF method based on a feature distribution weight of a multi-exposure image is completed.

[0081] Taking four input images of the multi-exposure cave image as an example, comparative simulation experiments are conducted according to the MEF method based on a feature distribution weight of a multi-exposure image in Embodiment 1 (hereinafter referred to as the present disclosure), Exposure Fusion Method (hereinafter referred to as No. 1), Multi-exposure Image Fusion: A Patch-wise Approach (hereinafter referred to as No. 2), A Multi-Exposure Image Fusion Based on the Adaptive Weights Reflecting the Relative Pixel Intensity and Global Gradient (hereinafter referred to as No. 3), and Multi-exposure Image Fusion via a Pyramidal Integration of the Phase Congruency of Input Images with the Intensity-based Maps (hereinafter referred to as No. 4), to verify the effects of the present disclosure. The existing MEF-structural similarity index (MEF-SSIM), average information entropy and average color saturation are used for evaluations, in which the higher MEF-SSIM score is an indication of the better quality of the fused image, the higher average information entropy is an indication of the more abundant information in the image, and the higher average saturation is an indication of the brighter color of the image. The MEF-SSIM scores for the present disclosure, No. 1, No. 2, No. 3 and No. 4 are compared.

[0082] Experimental results are shown in Table 1.

TABLE-US-00001 TABLE 1 MEF-SSIM scores of the present disclosure and other methods The present MEF No. 1 No. 2 No. 3 No. 4 disclosure MEF-SSIM score 0.980 0.975 0.977 0.981 0.982

[0083] As can be seen from Table 1, comparing the present disclosure with No. 1, No. 2, No. 3 and No. 4 in terms of the MEF-SSIM score, the present disclosure has the highest MEF-SSIM score of 0.982, indicating that the present disclosure achieves the better quality of fused image than other methods.

[0084] The average information entropies for the present disclosure, No. 3 and No. 4 are compared, with the result shown in Table 2.

TABLE-US-00002 TABLE 2 Average information entropies for the present disclosure and other methods The present MEF No. 3 No. 4 disclosure Average information entropy 3.231 3.220 3.238

[0085] As can be seen from Table 2, comparing the present disclosure with No. 3 and No. 4 in terms of the average information entropy, the present disclosure has the largest average information entropy of 3.238, indicating that the present disclosure includes more abundant information in the fused image.

[0086] The average color saturations for the present disclosure, No. 3 and No. 4 are compared, with the result shown in Table 3.

TABLE-US-00003 TABLE 3 Average color saturations for the present disclosure and other methods The present MEF No. 3 No. 4 disclosure Average color saturation 0.280 0.289 0.290

[0087] As can be seen from Table 3, comparing the present disclosure with No. 3 and No. 4 in terms of the average color saturation, the present disclosure has the largest average color saturation of 0.290, indicating that the present disclosure achieves the fused image with the brighter color.