PAN-SHARPENING METHOD BASED ON MULTIMODAL TEXTURE CORRECTION AND ADAPTIVE EDGE DETAIL FUSION

Abstract

A pan-sharpening method based on multimodal texture correction and adaptive edge detail fusion is provided, including: fusing upsampled low-resolution multispectral (LRMS) images with panchromatic images to obtain fused images; respectively extracting intensity components of the LRMS image and the fused image; inputting the intensity components and the panchromatic images into a multimodal texture correction model, and performing optimization solution on the multimodal texture correction model through optimization method to obtain texture-corrected images; extracting details of the texture-corrected images and applying edge protection to obtain first image details; extracting details of the upsampled LRMS image and applying edge protection to obtain second image details; performing adaptive fusion on the first image details and the second image details to obtain detail information; and adding the detail information to the upsampled LRMS image to obtain final high-resolution multispectral (HRMS) images.

Claims

1. A pan-sharpening method based on multimodal texture correction and adaptive edge detail fusion, comprising following steps: obtaining a low-resolution multispectral (LRMS) image and a panchromatic image, fusing an upsampled LRMS image with the panchromatic image to obtain a fused image, respectively extracting intensity components of the LRMS image and the fused image, inputting the intensity components and the panchromatic image into a multimodal texture correction model, and performing optimization solution on the multimodal texture correction model through an optimization method to obtain a texture-corrected image, wherein the multimodal texture correction model is constructed based on a variational optimization model; and performing detail extraction and edge protection on the texture-corrected image to obtain first image details; performing detail extraction and edge protection on the upsampled LRMS image to obtain second image details; performing adaptive fusion on the first image details and the second image details to obtain detail information, and adding the detail information to the upsampled LRMS image to obtain a final high-resolution multispectral (HRMS) image; wherein the multimodal texture correction model is: $T_C=\underset{T_C}{\arg \min }\frac{1}{2}{.Math.{\mathrm{DHT}}_C-I_0.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.{}^2\left({\mathrm{DHT}}_C\right)-{}^2{I}_0.Math.}_F^2+\frac{}{2}{.Math.T_C-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2+{.Math.{}^2{T}_C.Math.}_1$ wherein T.sub.C is the texture-corrected image, D represents a downsampling matrix, H represents a degradation filter, I.sub.0 represents the intensity component of the LRMS image, , , , , represent penalty parameters corresponding to different terms, .sup.2 is a Laplacian operator, P represents the panchromatic image, I.sub.net represents the intensity component of the fused image, | |F represents an Frobenius norm, and .Math..sub.1 represents a 1-norm; wherein the degradation filter H is obtained through an adaptive degradation filter algorithm, wherein the degradation filter H adopts a Gaussian filter H.sub.A, and the adaptive degradation filter algorithm is: $H_A=\underset{H_A}{\arg \min }\frac{1}{2}{.Math.{\mathrm{DH}}_A{T}_C-I_0.Math.}_F^2$ wherein DH.sub.AT.sub.C=DF.sup.1(H.sub.A (u, v) F(T.sub.C)); F(.Math.) represents a fast Fourier transform (FFT) operation, and F.sup.1(.Math.) represents an inverse fast Fourier transform (IFFT) operation; and a frequency domain expression H.sub.A (u, v) of the Gaussian filter H.sub.A is: $H_A\left(u,v\right)=e^{-\frac{D_C^2\left(u,v\right)}{2{}^2}}$ wherein D.sub.C (u, v) represents distance from a point (u, v) to a center of the frequency domain, represents standard deviation, and obtains an optimal value according to correlation and similarity indexes, and the optimal value of is .sub.best: ${}_{\mathrm{best}}=\underset{}{\arg \max }\frac{\left({\mathrm{DH}}_A{T}_C,I_0\right)+S\left({\mathrm{DH}}_A{T}_C,I_0\right)}{2}$ wherein (DH.sub.AT.sub.C, I.sub.0) is a correlation coefficient (CC) index between DH.sub.AT.sub.C and I.sub.0, and S (DH.sub.AT.sub.C, I.sub.0) is a structural similarity index measure (SSIM) index between the DH.sub.AT.sub.C and the I.sub.0.

2. The method according to claim 1, wherein: the intensity components of the LRMS image and the fused image are extracted by performing linear weighted summation on each band image of the LRMS image and each band image of the fused image.

3. The method according to claim 1, wherein the fused image is obtained by fusing the upsampled LRMS image with the panchromatic image through a pan-sharpening (A-PNN) model based on a target adaptive convolutional neural network.

4. The method according to claim 1, wherein the multimodal texture correction model is optimized and solved through an alternating direction method of multipliers (ADMM) model.

5. The method according to claim 1, wherein a process of extracting details from the texture-corrected image comprises: $D_{T_C}=T_C-T_{\mathrm{CL}}$ wherein D.sub.TC is image details of the texture-corrected image, T.sub.C represents the texture-corrected image, T.sub.CL is a low-resolution version of the texture-corrected image, $T_{\mathrm{CL}}={}_1{I}_{\mathrm{UP}}+\left(1-{}_1\right)T_{\mathrm{CD}}s.t.0<{}_1<1$ wherein .sub.1 represents a weight coefficient, I.sub.UP represents an intensity component of the upsampled LRMS image, and T.sub.CD represents an image of the texture-corrected image processed by the Gaussian filter; ${}_1=\sqrt{1-e^{-x_3}}s.t.x_3=\frac{x_1}{x_1+x_2}$ wherein .sub.3 represents a normalized weight, .sub.1 represents an influence coefficient of I.sub.UP, x.sub.2 represents an influence coefficient of T.sub.CD, a value of x.sub.1 is a mean value of correlation and similarity between the T.sub.C and the I.sub.UP, and a value of .sub.2 is a mean value of correlation and similarity between the T.sub.C and the T.sub.CD.

6. The method according to claim 1, wherein a process of adaptively fusing the first image details and the second image details comprises: enhancing the second image details to a same level as first image details according to a scale factor : $F_3^i={}_i{F}_2^i$ wherein F.sub.2 represents the second image details, F.sub.3 represents enhanced second image details, and superscript or subscript i represents a band label corresponding to an image; and fusing the enhanced second image details with the first image details to obtain detail information F: $F_i={}_2{F}_1+\left(1-{}_2\right)F_3^i$ wherein .sub.2 is a weight coefficient, .sub.2={square root over (1e.sup.x.sup.1)}, wherein x.sub.1 represents an influence coefficient of I.sub.UP, and a value of the x.sub.1 is a mean value of correlation and similarity between the T.sub.C and the I.sub.UP, and F.sub.1 represents the first image details.

7. The method according to claim 1, wherein a process of adding the detail information to the upsampled LRMS image comprises: $M_{\mathrm{HR}}^i=M_{\mathrm{UP}}^i+g_i\frac{M_{\mathrm{UP}}^i}{\frac{1}{B}\underset{i=1}{\overset{B}{.Math.}}{M}_{\mathrm{UP}}^i}{F}_i$ wherein g represents a scale factor of injected details, M.sub.UP is the upsampled LRMS image, B represents total number of bands, i represents a band label, superscript or subscript i represents a band label corresponding to the image, F represents the detail information, and MER is the HRMS image.

8. The method according to claim 1, wherein a scale factor g for injected details is: $g_i=\frac{{}^2\left(T_C\right)+\mathrm{cov}\left(T_C,M_{\mathrm{UP}}^i\right)}{{}^2\left(T_C\right)}$ wherein cov(.Math.) is a covariance function, .sup.2 is a variance function, T.sub.C represents the texture-corrected image, M.sub.UP is the upsampled LRMS image, and superscript or subscript i represents a band label corresponding to the image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings forming a part of the present application are used to provide a further understanding of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application, and do not constitute an improper limitation of the present application. In the accompanying drawings:

[0039] FIG. 1 is a block diagram of the method flow of the embodiment of the disclosure.

[0040] FIG. 2 is a schematic diagram of the iterative convergence result of the WorldView-3 dataset according to the embodiment of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] It should be noted that embodiments in the application and the features in the embodiments may be combined with each other if there is no conflict. The application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] It should be noted that the steps shown in the flowcharts of the accompanying drawings may be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in an order different from that here.

[0043] In order to solve the problems pointed out in the above technical background, the disclosure proposes a pan-sharpening method based on multimodal texture correction and adaptive edge detail fusion. In order to obtain a texture-corrected image T.sub.C highly correlated and similar to the multispectral (MS) image, a pan-sharpening (A-PNN) fusion method based on a target adaptive convolutional neural network is introduced. By constructing a multimodal texture correction model, intensity, gradient and deep plug-and-play correction constraints based on A-PNN are established between the texture-corrected image and the source image, and an adaptive degradation filter algorithm is proposed to ensure the accuracy of the establishment of these constraints. Since the obtained texture-corrected image may replace the panchromatic (PAN) image, and the MS image also contains part of spatial information, an adaptive edge detail fusion algorithm is proposed to adaptively extract the detail information of the texture-corrected image and the MS image respectively and apply edge protection. Since the MS image has less spatial information, its spatial information is enhanced in proportion and then adaptively fused. The fused spatial information is injected into the upsampled multispectral (UPMS) image to obtain the final HRMS image. A large number of experimental results show that compared with other methods, the algorithm proposed in the disclosure achieves better results in both subjective visual effects and objective evaluation indexes, and maintains high operation efficiency.

[0044] Related work and related technical basis involved in the disclosure are as follow:

Injection Model:

[0045] the injection model is commonly used in pan-sharpening methods. It generates HRMS images by injecting high-spatial-resolution spatial detail information from PAN images into the original UPMS images with high spectral resolution, so as to solve the problem that LRMS images lack a large amount of spatial information. It is assumed that the size of the LRMS image is LWB (that is, lengthwidthnumber of bands), and the size of the PAN image is LW, where L=L/r, W=W/r, and r represents the compression ratio. Then the sizes of the UPMS and HRMS images are LWB. The specific formula of the injection model may be uniformly expressed as:

[00012]$\begin{array}{cc}{M}_{HR}=M_{UP}+G{S}_D;& \left(1\right)\end{array}$

[0046] where M.sub.HR is the HRMS image, M.sub.UP is the UPMS image, G is the injection gain, and S.sub.D is the injected spatial detail information. Methods for extracting S.sub.D may be uniformly divided into CS-based methods and MRA-based methods. For CS-based methods, S.sub.D may be extracted using the following formula:

[00013]$\begin{array}{cc}{S}_D=P_I-I_{UP};& \left(2\right)\end{array}$ [0047] where P.sub.I represents the image obtained by histogram matching between the PAN and the intensity component of the UPMS image (I.sub.UP). Histogram matching ensures that the intensity and contrast of the PAN and LRMS images are within the same grayscale range, ensuring the accuracy of spatial information extraction. The formula of P.sub.I is as follows:

[00014]$\begin{array}{cc}{P}_I=\frac{{}_I}{{}_P}\left(P-{}_P\right)+{}_I;& \left(3\right)\end{array}$ [0048] where P represents the original PAN image, .sub.P and .sub.I represent the average values of P and I.sub.UP images respectively, .sub.P and .sub.I represent the variances of P and I.sub.UP images respectively. I.sub.UP is obtained by linearly weighting each band of M.sub.UP, and its formula is as follows:

[00015]$\begin{array}{cc}{I}_{\mathrm{UP}}\underset{i=1}{\overset{B}{.Math.}}{}_i{M}_{\mathrm{UP}}^i;& \left(4\right)\end{array}$ [0049] where represents the linear weighting coefficient, the superscript or subscript i represents the i-th band of the image, and B represents the total number of bands. For MRA-based methods, S.sub.D may be extracted using the following formula:

[00016]$\begin{array}{cc}\begin{array}{c}{S}_D=P-P_D\\ P_D=H_{LP}P\end{array};& \left(5\right)\end{array}$ [0050] where P.sub.D represents the degraded PAN image, which may be obtained by using a low-pass filter H.sub.LP on the PAN image P, and H.sub.LP has a blurring effect on P.

[0051] However, problems such as inaccurate injected spatial detail information still exist. Since the missing spatial detail information in the LRMS image is generally inferred from the PAN image, inaccurate inference and possible mismatching of spectral information during the fusion process make it impossible to maintain accurate spectral fidelity and spatial fidelity at the same time, which in turn leads to spectral and spatial distortions in the fused image.

Variational Optimization Model:

[0052] variational optimization methods have become popular in recent years, which may ensure that image spectral and spatial information as accurate as possible by establishing mathematical models. The established mathematical model may be regarded as a degradation model, in which the ideal HRMS image after fusion is recovered from the LRMS and PAN images, that is, the ideal HRMS image is an inverse process of degenerating into the source image. Therefore, variational optimization-based methods may retain the spatial and spectral information of LRMS and PAN images through various optimization algorithms, and finally restore the desired ideal HRMS image. To sum up, variational optimization methods generally establish an energy function among LRMS, PAN and the ideal HRMS image E(M.sub.HR), and the methods may be divided into three terms: the first term is the spectral fidelity term f.sub.spectral (M.sub.0, M.sub.HR), the second term is the spatial fidelity term f.sub.spatial (P, M.sub.HR), and the third term is the regularization prior term f.sub.prior (M.sub.HR). The specific formula is as follows:

[00017]$\begin{array}{cc}E\left(M_{\mathrm{HR}}\right)=f_{\mathrm{spectral}}\left(M_0,M_{\mathrm{HR}}\right)+f_{\mathrm{spatial}}\left(P,M_{\mathrm{HR}}\right)+f_{prior}\left(M_{HR}\right);& \left(6\right)\end{array}$ [0053] where M.sub.0 is the LRMS image. M.sub.0 may be obtained by blurring and downsampling the ideal HRMS image H.sub.MR. M.sub.HR may also be obtained by linearly weighted combination to get the PAN image P. Therefore, the energy function in formula (6) may be simplified to the following common form:

[00018]$\begin{array}{cc}E\left(M_{\mathrm{HR}}\right)={}_1.Math.\left({\mathrm{DH}}_{LP}{M}_{HR}-M_0\right).Math.+.Math.P-{\mathrm{CM}}_{HR}.Math.+{}_2{f}_{prior}\left(M_{HR}\right);& \left(7\right)\end{array}$ [0054] where .sub.1 and .sub.2 are penalty parameters, D represents a downsampling matrix, and C represents a linear weighted combination matrix. By optimizing and solving the above formula, M.sub.HR may finally be obtained.

[0055] Although variational optimization methods may retain relatively accurate spectral and spatial information at the same time, they depend on the accuracy of mathematical model establishment. Unreasonable variational optimization models will ignore the correlation and similarity between MS and PAN images, and the obtained spectral and spatial information may not match, which will lead to spectral and spatial distortions in the final HRMS image. In addition, the efficiency of most variational optimization models is relatively low.

[0056] The specific related method flow involved in the disclosure is described as follow: [0057] in order to solve the problems of poor correlation and similarity among LRMS, PAN and HRMS images, and inaccurate spatial information injected into UPMS images, a pan-sharpening method based on multimodal texture correction and adaptive edge detail fusion is proposed. This method may improve the spectral and spatial distortions of HRMS images.

[0058] The input end of the multimodal texture correction model is the intensity component I.sub.0 of the LRMS image, the PAN image and the intensity component of the image fused by A-PNN, and the output end is the texture-corrected image T.sub.C. Intensity constraints between I.sub.0 and T.sub.C images are corrected by establishing intensity correction priors. Gradient constraints among I.sub.0, PAN and T.sub.C images are corrected by establishing gradient correction priors. Intensity gradient constraints between I.sub.net and T.sub.C images are corrected by establishing deep plug-and-play correction priors based on A-PNN. These three correction priors form the basis of the multimodal texture correction model. In addition, an adaptive degradation filter algorithm is proposed, which may be used to obtain an accurate adaptive degradation filter H.sub.A in the intensity correction prior to degrade T.sub.C, so that the correlation and similarity between the degraded T.sub.C and I.sub.0 images are the highest. Finally, the multimodal texture correction model is optimized by alternating direction method of multipliers (ADMM) to obtain the texture-corrected image T.sub.C. Due to the high correlation and similarity between the texture-corrected image T.sub.C and the source images, the T.sub.C maintains the spectral information of the LRMS image unchanged while inheriting gradient information from the PAN image, and the intensity component I.sub.net of the image fused by A-PNN has more image features, which may further maintain the stability of texture information. Therefore, the texture-corrected image T.sub.C may be used to replace the PAN image for subsequent fusion operations.

[0059] After obtaining the texture-corrected image T.sub.C, the texture-corrected image T.sub.C and the multispectral MS image are fused through an adaptive edge detail fusion model to generate the final HRMS image; [0060] in the adaptive edge detail fusion model, since spatial detail information exists not only in the texture-corrected image T.sub.C that replaces the PAN image, but also in the multispectral MS image. Therefore, the detail information of the texture-corrected image T.sub.C is adaptively extracted and edge protection is applied, and the detail information in the UPMS image is extracted by using a Gaussian filter matching the modulation transfer function (MTF) and edge protection is applied. The detail information of the UPMS image with edge protection is enhanced to the same level as the texture-corrected image T.sub.C, and adaptively fused with the detail information of the texture-corrected image T.sub.C with edge protection to obtain spatial information with high correlation and similarity to the source image. The spatial information is injected into the UPMS image in an appropriate proportion to obtain the final HRMS image. The method flow block diagram of the disclosure is shown in FIG. 1. The specific process is shown in the following content.

[0061] Specifically, the multimodal texture correction model mainly includes an intensity correction prior term, a gradient correction prior term and a deep plug-and-play correction prior term based on A-PNN; where the relevant filters in the intensity correction prior term and the gradient correction prior term are determined by an adaptive degradation filter algorithm, and the multimodal texture correction model is optimized and solved by an optimization model algorithm to obtain the final texture-corrected image, and the specific content is as follows.

Intensity Correction Prior Term:

[0062] based on the spectral fidelity term in the variational optimization model in the above technical basis, the LRMS image may be obtained by blurring and downsampling the HRMS image, and the specific formula is as follows:

[00019]$\begin{array}{cc}{f}_{\mathrm{spectral}}^i=\frac{1}{2}{.Math.{\mathrm{DHM}}_{\mathrm{HR}}^i-M_0^i.Math.}_F^2;& \left(8\right)\end{array}$ [0063] where H is generally a Gaussian smoothing filter, and .Math..sub.F represents the Frobenius norm. In order to keep the intrinsic correlation and similarity between bands unchanged, the LRMS and ideal HRMS images of each band are linearly weighted and summed by formula (4) to obtain I.sub.0 and the intensity component I.sub.HR of the ideal HRMS image, and the specific formula is as follows:

[00020]$\begin{array}{cc}{f}_{\mathrm{spectral}1}=\frac{1}{2}{.Math.\mathrm{DH}\underset{i=1}{\overset{B}{.Math.}}{}_i{M}_{\mathrm{HR}}^i-\underset{i=1}{\overset{B}{.Math.}}{}_i{M}_0^i.Math.}_F^2=\frac{1}{2}{.Math.{\mathrm{DHI}}_{\mathrm{HR}}-I_0.Math.}_F^2.& \left(9\right)\end{array}$

[0064] Since I.sub.HR is unknown, it is assumed that T.sub.C is close to I.sub.HR and highly correlated. Therefore, the intensity correction prior term E.sub.intensity is as follows:

[00021]$\begin{array}{cc}{E}_{intensity}=\frac{1}{2}{.Math.\mathrm{DH}{T}_C-I_0.Math.}_F^2.& \left(10\right)\end{array}$

Gradient Correction Prior Term:

[0065] in the intensity correction prior model, T.sub.C maintains the invariance of spectral information, but spatial information is also required to be retained. Based on the spatial fidelity term in the variational optimization model in the above technical basis, the gradient information of the PAN image is retained by establishing a spatial fidelity term, and the specific formula is as follows:

[00022]$\begin{array}{cc}{f}_{\mathrm{spatial}1}=\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2;& \left(11\right)\end{array}$ [0066] where is a penalty parameter and .sup.2 is a Laplacian operator. Since the correction of the gradient information of the PAN image by T.sub.C may lead to deviations in the intensity correction between T.sub.C and I.sub.0 images, it is necessary to establish another spatial fidelity term to keep the intensity correction from deviating during the gradient correction process and further enhance the correlation and similarity between T.sub.C and I.sub.0 images, and the specific formula is as follows:

[00023]$\begin{array}{cc}{f}_{\mathrm{spatial}2}=\frac{}{2}{.Math.{}^2\left(DH{T}_C\right)-{}^2{I}_0.Math.}_F^2;& \left(12\right)\end{array}$ [0067] where is a penalty parameter. To sum up, the gradient correction prior term may be expressed as follows:

[00024]$\begin{array}{cc}{E}_{\mathrm{gradient}}=\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.{}^2\left({\mathrm{DHT}}_C\right)-{}^2{I}_0.Math.}_F^2.& \left(13\right)\end{array}$

Deep Plug-and-Play Correction Prior Term Based on A-PNN:

[0068] in order to generate more texture features, further improve the correlation and similarity between T.sub.C and I.sub.0, PAN images, and retain more spectral and spatial information, the PAN and UPMS images are fused by A-PNN to obtain an HRMS image, denoted as MS.sub.net. The intensity component I.sub.net of MS.sub.net is obtained by linearly weighting MS.sub.net through formula (4). A-PNN is a technology well known to those skilled in the art, which is a pan-sharpening method based on a target adaptive convolutional neural network. Based on the spectral fidelity term in the variational optimization model in the above technical basis, the intensity information of T.sub.C is corrected by establishing a spectral fidelity term between T.sub.C and I.sub.net, and the specific formula is as follows:

[00025]$\begin{array}{cc}{f}_{\mathrm{spectral}2}=\frac{}{2}{.Math.T_C-I_{net}.Math.}_F^2;& \left(14\right)\end{array}$ [0069] where is a penalty parameter. The gradient information of T.sub.C is corrected by establishing a spatial fidelity term between T.sub.C and I.sub.net, and the specific formula is as follows:

[00026]$\begin{array}{cc}{f}_{\mathrm{spatial}3}=\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2;& \left(15\right)\end{array}$ [0070] where is a penalty parameter. To sum up, the deep plug-and-play correction prior based on A-PNN may be expressed as follows:

[00027]$\begin{array}{cc}{E}_{\mathrm{DPP}}=\frac{}{2}{.Math.T_C-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2.& \left(16\right)\end{array}$

Multimodal Texture Correction Model:

[0071] in order to ensure the sparsity of the output texture map and reduce artifacts, in addition to combining the above intensity correction prior term, gradient correction prior term and deep plug-and-play correction prior term based on A-PNN, a total variation regularization term (TV) is also used, where

[00028]$\mathrm{TV}={.Math.{}^2{T}_C.Math.}_1.$ For this reason, the disclosure proposes a multimodal texture correction model, and the specific formula is as follows:

[00029]$\begin{array}{cc}{T}_C=\underset{T_C}{\arg \min }\frac{1}{2}{.Math.{\mathrm{DHT}}_C-I_0.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.{}^2\left({\mathrm{DHT}}_C\right)-{}^2{I}_0.Math.}_F^2+\frac{}{2}{.Math.T_C-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2+{.Math.{}^2{T}_C.Math.}_1;& \left(17\right)\end{array}$

[0072] where is a penalty parameter.

Adaptive Degradation Filter Algorithm:

[0073] in the model shown in formula (17), all are determined except the Gaussian filter H and the texture-corrected image T.sub.C. The texture-corrected image T.sub.C may be determined by algorithm 2, while H is difficult to determine. Therefore, the disclosure proposes an adaptive degradation filter algorithm, which uses a Gaussian filter as the degradation filter, defined as H.sub.A, and may be determined by the following formula:

[00030]$\begin{array}{cc}{H}_A=\underset{H_A}{\arg \min }\frac{1}{2}{.Math.{\mathrm{DH}}_A{T}_C-I_0.Math.}_F^2.& \left(18\right)\end{array}$

[0074] It may be known from the above formula that when the difference between the texture-corrected image DH.sub.AT.sub.C processed by downsampling and the degradation filter and the intensity component I.sub.0 of the LRMS image is the smallest, that is, when the correlation and similarity between the two reach the highest, H.sub.A at this time is the best degradation filter. Therefore, the adaptive degradation filter algorithm comprehensively considers the correlation and similarity between the two, which are measured by the correlation coefficient (CC) and structural similarity index measure (SSIM) respectively, and finally adaptively determines the best degradation filter. When the filter is processed in the spatial domain of the image, the convolution operation will greatly increase the computational complexity. When processing in the image frequency domain, the convolution operation is converted into an inner product operation, which will greatly reduce the computational complexity. Therefore, H.sub.A is selected to be calculated in the frequency domain, and the frequency domain expression of H.sub.A is:

[00031]$\begin{array}{cc}{H}_A\left(u,v\right)=e^{-\frac{D_C^2\left(u,v\right)}{2{}^2}};& \left(19\right)\end{array}$ [0075] where D.sub.C (u, v) represents the distance from the point (u, v) to the center of the frequency domain, and represents the standard deviation. After H.sub.A is converted to the frequency domain, T.sub.C also needs to be calculated in the frequency domain. Therefore, the fast Fourier transform (FFT) is used to convert T.sub.C into the frequency domain, and the inverse fast Fourier transform (IFFT) is used to convert H.sub.AT.sub.C back to the spatial domain, which is convenient for the subsequent correlation and similarity operations between DH.sub.AT.sub.C and I.sub.0. The specific formula is as follows:

[00032]$\begin{array}{cc}{\mathrm{DH}}_A{T}_C={\mathrm{DF}}^{-1}\left(H_A\left(u,v\right)F\left(T_C\right)\right);& \left(20\right)\end{array}$ [0076] where F(.Math.) represents the FFT operation, and F.sup.1(.Math.) represents the IFFT operation. To sum up, the key to determining H.sub.A is to determine the unknown parameter . Therefore, the best may be found by correcting the correlation and similarity between DH.sub.AT.sub.C and I.sub.0. The correlation is measured by the CC index, denoted as (DH.sub.AT.sub.C, I.sub.0) and the similarity is measured by the SSIM index, denoted as S (DH.sub.AT.sub.C, I.sub.0). The average rule is used to comprehensively consider these two indexes, and iterative processing is performed with different values, and the maximum value is taken from the final results. At this time, is the best value, denoted as .sub.best.

[0077] The specific formula is as follows:

[00033]$\begin{array}{cc}{}_{\mathrm{best}}=\underset{}{\arg \max }\frac{\left({\mathrm{DH}}_A{T}_C,I_0\right)+S\left({\mathrm{DH}}_A{T}_C,I_0\right)}{2}.& \left(21\right)\end{array}$

[0078] To sum up, the overall process of the adaptive degradation filter algorithm is shown in Algorithm 1.

Algorithm 1:

Algorithm 1: Adaptive Degradation Filter Algorithm

[0079] Input: texture-corrected image T.sub.C, intensity component I.sub.0 of the LRMS image, [0080] initializing: setting .sup.(0)=1, step size s=0.5, iteration step k=0, [0081] converting H.sub.A to the frequency domain by formula (19), [0082] calculating (DH.sub.AT.sub.C, I.sub.0) by formula (20), [0083] calculating .sup.(0) (DH.sub.AT.sub.C, I.sub.0) and S.sup.(0) (DH.sub.AT.sub.C, I.sub.0). [0084] Circulating: .sup.(k+1)=.sup.(k)+s, k=k+1, [0085] optimizing (DH.sub.AT.sub.C).sup.(K+1) by formula (20), [0086] optimizing .sup.(k+1) (DH.sub.AT.sub.C, I.sub.0) and S (k+1) (DH.sub.AT.sub.C, I.sub.0), [0087] calculating

[00034]$\begin{array}{cc}{}^{\left(k+1\right)}=\frac{{}^{\left(k+1\right)}\left({\mathrm{DH}}_A{T}_C,I_0\right)+S^{\left(k+1\right)}\left({\mathrm{DH}}_A{T}_C,I_0\right)}{2},& \mathrm{by}\mathrm{formula}\left(21\right)\end{array}$ [0088] until .sup.(k+1)<.sup.(k) is satisfied, break the loop. [0089] Output: .sub.best=.sup.(k), adaptive degradation filter H.sub.A.

Optimization Model Algorithm:

[0090] the optimization model algorithm uses ADMM for optimization, which is an optimization method that decomposes the original problem into multiple easy-to-handle subproblems. To facilitate the optimization process, different auxiliary variables A=H.sub.AT.sub.C, C=.sup.2T, namely B=DA, is introduced, then the model of formula (17) may be expressed as:

[00035]$\begin{array}{cc}\underset{A,B,C}{\min }\frac{1}{2}{.Math.B-I_0.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.{}^{2}B-{}^2{I}_0.Math.}_F^2+\frac{}{2}{.Math.T_C-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2+{.Math.C.Math.}_{1}s.t.A=H_A{T}_C,B+\mathrm{DA},C={}^2{T}_C.& \left(22\right)\end{array}$

[0091] The augmented Lagrangian function of the above formula may be expressed as:

[00036]$\begin{array}{cc}{E}_L\left(A,B,T_C,H_A,C,{}_1,{}_2,{}_3\right)=\frac{1}{2}{.Math.B-I_0.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.{}^{2}B-{}^2{I}_0.Math.}_F^2+\frac{}{2}{.Math.T_C-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C-{}^2{I}_{\mathrm{net}}.Math.}_F^2+{.Math.C.Math.}_1+{}_1^T\left(A-H_A{T}_C\right)+{}_2^T\left(B-\mathrm{DA}\right)+{}_3^T\left(C-{}^2{T}_C\right)+\frac{{}_1}{2}{.Math.A-H_A{T}_C.Math.}_F^2+\frac{{}_2}{2}{.Math.B-\mathrm{DA}.Math.}_F^2+\frac{{}_3}{2}{.Math.C-{}^2{T}_C.Math.}_F^2;& \left(23\right)\end{array}$ [0092] where .sub.1, .sub.2, .sub.3 are different Lagrange multipliers, and .sub.1, .sub.2, .sub.3 are different penalty parameters. To minimize the energy function of the above formula, A.sup.(k+1), B.sup.(k+1), TC.sup.(k+1), HA.sup.(k+1), C.sup.(k+1), .sub.1.sup.(k+1), .sub.2.sup.(k+1) and .sub.3.sup.(k+1) are optimized iteratively to finally obtain T.sub.C, where k is the number of iterations, and different parameters with the superscript k+1 represent the corresponding parameters in the (k+1)-th iteration process. The specific optimization process is as follows.

(1) Optimizing A.SUP.(k+1)

[0093] Fixing other variables, the subproblem of A (k+1) is as follows:

[00037]$\begin{array}{cc}{A}^{\left(k+1\right)}=\underset{A}{\arg \min }{\left({}_1^{\left(k\right)}\right)}^T\left(A^{\left(k\right)}-H_A^{\left(k\right)}{T}_C^{\left(k\right)}\right)+{\left({}_2^{\left(k\right)}\right)}^T\left(B^{\left(k\right)}-{\mathrm{DA}}^{\left(k\right)}\right)+\frac{{}_1}{2}{.Math.A^{\left(k\right)}-H_A^{\left(k\right)}{T}_C^{\left(k\right)}.Math.}_F^2+\frac{{}_2}{2}{.Math.B^{\left(k\right)}-{\mathrm{DA}}^{\left(k\right)}.Math.}_F^2.& \left(24\right)\end{array}$

[0094] Setting the derivative of A.sup.(k+1) to 0, that is, E.sub.L/A.sup.(k+1)=0, then A.sup.(k+1) is obtained by the following formula:

[00038]$\begin{array}{cc}{A}^{\left(k+1\right)}=\frac{-{}_1^{\left(k\right)}+D^T{}_2^{\left(k\right)}+{}_1{H}_A^{\left(k\right)}{T}_C^{\left(k\right)}+{}_2{D}^T{B}^{\left(k\right)}}{{}_{1}U+{}_2{D}^{T}D};& \left(25\right)\end{array}$ [0095] where U represents the identity matrix, and the superscript T represents the transpose operator.

(2) Optimizing B.SUP.(k+1)

[0096] Fixing other variables, the subproblem of B.sup.(k+1) is as follows:

[00039]$\begin{array}{cc}{B}^{\left(k+1\right)}=\underset{B}{\arg \min }\frac{1}{2}{.Math.B^{\left(k\right)}-I_0.Math.}_F^2+\frac{}{2}{.Math.{}^2{B}^{\left(k\right)}-{}^2{I}_0.Math.}_F^2+{\left({}_2^{\left(k\right)}\right)}^T\left(B^{\left(k\right)}-{\mathrm{DA}}^{\left(k+1\right)}\right)+\frac{{}_2}{2}{.Math.B^{\left(k\right)}-{\mathrm{DA}}^{\left(k+1\right)}.Math.}_F^2.& \left(26\right)\end{array}$

[0097] Setting the derivative of B.sup.(k+1) to 0, that is, E.sub.L/B.sup.(k+1)=0. However, due to the existence of the Laplacian operator, the computational complexity increases in the solution process. To improve computational efficiency, FFT and IFFT are used for fast calculation in the frequency domain, and then converted back to the spatial domain. Therefore, after A.sup.(k+1) is optimized, B.sup.(k+1) may be obtained by the following formula:

[00040]$\begin{array}{cc}{B}^{\left(k+1\right)}=F^{-1}\left(\frac{F\left(I_0+{\left({}^2\right)}^T{}^2{I}_0-{}_2^{\left(k\right)}+{}_2{\mathrm{DA}}^{\left(k+1\right)}\right)}{F\left(\left(1+{}_2\right)U+{\left({}^2\right)}^T{}^2\right)}\right).& \left(27\right)\end{array}$

(3) Optimizing T.sub.C.sup.(k+1)

[0098] Fixing other variables, the subproblem of T.sub.C.sup.(k+1) is as follows:

[00041]$\begin{array}{cc}{T}_C^{\left(k+1\right)}=\underset{T_C}{\arg \min }\frac{}{2}{.Math.{}^2{T}_C^{\left(k\right)}-{}^{2}P.Math.}_F^2+\frac{}{2}{.Math.T_C^{\left(k\right)}-I_{\mathrm{net}}.Math.}_F^2+\frac{}{2}{.Math.{}^2{T}_C^{\left(k\right)}-{}^2{I}_{\mathrm{net}}.Math.}_F^2+{\left({}_1^{\left(k\right)}\right)}^T\left(A^{\left(k+1\right)}-H_A^{\left(k\right)}{T}_C^{\left(k\right)}\right)+{\left({}_2^{\left(k\right)}\right)}^T\left(C^{\left(k\right)}-{}^2{T}_C^{\left(k\right)}\right)+\frac{{}_1}{2}{.Math.A^{\left(k+1\right)}-H_A^{\left(k\right)}{T}_C^{\left(k\right)}.Math.}_F^2+\frac{{}_3}{2}{.Math.C^{\left(k\right)}-{}^2{T}_C^{\left(k\right)}.Math.}_F^2.& \left(28\right)\end{array}$

[0099] Setting the derivative of T.sub.C.sup.(k+1) to 0, that is,

[00042]$E_L/T_C^{\left(k+1\right)}=0.$

Due to the existence of the Laplacian operator, FFT and IFFT are also used for solution. Therefore, after A.sup.(k+1) is optimized, T.sub.C.sup.(k+1) may be obtained by the following formula:

[00043]$\begin{array}{cc}{T}_C^{\left(k+1\right)}=F^{-1}\left(\frac{F\left(a\right)}{F\left(b\right)}\right)& \left(29\right)\end{array}$$a={\left({}^2\right)}^r{}^{2}P+I_{\mathrm{net}}+{\left({}^2\right)}^r{}^2{I}_{\mathrm{net}}+{\left(H_A^{\left(k\right)}\right)}^T{}_1^{\left(k\right)}+{\left({}^2\right)}^T{}_3^{\left(k\right)}+{{}_1\left(H_A^{\left(k\right)}\right)}^T{A}^{\left(k+1\right)}+{{}_3\left({}^2\right)}^T{C}^{\left(k\right)}$$b=\left(++{}_3\right){\left({}^2\right)}^T{}^2++{{}_1\left(H_A^{\left(k\right)}\right)}^T{H}_A^{\left(k\right)}.$

(4) Optimizing C.SUP.(k+1)

[0100] Fixing other variables, the subproblem of C.sup.(k+1) is as follows:

[00044]$\begin{array}{cc}\begin{array}{c}{C}^{\left(k+1\right)}=\underset{C}{\arg \min }{.Math.C^{\left(k\right)}.Math.}_1+{\left({}_3^{\left(k\right)}\right)}^T\left(C^{\left(k\right)}-{}^2{T}_C^{\left(k+1\right)}\right)+\frac{{}_3}{2}{.Math.C^{\left(k\right)}-{}^2{T}_C^{\left(k+1\right)}.Math.}_F^2\\ =\underset{C}{\arg \min }\frac{}{{}_3}{.Math.C^{\left(k\right)}.Math.}_1+\frac{1}{2}{.Math.C^{\left(k\right)}-\left({}^2{T}_C^{\left(k+1\right)}-\frac{{}_3^{\left(k\right)}}{{}_3}\right).Math.}_F^2\end{array}.& \left(30\right)\end{array}$

[0101] It is further simplified by using the SoftThresholding formula to obtain the following formula:

[00045]$\begin{array}{cc}\begin{array}{c}{C}^{\left(k+1\right)}=S_T\left({}^2{T}_C^{\left(k+1\right)}-\frac{{}_3^{\left(k\right)}}{{}_3},\frac{}{{}_3}\right)\\ =\mathrm{sgn}\left({}^2{T}_C^{\left(k+1\right)}-\frac{{}_3^{\left(k\right)}}{{}_3}\right)\max \left(.Math.{}^2{T}_C^{\left(k+1\right)}-\frac{{}_3^{\left(k\right)}}{{}_3}.Math.-\frac{}{{}_3},0\right)\end{array};& \left(31\right)\end{array}$ [0102] where sgn(.Math.) is the sign function, and max(.Math.) is the maximum function.
(5) Optimizing Lagrange multipliers .sub.1.sup.(k+1), .sub.2.sup.(k+1) and .sub.3.sup.(k+1)

[0103] Fixing other variables, the subproblem of .sub.1.sup.(k+1), .sub.2.sup.(k+1) and .sub.3.sup.(k+1) are as follows:

[00046]$\begin{array}{cc}\{\begin{array}{c}{}_1^{\left(k+1\right)}={}_1^{\left(k\right)}+{}^{\left(k+1\right)}\left(A^{\left(k+1\right)}-H_A^{\left(k+1\right)}{T}_C^{\left(k+1\right)}\right)\\ {}_2^{\left(k+1\right)}={}_2^{\left(k\right)}+{}^{\left(k+1\right)}\left(B^{\left(k+1\right)}-D{A}^{\left(k+1\right)}\right)\\ {}_3^{\left(k+1\right)}={}_3^{\left(k\right)}+{}^{\left(k+1\right)}\left(C^{\left(k+1\right)}-{}^2{T}_C^{\left(k+1\right)}\right)\end{array};& \left(32\right)\end{array}$ [0104] where is the step size required for gradient ascent, and the formula is as follows:

[00047]$\begin{array}{cc}{}^{\left(k+1\right)}={}^{\left(k\right)};& \left(33\right)\end{array}$ [0105] where is a penalty parameter, and generally >1 may accelerate the convergence speed. To sum up, the overall optimization process of the multimodal texture correction model is shown in Algorithm 2. where, in the iterative process, H.sub.A.sup.(T+1) is optimized by Algorithm 1, and when the relative change (RelCha) of T.sub.C in two consecutive iterations is less than the tolerance deviation , the iterative process is exited, and the T.sub.C image is finally obtained. The relative change discrimination formula is as follows:

[00048]$\begin{array}{cc}\mathrm{RelCha}=\frac{{.Math.T_C^{\left(k+1\right)}-T_C^{\left(k\right)}.Math.}_F}{{.Math.T_C^{\left(k\right)}.Math.}_F}<.& \left(34\right)\end{array}$

[0106] With the iteration, the relative change value RelCha gradually becomes smaller. Therefore, it is necessary to determine the parameter , which is slightly larger than RelCha, to balance the efficiency and accuracy of the model. For example, FIG. 2 shows the iterative convergence result of the test image in the WorldView-3 dataset. When the number of iterations reaches about 15, RelCha tends to converge and is close to 110.sup.4, that is, may be assigned as 110.sup.4.

Algorithm 2:

Algorithm 2: Optimization Algorithm of Multimodal Texture Correction Model

[0107] Input: panchromatic image P, intensity component I.sub.0 of the LRMS image, [0108] initializing: k=0, T.sub.C.sup.(0)=P, H.sub.A.sup.(0), .sub.1.sup.(0)=.sub.2.sup.(0)=.sub.3.sup.(0)=U, .sup.(0)=1 and =10.1 are obtained by the initialization of Algorithm 1. [0109] While RelCha>do

Circulating:

[0110] optimizing A.sup.(k+1) by the formula (25), [0111] optimizing B.sup.(k+1) by the formula (27), [0112] optimizing T.sub.C.sup.(k+1) by the formula (29), [0113] optimizing H.sub.A.sup.(k+1) by the Algorithm 1, [0114] optimizing C.sup.(k+1) by the formula (31), [0115] optimizing .sub.1.sup.(k+1), .sub.2.sup.(k+1) and .sub.3.sup.(k+1) by the formula (32),

[00049]${}^{\left(k+1\right)}={}^{\left(k\right)},k=k+1.$ [0116] Until RelCha is satisfied, break the loop. [0117] Output: texture-corrected image T.sub.C.

Adaptive Edge Detail Fusion Model:

[0118] adaptively extract T.sub.C image details and apply edge protection: [0119] after obtaining T.sub.C by Algorithm 2, the following formula is used to extract image details D.sub.TC.

[00050]$\begin{array}{cc}{D}_{T_C}=T_C-T_{\mathrm{CL}};& \left(35\right)\end{array}$ [0120] where T.sub.CL is the low-resolution version of the T.sub.C image. To extract details from T.sub.C more accurately, it may be known from formulas (2) and (5) that T.sub.CL may be obtained by two methods. The first method is to obtain the degraded image T.sub.CD of T.sub.C by applying Algorithm 1, which is similar to extracting details by MRA-based methods and better retains spectral information. The second method is to obtain I.sub.UP by formula (4), which is similar to extracting details by CS-based methods and better retains spatial information. Therefore, considering the advantages of these two methods comprehensively, D.sub.TC is adaptively extracted, and its algorithm design is as follows:

[00051]$\begin{array}{cc}\begin{array}{cc}{T}_{\mathrm{CL}}={}_1{I}_{\mathrm{UP}}+\left(1-{}_1\right)T_{\mathrm{CD}}& s.t.0<{}_1<1;\end{array}& \left(36\right)\end{array}$ [0121] where .sub.1 represents the weight coefficient to be determined. Since the accuracy of detail extraction by the two methods is affected by the correlation and similarity between source images, the influence coefficient of I.sub.UP may be set as .sub.1, and the influence coefficient of T.sub.CD may be set as X2, and their formulas are as follows:

[00052]$\begin{array}{cc}{x}_1=\frac{\left(T_C,I_{\mathrm{UP}}\right)+S\left(T_C,I_{\mathrm{UP}}\right)}{2}& \left(37\right)\end{array}$$x_2=\frac{\left(T_C,T_{\mathrm{CD}}\right)+S\left(T_C,T_{\mathrm{CD}}\right)}{2}.$

[0122] Since x.sub.1 and x.sub.2 do not satisfy the normalization constraint of .sub.1, .sub.1 is required to be positively correlated with x.sub.1 and x.sub.2 and within a reasonable range, so .sub.1 may be obtained by the following formula:

[00053]$\begin{array}{cc}\begin{array}{cc}{}_1=\sqrt{1-e^{-x_3}}& s.t.x_3=\frac{x_1}{x_1+x_2}\end{array}.& \left(38\right)\end{array}$

[0123] Substitute .sub.1 in the above formula into formula (36) to obtain T.sub.CL, and then substitute it into formula (35) to finally obtain D.sub.TC, completing the operation of adaptively extracting T.sub.C image details. To retain edge information during detail extraction, the following edge detection matrix formula E.sub.TC is used to extract edges:

[00054]$\begin{array}{cc}{E}_{T_C}=e^{-\frac{}{{.Math.T.Math.}^4+}};& \left(39\right)\end{array}$ [0124] where is the gradient operator, and and are modulation coefficients. Generally, let =110.sup.9, =110.sup.10. Therefore, the T.sub.C image detail information with edge protection, that is, the first image detail F.sub.1, is as follows:

[00055]$\begin{array}{cc}{F}_1=D_{T_C}{E}_{T_C}.& \left(40\right)\end{array}$

Extracting UPMS Image Details and Applying Edge Protection:

[0125] the following formula is used to extract the detail information D.sub.M of the UPMS image.

[00056]$\begin{array}{cc}{D}_M^i=M_{\mathrm{UP}}^i-M_{\mathrm{UPL}}^i;& \left(41\right)\end{array}$ [0126] where M.sub.UPL represents the low-resolution version of the UPMS image. Since M.sub.UPL is unknown, the MTF obtained from the MS sensor is introduced as an important index for extracting details of the UPMS image. Therefore, a Gaussian filter H.sub.MG matched with MTF is used to degrade the UPMS image to obtain the low-resolution version of the UPMS image, and the specific process is as follows:

[00057]$\begin{array}{cc}{M}_{\mathrm{UPL}}^i=H_{\mathrm{MG}}{M}_{\mathrm{UP}}^i;& \left(42\right)\end{array}$

[0127] The above formula into formula (41) is substituted to obtain the detail information of the UPMS image. At this time, it is necessary to use the edge detection matrix formula E.sub.M to perform edge protection on D.sub.M:

[00058]$\begin{array}{cc}{E}_M^i=e^{-\frac{}{{.Math.M_{\mathrm{UP}}^b.Math.}^4+}}.& \left(43\right)\end{array}$

[0128] Therefore, the detail information of the UPMS image with edge protection, that is, the second image detail F.sub.2, is as follows:

[00059]$\begin{array}{cc}{F}_2^i=D_M^i{E}_M^i.& \left(44\right)\end{array}$

Adaptive Edge Detail Fusion Process:

[0129] after extracting the detail information with edge protection from T.sub.C and UPMS images respectively, F.sub.1 and F.sub.2 may be fused. However, since the spatial resolution of the UPMS image is lower than that of T.sub.C, F.sub.2 contains less detail information than F.sub.1, and directly fusing them may lose detail information. To avoid this, the information of F.sub.2 is enhanced to the same level as F.sub.1 before fusion. The specific formula is as follows:

[00060]$\begin{array}{cc}{}_i=\underset{{}_i}{\arg \min }\frac{1}{2}{.Math.F_1-{}_i{F}_2^i.Math.}_F^2;& \left(45\right)\end{array}$ [0130] where is the scale factor. The linear regression model method is used to solve . Therefore, the spatial information F.sub.3 enhanced by is expressed as:

[00061]$\begin{array}{cc}{F}_3^i={}_i{F}_2^i.& \left(46\right)\end{array}$

[0131] At this time, F.sub.1 and F.sub.3 may be adaptively fused to obtain detail information F, and the specific algorithm is as follows:

[00062]$\begin{array}{cc}{F}_i={}_2{F}_1+\left(1-{}_2\right)F_3^i;& \left(47\right)\end{array}$ [0132] where .sub.2 is a weight coefficient. The weight distribution of detail information is affected by the correlation and similarity between the source images, that is, T.sub.C and UPMS. Therefore, formula (37) may be used to set the relationship x.sub.1 between T.sub.C and I.sub.UP, and at the same time, ensure that .sub.2 is within a reasonable range, and x.sub.1 is positively correlated with .sub.2. The specific formula is as follows:

[00063]$\begin{array}{cc}{}_2=\sqrt{1-e^{-x_1}}.& \left(48\right)\end{array}$ [0133] .sub.2 in the above formula into formula (47) is substituted to obtain the final detail information.

Final Injection of Spatial Edge Detail Information:

[0134] the detail information F in formula (47) is substituted into the following injection model to obtain the final HRMS image:

[00064]$\begin{array}{cc}{M}_{\mathrm{HR}}^i=M_{\mathrm{UP}}^i+g_i\frac{M_{\mathrm{UP}}^i}{\frac{1}{B}\underset{i=1}{\overset{B}{.Math.}}{M}_{\mathrm{UP}}^i}{F}_i;& \left(49\right)\end{array}$ [0135] where g represents the scale factor of injected details, which may be adaptively determined by the following formula:

[00065]$\begin{array}{cc}{g}_i=\frac{{}^2\left(T_C\right)+\mathrm{cov}\left(T_C,M_{\mathrm{UP}}^i\right)}{{}^2\left(T_C\right)};& \left(50\right)\end{array}$ [0136] where cov(.Math.) is a covariance function, and .sup.2 is a variance function.

Experimental Design and Conclusion for the Above Technical Scheme:

[0137] to illustrate the performance advantages and effectiveness of the method proposed in the disclosure, the method (Proposed) is compared with 8 methods including GSA, NIHS, BDSD-PC, SFIM, ATWT-W3, DMPIF, CDIF and A-PNN, and a large number of experiments are carried out using 3 datasets including QuickBird, WorldView-2 and WorldView-3. Each pair of images in each dataset includes one MS image and one PAN image. Among them, in the QuickBird dataset, the number of bands of the MS image is 4; while in the WorldView-2 and WorldView-3 datasets, the number of bands of the MS image is 8. All PAN images in the datasets contain only 1 band.

[0138] According to the Wald protocol, the original MS image is used as the reference image, that is, the ground truth (GT) image in this experiment. At this time, it is necessary to perform 4 downsampling degradation on the original MS and PAN images respectively. The degraded images may be used as the downscaled source images. The algorithm proposed in the disclosure is used to fuse the source images, and the fused image is compared with the GT image. The smaller the gap, the better the effect. Therefore, in this experiment, the size of each band of the GT image is cropped to 256*256, then the size of each band of the MS image is cropped to 6464, and the size of the PAN image is cropped to 256*256.

[0139] The specific information of these 3 datasets is summarized in this experiment, as shown in Table 1, which is the detailed information of the datasets used in this experiment.

TABLE-US-00001 TABLE 1 Resolution Satellite MS bands Sensor Size (m) QuickBird Blue (B), Green (G), Red MS 64 64 4 2.4 (R) and Near-infrared (NIR) PAN 256 256 0.61 WorldView- Coastal blue, B, G, R, Red MS 64 64 8 2 2 edge, NIR 1 and NIR 2 PAN 256 256 0.5 WorldView- MS 64 64 8 1.24 3 PAN 256 256 0.31

[0140] To evaluate and compare the image quality of different methods, a combination of subjective and objective evaluation criteria is adopted. 6 commonly used objective evaluation indexes are used for objective evaluation. Among them, the Q2n index (Q4 for 4-band datasets and Q8 for 8-band datasets) is selected to evaluate the spatial and spectral quality of images, the peak signal-to-noise ratio (PSNR) is used to measure the error degree between the reconstructed image and the reference image, the universal image quality index (UIQI) is used to more comprehensively evaluate the quality difference and similarity between the fused image and the reference image, the relative average spectral error (RASE) is used to evaluate the average spectral difference before and after image fusion, the overall dimensionless relative global error (ERGAS) is used to represent the distortion degree of image spatial and spectral information, and the spectral correlation coefficient (SCC) is used to measure the ability to retain image spectral information. For subjective evaluation, the fused MS images are visualized, and three bands of red (R), green (G) and blue (B) are extracted to display true-color fused images, which may more intuitively reflect the quality difference of images. Among the above evaluation indexes, the ideal values of Q2n, UIQI and SCC are 1, and the ideal value of PSNR is +, while RASE and ERGAS are ideally 0. All experiments in this section are run on a PC with an Inter Core i7-12700 CPU, a base speed of 2.10 GHz and a memory of 32 GB, and the experimental platform is MATLAB R2021b.

Comparative Experiments

QuickBird Dataset:

[0141] in the subjective evaluation of the QuickBird dataset, the subjective evaluation fusion results of the method proposed in the disclosure and various comparison methods are given, with the GT image as the reference image. After enlarging the local fusion results, it is able to be seen from the local enlargement that the GSA method shows too much detail information on the roof of the house. The images of BDSD-PC and ATWT-M3 methods are relatively blurred and dark. Although the SFIM method retains edge information well in some areas, it has the problem of dark brightness. The CDIF method maintains edge information well but produces artifacts. Although the DMPIF and A-PNN methods retain spatial information well, their edges produce redundant color information and suffer from serious spectral distortion. The result of the method proposed in the disclosure is the closest to the GT image, and retains spatial and spectral information well. The objective evaluation fusion results are shown in Table 2, which is the objective evaluation fusion result of the downscaled image in the QuickBird dataset, with the ideal values marked in brackets and the optimal results marked in bold black. It may be seen that compared with the other 8 methods, the method proposed in the disclosure achieves the optimal results in all evaluation indexes, and the time taken by this method is short.

TABLE-US-00002 TABLE 2 Fusion method Q4 PSNR UIQI RASE ERGAS SCC Time (s) GSA 0.7204 28.0864 0.8680 45.9107 11.9279 0.8384 0.09 NIHS 0.7359 30.3936 0.8389 37.2876 9.2502 0.7884 0.02 BDSD-PC 0.7787 31.0244 0.8727 34.3589 8.8712 0.8241 0.11 SFIM 0.8228 31.9209 0.8953 31.4609 7.7403 0.8574 0.01 ATWT-M3 0.7488 30.3354 0.8406 37.5634 9.2636 0.8173 0.12 DMPIF 0.6629 30.2939 0.8904 36.4980 9.3122 0.8485 4.14 CDIF 0.8426 32.0266 0.9133 31.0258 7.5931 0.7707 32.31 A-PNN 0.8315 31.5654 0.9071 32.5016 8.0506 0.7775 0.24 Proposed 0.8579 32.5524 0.9272 28.5341 7.0273 0.8595 0.66

WorldView-2 Dataset:

[0142] in the subjective evaluation of fusion results of various comparison methods in the WorldView-2 dataset. After enlarging the local fusion results, it is able to be seen from the enlarged local area that the image definition of GSA, BDSD-PC and ATWT-M3 methods is poor, resulting in serious spatial distortion, and the color is dark. In the NIHS method, some areas have the problem of excessive injection of spatial information. Compared with the GT image, the SFIM method still has a certain gap in spatial information. The CDIF method has serious problems of image spatial distortion and spectral distortion. The image definition of the DMPIF method has a gap compared with the GT image, and the image has serious artifacts. In the A-PNN method, the spectrum of some areas is distorted, and the retention of spatial information is poor. The method proposed in the disclosure is the closest to the GT image, and its visual effect is better than other comparison methods. The objective evaluation fusion results are shown in Table 3, which is the objective evaluation fusion result of the downscaled image in the WorldView-2 dataset. Obviously, compared with the other 8 methods, the method proposed in the disclosure achieves the best results in all evaluation indexes, and the running time is also short.

WorldView-3 Dataset:

[0143] in the subjective evaluation of fusion results of various methods in the WorldView-3 dataset. After enlarging the local fusion results, it is able to be seen from the enlarged local area that compared with the GT image, the roof of the house in the GSA method has a darker color. The image of the NIHS method produces certain artifacts, which affects the quality of image spatial information. The images of BDSD-PC, SFIM and ATWT-M3 methods are relatively blurred. Although the CDIF method retains spectral information well, its detail information retention is poor, resulting in serious spatial distortion. The color of DMPIF and A-PNN methods changes greatly, resulting in serious spectral distortion. The method proposed in the disclosure is the closest to the GT image, and achieves the best subjective visual effect. The objective evaluation fusion results are shown in Table 4, which is the objective evaluation fusion result of the downscaled image in the WorldView-3 dataset. It may be seen that the method of the disclosure achieves the best results in all evaluation indexes, and the running time is also short.

TABLE-US-00003 TABLE 3 Fusion method Q8 PSNR UIQI RASE ERGAS SCC Time (s) GSA 0.7415 23.0250 0.8260 28.4076 6.9257 0.8930 0.04 NIHS 0.8718 26.5331 0.9432 19.2262 4.7072 0.8983 0.01 BDSD-PC 0.8484 25.5758 0.9340 21.0005 5.3739 0.8675 0.10 SFIM 0.8924 26.9918 0.9521 18.0386 4.4243 0.9111 0.01 ATWT-M3 0.8262 25.1100 0.9234 22.9734 5.5593 0.8554 0.25 DMPIF 0.8910 27.1957 0.9575 17.0660 4.2016 0.9054 4.47 CDIF 0.8407 24.9159 0.9321 22.7995 5.5670 0.6384 32.67 A-PNN 0.9149 27.7784 0.9617 16.2140 4.0000 0.9143 0.19 Proposed 0.9483 29.3102 0.9732 13.2903 3.3109 0.9412 0.67

WorldView-3 Dataset:

[0144] in the subjective evaluation of the fusion results of various methods on the WorldView-3 dataset. After enlarging the local fusion results, it is able to be seen from the enlarged local area that, compared with the GT image, the color of the roof in the GSA method is darker. The image processed by the NIHS method has certain artifacts, which affects the quality of the spatial information of the image. The images processed by the BDSD-PC, SFIM and ATWT-M3 methods are relatively blurred. Although the CDIF method retains spectral information well, it retains detail information poorly and causes serious spatial distortion. The DMPIF and A-PNN methods result in significant color changes and serious spectral distortion. The method proposed in the present disclosure is the closest to the GT image and achieves the best subjective visual effect. The objective evaluation fusion results are shown in Table 4, which presents the objective evaluation fusion results of the downscaled images in the WorldView-3 dataset. It may be seen that the method of the present disclosure yields the best results in all evaluation indexes and has a relatively short running time.

TABLE-US-00004 TABLE 4 Fusion method Q8 PSNR UIQI RASE ERGAS SCC Time (s) GSE 0.8283 29.9868 0.8908 17.1739 4.0152 0.9135 0.04 NIHS 0.7839 29.8210 0.8978 17.8321 4.1553 0.8691 0.01 BDSD-PC 0.8185 30.3303 0.9203 16.1767 3.9888 0.8998 0.10 SFIM 0.8700 31.3094 0.9322 14.8694 3.4633 0.9081 0.02 ATWT-M3 0.8025 29.6295 0.8928 18.7549 4.3115 0.8640 0.42 DMPIF 0.8684 31.8053 0.9511 13.2660 3.1358 0.9279 4.64 CDIF 0.8573 30.5537 0.9294 15.9662 3.7505 0.7900 36.70 A-PNN 0.8937 31.0437 0.9386 14.1669 3.4508 0.8945 0.30 Proposed 0.9206 32.8589 0.9579 11.5778 2.8134 0.9308 1.03

RELATED CONCLUSIONS

[0145] since MS and PAN images are obtained by different sensors, this pair of source images usually have low correlation and similarity, and direct fusion may lead to serious spectral distortion and spatial distortion. Secondly, to obtain an ideal HRMS image, it is necessary to inject the spatial information of the PAN image into the UPMS image. However, inaccurate injected spatial information will lead to low spatial resolution of the HRMS image. To solve these main problems, the disclosure proposes a model based on multimodal texture correction and adaptive edge detail fusion. To obtain T.sub.C hat is highly correlated and similar to MS and accurately inherits PAN spatial information, intensity constraints between T.sub.C and I.sub.0, gradient constraints between T.sub.C and PAN, I.sub.0, and deep plug-and-play constraints between T.sub.C and I.sub.net based on A-PNN are established, and an adaptive degradation filter algorithm is proposed to accurately maintain the constraints of the model. Finally, a multimodal texture correction model is constructed, which uses the ADMM algorithm to solve T.sub.C to replace the function of the PAN image. Since spatial detail information exists not only in T.sub.C but also in MS images, an adaptive edge detail fusion model is proposed, which extracts the detail information of T.sub.C and UPMS images respectively and applies edge protection. To extract detail information more accurately, an algorithm for adaptively extracting T.sub.C is used to extract details, and a Gaussian filter matched with MTF is used to extract UPMS images. The detail information of T.sub.C with edge protection is adaptively fused with the enhanced detail information of UPMS images with edge protection. Finally, the injection model is used to inject the fused spatial information into the UPMS image to obtain the final HRMS image. In comparative experiments, the performance advantages of the algorithm of the disclosure are illustrated, and parameter analysis and ablation studies prove the effectiveness of the algorithm of the disclosure. The final results show that the algorithm proposed in the disclosure may obtain better fusion results.

[0146] In the multimodal texture correction model, since iterative optimization is performed in 2D images, the solution efficiency is greatly improved, and the 3 set correction prior terms may well retain spatial and spectral information. However, the model still has shortcomings. There are unknown parameters in the correction prior terms that need to be determined through experiments, which may consume a lot of computing resources and time. In the adaptive edge detail fusion model, to obtain accurate spatial information, the edge detail information of T.sub.C and UPMS is comprehensively considered. However, problems such as the amount of injected spatial information and the ratio of UPMS spectral information to injected spatial information still exist. Therefore, our future work will focus on adaptively determining other unknown parameters in the pan-sharpening model and exploring more appropriate injection model methods to improve the overall performance and efficiency.

[0147] The above are only optional specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art may easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.