MULTIMODAL MEDICAL IMAGE FUSION METHOD BASED ON DARTS NETWORK

Abstract

A multimodal medical image fusion method based on a DARTS network is provided. Feature extraction is performed on a multimodal medical image by using a differentiable architecture search (DARTS) network. The network performs learning by using the gradient of network weight as a loss function in a search phase. A network architecture most suitable for a current dataset is selected from different convolution operations and connections between different nodes, so that features extracted by the network have richer details. In addition, a plurality of indicators that can represent image grayscale information, correlation, detail information, structural features, and image contrast are used as a network loss function, so that the effective fusion of medical images can be implemented in an unsupervised learning way without a gold standard.

Claims

1. A multimodal medical image fusion method based on a differentiable architecture search (DARTS) network, characterized by comprising: S1. performing a network architecture search on a preset DARTS model by using multimodal medical image data to obtain a DARTS network architecture applicable to the multimodal medical image data, wherein the DARTS network architecture comprises one or more cells connected in series, each cell comprises a plurality of nodes, each of the nodes is connected to outputs of two preceding cells or other nodes in a current cell by means of different convolution operations, and channels of the nodes are combined into an output of the cell; S2. constructing a multimodal medical image fusion network, wherein the multimodal medical image fusion network comprises a multi-channel DARTS network module, a feature fusion module, and an upsampling module, wherein the multi-channel DARTS network module is formed by a plurality of parallel DARTS network architectures, the DARTS network architecture is configured to downsample an input image to obtain a corresponding feature map, the feature fusion module is configured to perform feature fusion on a feature map outputted by a dual-channel DARTS network module, and the upsampling module is configured to perform convolution upsampling on a fused feature to obtain a fusion image with the same size as the input image; S3. training the multimodal medical image fusion network by using the multimodal medical image data through an unsupervised learning method; and S4. inputting a multimodal medical image to be fused into a trained multimodal medical image fusion model to obtain a fusion result.

2. The multimodal medical image fusion method based on a DARTS network according to claim 1, wherein structural similarity (SSIM), multi-scale structural similarity (MS-SSIM), edge preservation, sum of the correlations of differences (SCD), mutual information (MI), and structural representation-based mutual information (SR-MI) are used as a loss function for training the multimodal medical image fusion network.

3. The multimodal medical image fusion method based on a DARTS network according to claim 2, wherein the loss function for training the multimodal medical image fusion network is:
Loss=(L.sub.SCD+L.sub.MI)+λ.sub.1.Math.(L.sub.SSIM−L.sub.MS-SSIM+L.sub.Qps.sub.1s.sub.2/F)+λ.sub.2.Math.L.sub.SR-MI wherein L.sub.SSIM represents an SSIM loss, L.sub.SCD represents an SCD loss, L.sub.MI represents an MI loss, L.sub.MS-SSM represents an MS-SSIM loss, L.sub.Q.sub.ps.sub.1s.sub.2/F represents an edge preservation loss, L.sub.SR-MI represents an SR-MI loss, and λ.sub.1 and λ2 are respectively weights of the groups of loss functions, respectively.

4. The multimodal medical image fusion method based on a DARTS network according to claim 1, wherein the different convolution operations in step Si comprise: a depthwise separable convolution with a convolution kernel size of 3, a depthwise separable convolution with a convolution kernel size of 5, a standard convolution with a convolution kernel size of 3, a standard convolution with a convolution kernel size of 5, a dilated convolution with a convolution kernel size of 3, a dilated convolution with a convolution kernel size of 5, and skip connections.

5. The multimodal medical image fusion method based on a DARTS network according to claim 4, wherein the DARTS network architecture comprises one cell.

6. The multimodal medical image fusion method based on a DARTS network according to claim 5, wherein the cell comprises four nodes.

7. The multimodal medical image fusion method based on a DARTS network according to claim 6, wherein a convolution step size in the cell is set to 1, and a padding mode is used to make the feature map have a size consistent with that of the input image.

8. The multimodal medical image fusion method based on a DARTS network according to claim 1, wherein the feature fusion module is configured to perform feature fusion on the feature map outputted by the dual-channel DARTS network module, and specifically fusion of the feature map is implemented by means of channel combination.

9. The multimodal medical image fusion method based on a DARTS network according to claim 1, further comprising performing data augmentation on the multimodal medical image data.

10. The multimodal medical image fusion method based on a DARTS network according to claim 9, wherein the data augmentation comprises translation, rotation, and non-rigid deformation.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0025] FIG. 1 is a structural diagram of a multimodal medical image fusion method based on a DARTS network according to the present invention;

[0026] FIG. 2 shows a cell structure of a DARTS model in a CIFAR-10 dataset;

[0027] FIG. 3(a) shows a source CT image used in an embodiment of the present invention and a compared method;

[0028] FIG. 3(b) shows a source MR image used in an embodiment of the present invention and a compared method;

[0029] FIG. 4(a) shows a fused image obtained through NSCT-SR in Compared Method 1;

[0030] FIG. 4(b) shows a fused image obtained through DTCWT-SR in Compared Method 2;

[0031] FIG. 4(c) shows a fused image obtained through NSST-PAPCNN in Compared Method 3;

[0032] FIG. 4(d) shows a fused image obtained through DenseFuse in Compared Method 4;

[0033] FIG. 4(e) shows a fused image obtained through VIFNet in Compared Method 5;

[0034] FIG. 4(f) shows a fused image obtained through Nestfuse in Compared Method 6; and

[0035] FIG. 5 shows a fused image obtained through the method according to the present invention.

DETAILED DESCRIPTION

[0036] To make the purpose, technical solution, and advantages of the present invention clearer, the present invention is further described in detail below in connection with the accompanying drawings and embodiments. It should be appreciated that the specific embodiments described here are used merely to explain the present invention and are not used to define the present invention.

[0037] In addition, the technical features involved in various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict between them.

[0038] Referring to FIG. 1, a multimodal medical image fusion method based on a DARTS network provided in the present invention includes the following steps:

[0039] S1. Performing a network architecture search on a preset DARTS model by using multimodal medical image data to obtain a DARTS network architecture applicable to the multimodal medical image data, wherein the DARTS network architecture includes a plurality of cells connected in series, each cell includes a plurality of nodes, each of the nodes is connected to outputs of two preceding cells or other nodes in a current cell by means of different convolution operations, and channels of the nodes are combined into an output of the cell.

[0040] In the embodiments of the present invention, the DARTS network is a DARTS model pretrained by using a data set CIFAR-10. However, a fully connected layer part and a classifier part of the DARTS network are removed. To ensure the extraction of image features and at the same time greatly reduce network parameters, a quantity of cells is set to 1, and a quantity of nodes in each cell is set to 4. A specific structure is shown in FIG. 2. The nodes are connected by a depthwise separable convolution, a skip connection, and a dilated convolution. In addition to the convolution operations described above, optional convolution operations for node connection may include a depthwise separable convolution with a convolution kernel size of 5, a standard convolution with a convolution kernel size of 3, and a standard convolution with a convolution kernel size of 5. At present, these convolution modes can have good representation for image features in an end-to-end network architecture.

[0041] A downsampling output is a channel combination of the four nodes, a convolution step size in the cell is set to 1, and a padding mode is used to make the feature map have a size consistent with that of the input image so as to avoid an information loss.

[0042] S2. Constructing a multimodal medical image fusion network, wherein the multimodal medical image fusion network includes a multi-channel DARTS network module, a feature fusion module and an upsampling module. Here, the multi-channel DARTS network module is formed by a plurality of parallel DARTS network architectures. The DARTS network architecture is configured to downsample an input image to obtain a corresponding feature map. The feature fusion module is configured to perform feature fusion on a feature map outputted by a dual-channel DARTS network module. Optionally, in the present invention, the fusion of the feature map is implemented by means of channel combination. Channel combination may perform weighted fusion on features of source images through subsequent convolution. In a case that a network loss function is appropriately designed, the feature fusion implemented by means of channel combination has a better effect, so that useful information of the source images can be retained to a greater extent. The upsampling module is configured to perform convolution upsampling on a fused feature to obtain a fusion result having the same size as the input image. In an upsampling operation, a padding mode with a convolution of 3×3 and a step size of 1 is used to reduce a channel quantity of a convolutional neural network from 128 to 1.

[0043] A two-channel DARTS network shown in FIG. 1 is only used as an implementable example for ease of showing the overall structure of the network. According to a quantity of modalities of a fused image, a plurality of parallel channels may further be set.

[0044] S3. Training the multimodal medical image fusion network by using the multimodal medical image data through an unsupervised learning method.

[0045] To implement network training, the loss function used in the present invention includes SSIM MS-SSIM, edge preservation Q.sub.p.sup.s.sup.1.sup.s.sup.2.sup./F, SCD, MI, and SR-MI.

[0046] In the present invention, the foregoing evaluation indicators are combined in the following manner into a loss function:

Loss=(L.sub.SCD+L.sub.MI)+λ.sub.1∜(LSSIM+L.sub.MS-SSIM+L.sub.Q.sub.ps.sub.1s.sub.2/F)+λ.sub.2.Math.L.sub.SR-MI.

[0047] The foregoing loss function includes three groups of functions: The first group of MI and SCD is related to image grayscale information and a correlation, the second group is related to an image edge and structure information, and the third group is an additional item for constraining an edge, where λ.sub.1 and λ.sub.2 are respectively weights of the groups of loss functions.

[0048] SCD reflects a sum of correlations of differences between the fused image and the source images, and a calculation formula of SCD is as follows:

L.sub.SCD=−(R(F−S.sub.2, S.sub.1)+R(F−S.sub.1, S.sub.2))

[0049] where S.sub.1, S.sub.2 denote two source images and F denotes the fused result; R represents the calculation of a correlation, and a specific formula is as follows:

[00001]$R\left(S,F\right)=\frac{{.Math.}_m{.Math.}_n{S}_{m,n}{F}_{m,n}}{{\left({.Math.}_m{.Math.}_n{S}_{m,n}^2\right)}^{\frac{1}{2}}{\left({.Math.}_m{.Math.}_n{F}_{m,n}^2\right)}^{\frac{1}{2}}}$

[0050] MI represents the amount of information of the two sources images included in the fused image, and a specific formula of MI is:

L.sub.MI=−(MI(S.sub.1,F)+MI(S.sub.2,F))

[0051] where a formula of the MI is as follows:

[00002]$\mathrm{MI}\left(S,F\right)=\underset{f\in F}{.Math.}\underset{s\in S}{.Math.}p\left(s,f\right)\log \left(\frac{p\left(s,f\right)}{p\left(s\right)p\left(f\right)}\right)$

[0052] where p(s) and p(f) are edge distributions, and p(s, f) is a joint distribution of s and f.

[0053] A formula of the SSIM loss is as follows:

[00003]$L_{\mathrm{SSIM}}=-\frac{\mathrm{SSIM}\left(S_1,F\right)+\mathrm{SSIM}\left(S_2,F\right)}{2}$

[0054] where the SSIM is calculated by using information such as brightness, contrast, and structure of two images, and a specific formula is as follows:

[00004]$\mathrm{SSIM}\left(S,F\right)=\frac{\left(2{\mu }_S{\mu }_F+c_1\right)\left(2{\sigma }_{SF}+c_2\right)}{\left({\mu }_S^2+{\mu }_F^2+c_1\right)\left({\sigma }_S^2+{\sigma }_F^2+c_2\right)}$

[0055] where μ.sub.s and μ.sub.f are respectively average values of S and F. σ.sub.S.sup.2 and σ.sub.F.sup.2 are respectively variances of S and F, σ.sub.SF is a covariance of S and F, and c.sub.1 and c.sub.2 are relatively small constants.

[0056] MS-SSIM is an indicator obtained by shrinking an image by different scales with a power of 2 as a factor and then calculating SSIMs on different scales.

[0057] Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F uses a local metric to estimate a preservation degree of significant information (i.e., edge information) from an input in a fused image, and a calculation formula is as follows:

[00005]$$\begin{gathered} L_{Q_{\underset{\_}{p}}^{S_1{S}_2/F}}=-Q_{\underset{¯}{p}}^{S_1{S}_2/F} \\ {Q}_{\underset{¯}{p}}^{S_1{S}_2/F}=\frac{{.Math.}_{n=1}^N{.Math.}_{m=1}^M{Q}^{S_{1}F}\left(n,m\right)w^{S_1}\left(n,m\right)+Q^{S_{2}F}\left(n,m\right)w^{S_2}\left(n,m\right)}{{.Math.}_{i=1}^N{.Math.}_{j=1}^M\left(w^{S_1}\left(i,j\right)+w^{S_2}\left(i,j\right)\right)} \end{gathered}$$

[0058] where Q.sup.SA.sup.1F and Q.sup.S.sup.2F respectively represent the degree of preservation of the fused image with respect to edges of the two source images, and a calculation formula thereof is as follows:

Q.sup.SF(n, m)=Q.sub.g.sup.SF(n,m)Q.sub.α.sup.SF(n, m)

[0059] where Q.sub.g.sup.SF and Q.sub.α.sup.SF respectively represent a length preservation value for an edge and a direction preservation value for an edge, and calculation formulas thereof are as follows:

[00006]$Q_{\alpha }^{SF}\left(n,m\right)=\frac{{\Gamma }_{\alpha }}{1+e^{{\kappa }_{\alpha }\left(A^{SF}\left(n,m\right)-{\sigma }_{\alpha }\right)}}{Q}_g^{\mathrm{SF}}\left(n,m\right)=\frac{{\Gamma }_g}{1+e^{{\kappa }_g\left(G^{SF}\left(n,m\right)-{\sigma }_g\right)}}$

[0060] where Γ.sub.α, Γ.sub.g, κ.sub.α, κ.sub.g, σ.sub.g, and σ.sub.α are constants, and A.sup.SF and g.sup.SF are respectively a direction related value and a length related value of a source image S and a fused image F, and calculation formulas thereof are as follows:

[00007]$A^{SF}\left(n,m\right)=1-\frac{.Math.{\alpha }_S\left(n,m\right)-{\alpha }_F\left(n,m\right).Math.}{\pi /2}{G}^{SF}\left(n,m\right)=\{\begin{array}{c}\frac{g_F\left(n,m\right)}{g_S\left(n,m\right)}\mathrm{if}{g}_S\left(n,m\right)>g_F\left(n,m\right)\\ \frac{g_S\left(n,m\right)}{g_F\left(n,m\right)}\mathrm{otherwise}\end{array}$

[0061] where g.sub.s, α.sub.s, g.sub.F, and α.sub.F are respectively edge lengths and angles of the source image and the fused image. In a calculation formula of Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F, w.sup.S.sup.1(n,m)=[g.sub.S.sub.1(n,m)].sup.L, and w.sup.S.sup.2 (n, m)=[g.sub.s.sub.2 (n, m)].sup.L (where L is a constant), which are used as the weights of the fused image with respect to the edge preservation of the source image for calculating Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F.

[0062] SR-MI represents mutual information based on a structural representation result. The structural representation result is obtained by using a principal component analysis network (PCANet).

[0063] In the embodiments of the present invention, brain CT and MR images in the data set ATLAS (which comes from the website hosted by Harvard Medical School: http://www.med.harvard.edu/aanlib/home.html, and has collected brain CT, Mill, PET, SPECT images under normal conditions and in different disease states) are chosen, and translated, rotated, and non-rigid transformed, and 30000 pairs of obtained images are used as a training set. In addition, brain CT and MR images that do not belong to the training set are selected to construct a test set, to test the multimodal medical image fusion network.

[0064] S4. Inputting a multimodal medical image to be fused into a trained multimodal medical image fusion model, to obtain a fusion result.

[0065] To prove the superiority of the method in the present invention, a comparative experiment is performed by using six compared algorithms and the method of the present invention. The six compared algorithms are introduced:

[0066] 1. NSCT-SR: an image fusion algorithm based on the non-subsampled contourlet transform (NSCT) and sparse representation.

[0067] 2. DTCWT-SR: an image fusion algorithm based on the dual-tree complex wavelet transform (DTCWT) and sparse representation.

[0068] 3. NSST-PAPCNN: a medical image fusion method based on a parameter adaptive neural network in the non-subsampled shearlet transform (NSCT) field.

[0069] 4. DenseFuse: an image fusion algorithm that uses a DenseNet-based encoding and decoding structure, performing fusion by using fusion strategies such as addition and L1 regularization.

[0070] 5. VIFNet: a depthwise learning image fusion method using unsupervised DenseNet encoding and convolutional decoding and a MSE and a weighted SSIM as the loss function.

[0071] 6. NestFuse: a depthwise learning image fusion method based on a novel NEST network architecture.

[0072] Some objective evaluation indicators (EN, FMI_pixel, MI, MS_SSIM, Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F, and SCD) are chosen to evaluate a fusion result. In the foregoing evaluation indicators, FMI_pixel represents pixel-level feature MI, and is MI calculation performed based on an image feature. The image feature may be represented by using a gradient, a Sobel edge operator, and the like.

[0073] EN (Entropy) represents information entropy. The larger the value of EN is, the richer the information included in an image is. The formula is as follows:

[00008]$\begin{array}{cc}H\left(X\right)=\underset{i}{.Math.}P\left(x_i\right)I\left(x_i\right)=-\underset{i}{.Math.}P\left(x_i\right){\log }_{b}P\left(x_i\right)& \end{array}$

H(X) represents information entropy of an image, P(x.sub.i) represents a probability that a pixel with a grayscale value of i appears, and I(x.sub.i) represents a non-deterministic function of the probability P(x.sub.i).

[0074] Fusion evaluation results of the same pair of images in ATLAS using different methods are shown in Table 1. As can be seen from the table, DARTS image fusion has clearly higher SCD, Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F, and EN indicators than other methods, and has certain advantages in indicators such as MI and MS_SSIM, indicating that the method has kept information of two source images to a relatively high degree, and has certain advantages in structural aspects such as edge information.

TABLE-US-00001 TABLE 1 Evaluation Indicators NSCT-SR DTCWT-SR NSST-PAPCNN DenseFuse VIFNet Nestfuse DARTS EN 5.1348 4.9061 5.4184 4.6565 4.7000 4.6545 5.5156 FMI_pixel 0.8880 0.8980 0.8850 0.8865 0.8895 0.8864 0.8895 MI 2.6880 2.9077 2.6672 3.1449 3.3285 3.3194 3.3920 MS_SSIM 0.5318 0.4827 0.5746 0.5617 0.5446 0.5865 0.5960 Q.sub.p.sup.S.sup.1.sup.S.sup.2.sup./F 0.5292 0.5257 0.5074 0.5006 0.5254 0.4480 0.5792 SCD 1.2110 1.1112 1.4875 1.4059 1.1731 1.2392 1.6352

[0075] To show the superiority of the present invention to other methods, the embodiments of the present invention further provide diagrams of visual effects of corresponding fused images using the method of the present invention and various compared methods. FIG. 3(a) shows a source CT image. FIG. 3(b) shows a source MR image. FIG. 4(a) shows a fused image obtained by using Compared Method 1. FIG. 4(b) shows a fused image obtained by using Compared Method 2. FIG. 4(c) shows a fused image obtained by using Compared Method 3. FIG. 4(d) shows a fused image obtained by using Compared Method 4. FIG. 4(e) shows a fused image obtained by using Compared Method 5. FIG. 4(f) shows a fused image obtained by using Compared Method 6. And FIG. 5 shows a fused image obtained through the method according to the present invention. As can be seen from the figures, the method in the present invention has excellent performance in the aspect of contrast, and especially has rich details and edge information. This also indicates that the method of the present invention has the advantage that an obtained image has more details and a large amount of information.

[0076] It can be easily understood by those skilled in the art that the foregoing description is only preferred embodiments of the present invention and is not intended to limit the present invention. All the modifications, identical replacements and improvements within the spirit and principle of the present invention should be in the scope of protection of the present invention.