METHOD FOR PREDICTING MORPHOLOGICAL CHANGES OF LIVER TUMOR AFTER ABLATION BASED ON DEEP LEARNING

Abstract

A method for predicting the morphological changes of liver tumor after ablation based on deep learning includes: obtaining a medical image of liver tumor before ablation and a medical image of liver tumor after ablation; preprocessing the medical image of liver tumor before ablation and the medical image of liver tumor after ablation; obtaining a preoperative liver region map, postoperative liver region map, and postoperative liver tumor residual image map; obtaining a transformation matrix by a Coherent Point Drift (CPD) algorithm and obtaining a registration result map according to the transformation matrix; training the network by a random gradient descent method to obtain a liver tumor prediction model; using the liver tumor prediction model to predict the morphological changes of liver tumor after ablation. The method provides the basis for quantitatively evaluating whether the ablation area completely covers the tumor and facilitates the postoperative treatment plan for the patient.

Claims

1. A method for predicting morphological changes of liver tumor after ablation based on deep learning, comprising: 1) obtaining a data set comprising a medical image of the liver tumor before the ablation and a medical image of the liver tumor after the ablation; 2 pre-processing the medical image of the liver tumor before the ablation and the medical image of the liver tumor after the ablation; 3) obtaining a preoperative liver region map and a preoperative liver tumor region map from a medical image map before the ablation, and obtaining a postoperative liver region map, a postoperative ablation region map, and a postoperative liver tumor residual image map from a medical image map after the ablation; 1 registering the preoperative liver region map and the postoperative liver region map by a Coherent Point Drift (CPD) algorithm, and obtaining a transformation matrix; obtaining a registration result map of the medical image map after the ablation corresponding to the preoperative liver region map and the preoperative liver tumor region map according to the transformation matrix; 5) using the medical image map before the ablation, the preoperative liver region map, the preoperative liver tumor region map and the registration result map as an input of U-net network, and using the postoperative liver tumor residual image map as a real training label; training the U-net network by a random gradient descent method to obtain a liver tumor prediction model; and 6) using the liver tumor prediction model to predict the morphological changes of the liver tumor after the ablation; wherein step 3 comprises: marking the preoperative liver region map, the preoperative liver tumor region map, and the postoperative liver region map by a maximum flow/minimum cut algorithm; introducing a potential energy field function based on a global and local region representation as a constraint in a segmentation process, and establishing an adaptive hybrid variational model; using the maximum flow/minimum cut algorithm to solve an energy equation minimization: determining a target region selectively according to tray information, boundary gradient, texture information and local context information in different image regions: wherein step 4 comprises: obtaining liver data point set of the preoperative liver region map and the postoperative liver region map, making the liver data point set of the preoperative liver region map X.sub.i=(x.sub.1, . . . , x.sub.N).sup.T as a target point set, and making the liver data point set of the postoperative liver region man Y.sub.i=(y.sub.1, . . . , y.sub.M).sup.T as a model point set, wherein the target point set is a data set of Gaussian mixture model, and the model point set is a kernel point set of the Gaussian mixture model; N and M respectively represent a number of points in the target point set and a number of the model point set; a probability density function of the Gaussian mixture model is: $p\left(x.Math.m\right)=\frac{1}{{\left(2.Math.{\sigma }^2\right)}^{D/2}}{\exp }^{\frac{{.Math.x-y_m.Math.}_2^2}{2{\sigma }^2}}$ $p\left(x\right)=\omega \frac{1}{N}+\left(1-\omega \right)\underset{m=1}{\overset{M}{.Math.}}\frac{1}{M}p\left(x.Math.m\right);$ wherein P(x|M) is a probability density basis function of the Gaussian mixture model, a represents a standard deviation of the probability density function, C represents a weight value of overflow points, and a value range is 0-1, X is a translation variable; calculating a minimum negative logarithm likelihood function:
E(θ,σ.sup.2)=−Σ.sub.n=1.sup.N log Σ.sub.m=1.sup.M+1P(m)p(x|m), θ represents transformation parameters, and a represents the standard deviation of the probability density function; obtaining a derivative according to a gradient descent method: $Q\left(\theta ,{\sigma }^2\right)=\frac{1}{2{\sigma }^2}\underset{n=1}{\overset{N}{.Math.}}\underset{m=1}{\overset{M}{.Math.}}{P}^{\mathrm{old}}\left(m.Math.x_n\right){.Math.x_n-\tau \left(y_m,\theta \right).Math.}^2+\frac{N_{p}D}{2}\log {\sigma }^2,$ wherein $P\left(m\right)=\frac{1}{M},$ $N_P={.Math.}_{n=1}^N{.Math.}_{m=1}^M{P}^{\mathrm{old}}\left(m.Math.x_n\right)\le N,$ $P^{\mathrm{old}}\left(m.Math.x_n\right)=\frac{{\exp }^{-\frac{1}{2}{.Math.\frac{x_n-\tau \left(y_m,{\theta }^{\mathrm{old}}\right)}{{\sigma }^{\mathrm{old}}}.Math.}^2}}{{.Math.}_{k=1}^M{\exp }^{-\frac{1}{2}{.Math.\frac{x_n-\tau \left(y_k,{\theta }^{\mathrm{old}}\right)}{{\sigma }^{\mathrm{old}}}.Math.}^2}+c},$ τ represents one of rigid, affine and non-rigid transformations, and $c={\left(2.Math.{\sigma }^2\right)}^{D/2}\frac{\omega }{1-\omega }\frac{M}{N};$ solving optimal parameters for the transformation parameters of the minimum negative logarithm likelihood function by iteration with an Expectation-Maximization algorithm; and calculating a location of the preoperative liver region map corresponding to the medical image map after the ablation according to selected point cloud data and the optimal parameters for the transformation parameters, wherein the selected point cloud data comprise the liver data point set of the preoperative liver region map X.sub.i=(x.sub.1, . . . , x.sub.N).sup.T and the liver data point set of the postoperative liver region map Y.sub.i=(y.sub.1, . . . , y.sub.M).sup.T.

2. The method according to claim 1, wherein the medical image comprises an image obtained from computed tomography (CT) and magnetic resonance imaging MRI.

3. The method according to claim 1, wherein step 2 comprises dividing the data set into groups according to liver influencing factors, then reading the medical image of the liver tumor before the ablation and the medical image of the liver tumor after the ablation, and processing the medical image of the liver tumor before the ablation and the medical image of the liver tumor after the ablation by Gaussian de-noising, gray histogram equalization, image contrast enhancement, rotation, flipping and data standardization.

4. The method according to claim 3, wherein the liver influencing factors comprise a liver state, a tumor type, and a pathological type.

5. (canceled)

6. (canceled)

7. The method according to claim 1, wherein step 5 comprises: composing input data comprising four channels of image data with the medical image before the ablation, the preoperative liver region map, the preoperative liver tumor region map, and the registration result map, and then inputting into the U-net net work for coding to obtain a pixel classification probability map; determining pixels with probability ≥0.5 in the pixel classification probability map as the liver tumor, and pixels with probability ≤0.5 as background, and finally obtaining a predicted liver tumor region.

8. The method according to claim 7, wherein the step of obtaining the pixel classification probability map comprises: composing the input data comprising the four channels of the image data with the medical image before the ablation, the preoperative liver region map, the preoperative liver tumor region map, and the registration result map, and then inputting the input data into the U-net network for coding; convoluting the input data twice, a number of output channels of each convolution is 64, and obtaining a feature map f1 for maximum pooling; convoluting twice, the number of output channels of each convolution is 128, and obtaining a feature map f2, for the maximum pooling; convoluting twice, the number of output channels of each convolution is 256, and obtaining a feature map 3, for the maximum pooling; convoluting twice, the number of output channels of each convolution is 512, and obtaining a feature map f4, for the maximum pooling; convoluting twice, the number of output channels of each convolution is 1024, and obtaining a feature map the for an encoding process; decoding the feature map f5, comprising sampling the feature map f5 and performing concat with the feature map f4, after convoluting twice, the number of output channels of each convolution is 512, and obtaining a feature map f4_1; up sampling the feature map f4_1, and performing concat with the feature map f3, after convoluting twice, the number of output channels of each convolution is 256, and obtaining a feature map f3_1; up sampling the feature map f3_1, and performing concat with the feature map f2, after convoluting twice, the number of output channels of each convolution is 128, and obtaining a feature map f2_1; up sampling the feature map f2_1, and performing concat with the feature map f1, after convoluting twice, the number of output channels of each convolution is 64, and after convoluting once, the number of output channels is 2 to obtain the pixel classification probability map; wherein the concat is to connect two feature maps in L channel dimension.

9. The method according to claim 8, wherein each of the convoluting steps is based on X.sub.j.sup.l=Σ.sub.i∈M.sub.jx.sub.i.sup.l−1*k.sub.ij.sup.l+b.sub.j.sup.l), wherein x represents an input characteristic channel of an accretion layer, X represents an output characteristic channel of the accretion layer, k represents a parameter of a convolution layer, b represents an offset term of the convolution layer, the symbol * represents a convolution operator, l represents a number of layers, i represents i.sup.th neuron node of l−1 layer, j represents j.sup.th neuron node of l layer, M.sub.j represents a set of selected input characteristic graphs, x.sub.i.sup.l−1 refers to an output of the l−1 layer as an input of the l layer, ƒ represents an activation function, and the maximum pooling is to select a maximum value in a region to represent characteristics of the region.

10. The method according to claim 9, wherein step 5 comprises constructing Dice loss function in a process of training the U-net network to alleviate an imbalance of background and foreground pixels, wherein the Dice loss function is as follows: $d=1-2\frac{\mathrm{pt}}{p+t},$ wherein p represents the predicted liver tumor region and t represents a real liver tumor region.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0030] FIG. 1 is a flow chart of the present invention.

[0031] FIG. 2 is an interactive segmentation result diagram of the present invention.

[0032] FIG. 3 is the CPD registration result diagram of the present invention.

[0033] FIG. 4 is a U-net network structure diagram of the present invention.

[0034] FIG. 5 is the result diagram of the tumor deformation correction in the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] As shown in FIG. 1, this embodiment provides a method for predicting the morphological changes of liver tumor after ablation based on deep learning, including the following steps:

[0036] Step 1, CT/MRI scanning sequence images of liver tumor before and after ablation are obtained.

[0037] Step 2, the medical images before and after ablation are preprocessed as follows: the data set is divided into groups according to the influencing factors of liver, then CT/MRI scanning sequence images before and after ablation is read, and CT/MRI scanning sequence images is processed by Gaussian de-noising, gray histogram equalization, image contrast enhancement, rotation, flipping and data standardization, to increase the diversity of samples and accelerate the convergence speed of the network, wherein the liver influencing factors include liver state, tumor type, pathological type and other factors.

[0038] Step 3, the preoperative liver region map and preoperative liver tumor region map are obtained from the medical image map before ablation, the postoperative liver region map, postoperative ablation region map and postoperative liver tumor residual image map are obtained from the medical image map after ablation. As shown in FIG. 2, specifically: the preoperative liver region map, preoperative liver tumor region map and postoperative liver region map are marked by the maximum flow/minimum cut (Max-Flow/Min-Cut) algorithm. In the segmentation process, the potential energy field function based on global and local region representation is introduced as a constraint, and the adaptive hybrid variational model is established. The maximum flow/minimum cut algorithm is used to solve the energy equation minimization. The target region is determined selectively according to the gray information, boundary gradient, texture information and local context information in different image regions. The simple interaction on single-layer data can quickly complete 3D segmentation, and the doctor can modify the segmentation result manually to complete image annotation efficiently and accurately.

[0039] Step 4, the CPD algorithm is used to register the preoperative liver region map and the postoperative liver region map, and the transformation matrix is obtained. According to the transformation matrix, the registration result map of the preoperative liver region map and the preoperative liver tumor region map corresponding to the medical image map after ablation are obtained, specifically: [0040] Obtaining the liver data points set of the preoperative liver region map and the postoperative liver region map, making the preoperative liver data points set of the preoperative liver region map X.sub.i=(X.sub.1, . . . , x.sub.N).sup.T as the target point set, and making the postoperative liver data points set of the postoperative liver region map Y.sub.i=(Y.sub.1, . . . , Y.sub.M).sup.T as the target point set. The target point set is the data set of Gaussian mixture model, and the template point set is the kernel point set of Gaussian mixture model. N and M represent the number of target point set and template point set respectively. The probability density function of Gaussian mixture model is

[00006]$p\left(x.Math.m\right)=\frac{1}{{\left(2.Math.{\sigma }^2\right)}^{D\text{/2}}}{\exp }^{\frac{{.Math.x-y_m.Math.}_2^2}{2{\sigma }^2}},$$p\left(x\right)=\omega \frac{1}{N}+\left(1-\omega \right)\underset{m=1}{\overset{M}{.Math.}}\frac{1}{M}p\left(x.Math.m\right);$

wherein p(X|M) is the probability density basis function of Gaussian mixture model, a represents a standard deviation of the Gaussian probability density function, w represents the weight value of overflow point, and the value range is 0-1, X is translation variable. [0041] Calculating the minimum negative logarithm likelihood function:

E(θ,σ.sup.2)=−Σ.sub.n=1.sup.N log Σ.sub.m=1.sup.M+1P(m)p(x|m),

[0042] wherein θ represents transformation parameters, and a represents the standard deviation of the Gaussian probability density function.

[0043] According to the gradient descent method, the derivative can be obtained:

[00007]$Q\left(\theta ,{\sigma }^2\right)=\frac{1}{2{\sigma }^2}\underset{n=1}{\overset{N}{.Math.}}\underset{m=1}{\overset{M}{.Math.}}{P}^{\mathrm{old}}\left(m.Math.x_n\right){.Math.x_n-\tau \left(y_m,\theta \right).Math.}^2+\frac{N_{p}D}{2}\log {\sigma }^2,$

wherein

[00008]$P\left(m\right)=\frac{1}{M},$$N_P={.Math.}_{n=1}^N{.Math.}_{m=1}^M{P}^{\mathrm{old}}\left(m.Math.x_n\right)\le N,$$P^{\mathrm{old}}\left(m.Math.x_n\right)=\frac{{\exp }^{-\frac{1}{2}{.Math.\frac{x_n-\tau \left(y_m,{\theta }^{\mathrm{old}}\right)}{{\sigma }^{\mathrm{old}}}.Math.}^2}}{{.Math.}_{k=1}^M{\exp }^{-\frac{1}{2}{.Math.\frac{x_n-\tau \left(y_k,{\theta }^{\mathrm{old}}\right)}{{\sigma }^{\mathrm{old}}}.Math.}^2}+c},$

r represents one of rigid, affine and non-rigid transformations, and

[00009]$c={\left(2.Math.{\sigma }^2\right)}^{D/2}\frac{\omega }{1-\omega }\frac{M}{N}.$

[0044] The optimal parameters of the model of the minimum negative logarithm likelihood function are solved by Expectation-Maximization algorithm (EM). Finally, according to the selected point cloud data and transformation parameters, the location of the preoperative liver region map corresponding to CT/MRI scan sequence image after operation is calculated.

[0045] Through the registration of liver, the registration of liver tumor ablation area and liver tumor can be achieved indirectly. The registration result is shown in FIG. 3, 1 is the inactivated tumor residual image, 2 is the preoperative tumor registration transformation result, and 3 is the postoperative ablation area. It can be seen that the preliminary registration result is different from the actual tumor morphology.

[0046] Step 5, the medical image map before ablation, the preoperative liver region map, the preoperative liver tumor region map and the registration result map are used as the input of U-net network, and the postoperative liver tumor residual image map is used as the real training label. The network is trained by the random gradient descent method to obtain the liver tumor prediction model. The prediction model of liver tumor obtained is as follows: the medical image before ablation, the preoperative liver region map, the preoperative liver tumor region map and the registration result map are composed of four channels of image data, and then input into U-net for coding to obtain the pixel classification probability map. The pixels with probability ≥0.5 in the pixel classification probability map are determined as liver tumors, and the pixels with probability <0.5 are determined as background, and finally the predicted liver tumor region is obtained.

[0047] As shown in FIG. 4, a pixel classification probability map is obtained, specially: the medical image map before ablation, the preoperative liver region map, the preoperative liver tumor region map and the registration result map are composed of four channels of image data, and then input into U-net for coding. The input data is convoluted twice, and the number of output channels of each convolution is 64, and the feature map f1 is obtained, which is performed for maximum pooling. After two convolutions, the number of output channels of each convolution is 128, and the feature map f2 is obtained, which is performed for maximum pooling. After two convolutions, the number of output channels of each convolution is 256, and the feature map f3 is obtained, which is performed for maximum pooling. After two convolutions, the number of output channels of each convolution is 512, and the feature map f4 is obtained, which is performed for maximum pooling. After two convolutions, the number of output channels of each convolution is 1024, the feature map f5 is obtained, which is the encoding process. Then, f5 is decoded, that is, f5 is up sampled and performed concat with f4. After two convolutions, the number of output channels of each convolution is 512, and the feature map f4_1 is obtained, f4_1 is up sampled and performed concat with f3. After two convolutions, the number of output channels of each convolution is 256, and the feature map f3_1 is obtained. F3_1 is up sampled and performed concat with f2. After two convolutions, the output channel of each convolution is 128, and the feature map f2_1 is obtained. F2_1 is up sampled and performed concat with f1. After two convolutions, the number of output channels of each convolution is 64, and after another convolution, the number of output channels is 2 to obtain the pixel classification probability map; the concat is to connect the two feature maps in the channel dimension.

[0048] The convolution is specifically as follows: X.sub.j.sup.l=ƒ(Σ.sub.i∈M.sub.jx.sub.i.sup.l−1*k.sub.ij.sup.l+b.sub.j.sup.l), wherein x represents an input characteristic channel of the accretion layer, X represents an output characteristic channel of the accretion layer, k represents a parameter of the convolution layer, b represents an offset term of the convolution layer;

[0049] the symbol * represents the convolution operator; l represents the number of layers, i represents the i.sup.th neuron node of l−1 layer; j represents the j.sup.th neuron node of l layer; M.sub.j, represents the set of selected input characteristic graphs; x.sub.i.sup.l−1 refers to the output of l−1 layer as the input of l layer; ƒ represents the activation function; the maximum pooling is to select the maximum value in a region to represent the characteristics of the region.

[0050] Dice loss function is constructed in the process of network training to alleviate the imbalance of background and foreground pixels. The Dice loss function is as follows:

[00010]$d=1-2\frac{\mathrm{pt}}{p+t},$

wherein p represents the predicted liver tumor region and t represents the real liver tumor region. The loss of a back propagation in the training process includes the loss calculated by the deviation between the predicted tumor location and the real location.

[0051] Step 6, the morphological changes of liver tumor after ablation is predicted with the liver tumor prediction model. According to the grouping of data sets in Step 2, multiple U-net network models can be obtained, and then input into the corresponding network model according to the basic liver morphology, tumor type and pathological type, and predict the tumor changes.

[0052] According to the changes of tumor and the actual tumor, the tumor morphology is obtained by modifying Step 4. The result of tumor deformation correction is shown in FIG. 5. 5 is the residual image of inactivated tumor, 4 is the result of preoperative tumor registration transformation, 6 is the ablation area after operation. It can be seen that the modified tumor deformation is basically consistent with the actual shape after inactivation.

[0053] The invention provides a method for predicting the morphological changes of liver tumor after ablation based on deep learning, by which solves the problem of curative effect evaluation after ablation. It predicts the morphological changes of liver tumor after ablation with CPD point set registration and U-net network. Through the registration of liver contour before and after ablation, the transformation relationship is obtained, and then the location of preoperative liver tumor in postoperative CT/MRI image is obtained. Finally, the morphological changes of liver tumor after ablation are predicted by U-net network. The invention can predict the morphological changes of liver tumor after ablation, provide the basis for quantitatively evaluating whether the ablation area completely covers the tumor, facilitate the doctor to accurately evaluate the postoperative curative effect, and lay the foundation for the follow-up treatment plan of the patient.

[0054] The invention is not limited to the above optional embodiments, and anyone can obtain various other forms of products under the enlightenment of the invention. The above specific embodiments should not be understood as limiting the scope of protection of the invention. The scope of protection of the invention should be defined in the claims, and the specification can be used to explain the claims.