Method for automatically recognizing liver tumor types in ultrasound images

Abstract

The disclosure relates to a method for automatically recognizing liver tumor types in ultrasound images. The method specifically comprises: using a plurality of Regions of Interest (ROIs) to represent a CEUS image; different lesions are distinguished by the performance and changes of the ROI in time and space; representing a space-time relationship between ROIs by establishing a model in time and space at the same time; and determining, by the model, a relatively appropriate ROI and relevant parameters of the model according to existing CEUS lesion samples by means of an iterative learning method. After giving a sample, an appropriate ROI can be determined and a reference diagnosis for the lesion can be given by removing part of inappropriate ROIs in advance and by means of a rapid search method.

Claims

1. A method for automatically recognizing liver tumor types in ultrasound images, comprising: S1: using a model to represent a case, and using a local classifier to represent possible change forms of a lesion; S2: inputting ultrasound images of a group of cases and the type of liver cancer in each case as a training sample; S3: initializing the values of model parameters all to 0, or randomly initializing the values with a Gaussian probability distribution which has the expectation of zero; S4: based on the model parameters, in an ultrasound image video of a case, using a dynamic programming algorithm to search for an optimal Region of Interest (ROI) location, size, and time for each local classifier, so that when a model determines that the lesion class of the training sample is correct, a maximum score is obtained; S5: using a graph cut algorithm to determine a specific change form and ROI of the case; S6: based on the ROI determined in S5, using the training sample in S2 as an input, using a cutting-plane algorithm to train it, and using outputs of the algorithm as model parameters to obtain possible locations and sizes of lesions in the case and the number of local classifiers; S7: repeating S4 to S6 to acquire the lesion type of each case of a training sample data type, judging the correctness of the acquired lesion type, and when the number of judgment errors is fixed or the number of repetition steps reaches a preset value, obtaining training model parameters; S8: using the training model parameters acquired in S7 to determine an optimal ROI location, size, and time for all local classifiers in an ultrasound image video of a case to be detected, using a graph cut algorithm to determine a specific change form and ROI of the case to be detected, and using a model to obtain a probability score for a lesion according to the determined change form and ROI; and S9: repeating S2 to S8 for all lesion types in the case to be detected, and obtaining a probability score for one lesion by each repetition, a lesion type corresponding to the highest probability score being the lesion type of the case to be detected.

2. The method according to claim 1, wherein an ROI is respectively extracted for an input case ultrasound image at three phases namely an arterial phase, a portal phase and a delayed phase, the three ROIs being used to express FLLs; appearance features of an ROI interior, appearance features of an ROI boundary and appearance features of an ROI periphery are respectively extracted on the ROI interior, the ROI boundary and the ROI periphery of the ROI at each phase, and an average grayscale difference between the ROI interior and the ROI boundary and an average grayscale difference between the ROI interior and the ROI periphery are also acquired in the ROI at each phase; and when regional features of the ROI interior, the ROI boundary and the ROI periphery are extracted, the contrast, correlation, energy, and identity of a regional grayscale co-occurrence matrix are used as appearance features.

3. The method according to claim 2, wherein in S3, pruning is used first, and then an optimal ROI is searched by using a dynamic programming method, a specific process being as follows: a pruning process comprises time and space pruning; the time pruning is: calculating the grayscale feature of a grayscale co-occurrence focus in a certain frame in an ultrasound image video, and making a difference between vector frames to obtain a change value used to represent each frame; using a series of values obtained within a period of time to represent the degree of change of each frame within the period of time; arranging this series of values in a time sequence, and extracting the local maximum points to obtain a frame most dramatically changing within a period of time; selecting these frames corresponding to the local maximum points to obtain candidate frames of a lesion region; and in the remaining candidate frames, removing some unimportant regions based on experience; the space pruning is mainly achieved by using a priori of significance and location, comprising: calculating a saliency map of the entire image first, normalizing the saliency map, and averaging salient values in a region to obtain a salient value of the region, the priori of location being image-centered Gaussian distributions; and multiplying the values obtained from priori information of these two parts to obtain the probability that a certain region is an FLL region, and selecting a region of the probability is greater than the threshold as a candidate ROI; dynamic programming is used to search an ROI; after time and space pruning, a different number of frames will be retained within three different phases, and a different number of candidate ROIs will be retained within each frame; in the timing constraint, two ROIs adjacent to each other in a chronological order must be close to each other at a spatial location; the dynamic programming method is: finding an ROI within a previous phase, so that the ROI is in the vicinity of a current ROI, and its score plus timing relationship score of the current ROI can reach a maximum value; and searching a local classifier, so that an appearance score of the current ROI is maximized.

4. The method according to claim 3, wherein in S3, after searching for an optimal ROI, selection of the local classifier for the case will also be adjusted, the adjustment mode being: increasing, decreasing, or maintaining the number of local classifiers; in this process, the goal is to maximize a local classifier score selected for each case while maximizing the similarity measure of two ROIs under the same local classifier.

5. The method according to claim 4, wherein a specific mode to achieve the above goal is: using a Euclidean distance between two ROI features as a measure of similarity therebetween, and using a graph cut algorithm to solve the problem by converting into a graph label problem.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a diagram of a space-time AND-OR model;

(2) FIG. 2 is a diagram of model parameters;

(3) FIG. 3 is a diagram of space pruning;

(4) FIG. 4 is a diagram of dynamic programming reasoning in a model reasoning method;

(5) FIG. 5 is a reconfiguration diagram of a model structure; and

(6) FIG. 6 is a partial result diagram of the method of the disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

(7) The disclosure will be further described below with reference to the drawings, but the embodiments of the disclosure are not limited thereto.

(8) The disclosure mainly provides a method for automatically recognizing liver tumor types in ultrasound images, comprising two parts: a method for representing a liver tumor in an ultrasound image; and a method for automatically determining the location, time and size of a lesion in an ultrasound image and recognizing the lesion.

(9) For the convenience of description, key terms are defined as follows.

(10) Vector is a set of sequentially arranged numbers that can be represented in a computer programming language, such as an array in C language.

(11) Model is a set of rules that can take a video of a case as an input and output it as a possible value for a particular lesion. In this method, the model divides the input into a vector with a certain structure according to the rules, and multiplies model parameter vectors by elements to obtain a possible value. The larger the value is, the more likely it belongs to that type of lesion.

(12) Model parameter is a vector. By inputting ultrasound image videos of multiple cases and corresponding lesion types by a user, a specific value of a model vector may be obtained by using a learning method in the disclosure.

(13) ROI is a region on a video frame that consists of all pixels in this portion of a frame of image. In this method, these regions are identified as lesion regions.

(14) Local classifier is a part of the model. An ROI is input, and a value indicating whether the ROI belongs to a specific change form of a certain lesion. The larger the value is, the more likely it belongs to that specific change.

(15) For an ultrasound video of a case, this method comprises: automatically extracting an ROI in a certain frame at three imaging phases as the location of a lesion, and representing it with an AND-OR model structure. The model structure is as shown in FIG. 1. The three imaging phases in FIG. 1 are combined to represent a case class.

(16) An imaging mechanism of ultrasound images makes each ROI change considerably at different times and in different cases. At the same time, internal echo and posterior echo enhancements will be generated in the ultrasound image, thereby making it more important to refer to an enhancement pattern of surrounding tissues when extracting the features of the ROI.

(17) This method extracts the features of the ROI in the following ranges: an ROI interior, used to describe a lesion itself; an ROI boundary, used to describe the shape of the lesion; and a normal tissue portion within a certain range of an ROI periphery, used to describe a posterior echo enhancement pattern of this ultrasound image. In addition, an internal echo pattern of the lesion is obtained by making a difference between an average grayscale of a lesion region and surrounding and an average grayscale of a lesion region edge and a lesion interior.

(18) When extracting the features of the three regions of the ROI, the contrast, correlation, energy, and identity of a region grayscale co-occurrence matrix are used as appearance features in this method. Thus, a feature vector of the ROI is represented by five parts: appearance features of an ROI interior, appearance features of an ROI boundary, appearance features of an ROI periphery, an average grayscale difference between the ROI interior and the ROI boundary, and an average grayscale difference between the ROI interior and the ROI periphery.

(19) The other features of the case consist of a vector obtained by making a difference between appearance feature vectors of two ROIs and a Euclidean distance from spatial coordinates of the two regions. This part correspondingly represents timing features of the case.

(20) All the appearance features and the timing features are spliced together to obtain a representation mode as shown in FIG. 2, where a circle with horizontal lines represents a selected local classifier, and a hollow circle represents an unselected local classifier, corresponding features being set as 0.

(21) The above method may convert a piece of ultrasound image video into features, represented by a vector. This vector will be used for ultrasound image recognition.

(22) After obtaining the vector representation of the ultrasound image, the following steps are needed to recognize a liver cancer lesion region in the ultrasound image.

(23) 1. A user inputs ultrasound images of a group of cases and the type of liver cancer in each case as training data.

(24) 2. An initial value is set for a model parameter.

(25) 3. The model parameter is used to determine an optimal ROI location, size and time for all local classifiers in an ultrasound image video of each case.

(26) 4. A graph cut algorithm is used to determine a specific change form and ROI of each case.

(27) 5. The model parameter is trained by using an SVM method based on these ROIs.

(28) 6. S3 to S5 are repeated; until the lesion type of the training data is judged, it is determined that the number of judgment errors is no longer reduced; or until the number of repetition steps reaches a preset value, a model parameter is obtained.

(29) 7. A case is given, and the model parameter and method obtained above are used to obtain the ROI location, size, time and change pattern of the case.

(30) 8. The above steps are repeated for all possible lesion types, the most likely type being a lesion type recognized by this method.

(31) Herein, model parameters in S2 may be all initialized to 0, or randomly initialized with zero expectation.

(32) In S3 to S5, a Latent Structural SVM method is used to optimize a model. While obtaining the model parameters, the possible location and size of a lesion and number of local classifiers in the training data are determined. Iterative training is performed between an objective function of the optimized model and the determined ROI.

(33) In S3, the method searches for the size, location and frame of an ROI with the highest score for each local classifier when calculating a hidden variable via a fixed model parameter. During searching for the ROI, this method uses pruning first and then uses dynamic programming.

(34) In an ultrasound image, two frames generally appear to be very smooth, and when a lesion region starts to show an enhancement pattern, the grayscale will generally have a large change with the nearby frame in a time sequence. Therefore, in some periods of time, when a lesion region enhancement pattern is obvious, those frames that have the greatest change in grayscale are more likely to be selected to be used as temporal candidate frames.

(35) Specifically speaking, for a certain frame in the ultrasound image, grayscale features of a grayscale co-occurrence focus are calculated on this frame, and a difference is made between vector frames to obtain a value used to represent the degree of change of each frame. By arranging this string of values in a chronological order and extracting the local maximum points from it, a frame most dramatically changing within a period of time may be obtained. By selecting these frames, candidate frames of the lesion region may be obtained.

(36) After the time pruning in the previous step, less important regions are removed in the remaining candidate frames. In this step of pruning, the significance and location priori are mainly used. First, the saliency map of the entire image is calculated, and normalized, and the salient values in the region are averaged to obtain a salient value of the region. The second location priori is image-centered Gaussian distribution. The values obtained from the priori information of these two parts are multiplied to obtain the probability that a certain region is an FLL region, as shown in FIG. 3. A threshold is set, and a region having a probability greater than the threshold is selected as a candidate ROI, where a circle shown in FIG. 4 is a candidate ROI.

(37) Dynamic programming is then used to search an ROI; after time and space pruning, a different number of frames will be retained within three different phases, and a different number of candidate ROIs (circle) will be retained within each frame, as shown in FIG. 4. In the timing constraint of the method model, two ROIs adjacent to each other in a chronological order must be close to each other at a spatial location. A dynamic programming algorithm in this method is: finding an ROI within a previous phase, so that the ROI is in the vicinity of a current ROI, and its score plus timing relationship score of the current ROI can reach a maximum value; and searching a local classifier, so that an appearance score of the current ROI is maximized. As shown in FIG. 4, a solid circle is an ROI selected by the dynamic programming algorithm.

(38) After finding the optimal ROI, the method will adjust selection of the local classifier for each case, and increase, decrease, or maintain the number of local classifiers, thereby ensuring that the number of ROIs in each local classifier, i.e., a sub-class, is not too small, so as to achieve the purpose of reconfiguring the model structure. In this process, the goal of the method is to make the local classifier scores for each case selected as high as possible while making the ROIs as similar as possible between the cases where the same local classifier is selected. This method specifically maximizes the model score while maximizing the similarity measure of the two ROIs under the same local classifier, and makes the number of local classifiers as few as possible to prevent a situation that a local classifier is provided for a case during actual operation. Here, a Euclidean distance between two ROI features is specifically used as a measure of similarity therebetween, and a graph cut algorithm is used to solve the problem by converting into a graph label problem.

(39) After the problem is solved, the label of each node is the corresponding local classifier selected by the node or in the sample, and the ROI selected by the local classifier is the optimal ROI. At the same time, the number of local classifiers may also be determined, thereby completing the reconfiguration of the AND-OR model. As shown in FIG. 5, the model in (a) is reconfigured as the model in (b). In the first stage, a local classifier is added to represent a newly discovered lesion imaging pattern.

(40) Finally, in S7 and S8, all possible classes are traversed, and the largest class is taken out as a recognition result, as shown in FIG. 6. At the same time, the found ROI is used as an analysis result of this method, and it can provide more information and reference for the aided diagnosis of a doctor.

(41) The above-described embodiments of the disclosure do not limit the protection scope of the disclosure. Any modifications, equivalent replacements and improvements made within the spirit of the disclosure shall fall within the protection scope of the claims of the disclosure.