Verifying and/or improving the consistency of labels within the scope of medical image processing
11288805 · 2022-03-29
Assignee
Inventors
- Markus Michael Geipel (Munich, DE)
- Florian Büttner (Munich, DE)
- Gaby Marquardt (Hausen, DE)
- Daniela Seidel (Baiersdorf, DE)
- Christoph Tietz (Ottobrunn, DE)
Cpc classification
G06F18/214
PHYSICS
G16H50/70
PHYSICS
International classification
Abstract
A computer-implemented method and a data processing apparatus provide and apply a trained probabilistic graphical model for verifying and/or improving the consistency of labels within the scope of medical image processing. Also provided are a computer-implemented method for verifying and/or improving the consistency of labels within the scope of medical imaging processing, a data processing apparatus embodied to verify and/or improve the consistency of labels within the scope of medical image processing, and a corresponding computer program product and a computer-readable medium.
Claims
1. A computer implemented method using a trained probabilistic graphical model to adapt hidden labels within the scope of medical image processing of cells, the trained probabilistic graphical model stored in a memory and executable on a processor, the method including the steps of: providing hidden labels based on similarities between at least two data points of images as training data and expert-generated labels associated with this training data, the images comprising image information of respectively one cell; linking the hidden labels and the expert-generated labels, the linking comprising a connecting of all hidden labels among themselves and a connecting of each hidden label to a respective expert-generated label; and adapting or confirming the hidden labels based on the trained probabilistic graphical model, the adapting or confirming including linking an adapted or confirmed hidden label with an image of a cell and storing in the memory the image of the cell together with the linked adapted or confirmed hidden label.
2. A computer-implemented method for producing the trained probabilistic graphical model of claim 1 for verifying or improving a consistency of labels within a scope of medical image processing, the method including the following steps: providing a set of images as training data, the images comprising image information of respectively one cell; calculating a feature space for the training data with the aid of a deep convolutional neural network (DCNN) executing on a processor, with an image corresponding to a feature vector in the feature space; calculating similarities between at least two data points of the training data based on the feature space; providing expert-generated labels for cells, the images of which were provided as training data; calculating the hidden labels based on similarities between at least two data points of the training data and the expert-generated labels; and producing a conditional random field (CRF) model as the trained probabilistic graphical model.
3. A non-transitory computer program product with a computer program, which is directly loadable into a memory device of a computer, comprising program sections for carrying out all steps of a method as claimed in claim 2 when the computer program is executed on the computer.
4. The computer-implemented method as claimed in claim 2, wherein the calculation of the similarities between at least two data points comprises creation of a similarity graph.
5. A data processing apparatus, embodied to verify or improve a consistency of labels within a scope of medical image processing, wherein the apparatus has at least one processor and a memory, wherein the at least one processor is configured to load and execute program code from the memory and, in response to execution of the program code, carry out the following steps: providing an image of a cell; providing an available expert-generated label for this image; calculating a feature space for the image; applying the method as claimed in claim 1 and the trained probabilistic graphical model of claim 1 on the calculated features of the image; linking a label to the image, the linking comprising a correction or confirmation of the already available expert-generated label for this image; and outputting or storing the image together with the corrected or confirmed linked label.
6. The computer-implemented method as claimed in claim 1, wherein the adapted or confirmed hidden labels, which were obtained based on the probabilistic graphical model as claimed in claim 1, are used to repeat the method as claimed in claim 1.
7. The computer-implemented method as claimed in claim 1, wherein the linking of the calculated label and the expert-generated label is carried out based on a conditional probability of the correctness of the expert-generated label.
8. A computer-implemented method for producing a trained probabilistic graphical model for verifying or improving a consistency of labels within a scope of medical image processing, wherein a cell can represent different development stages of the cell, comprising the following steps: providing a set of images as training data, the images comprising image information of respectively one cell; calculating a feature space for the training data with the aid of a deep convolutional neural network (DCNN), with an image corresponding to a feature vector in the feature space; calculating similarities between at least two data points of the training data based on the feature space; calculating a pseudotime, which orders the at least two data points as per a development sequence; providing expert-generated labels for cells whose images were provided as training data; calculating hidden labels based on the development sequence reflected in the pseudotime and the expert-generated labels; and producing and storing in a memory a probabilistic graphical model for adapting the hidden labels, the probabilistic model being a hidden Markov model (HMM) in the case of a linear development sequence and a hidden Markov tree (HMT) in the case of a dichotomous development sequence.
9. A computer-implemented method, wherein the probabilistic graphical model of claim 8 is used to adapt hidden labels within the scope of medical image processing of cells, which can represent different cell development stages, including the steps of: providing hidden labels based on similarities between at least two data points of a set of images as training data, the images comprising image information of respectively one cell, and a pseudotime inferred therefrom, and expert-generated labels associated with the training data; linking the hidden labels and the expert-generated labels, the linking comprising a connection of all labels by way of directed edges; and adapting or confirming the hidden labels based on the probabilistic graphical model.
10. The computer-implemented method as claimed in claim 9, wherein the adapted or confirmed hidden labels, which were obtained based on the probabilistic graphical model as claimed in claim 9, are used to repeat the method as claimed in claim 9.
11. The computer-implemented method as claimed in 9, wherein the linking of the calculated label and the expert-generated label is carried out based on a conditional probability of the correctness of the expert-generated label.
12. The computer-implemented method as claimed in claim 8, wherein the calculation of the similarities between at least two data points comprises creation of a similarity graph.
13. A non-transitory computer-readable medium, on which program sections that are readable and executable by a computer are stored in order to carry out all steps of the method as claimed in claim 8 when the program sections are executed by the computer.
14. A data processing apparatus for adapting hidden labels within a scope of medical image processing of cells based on a trained probabilistic graphical model, comprising: a first unit for (i) providing hidden labels based on similarities between at least two data points of a set of images as training data and expert-generated labels associated with this training data, the images comprising image information of respectively one cell, or (ii) providing hidden labels based on similarities between at least two data points of a set of images as training data, the images comprising image information of respectively one cell, and a pseudotime inferred therefrom, and expert-generated labels associated with the training data; a second unit for (i) linking the hidden labels and the expert-generated labels, the linking comprising a connecting of all hidden labels among themselves and a connecting of each hidden label to a respective expert-generated label, or (ii) linking the hidden labels and the expert-generated labels, the linking comprising connecting each hidden label to a respective expert-generated label; and a third unit for receiving adapted hidden labels based on the probabilistic graphical model.
15. A computer-implemented method for verifying or improving a consistency of labels within a scope of medical image processing, including the steps of: providing an image of a cell to a first processor; providing an available expert-generated label to the first processor; using background information in respect of the image of the cell to define a topology of a hidden Markov tree (HMT), a start probability, and an emission matrix; learning the feature representations of the image using a neural network executing on a first processor; choosing a suitable pseudotime inference algorithm and calculating pseudotimes based on feature vectors; sorting the available expert-generated labels according to increasing pseudotime; configuring the hidden Markov tree (HMT), in which the sorted expert-generated labels are observed information; learning parameters of transition matrices using a generalized EM (Expectation-Maximization) algorithm executing on the first or another processor; applying a generalized Viterbi algorithm in order to infer most probable true labels, the Viterbi algorithm executing on the first or another processor; identifying images with inconsistent labels by virtue of comparing actual the true labels with the expert-generated labels using the first or another processor; and outputting and storing in a memory images with inconsistent labels, which additionally contain proposed labels.
16. The computer-implemented method as claimed in claim 15, wherein a correction of an already available expert-generated label leads to a feedback query with an expert in respect of a labeling discrepancy.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
DETAILED DESCRIPTION OF THE INVENTION
(12)
(13)
(14) Subsequently, hidden labels 13 can be inferred after the application of the CRF. This leads to possible correction of the hidden labels 14. These are then supplied to a module for machine learning 4, which can finally generate a classification model 5.
(15)
(16)
(17) The imaged cells correspond to an ordered sequence and differ from arbitrary classes by inherent ordering principles, such as similar structure or similar size, but differ in terms of the morphology of the cell nucleus, etc.
(18)
(19) Subsequently, labeling borders can be inferred in pseudotime 53 following the application of the HMM. This leads to possible correction of the hidden labels 14. These are then supplied to a module for machine learning 4, which can finally generate a classification model 5.
(20)
(21)
(22)
(23)
(24)
(25)
(26) The identified discrepancy with the presumed assessment error between the labels can subsequently be provided to the expert in form of feedback such that a reevaluation or verification is possible. Among other things, this renders an efficient assistance system for cell classification implementable.
EXAMPLES
Example 1
(27) Pseudotime Inference
(28) The pseudotime of a cell describes the developmental progress of the cell along a dynamic process such as cell differentiation. The greater the pseudotime of a cell, the more mature the cell is. Pseudotime inference algorithms can be used to create a pseudotemporal ordering for all cells in a population. Pseudotime inference algorithms are usually applied to single-cell gene expression similarity measurements (Haghverdi et al., 2016, Nature Methods, 13(10), 845-848), where adjacent cells have higher expression similarity. These algorithms can be applied to medical images by interpreting the pixels of a cell image as information in respect of this cell, similar to gene expression data, to obtain an ordering of the cells along trajectories. There are a multiplicity of pseudotime inference methods to date, which differ in terms of the requirement of existing prior information, scalability, and type of topology (Saelens et al., 2019, Nature Biotechnology, 37, 547-554). Most pseudotime inference methods consist of two parts. The first part is the calculation of a low-dimensional representation from the given expression data of the cells, and the second part is the ordering of the cells along an inferred trajectory. Here, use was made of the SCORPIUS (Cannoodt, 2016, SCORPIUS improves trajectory inference and identifies novel modules in dendritic cell development, bioRxiv:10.1101/079509v2) and STREAM (Chen et al., 2019, Nature Communications, 10, 1, 1903) algorithms. SCORPIUS shows very good performance for linear datasets, while STREAM is well-suited to datasets with tree-like topologies. Given the expression profiles of the cells, SCORPIUS obtains a low-dimensional representation using multi-dimensional scaling (MDS). Next, SCORPIUS applies k-means clustering and sets the initial trajectory by connecting the cluster centers. The final trajectory results from an iterative refinement through the principal curves algorithm. The pseudotime is calculated by projecting the low-dimensional representations onto the trajectory. Similarly, STREAM first determines relevant features and then performs dimensionality reduction using modified locally linear embedding (MLLE). In the new embedding, an implementation of elastic principal graphs (ElPiGraph) (Albergante et al., 2018, Robust and scalable learning of complex dataset topologies via ElPiGraph, arXiv:1809.07580v2) is used to infer the trajectory and branching points. ElPiGraph approximates datasets with complex topologies by minimizing the elastic energy of the embedding and applying graph transformations. The cells are then projected onto the resulting tree according to their pseudotimes and their assigned branches (see also Chen et al., 2019, Nature Communications, 10, 1, 1903).
Example 2
(29) Hidden Markov Trees
(30) Hidden Markov trees are used to describe the differentiation process of the cells, which is a stochastic process following the Markov assumption (Abkowitz et al., 1996, Nature Medicine, 2, 2, 190-197). There is one root cell type, and all other cell types develop therefrom and can be mapped onto a tree-like topology reflecting their respective progeny. Assuming that the topology of the dataset, i.e., the shape of the Markov tree, is known, the following applies:
(31) Definition 1: A tree
(32) Define
(33) Definition 2: Let
(34) (i)
(35) (ii) the distribution of the observed variable Xt depends only on the hidden variable Zt for all t∈{1, . . . , T}.
(36) For the application to cell image labels, the variable Xt corresponds to the noisy (observed) expert label, and Zt represents the true (unobservable) labels of the image, which may be different from the expert labels. The sequence of images is sorted by increasing pseudotime, which has been calculated before by a suitable pseudotime inference algorithm. Let K be the number of cell types and T be the number of images in the dataset.
(37) Definition 3: The hidden Markov tree (
(38) The following definitions apply for 2≤t≤T, 1≤k, {tilde over (k)}≤K:
π.sub.k:=(Z.sub.1=k),
A.sub.kl.sup.(t):=(Z.sub.t=l|Z.sub.p(t)=k),
B.sub.k{tilde over (k)}:=(X.sub.t={tilde over (k)}|Z.sub.t=k),
(39) where p(t) denotes the parent of node t.
(40) π denotes the start probability, A.sup.(t) denotes the transition matrix at node t, and B denotes the emission matrix. If the transition matrix A.sup.(t) is independent of t, the model is called homogeneous; otherwise, the model is called inhomogeneous. The transition matrix A.sup.(t) describes the probability of staying in the present cell type or changing to a child cell type. The emission matrix B represents the expert labeling error model, where B.sub.kn is the probability that the expert predicts label {tilde over (k)} when the true cell type of the cell in the image is k.
(41) A hidden Markov model (HMM) is a special case of an HMT, where the underlying topology is a chain.
(42) Time-Dependent Transition Matrices
(43) The following information is used to set up the parametric transition matrices. The topology of the dataset is known, and following the Markov assumption of blood cell differentiation (Abkowitz et al., 1996, Nature Medicine, 2, 2, 190-197), it is only possible for a cell to stay in the same cell type or to transition to one of the child cell types. There is no way to skip one cell type or to go back to a previous cell type. Once one of the end stages is reached, there are no transitions anymore. Standard homogeneous HMMs/HMTs are based on the assumption that the transition between states is independent of t, which would correspond to cells sampled uniformly across the development trajectory. However, in practice, these samples (i.e., the labeled cells) are from arbitrary points on the development trajectory, which is reflected by large variation in pseudotime difference between neighboring cells. This difference directly affects the probability of a cell to transition to a different cell type: the larger the pseudotime difference between two cells, the greater is the likelihood for a transition (and the lower is the likelihood of the cell to remain in the same cell type). Consequently, the entries of the transition matrix at node t should not only depend on the cell type of the previous cell, but also on the time difference between the present cell and the previous cell. To model the dependency of the transition matrix on the pseudotime, the algorithms for HMMs and HMTs were extended to the inhomogeneous case and appropriate parametric transition matrices were derived.
(44) The following definitions apply:
(45) yt∈R≥0 as the pseudotime difference between cell t−1 and cell t, after they have been ordered by increasing pseudotime. To find reasonable entries for the transition matrices, the transition probabilities at node t are defined as follows:
(46)
(47) Here, P (Zt=l|Zt−1=k)=: p.sub.kl∈[0, 1] is the transition probability from cell type k to cell type l. Let p.sub.kl be a constant independent of t, with condition
Σ.sub.l=1.sup.kpkl=1 for all k.
(48) The support of yt is known to be [0, ∞) for the probability P (yt|Zt=l, Zt−1=k). Since there is no more information about the distribution of the pseudotime difference, the maximum entropy probability distribution is used. The least informative distribution for a random variable with support [0, ∞) and mean 1/λ is the exponential distribution with rate λ. Let the rate λ be dependent on the cell types k and l.
(49) Then, for each possible transition in the cell lineage tree, the entry in the transition matrix after normalization has the form
(50)
(51) for p.sub.kl∈[0, 1] and λ.sub.kl>0.
(52) The parameters in this formula are learned using the generalized EM algorithm (Neal and Hinton, 1998, A view of the EM algorithm that justifies incremental, sparse, and other variants, Learning in Graphical Models, 355-368) since the corresponding objective function is intractable.
(53) The generalized Viterbi algorithm (Durand et al., 2004, IEEE Transactions on Signal Processing, 52(9), 2551-2560) then computes the most probable hidden variables arg max .sub.Zl: T P (Z1: T X1: T).
Example 3
(54) The TIMELY Algorithm
(55) TIMELY combines pseudotime inference methods with inhomogeneous HMTs. The pseudotime inference algorithm establishes an intrinsic ordering of the cells based on morphology, and the HMT then finds inconsistent labels and proposes correct labels of the cells corresponding to the true cell types. The input of TIMELY is a set of images together with noisy expert labels. First, a network (convolutional network) is used to learn meaningful feature representations of the cell images that are consistent with the morphology of the cells. The convolutional network consists of three convolutional layers with 32 filters each, where the filter size is 3×3. After each convolutional layer, there is a max-pooling layer with a pooling size of 2×2. A bottleneck of 50 units, which provides the resulting feature vectors, is followed by two dense layers with 30 hidden units each and an output layer.
(56) As an alternative, unsupervised methods such as autoencoders were also explored to learn feature representations of the images so that the training is not affected by noisy labels. This yielded qualitatively similar findings.
(57) Next, a suitable pseudotime inference method was applied to calculate the pseudotimes. The cells were ordered according to increasing pseudotime. SCORPIUS or STREAM was used, depending on the topology of the data. The sorted expert labels served as the observed information in the HMT, and the hidden labels are the true cell types to be determined. The background information about the dataset can be used to fix the start probabilities n and the emission matrix B, while the parameters of the transition matrices are learned by the generalized EM algorithm. Through the generalized Viterbi algorithm, the most probable true labels and the estimated cell type borders were found, which are unique due to the Markov assumption (Abkowitz et al., 1996, Nature Medicine, 2, 2, 190-197).
(58) Any inconsistencies between the true labels and the expert labels are potential mistakes by the expert. Hematologists can reconsider the affected images and, if necessary, correct the labels of the cells. The method is summarized in Algorithm 1 (see below). TIMELY was implemented in Python, and the library SciPy is used for maximizing the objective function in the generalized EM algorithm.
(59) Algorithm 1: TIMELY
(60) Input: Images and noisy expert labels
(61) Output: Images with inconsistent labels and proposed labels
(62) 1: Use background information about the dataset to define the topology of the HMT, the start probabilities n, and the emission matrix B.
(63) 2: Learn feature representations of the images using a neural network.
(64) 3: Choose a suitable pseudotime inference algorithm and calculate the pseudotimes on the basis of the feature vectors.
(65) 4: Sort the corresponding expert labels according to increasing pseudotime.
(66) 5: Set up an HMT, where the sorted expert labels are the observed information.
(67) 6: Learn the parameters in the transition matrices A(t) using the generalized EM algorithm.
(68) 7: Apply the generalized Viterbi algorithm to infer the most probable true labels.
(69) 8: Identify images with inconsistent labels by comparing the true labels with the expert labels.
Example 4
(70) Baseline Methods
(71) TIMELY was compared to three baseline methods. As explained above, most algorithms are robust to noisy data labels, find and remove noisy labels, or model label noise explicitly, but they do not propose new labels.
(72) The algorithms k-nearest neighbors (k-NN) and k-nearest centroid neighbors (k-NCN) (Sanchez et al., 1997, Pattern Recognition Letters, 18, 11-13, 1179-1186) find neighbors for each instance for a given distance measure. A commonly used distance measure for k-NN is the Euclidean distance, while, for k-NCN, instances were added to the set of nearest neighbors for which the centroid of the new set is nearest to the considered instance. The label of the considered instance is then obtained by a majority vote. If the majority vote yields a different label than the original label of the instance, or if there is a tie, the instance might be incorrectly labeled.
(73) To compare this method with other methods that also propose corrections, these two methods were extended with generalized editing (Koplowitz and Brown, 1981, Pattern Recognition, 13, 3, 251-255), i.e., numbers k and k′ with (k+1)/2≤k′≤k were chosen for k-NN and k-NCN. For each instance, if there are at least k′ nearest neighbors from a different cell type, the cell type of the instance is changed to that type. Unlike in Koplowitz and Brown, 1981, no samples were deleted. For both methods, k=3 and k′=2 were chosen, which are common values in the literature (Saez et al., 2015 Journal of Medical Informatics & Technologies, 24, pp. 123-130).
(74) TIMELY was also compared to cleanlab (Northcutt et al., 2019, Confident learning: estimating uncertainty in dataset labels, arXiv:1911.00068v1), which is based on confident learning Northcutt et al., 2017, Learning with confident examples: rank pruning for robust classification with noisy labels, in Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence. AUAI Press) and finds labeling errors. It estimates the noise rates by calculating the joint distribution between noisy and uncorrupted labels and then prunes inconsistent samples.
Example 5
(75) Simulation Data
(76) Since expert labels from real-world datasets are often noisy, the ground truth labels of the images are unknown. For comparing the algorithm described herein to other methods in finding inconsistent labels, three datasets with different noise levels that mimic the cell differentiation setting were simulated. Each dataset consists of 250 samples from five cell types, where the underlying topology is a chain. The process of simulating the datasets is the following:
(77) 1. Let X∈R.sup.2×250, where X is normally distributed.
(78) 2. Sort the columns of X by increasing X.sub.lj, 1≤j≤250.
(79) 3. Define the corresponding ground truth labels Y∈R.sup.250, where the entries Y.sub.50(i−1)+1:50 are i for i∈{1 . . . 5}.
(80) 4. Apply mapping P to project X to a higher-dimensional space with {tilde over (X)}=PX∈R.sup.k×250. k=50 is chosen to be consistent with the real-world datasets.
(81) 5. Add noise level l∈{10, 20, 30} to the ground truth labels Y by randomly changing 1% of the entries in Y to different labels.
(82) The steps 1 to 4 are repeated for each noise level.
(83) The idea is that the samples have a low-dimensional ordering, corresponding to the pseudotemporal ordering, which can be retrieved by dimensionality reduction of the higher-dimensional feature vectors.
Example 6
(84) Simulation Results
(85) The results of the comparison is shown in Table 1. The methods k-NN+edit and k-NCN+edit modify the labels during application, while k-NN, k-NCN, and cleanlab only find possible labeling errors. TIMELY finds labeling errors and proposes new labels without changing them directly.
(86) The proposed labels are compared with the ground truth labels to calculate the accuracy. The selected items are the instances that the algorithm marked as labeling errors. While TIMELY finds errors in a magnitude that is similar to the noise level, the other methods mostly find too many errors, without increasing the recall. Only in one case does k-NCN have a higher recall than TIMELY. The method according to the invention has the highest accuracy, precision, recall, and F1 score in all the other cases. Editing in k-NN and k-NCN often improves the F1 score compared to the versions without editing. However, editing of labels during application influences the classification of subsequent samples, and so the accuracy drops if there are too many false positives.
(87) TABLE-US-00001 TABLE 1 Comparison to baseline methods for simulation data. TIMELY outperforms all baseline methods in terms of accurately identifying and correcting noisy labels. k − k − Noise TIME- k − NN + k − NCN + level Metric LY NN edit NCN edit cleanlab 10 Accuracy 0.984 — 0.920 — 0.944 — Selected 0.108 0.164 0.152 0.152 0.144 0.112 items Precision 0.889 0.561 0.579 0.658 0.667 0.679 Recall 0.960 0.920 0.880 1.000 0.960 0.760 F.sub.1 Score 0.923 0.697 0.698 0.794 0.787 0.717 20 Accuracy 0.992 — 0.932 — 0.820 — Selected 0.192 0.292 0.236 0.324 0.284 0.256 items Precision 1.000 0.658 0.797 0.556 0.634 0.625 Recall 0.960 0.960 0.940 0.900 0.900 0.800 F.sub.1 Score 0.980 0.781 0.863 0.687 0.744 0.702 30 Accuracy 0.972 — 0.792 — 0.712 — Selected 0.300 0.416 0.340 0.484 0.404 0.272 items Precision 0.987 0.673 0.706 0.537 0.604 0.809 Recall 0.987 0.933 0.800 0.867 0.813 0.733 F.sub.1 Score 0.987 0.782 0.750 0.663 0.693 0.769
Example 7
(88) Application to Real Data
(89) TIMELY was applied to two image datasets of stained white blood cells. All images were generated by a digital microscope (Cellavision, Siemens Healthineers AG) and labeled by an expert. Due to the challenges in manual labeling described above, the labels are noisy and partly incorrect. For the preparation of the images, a thin blood film was applied on a glass slide and stained. A digital microscope then located the blood cells and created corresponding images. The datasets contained images from a plurality of patients. TIMELY was applied to the whole dataset to first find the ordering of the images. Then, it suggested a label for each image. For a new patient, images from the same developmental tree can be mapped onto the already calculated tree, and consistent labels can be read off the tree directly by making use of the already computed transition borders.
(90) Cell Lineage
(91) Datasets
(92) The first dataset consisted of 1000 cell images that contained five cell types of the granulopoiesis development line. The topology was a linear chain. There were 200 images, labeled by an expert as belonging to each of the cell types promyelocyte (PMY), myelocyte (MY), metamyelocyte (MMY), band neutrophil (BNE), and segmented neutrophil (SNE).
(93) Parameters in HMM
(94) Available background knowledge about the dataset was used to fix the start probabilities n and the emission matrix B. The dataset has five cell types, and the root type in the development process is known to be PMY. Thus, the start probabilities could be fixed as follows:
(95) π:=(0.9 0.025 0.025 0.025 0.025).sup.T
(96) The first cell should be in the first cell type with high probability and in the other cell types with low probability.
(97) The constant emission matrix B is based on estimations of an expert who could realistically estimate the probability of labeling errors. The emission matrix for the first dataset is as follows:
(98)
(99) The more mature cell types band neutrophil and segmented neutrophil are fairly easy for humans to differentiate, while the first three cell types, especially myelocytes, are more difficult to label.
(100) Pseudotime Inference
(101) The SCORPIUS algorithm was used to compute the pseudotimes. Diffusion maps (Coifman and Lafon, 2006, Applied and Computational Harmonic Analysis, 21(1), 5-30) for dimensionality reduction were used before SCORPIUS was applied. Subsequently, SCORPIUS directly inferred the trajectory without performing MDS.
(102) Visualization Tools
(103) Following the parameter optimization, the HMM found unique transition borders between the cell types. A visualization tool for viewing the images was provided (see also
(104) Inconsistent Labels
(105) The percentage of consistent labels, where the hidden labels and expert labels coincide, is 72% according to the HMM. This means that there are 280 images with potentially wrong labels. By way of a confusion matrix, it was possible to show that the consistency is particularly low for myelocytes and metamyelocytes. Overall, the tendency of the values was similar to the expert's estimation of the emission matrix, as shown above.
(106) Experiments have shown that the results are quite robust with respect to the emission matrix, and so small changes in the estimations will not significantly affect the results.
(107) The 280 inconsistent images were passed to an expert for reclassification. For 128 of these images (45.7%), the expert confirmed the previous labels. For the remaining 152 cells, the expert either relabeled them as the cell types proposed by the HMM, or they could not assign a label with high confidence, meaning that up to 54.3% of the inconsistent images might have wrong labels. Most of the reclassifications related to the first three cell types in the development line, where changes in the morphology can be very subtle.
(108) Cell Lineage Trees
(109) Dataset
(110) The second dataset consisted of 1821 cell images in ten classes, which are part of a development process with branching points. There were 200 images labeled by an expert as belonging to each of the promyelocyte (PMY), myelocyte (MY), metamyelocyte (MMY), band neutrophil (BNE), segmented neutrophil (SNE), blast (BL), basophil (BA), eosinophil (EO), and lymphocyte (LY) cell types. There were only 21 images for the last class plasma cell (PC). Eosinophils and basophils also have myelocytes, metamyelocytes, and band neutrophils as precursors. However, these exhibit different staining behavior than the precursors of the segmented neutrophils. Because those cell types are quite rare in the blood, they were not included in the dataset.
(111) Parameters in HMT
(112) The root cell should be a blast cell so that the entry for blast is very high in π. The constant emission matrix B is again based on discussions with an expert and is a consistent extension of the emission matrix shown above. The five additional cell types should not be too difficult to differentiate from the cell types of the first dataset because they are part of different development lines. Only the blasts have some similarities to the promyelocytes, which are descendants of the blasts. The end stages of segmented neutrophil, basophil, and eosinophil should be easy for experts to classify.
(113) Pseudotime Inference
(114) The STREAM algorithm was used to infer a reasonable tree for the dataset. Two of the three possible branching points matched the branching points from the cell lineage tree. However, the last branching point, where the eosinophils branch off from the metamyelocytes, is different. In general, the eosinophils are far away from the other cell types in the feature space following the dimensionality reduction. The connection point to the remaining tree might not be correct. A further reason could be that the precursor cells of segmented neutrophils and eosinophils look alike. Eosinophils have the same progenitor stages as the neutrophils, which are only stained in a different color. The algorithm might also identify the metamyelocytes as a previous development stage of the eosinophils. The range of the pseudotimes is still plausible for all cell types however.
(115) Inconsistent Labels
(116) The percentage of consistent labels according to the HMT is 69%, meaning that there are 564 images with potentially wrong labels. The blasts and promyelocytes seem to be mixed up often, while basophils and eosinophils have high agreement between hidden labels and expert labels, presumably on account of their distinct staining colors. The agreement for lymphocytes is also very high since cells from different development lines are usually easier to differentiate.
(117) The 564 inconsistent images were given to an expert for reclassification. The expert confirmed their previous labels for 341 images, and so up to 40.1% of the inconsistent images might have wrong labels. Most reclassifications affected promyelocytes, myelocytes, and metamyelocytes, which represent the first three cell types in the granulopoiesis line. The cells were mostly classified as the progenitor of the cell type determined by the experts.
(118) Summary of the Results
(119) As a method according to the invention, TIMELY, a human-centered approach for increasing labeling consistency within the scope of medical imaging for cell type classification, was introduced.
(120) TIMELY takes as input cell microscopy images together with noisy expert-generated labels, identifies inconsistent labels and proposes alternative, consistent labels on the basis of a two-step procedure.
(121) In the first step, TIMELY establishes an intrinsic order between cells with the aid of a pseudotime inference algorithm. In the second step, TIMELY creates a Markov model on the basis of the ordered cells and their noisy labels. An HMM or an HMT is used, depending on the complexity of the topology of the dataset. Pseudotime estimations are combined with interpretable HMTs in order to establish a system that assists an annotating hematologist or histologist, for example, with generating more consistent cell classifications. By sorting the cells according to the pseudotime, the annotating hematologist or histologist is able to consider each cell in the neighborhood of cells that have a similar morphology. This assists them in making more consistent decisions. Moreover knowledge in the art is transparently and explicitly encoded in form of differentiation hierarchies, start probabilities (see above), and an expert-driven emission matrix (see above), reflecting prior experience on the likelihood of labeling errors.
(122) Taken together, this allows, for example, a hematologist or histologist to develop an intuitive understanding as to why specific cells are suggested as being inconsistently labeled and helps an easier adoption in practice.
(123) Manually labeling cells is also a time-consuming process, and the method according to the invention can be applied to reduce the time experts spend on this task.
(124) As soon as the parameters of an HMT are optimized, new images from the same developmental tree can be mapped onto the already calculated tree, and consistent labels can be read directly off the tree by virtue of making use of the already computed transition borders.
(125) An additional, exemplary use case of TIMELY is the application to automatically generated labels since they are often noisy. Moreover, the classification algorithm does not include all possible cell types. These labels would then serve as the observed information of the HMT, and only the inconsistent labels will be given to the expert for reclassification.
REFERENCE SIGNS
(126) 1 Image data 2 Expert-generated labels 3 Feature calculation 4 Machine learning 5 Classification model 11 Calculation of the similarity graph 12 Creation of a probabilistic model in the form of a conditional random field (CRF) 13 Inference of hidden labels 14 Correction of the hidden labels 21 Hidden label of cell A 22 Expert-generated label of cell A 23 Hidden label of cell B 24 Expert-generated label of cell B 25 Hidden label of cell C 26 Expert-generated label of cell C 27 Hidden label of cell D 28 Expert-generated label of cell D 29 Hidden label of cell E 30 Expert-generated label of cell E 31 Hidden label of cell F 32 Expert-generated label of cell F 33 Hidden label of cell G 34 Expert-generated label of cell G 35 Linking of the hidden labels among themselves 36 Linking of the hidden labels and the expert-generated labels 41 Transition of cell development stages 42 Myeloblast 43 Promyelocyte 44 Myelocyte 45 Metamyelocyte 46 Band segmented neutrophil 51 Calculating the pseudotime 52 Creating a probabilistic model in the form of a hidden Markov model (HMM) 53 Inferring label borders in pseudotime 61 Convolution 62 Max pooling 63 Compression 64 Representation in the feature space 71 Transformation 72 Feature space 73 Cells within the scope of their development in the feature space 81 Cells in pseudotime 91 Distance to the next cell in pseudotime 101 Coding in the emission probability of the HMM 102 Coding by the transition matrix of the HMM 111 Estimated class or development stage border 112 Error by the expert