A METHOD TO DETERMINE A DEGREE OF ABNORMALITY, A RESPECTIVE COMPUTER READABLE MEDIUM AND A DISTRIBUTED CANCER ANALYSIS SYSTEM
20210142908 · 2021-05-13
Inventors
Cpc classification
G16H50/20
PHYSICS
G06F21/00
PHYSICS
H04L9/3239
ELECTRICITY
G06V10/454
PHYSICS
G06N3/042
PHYSICS
G16H10/00
PHYSICS
G16H10/40
PHYSICS
G06F18/241
PHYSICS
International classification
G16H50/20
PHYSICS
G16H10/40
PHYSICS
Abstract
Current cancer screening methods are not suitable to be applied on a broad scale and are not transparent to the patient. The problem is solved by a method to determine a degree of abnormality, the method comprising the following steps: receiving a whole slide image, the whole slide image depicting at least a portion of a cell, classifying at least one image tile of the whole slide image using a neural network to determine a local abnormality degree value associated with the at least one image tile, the local abnormality degree value indicating a likelihood that the associated at least one segment depicts at least a part of a cancerous cell, and determining a degree of abnormality for the whole slide image based on the local abnormality degree value for the at least one image tile.
Claims
1. A method to determine a degree of abnormality, the method comprising the following steps: a) receiving a whole slide image (11, w, 722), the whole slide image (11, w, 722) depicting at least a portion of a cell, in particular a human cell; b) classifying at least one image tile (13, 601, 721, 721′, 721″) of the whole slide image (11, w, 722) using a neural network (600) to determine a local abnormality degree value (15, a_j, 519, 719, 719′, 719″) associated with the at least one image tile (13, 601, 721, 721′, 721″), the local abnormality degree value (15, a_j, 519, 719, 719′, 719″) indicating a likelihood that the associated at least one segment depicts at least a part of a cancerous cell; and c) determining a degree of abnormality (17) for the whole slide image (11, w, 722) based on the local abnormality degree value (15, a_j, 519, 719, 719′, 719″) for the at least one image tile (13, 601, 721, 721′, 721″).
2. The method of claim 1, characterized in that the degree of abnormality (17) for the whole slide image (11, w, 722) is indicated by a function, in particular a max-function, a statistical aggregation of the local abnormality degree value (15, a_j, 519, 719, 719′, 719″) or an average function dependent on the local abnormality degree value (15, a_j, 519, 719, 719′, 719″).
3. The method of claim 1, characterized by: Segmenting the whole slide image (11, w, 722) into a plurality of image tiles (13, 23, 502, t_1, . . . , t_n), the size of each image tile being equal, in particular using a k-means clustering algorithm.
4. The method of claim 1, characterized in that the neural network (600) is implemented as a convolutional neural network (600), the neural network (600) comprising: at least fifty layers, at least twenty pooling layers, at least fourty convolutional layers, at least twenty kernels in each convolutional layer, at least one fully connected layer (608), a softmax-classifier layer (608) and/or neurons using logit and/or logistic functions as activating functions.
5. The method of claim 1, characterized by training the neural network (600) with training data (26, 509) stored in a knowledgebase (25, 507), the training data (26, 509) including a plurality of tuples, each tuple indicating an image tile (13, 601, 721, 721′, 721″), a training abnormality value (15, 519, 719, 719′, 719″, a_j) and a likelihood value (l_j).
6. The method of claim 1, characterized by Receiving an update whole slide image (11, 21, 501, 722, w) to update the knowledgebase (25, 507); Segmenting the update whole slide image (11, 21, 501, 722, w) into a plurality of image tiles (13, 601, 721, 721′, 721″); Determining a training abnormality degree value (a_w) for each image tile of at least one subset of the plurality of image tiles (C), in particular by a human expert (E); Updating the knowledgebase (25, 507) with the subset of the plurality of image tiles (C) and the associated training abnormality degree values (a_w) if it is determined that adding the subset of the plurality of image tiles (C) and the associated training abnormality degree values (a_w) improves the accuracy of the neural network when trained with the updated knowledgebase (25, 507).
7. The method of any of claim 6, characterized in that updating the knowledgebase (25, 507) further comprises: Computing a predicted abnormality degree value (a_j) and an associated likelihood value (l_j) for each of the plurality of image tiles (13, 601, 721, 721′, 721″) using the neural network; Determining a priority value (p_j) based on the predicted abnormality degree value (a_j) and the associated likelihood value (l_j) for each of the plurality of image tiles (13, 601, 721, 721′, 721″); Determining the subset of the image tiles (C) based on the determined priority (p_j) values.
8. The method of claim 1, characterized in that determining that adding the subset of the image tiles (C) and the associated training abnormality degree values (a_w) improves the accuracy of the neural network when trained with the updated knowledgebase (25, 507) comprises: Updating a validation database (515) with the subset of the image tiles (C) and the associated training abnormality degree values (a_w), the validation database in particular including at least a subset of the knowledge base; Training the neural network (600) using the validation database (515); Using the neural network (600) trained on the validation database (515) to predict the data in an independent validation cohort (516) to compute a first accuracy value; Using the neural network trained (600) on the knowledgebase (25, 517) to predict the data in the independent validation cohort (516) to compute a second accuracy value; Comparing the first and the second accuracy values to determine whether adding the subset of the image tiles (C) and the associated training abnormality degree values (a_w) improves the accuracy of the neural network (600) when trained with the updated knowledgebase (25, 507).
9. The method of claim 1, characterized by Storing the training abnormality degree value (a_w) for each of at least a subset of the image tiles (C) determined by a human expert (E) in a block (710, 710′, 710″) of a blockchain (700).
10. The method of any of claim 9, characterized in that the blockchain (700) indicates the image tiles (721, 721′, 721″), the associated training abnormality degree values (719, 719′, 719″) and the expert (714, 714″, 714″) determining the abnormality degree values (719, 719′, 719″) of the training data (509) stored in the knowledgebase (517).
11. The method of claim 9, characterized in that each block (710, 710′, 710″) of the blockchain (700) comprises a header (711, 711′, 711″) comprising a hash value (712, 712′, 712″) of the header of a previous block and a hash value of a the root node of a merkle tree (713, 713′, 713″), the merkle tree indicating all data stored in the knowledgebase (25, 517).
12. The method of claim 9, characterized in that each block (710, 710′, 710″) of the blockchain (700) comprises a data block (715, 715′, 715″) comprising a link to an associated image tile (717, 717′, 717″), a link to an associated whole slide image (718, 718′, 718″) and a training abnormality degree value (719, 719′, 719″) determined by the expert (E, E′, E″) and associated with the respective image tile (717, 717′, 717″).
13. A computer readable medium storing instructions that when executed by at least one processor cause the at least one processor to implement a method according to claim 1.
14. A distributed cancer analysis system (500), comprising the following components: a segmentation entity (511) adapted to receive a whole slide image (501), the whole slide image (501) depicting at least a portion of a cell, in particular a human cell; a computation entity (512) adapted to compute a degree of abnormality (519) for the whole slide image (501) using a neural network (600).
15. The distributed cancer analysis system (500) according to claim 14, characterized in that the computation entity is adapted to compute the degree of abnormality (17) for the whole slide image (11, w, 722) using a function, in particular a max-function, a statistical aggregation of local abnormality degree values (15, a_j, 519, 719, 719′, 719″) associated with the at least one images tile (13, 601, 721, 721′, 721″) and/or an average function dependent on local abnormality degree values (15, a_j, 519, 719, 719′, 719″) associated with the at least one image tile (13, 601, 721, 721′, 721″).
16. The distributed cancer analysis system (500) according to claim 14, characterized by: a knowledgebase (517) comprising training data (509) wherein the computation entity (512) is further adapted to train the neural network (600) using the training data (509).
17. The distributed cancer analysis system (500) according to claim 14, characterized by: a communication interface (514) adapted to distribute at least a subset of a plurality of image tiles (C) to at least one expert (E), wherein the segmentation entity (512) is further adapted to segment the whole slide image (501) into the plurality of images tiles (504)
18. The distributed cancer analysis system (500) according to claim 14, characterized by a priorization entity (513) adapted to determine a subset of the image tiles (C) to be transmitted to the at least one expert (E, E′, E″) by the communication interface (514) wherein the priorization entity (513) is further adapted to determine the subset of candidate image tiles (C) based on a computed priority value for each image tile (502).
19. The distributed cancer analysis system (500) according to claim 14, characterized by, a testing entity (518) adapted to: receive validation training data (506) of a validation database (515), the validation training data (506) comprising at least one validation set (505, 505′, 505″), the validation database (515) in particular including at least a subset of the knowledge base, determined by the at least one expert (E, E′, E″); training the neural network using the received validation data (506); using the neural network trained on the validation data to predict the data in an independent validation cohort to compute a first accuracy value; using the neural network trained on the knowledgebase to predict the data in an independent validation cohort to compute a first accuracy value; comparing the first and the second accuracy values to determine whether adding the subset of the image tiles (C) and associated training abnormality degree values (a_w) improves the accuracy of the neural network when trained with the updated knowledgebase (25, 507).
20. The distributed cancer analysis system (500) of claim 14, characterized by a blockchain (700) adapted to store the training abnormality degree value (719, 719′, 719″) for each of at least a subset of the image tiles (C) in a block (710, 710′, 710″), the training abnormality degree value determined by a human expert (E, E′, E″).
21-22. (canceled)
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0105] In the following, embodiments of the invention are described with respect to the figures, wherein
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112]
[0113]
[0114]
[0115]
DETAILED DESCRIPTION
[0116]
[0117] In a next step, the image tiles 13 are processed in a prediction phase 14, wherein a local abnormality degree value 15 is computed for each image tile. In the present embodiment, this local abnormality degree value 15 is computed by a convolutional neural network. Importantly, each image tile 13 may be processed in parallel. The architecture of the neural network will be explained in detail with respect to
[0118] Having determined a local abnormality degree value 15 for each image tile 13, a degree of abnormality 17 is computed during an evaluation phase 16. Thus, the abnormality degree value 17 is based on the plurality of image tiles 13. In the present embodiment, the degree of abnormality 17 is merely the maximum value of the different local abnormality degree values 15 for each image tile. This is due to the fact that for a human cell to be cancerous, it may be sufficient that a single part of the human cell shows a cancerous characteristic.
[0119]
[0120] In the embodiment shown in
[0121]
[0122]
[0123] For example, the neural network may predict a tuple (a_1, l_1)=(4, 30%). Thus, the neural network has predicted a Gleason grade of 4 and a likelihood of 30% for the image tile t_1. The same procedure is repeated for each image tile, resulting in an array with the size being equal to the number of image tiles t_1 to t_n generated during the partitioning phase 110.
[0124] A complete list is generated in the list creation phase 130, wherein all predicted abnormality and likelihood values are grouped together into a list L_w along with a pointer to their respective image tiles.
[0125]
[0126] Thus, for the above-mentioned tuple (4, 30%) the resultant priority value is computed as follows:
[0127] The same process is repeated for each tuple stored in L_w. The results are grouped together in a list of priority tuples S_w, wherein each tuple has the form (t_j, a_j, l_j, p(t_j,l_j)).
[0128] In a next candidate elimination phase 230, the list S_w is filtered, filtering out all tuples with a low priority. For example, all entries in the list S_w are eliminated which have a priority of lower than 0.5. The resultant list C is then distributed to experts E during a distribution phase 240. The experts E may be human experts in the field, for example pathologists. The human experts E may be located around the world and thus distribution during the distribution phase 240 may be done using an Internet or any other electronic communication means. Importantly, only the list C is distributed to the experts and not the entire list S_w.
[0129]
[0130]
[0131] Consequently, a first accuracy value is computed for the independent validation cohort using the neural network trained on the knowledge base 25. A second accuracy value is then computed for the neural network trained on the validation database VD. Finally, the first and the second accuracy values may be compared, indicating, which training data leads to better prediction results on the independent validation cohort. The accuracy may be computed as
[0132] Consequently, in the determination phase 420, it may be determined, which training data leads to better results. If the validation database VD leads to a better accuracy value, the method proceeds with the yes-branch, continuing with the network replacement phase 430. In the network replacement phase 430, the knowledge base 25 is replaced by the validation database VD. Also, the neural network used to compute the predictions in phase 120 is replaced by the neural network trained on the validation database VD. If the determination phase 420 finds that the knowledge base 25 leads to better results than the neural network trained on the validation database, the no-branch is used and the process ends.
[0133]
[0134] Based on the priority values, the priorization entity 513 determines a list of candidate image tiles 504 by comparing the computed priority values with a threshold value, e.g. 0.7 or 0.5. The list of candidate image tiles 504 is sent to a communication interface 514, which is communicatively connected to three experts E, E′, E″. The experts E, E′, E″ are located outside of the analysis system 510 and may be located around the world. Each expert E, E′, E″ processes the received list of candidate image tiles 504 to produce respective validation sets 505, 505′, 505″. That is, each expert E, E′, E″ validates the predicted abnormality degree values for the image tiles or changes the values and thus creates training abnormality degree values. Having reviewed the list of candidate image tiles 504, the experts E, E′, E″ send the results of the review process as validation sets 505, 505′, 505″ to the validation database 515. The validation database 515 comprises the training data 509 of the knowledge base 517 with the additional data obtained by the experts E, E′, E″.
[0135] The validation database 515 sends validation training data 506 to a testing entity 518 which is comprised in the analysis system 510. Using the validation training data 506, the testing entity 518 trains a neural network and uses the trained neural network to predict the data in an independent validation cohort 516. The independent validation cohort 516 also sends its validation data 508 to the testing entity 518. The testing entity 518 is further adapted to compute a first accuracy value for the neural network trained on the validation training data 506. The computed accuracy value is then compared to an accuracy value of the neural network trained on the training data 509 of the knowledge base 517. In case the accuracy value achieved the neural network trained on the validation training data 506 is greater than the accuracy value of the neural network trained on the training data 509 of the knowledge base, the validation training data 506 replaces the data in the knowledge base 517. Also, the neural network trained on the validation training data 506 is then used by the computation entity 512 to process image tiles.
[0136]
[0137] Moreover, a kernel size determines the size of the patch that is scanned by the kernel. Thus, depending on the size of the input image tile 601, the kernel size and the stride, the size of the feature maps 603 in the first convolutional layer is determined. Each feature map 603 represents a feature detector. For example, a first feature map may be adapted to detect corners. Consequently, the resultant feature map 603 is a map of corners detected in the input image tile 601. A second feature map may indicate edges.
[0138] In the next layer of the convolutional neural network 600, sub sampling 604 generates four second feature maps 605. In this layer, the feature maps 603 of the previous layer are subsampled in order to generate a more compact representation of the input image tile 601. This is in particular useful to reduce the size of the convolutional neural network in order to increase training and prediction speed. From the four second feature maps 605 onward, another convolution 606 generates a greater plurality of third feature maps 607 in the same fashion as described before. From the generated third feature maps 607 the output is input of a feedforward neural network, which is fully connected and comprises two layers 608 and 609 in the described embodiment.
[0139] Importantly, the last layer 609 of the neural network 600 is a so-called soft-max layer, wherein the input image tile 601 is classified into one of many classes.
[0140] Each layer in the convolutional neural network 600 is built from a great number of neurons, i.e. activation functions, having weights. Depending on the weight and value of an input, the output of the neuron is activated or left deactivated. Possible activation functions include for example a logit, arc tan, or Gaussian functions. The training of the neural network 600 is conducted using the back propagation algorithm and using training data to determine the weights associated with the activation functions.
[0141] Many different architectures of convolutional neural networks are possible to implement the inventive aspects of the present application. For example, the number of kernels may be varied in the different convolutional layers. Also, the number of layers can be varied. Possible architectures include the VGG-net, RES-net, general adversial networks, google LeNet with inception modules.
[0142] The training of the convolutional neural network may be conducted in a cloud service such that the computation is spread across multiple machines, using parallelism to increase the training speeds.
[0143]
[0144]
[0145] For example, block 710′ comprises a header 711′ and a data block 715′. The header 711′ comprises a hash value 712′, storing the hash value of the header of the previous block 710. As a result, the block that precedes the block 710′ in the blockchain 700 is uniquely identifiable. Moreover, header 711′ comprises a Merkle root value 713′. The Merkle root value 713′ is the hash value of the root node of a Merkle tree. The Merkle tree may be used to identify all training abnormality values used to train a neural network.
[0146] The data block 715′ comprises a link to a data block 716′, which is stored outside of the blockchain 700. Also, the data block 715′ stores a link to an image tile 717′, to which the abnormality degree value 719′ is associated to. In addition, the data block 715′ stores the training abnormality degree value 719′ as well as a link to the whole slide image 718′ of which the image tile 721 is a part of Even more, the data block 715′ stores a link to the expert E, E′, E″ that evaluated the linked image tile of the linked whole slide image and determined the abnormality degree value stored in the block 710′.
[0147] Furthermore, the blockchain 700 can be configured to only allow experts E, E′, E″ to insert data into the block chain 700, which received a proper education and which have the necessary qualifications. This can be implemented using scripting mechanisms of the blockchain technology. The scripts may define conditions that need to be fulfilled before an entry may be added to the blockchain. These mechanisms are also known as smart contracts.
[0148] It is worth pointing out that blockchain technology does not rely on a single central server but is a distributed data structure shared among all peers of a peer-to-peer network. Data is added to the blockchain using a trust mechanism, which is well-known in the art. As a result, all peers in the network usually accept the current longest chain as the most trustworthy. Moreover, the content of the blockchain 700 is publicly available and thus any user may identify all records that are stored in the blockchain 700. This allows a user/patient/doctor to review all the experts that have contributed to the blockchain 700 and their reviews of image tiles stored in the blockchain 700. This provides an unprecedented degree of transparency in the diagnostic process.
REFERENCE NUMERALS
[0149] 10 method to determine abnormality value [0150] 11 whole slide image [0151] 12 Segmentation phase [0152] 13 image tiles [0153] 14 prediction phase [0154] 15 local abnormality degree value for each tile [0155] 16 evaluation phase [0156] 17 degree of abnormality [0157] 20 cancer analysis system [0158] 21 whole slide image [0159] 22 segmentation entity [0160] 23 set of image tiles [0161] 24 computation entity [0162] 25 knowledgebase [0163] 26 training data [0164] 27 result [0165] 30 image processing entity [0166] 100 prediction phase [0167] 103 pairs of abnormality and likelihood [0168] 104 list of pairs L_w [0169] 110 partitioning phase [0170] 120 abnormality prediction phase [0171] 130 List creation [0172] 200 priorization phase [0173] 201 pairs of priorities and [0174] 210 Prioritize [0175] 220 sorting phase [0176] 230 candidate elimination phase [0177] 240 distribution phase [0178] 300 decision making phase [0179] 310 review phase [0180] 320 abnormality determination phase [0181] 330 KB extension phase [0182] 400 improvement phase [0183] 410 accuracy comparison phase [0184] 420 Determination phase [0185] 430 Network replacement phase [0186] 440 Leave network [0187] 500 DCAS [0188] 501 whole slide image [0189] 502 set of image tiles [0190] 503 list of abnormality degree and likelihood pairs [0191] 504 list of candidate image tiles [0192] 505, 505′, 505″ validation set [0193] 506 validation training data [0194] 507 independent validation data [0195] 508 validation data [0196] 509 training data [0197] 510 analysis system [0198] 511 segmentation entity [0199] 512 computation entity [0200] 513 priorization entity [0201] 514 communication interface [0202] 515 Validation Database [0203] 516 independent validation cohort [0204] 517 knowledgebase [0205] 518 testing entity [0206] 519 degree of abnormality [0207] 600 convolutional Neural Network [0208] 601 Input image tile [0209] 602 first convolution [0210] 603 eight first feature maps [0211] 604 subsampling [0212] 605 four second feature maps [0213] 606 second convolution [0214] 607 16 third feature maps [0215] 608 feed forward layer/fully connected layer [0216] 609 output layer [0217] 700 blockchain [0218] 710, 710′, 710″ block [0219] 711, 711′, 711″ header [0220] 712, 712′, 712″ hash value of previous header [0221] 713, 713′, 713″ Merkle root [0222] 714, 714′, 714″ WID hash [0223] 715, 715′, 715″ Data block [0224] 716, 716′, 716″ link to data block [0225] 717, 717′, 717″ link to image tile [0226] 718, 718′, 718″ link to whole slide image [0227] 719, 719′, 719″ abnormality degree [0228] 720, 720′, 720″ data block [0229] 721, 721′, 721″ image tile [0230] 722 whole slide image [0231] E, E′, E″ expert [0232] w whole slide image [0233] t_1, . . . , t_j . . . , t_n sequence of image tiles [0234] l_j likelihood [0235] a_j abnormality degree [0236] p priority [0237] L_w List of abnormal events for whole slide image [0238] S_w List [0239] C subset of S_w [0240] V list of experts assignments [0241] a_w final abnormality degree value [0242] VB Validation database