Automated identification of tumor buds

Abstract

Automated image analysis methods to identify and quantify tumor buds in a high resolution image of a section of a tumor that is stained using either pan-cytokeratin AE1/3 or hematoxylin and eosin (H&E) are disclosed. The methods may be used to aid and/or replace manual visual inspection for tumor buds and may be used to predict a clinically relevant outcome or treatment in some cases. The disclosed methods may be used for many different cancer types, such as colorectal cancer.

Claims

1. A method for identifying tumor buds in an image of a section of a tumor stained only with a immunohistochemical stain for pan-cytokeratin AE1/3, the method comprising: receiving the image, wherein the image is a magnified view of the section of the tumor stained only with the immunohistochemical stain for pan-cytokeratin AE1/3 and is comprised of pixels; segmenting the image to identify tissue areas, wherein the segmenting comprises thresholding the image to identify each pixel as corresponding to tissue or as not corresponding to tissue; determining a size of each identified tissue area; comparing the size of each identified tissue area to a lower threshold and an upper threshold to identify candidate areas in the image as tissue areas that have a size between the upper threshold and the lower threshold; and counting the number of cells in each candidate area of the image, wherein counting the number of cells comprises detecting nuclei in the candidate areas and counting nuclei in each candidate area to determine the number of cells in each candidate area; identifying candidate areas with nuclei; analyzing a shape and an intensity of the candidate areas with nuclei; and determining that the candidate area with nuclei is a tumor bud based the candidate area having between one and five cells and candidate area's shape and intensity.

2. The method according to claim 1, wherein the thresholding of the image to identify each pixel as corresponding to a tissue or as not corresponding to a tissue comprises: converting the image to a gray-scale image; and comparing each pixel's value to a threshold that is computed using Otsu's method; creating a binary image that indicates tissue areas based on the comparison; and determining areas in the image corresponding to tissues based on the binary image.

3. The method according to claim 2, wherein the thresholding of the image to identify each pixel as corresponding to a tissue or as not corresponding to a tissue further comprises: applying, after creating the binary image, morphological operations to the binary image.

4. The method according to claim 3, wherein the morphological operations comprise one or more of a dilation, a filling, and an erosion.

5. The method according to claim 1, wherein determining a size of each identified tissue area comprises: counting a number of pixels within in each identified tissue area.

6. The method according to claim 5, wherein the lower threshold is 16 pixels and the upper threshold is 4096 pixels.

7. The method according to claim 1, wherein the comparing the size of each identified tissue area to a lower threshold and an upper threshold to identify candidate areas in the image as tissue areas that have a size between the upper threshold and the lower threshold further comprises: determining that an identified tissue area having a size smaller than the lower threshold is noise caused by errors in the sectioning or staining of the tumor; and eliminating the noise from the candidate areas.

8. The method according to claim 1, wherein the comparing the size of each identified tissue area to a lower threshold and an upper threshold to identify candidate areas in the image as tissue areas that have a size between the upper threshold and the lower threshold further comprises: determining that an identified tissue area having a size larger than the upper threshold is fat; and eliminating the fat from the candidate areas.

9. The method according to claim 1, further comprising: identifying a candidate area that has more than five nuclei as part of a main tumor.

10. The method according to claim 1, further comprising: presenting results of the identified tumor buds, wherein the results comprise one or more of: an indication of each tumor bud in the image, and a count of the number of tumor buds.

11. The method of claim 10, wherein presenting results of the identified tumor buds further comprises presenting a distance of each identified tumor bud to a tumor front.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is an example image of a pan-cytokeratin AE1/3 stained section of a tumor under magnification.

(2) FIG. 2 is a flow chart depicting an exemplary method for identifying tumor buds in an image of a pan-cytokeratin AE1/3 stained section of a tumor.

(3) FIGS. 3A-C are images corresponding intermediate steps in the method of FIG. 1, wherein FIG. 3A is a binary image depicting tissue areas corresponding to the image of FIG. 1, FIG. 3B depicts detected nuclei within the tissue areas, and FIG. 3C is a binary image depicting identified tumor buds corresponding to the image of FIG. 1.

(4) FIG. 4 illustrates exemplary results of tumor bud identification, wherein the image of FIG. 1 is shown with indications of identified tumor buds.

(5) FIG. 5 is a flowchart depicting an exemplary method for identifying tumor buds in an image of an H&E stained section of a tumor.

DETAILED DESCRIPTION

(6) Recent findings have suggested the use of pan-cytokeratin (AE1/3) immunostaining for the evaluation of tumor budding is feasible in daily diagnostic practice [20]. A study by the Swiss Association of Gastrointestinal Pathology in 2016 found that tumor budding counts are three to six times greater upon pan-cytokeratin staining compared to the standard H&E staining. The same study also found that inter-observer reproducibility was markedly improved with pan-cytokeratin staining compared to H&E. Koelzer [20] and Rieger et al. [21][8] presented a comprehensive assessment of tumor budding by cytokeratin staining in colorectal cancer. Koelzer [20] also concluded that assessment of tumor budding on pan-cytokeratin slides is feasible in a large pathology institute and leads to expected associations with clinic-pathological features. AE1/3 specifically highlights tumor cells while de-emphasizing normal cells, making it very useful in computer assisted tumor budding assessment.

Tumor Bud Identification in Pan-Cytokeratin AE1/3 Images

(7) An image of a pan-cytokeratin AE1/3 stained section of a tumor is shown in FIG. 1. As shown, the cytoplasm of cells is stained brown (dark), nuclei are counterstained blue (light), and a background region is white. A method 200 for identifying tumor buds in the image (FIG. 1) generally consists of the operations of tissue segmentation to identify tissue areas 210, filtering the identified tissue areas by size to identify candidate areas 220, counting nuclei (i.e., cells) in the candidate areas 230, and identifying tumor buds based on the number of cells in the candidate areas 240.

(8) Tissue segmentation to identify tissue areas may use thresholding to distinguish nuclei tissue from the background and from debris (e.g., necrotic tissues). Additionally, noise and small spurious segments from debris may be removed in the segmentation process. This removal is especially important when debris or necrotic tissues are present in the image. This noise removal may use an adjustable threshold, but care should be taken that it is not selected too large because potential tumor buds may be removed.

(9) Segmentation may use an automatic thresholding in which the threshold is computed using Otsu's method. In a typical embodiment, initial parameters are provided to an Otsu thresholding algorithm. The initial parameters provided to the Otsu thresholding algorithm may include an Otsu threshold of t=64 pixels and an Otsu weight w=1.25. These parameters may be selected optimally as a study requires.

(10) Additionally, identifying tissue areas 200 may require morphological operations (e.g., dilation, filling, and erosion) to correct for thresholding errors and provide the identification of tissue areas.

(11) FIG. 3A illustrates the results of tissue segmentation (i.e., identification of tissue areas) for the image of FIG. 1. As shown, the results include a binary image, wherein pixels corresponding to tissues area white and pixels not corresponding to tissue (e.g., debris, background, etc.) are black. This binary image may be used to determine the areas in the image (FIG. 1) that correspond to tissues.

(12) The tissue areas may be filtered in terms of size to identify candidate areas likely to contain tumor buds (i.e., candidate areas). The size of each candidate areas should not be too small or too large. Accordingly, two threshold values n.sub.1 and n.sub.2 are used to identify candidate areas from the tissue areas. Regions smaller than n.sub.1 are likely to be noise (e.g., due to imperfect staining), while regions larger than n.sub.2 are likely to be another tissue component (e.g., fat). In a typical implementation, thresholds of n.sub.1=16 and n.sub.2=4096 pixels may be used to identify the candidate areas from the tissue areas. In the example shown in FIG. 3A, the tissue areas are all identified as candidate areas. These threshold values may be optimized based on the application.

(13) To identify tumor buds, cells within each candidate area may be counted and compared to a range. Because nuclei indicate a cell, nuclei detection forms the basis of cell counting. FIG. 3B shows the candidate areas of FIG. 3A with detected nuclei (shown as light gray).

(14) As mentioned previously, AE1/3 staining highlights the tumor budding cells while de-emphasizing non-epithelial cells, hence the detected nuclei from the operations of segmentation and nuclei detection are all tumor cells. After nuclei detection, cell counting (i.e., nuclei counting) within the candidate areas is performed and candidate areas with a cell count in a range of one to five cells may be identified as tumor buds. Additionally, tissue segments with more than five cells may be identified as part of the main tumor. The number of nuclei used to identify a tumor bud may be optimized based on the application.

(15) Tissue cells with no visible nuclei are not immediately classified tumor buds but are considered exceptions that require further analysis. Accordingly, tissue segments with no detected cells (i.e., nuclei), can be further analyzed in terms of a shape and intensity of the tissue. For example, tissue regions having no visible nuclei that also have a high average intensity and a high degree of circularity may be identified as tumor buds.

(16) FIG. 3C illustrates the tumor buds identified from an analysis of FIG. 3B. The tumor buds are candidate areas that meet the cell count and size/shape criteria described above. Using this binary image of tumor buds, results may be generated and presented.

(17) FIG. 4 illustrates the image of FIG. 1 with indications of tumor buds. Additional results may also be generated. The results may include a count of the number of tumor buds and the relative location of tumor buds. The results may be presented to a user and may guide the user in a visual analysis of the tumor section. Further, in some cases, the results may replace a manual analysis of the image (or slide) by a human.

(18) While AE1/3 staining facilitates tumor budding evaluation, it is not used routinely in evaluating colorectal cancer. College of American Pathologists Cancer Protocol and International Tumor Budding Consensus Conference (ITBCC, 2016) recommend the following [17]:

(19) “Tumor budding counts should be done on H&E sections. In cases of obscuring factors like inflammation, immunohistochemistry for keratin can be obtained to assess the advancing edge for tumor buds, but the scoring should be done on H&E sections.”

(20) Accordingly, the present disclosure also embraces a method for detecting tumor buds in images of hematoxylin and eosin (H&E) stained slides.

Tumor Bud Identification in Hematoxylin and Eosin (H&E) Images

(21) A relative distribution of normalized color features can be used to identify tumor bud regions with high sensitivity and relatively low specificity. The identification method 500 consists of initially training 510 a machine learning classifier to classify textural and spatial features that correspond to tumor buds in a set of training images. Each training image is a magnified image of an H&E stained section of a tumor with tumor buds. After the machine learning classifier is trained, the machine learning classifier may be applied 540 to determine if textural and spatial features extracted 530, from an image received 520 for tumor bud identification, correspond to tumor buds.

(22) In the method, color deconvolution is first applied to separate the hematoxylin from the eosin images. Textural and spatial features (i.e., represented as quantitative vectors) are then be extracted and input to the machine learning classifier (e.g., a support vector machine or a neural network) to detect tumor bud regions. Local binary patterns (LBP), histogram of oriented gradients (HOG) and the multiresolution Shearlet transform (i.e., an extension of wavelet transform) are among the suitable techniques to be used to extract textural and shape features from the histology images. Deep learning approach may also be used depending on the number of training images available. For example, if the number of training samples are small deep learning may not provide optimal results. In some implementations, the candidate regions are further refined by analyzing the area and distance of the regions to the tumor front.

Results from an Example Implementation of the Method for Identifying Tumor Buds in AE1/3 Images

(23) Fifteen cases from Ohio Colorectal Cancer Prevention Initiative (OCCPI) cohort have been identified for digital image analysis, double stained with both AE1/3 and H&E. High power field images (40× magnification) with varying number of tumor buds (minimum 1 tumor bud region and maximum up to 20 tumor bud regions) have been captured. Five HPF images with comprehensive ground truths are selected, each from three of the 15 AE1/3 cases. Each case corresponds to high power field image at 400 times magnification, with varying number of tumor buds (i.e., minimum 1 tumor bud region and maximum up to 18 tumor bud regions). The ground truth for each case was provided by the collaborating pathologists. The ground truth was annotated by cross-marking with their corresponding H&E images. Table 1 shows the detail summary of the experimental results.

(24) TABLE-US-00001 TABLE 1 Summary of Experimental Results Images # TB TP FN FP Img01 5 5 0 2 Img02 1 1 0 0 Img03 5 4 1 2 Img04 2 2 0 0 Img05 2 2 0 0 Img06 4 3 1 0 Img07 6 6 0 4 Img08 2 2 0 2 Img09 5 5 0 0 Img10 18 17 1 4 Img11 7 6 1 0 Img12 5 4 1 0 Img13 5 5 0 0 Img14 6 5 1 0 Img15 8 8 0 0 Total 81 75 6 14

(25) The first column in Table 1 lists the ID of the HPF cases, the second column shows the total number of tumor budding regions marked by the pathologists, while columns three to five shows the true positive (TP), false negative (FN), and false positive (FP) of the proposed tumor bud detection system.

(26) From the results, a sensitivity of 92.6% and specificity of 84.3% may be computed for ground truth. These results demonstrate the reliability of the proposed automated system.

(27) The techniques disclosed may reveal associations and correlations with a variety of clinically relevant outcomes. Accordingly, it is envisioned that classification and clustering tools (heatmaps, etc.) may be used for this purpose.

(28) Tumor recurrence, tumor progression, and the association with selected features/clinical covariates and or groups of features may be modeled using logistic regression or Cox regression models (with random subject effects). Associations between a time to recurrence and a progression of different features or feature clusters may be revealed using Cox regression models.

(29) Predictive models can be developed for time to recurrence and time to progression. For example, predictive models may be developed through a stepwise approach using a Bayesian information criterion (BIC) or an Akaike information criterion (AIC). Alternatively, predictive models may be developed using a K-fold or a penalized likelihood approach.

(30) Logistic regression models can be used to study association with tumor budding scores. Because the number of features is often large compared to the number of samples, feature selection methods can be utilized to control the number of false positives. The significance level can be adjusted by controlling the mean number of false positives [22, 23].

(31) Multivariate prediction models, using features identified by above analyses, can be developed incorporating important covariates, such as age, race, tumor characteristics, or other known prognostic factors. To overcome the over-fitting problem with a large number of features penalized likelihood maximization (e.g., least absolute shrinkage and selection operator (LASSO)) and penalized risk minimization approaches can be used together with cross-validation methods for building prediction models of clinical outcomes.

(32) In the specification and/or figures, typical embodiments have been disclosed. The present disclosure is not limited to such exemplary embodiments. The use of the term “and/or” includes any and all combinations of one or more of the associated listed items. The figures are schematic representations and so are not necessarily drawn to scale. Unless otherwise noted, specific terms have been used in a generic and descriptive sense and not for purposes of limitation.

(33) Those skilled in the art will also appreciate that various adaptations and modifications of the preferred and alternative embodiments described above can be configured without departing from the scope and spirit of the disclosure. Therefore, it is to be understood that, within the scope of the appended claims, the disclosure may be practiced other than as specifically described herein.

REFERENCES

(34) [1] M. Boxberg, M. Jesinghaus, C. Dorfner, C. Mogler, E. Drecoll, A. Warth, et al., “Tumour budding activity and cell nest size determine patient outcome in oral squamous cell carcinoma: proposal for an adjusted grading system,” Histopathology, vol. 70, pp. 1125-1137, 2017. [2] P. V. Angadi, P. V. Patil, K. Hallikeri, M. Mallapur, S. Hallikerimath, and A. D. Kale, “Tumor budding is an independent prognostic factor for prediction of lymph node metastasis in oral squamous cell carcinoma,” International journal of surgical pathology, vol. 23, pp. 102-110, 2015. [3] X. Li, B. Wei, C. Sonmez, Z. Li, and L. Peng, “High tumor budding count is associated with adverse clinicopathologic features and poor prognosis in breast carcinoma,” Human Pathology, 2017. [4] M. Jesinghaus, M. Boxberg, B. Konukiewitz, J. Slotta-Huspenina, A. M. Schlitter, K. Steiger, et al., “A Novel Grading System Based on Tumor Budding and Cell Nest Size Is a Strong Predictor of Patient Outcome in Esophageal Squamous Cell Carcinoma,” The American Journal of Surgical Pathology, vol. 41, pp. 1112-1120, 2017. [5] K. Che, Y. Zhao, X. Qu, Z. Pang, Y. Ni, T. Zhang, et al., “Prognostic significance of tumor budding and single cell invasion in gastric adenocarcinoma,” OncoTargets and therapy, vol. 10, p. 1039, 2017. [6] S. Olsen, J. Linda, R. C. Fields, Y. Yan, and I. Nalbantoglu, “Tumor Budding in Intestinal Type Gastric Adenocarcinoma is Associated with Nodal Metastasis and Recurrence,” Human Pathology, 2017. [7] B. Huang, J. Cai, X. Xu, S. Guo, and Z. Wang, “High-Grade Tumor Budding Stratifies Early-Stage Cervical Cancer with Recurrence Risk,” PloS one, vol. 11, p. e0166311, 2016. [8] N. Satabongkoch, S. Khunamornpong, T. Pongsuvareeyakul, J. Settakorn, K. Sukpan, A. Soongkhaw, et al., “Prognostic Value of Tumor Budding in Early-Stage Cervical Adenocarcinomas,” Asian Pacific journal of cancer prevention: APJCP, vol. 18, p. 1717, 2017. [9] Y. H. Lai, L. C. Wu, P. S. Li, W. H. Wu, S. B. Yang, P. Xia, et al., “Tumour budding is a reproducible index for risk stratification of patients with stage II colon cancer,” Colorectal Disease, vol. 16, pp. 259-264, 2014. [10] A. Mezheyeuski, I. Hrynchyk, M. Karlberg, A. Portyanko, L. Egevad, P. Ragnhammar, et al., “Image analysis-derived metrics of histomorphological complexity predicts prognosis and treatment response in stage II-III colon cancer,” Scientific reports, vol. 6, p. 36149, 2016. [11] I. Zlobec, M. Hadrich, H. Dawson, V. Koelzer, M. Borner, M. Mallaev, et al., “Intratumoural budding (ITB) in preoperative biopsies predicts the presence of lymph node and distant metastases in colon and rectal cancer patients,” British journal of cancer, vol. 110, p. 1008, 2014. [12] R. Cappellesso, C. Luchini, N. Veronese, M. L. Mele, E. Rosa-Rizzotto, E. Guido, et al., “Tumor Budding as a Risk Factor for Nodal Metastasis in Pt1 Colorectal Cancers: A Meta-Analysis,” Human Pathology, 2017. [13] R. P. Graham, R. A. Vierkant, L. S. Tillmans, A. H. Wang, P. W. Laird, D. J. Weisenberger, et al., “Tumor Budding in Colorectal Carcinoma: Confirmation of Prognostic Significance and Histologic Cutoff in a Population-based Cohort,” The American journal of surgical pathology, vol. 39, pp. 1340-1346, 2015. [14] V. H. Koelzer, I. Zlobec, M. D. Berger, G. Cathomas, H. Dawson, K. Dirschmid, et al., “Tumor budding in colorectal cancer revisited: results of a multicenter interobserver study,” Virchows Archiv, vol. 466, pp. 485-493, 2015. [15] V. H. Koelzer, I. Zlobec, and A. Lugli, “Tumor budding in colorectal cancer—ready for diagnostic practice,” Human pathology, vol. 47, pp. 4-19, 2016. [16] F. Petrelli, E. Pezzica, M. Cabiddu, A. Coinu, K. Borgonovo, M. Ghilardi, et al., “Tumour budding and survival in stage II colorectal cancer: A systematic review and pooled analysis,” Journal of gastrointestinal cancer, vol. 46, pp. 212-218, 2015. [17] A. Lugli, R. Kirsch, Y. Ajioka, F. Bosman, G. Cathomas, H. Dawson, et al., “Recommendations for reporting tumor budding in colorectal cancer based on the International Tumor Budding Consensus Conference (ITBCC) 2016,” Modern Pathology, 2017. [18] F. Grizzi, G. Celesti, G. Basso, and L. Laghi, “Tumor budding as a potential histopathological biomarker in colorectal cancer: Hype or hope?,” World Journal of Gastroenterology: WJG, vol. 18, p. 6532, 2012. [19] R. L. Siegel, K. D. Miller, S. A. Fedewa, D. J. Ahnen, R. G. Meester, A. Barzi, et al., “Colorectal cancer statistics, 2017,” CA: a cancer journal for clinicians, vol. 67, pp. 177-193, 2017. [20] V. H. Koelzer, N. Assarzadegan, H. Dawson, B. Mitrovic, A. Grin, D. E. Messenger, et al., “Cytokeratin-based assessment of tumour budding in colorectal cancer: Analysis in stage II patients and prospective diagnostic experience,” The Journal of Pathology: Clinical Research. [21] G. Rieger, V. H. Koelzer, H. E. Dawson, M. D. Berger, M. Hädrich, D. Inderbitzin, et al., “Comprehensive assessment of tumour budding by cytokeratin staining in colorectal cancer,” Histopathology, vol. 70, pp. 1044-1051, 2017. [22] A. Gordon, G. Glazko, X. Qiu, and A. Yakovlev, “Control of the mean number of false discoveries, Bonferroni and stability of multiple testing,” The Annals of Applied Statistics, pp. 179-190, 2007. [23] E. L. Korn and B. Freidlin, “A note on controlling the number of false positives,” Biometrics, vol. 64, pp. 227-231, 2008.