SYSTEM FOR PROCESSING A WHOLE SLIDE IMAGE, WSI, OF A BIOPSY

Abstract

A system and method is provided for processing a whole slide image, WSI, of a biopsy such as a core needle biopsy or a vacuum assisted biopsy. A skeleton of the shape of the detected tissue is created and a skeleton path is determined. An image is generated comprising a line representing the skeleton path with different line representations along the line representing different tissue pathologies. A pathology summary for the overall line is also prepared. This provides the information desired by a pathologist in a most convenient format and representation.

Claims

1. A system for processing a whole slide image, WSI, of a biopsy, comprising a processor which is adapted to: detect tissue present in the WSI relating to one biopsy; create a skeleton of the shape of the detected tissue; select a skeleton path as the longest continuous path through the tissue; determine tissue pathologies of the detected tissue; and generate an image comprising: a line representing the skeleton path with different line representations along the line representing different tissue pathologies; and a pathology summary for the overall line, wherein the pathology summary comprises a biopsy length and a fraction of a line occupied by a particular type of tissue pathology.

2. The system as claimed in claim 1, wherein the different representations comprise different colors or line thickness.

3. (canceled)

4. The system as claimed in claim 3, wherein the particular type of tissue pathology is tumor tissue.

5. The system as claimed in claim 1, wherein generating the line comprises overlaying the detected tissue pathologies over the skeleton path, and selecting a representative tissue pathology for a point along the skeleton path.

6. The system as claimed in claim 5, wherein the representative tissue pathology is selected to be tumor tissue if tumor tissue is present at or perpendicular to the respective point along the line.

7. The system as claimed in claim 1, comprising a first machine learning algorithm (ML1) for tissue detection.

8. The system as claimed in claim 7, comprising one or more second machine learning algorithms (ML2) for detecting tissue pathologies.

9. The system as claimed in claim 1, comprising a display for displaying the image, with the pathology summary alongside the line.

10. A computer-implemented method for processing a whole slide image, WSI, of a biopsy, comprising: detecting tissue present in the WSI relating to one biopsy; creating a skeleton of the shape of the detected tissue; selecting a skeleton path as the longest continuous path through the tissue; determining tissue pathologies of the detected tissue; and generating an image comprising: a line representing the skeleton path with different line representations along the line representing different tissue pathologies; and a pathology summary for the overall line, wherein the pathology summary comprises a biopsy length and a fraction of the line occupied by a particular type of tissue pathology.

11. The method as claimed in claim 10, comprising generating an image with different colors of the line representing different tissue pathologies.

12. (canceled)

13. The method as claimed in claim 10, comprising generating the image by overlaying or juxtaposing the determined tissue pathologies over the skeleton path, and selecting an associated tissue pathology for a point along the skeleton path.

14. The method as claimed in claim 10, comprising displaying the image, with the pathology summary alongside the line.

15. A non-transitory computer readable medium comprising a computer program such that when said program is run on a computer, the computer: detects tissue present in the WSI relating to one biopsy; creates a skeleton of the shape of the detected tissue; selects a skeleton path as the longest continuous path through the tissue; determines tissue pathologies of the detected tissue; and generates an image comprising: a line representing the skeleton path with different line representations along the line representing different tissue pathologies; and a pathology summary for the overall line, wherein the pathology summary comprises a biopsy length and a fraction of the line occupied by a particular type of tissue pathology.

16. The computer readable medium of claim 15, wherein the computer generates an image with different colors of the line representing different tissue pathologies.

17. The computer readable medium of claim 15, wherein the computer generates the image by overlaying or juxtaposing the determined tissue pathologies over the skeleton path, and selecting an associated tissue pathology for a point along the skeleton path.

18. The computer readable medium of claim 15, wherein the computer displays the image, with the pathology summary alongside the line.

19. The computer readable medium of claim 15, wherein when the computer generates the line the computer overlays the detected tissue pathologies over the skeleton path, and selects a representative tissue pathology for a point along the skeleton path.

20. The computer readable medium of claim 19, wherein the representative tissue pathology is selected to be tumor tissue if tumor tissue is present at or perpendicular to the respective point along the line.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0040] For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0041] FIG. 1 shows a whole slide image of a single core needle biopsy and an area outline;

[0042] FIG. 2 shows the area outline and a skeleton;

[0043] FIG. 3 shows the skeleton and a trimmed skeleton forming a skeleton path;

[0044] FIG. 4 shows the skeleton path and the skeleton path overlaid within the area outline;

[0045] FIG. 5 shows the area outline with the skeleton path and the skeleton path combined with the image of the pathology;

[0046] FIG. 6 shows the skeleton path combined with the image of the pathology and shows how the pathology information is projected onto the skeleton path;

[0047] FIG. 7 shows a pathology summary for the overall line which can be derived from analysis of the information of FIG. 6;

[0048] FIG. 8 shows an example of a WSI with multiple samples;

[0049] FIG. 9 shows how the image of FIG. 8 is modified using the approach of the invention described above applied to a first sample;

[0050] FIG. 10 shows a system for processing a whole slide image of a biopsy; and

[0051] FIG. 11 shows a computer-implemented method for processing a whole slide image of a biopsy.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] The invention will be described with reference to the Figures.

[0053] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. It should be understood that the Figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the Figures to indicate the same or similar parts.

[0054] The invention provides a system and method for processing a whole slide image, WSI, of a biopsy such as a core needle biopsy or a vacuum assisted biopsy. A skeleton of the shape of the detected tissue is created and a skeleton path is determined. An image is generated comprising a line representing the skeleton path with different line representations along the line representing different tissue pathologies. A pathology summary for the overall line is also prepared. This provides the information desired by a pathologist in a most convenient format and representation.

[0055] The invention relates to elongate biopsies wherein pathology information at different positions along the elongate length of the biopsy is of interest. As one example of the type of biopsy to which the invention may be applied, FIG. 1 shows on the left a whole slide image 10, WSI, of a single core needle biopsy. There may be multiple biopsies in one WSI, but the invention will be illustrated with a single biopsy in the WSI.

[0056] The steps carried out in the processing of the WSI will be discussed. These steps are carried out as image analysis steps of the WSI.

[0057] The first step is to detect all tissue present on the WSI. This creates an area outline 12 as shown in the right image in FIG. 1. If there are multiple biopsies, separate tissue regions are identified that correspond to individual biopsies, equivalent to independent samples.

[0058] For the identified biopsies, a skeleton 20 of the biopsy is identified as shown in FIG. 2. The left image of FIG. 2 shows the area outline 12 and the right image shows the resulting skeleton 20. The skeleton is a series of nodes and vertices joining the nodes which represent identifiable branches in the shape of the tissue sample.

[0059] The skeleton is then trimmed, to remove any branches of the skeleton that differ from the longest possible path. The left image of FIG. 3 shows the skeleton 20 and the right image shows the trimmed skeleton 30. The trimmed skeleton defines a skeleton path 32 which is the longest continuous path through the tissue. It follows a median line, i.e. each point along the skeleton path 32 is at the center of the width of the sample at that point along the length of the sample.

[0060] The skeleton path 32 thus defines the full extent of the biopsy.

[0061] The left image of FIG. 4 shows the skeleton path 32 and the right image shows the skeleton path 32 overlaid within the area outline 12.

[0062] The relevant pathology or pathologies in each sample (i.e. within the area outline 12) is then identified.

[0063] The left image of FIG. 5 shows the area outline 12 with the skeleton path 32 and the right image shows the skeleton path 32 combined with the image 50 of the pathology. Thus, the points along the skeleton path 32 are aligned with the associated pathology information.

[0064] The left image of FIG. 6 shows the skeleton path combined with the image 50 of the pathology and the right image shows how the pathology information is projected onto the skeleton path.

[0065] The result is a line 60 representing the skeleton path but with different line representations along the line representing different tissue pathologies. For example, the regions 62 may correspond to areas of tumor tissue (e.g. indicated red) whereas the other regions are free of tumor tissue (e.g. indicated green).

[0066] The determination of whether tumor tissue or normal tissue is represented along the line may be based on a most sensitive detection approach, by which any tumor tissue detected along a path perpendicular to a point along the line 60 results in that point indicating tumor tissue. If there are multiple pathologies being detected, a weighting scheme may determine which pathology is identified at the point along the line 60.

[0067] FIG. 7 shows a pathology summary for the overall line which can be derived from analysis of the information of FIG. 6. This shows the percentage coverage (as a percentage of the line length, 80% in this example), the total biopsy length (7 mm in this example) and the total corresponding physical length of the tumor tissue (5.6 mm in this example).

[0068] This example has only two classes (tumor or normal), but there may be more.

[0069] FIG. 8 shows an example of a WSI with multiple samples.

[0070] FIG. 9 shows how the image is modified using the approach described above applied to a first sample 90 within the WSI. This image is displayed on a display device.

[0071] The line 60 is overlaid over the WSI. It has different colors or different thickness (or other differences such as markers, dot patterns etc.) along its length to represent the different tissue pathologies. In this example the area outline 12 is also represented.

[0072] The pathology summary is displayed alongside the line, for example tumor lengths 92 for individual tumor portions along the line length as well as an overall summary 94 (such as in FIG. 7).

[0073] The pathologist may zoom in to inspect a particular area of the WSI. The line 60 may then, for example, disappear so that the inspection is not obstructed.

[0074] FIG. 10 shows a system for processing a whole slide image, WSI, of a biopsy such as a core needle biopsy. The system comprises a processor 100 which is adapted to carry out the analysis steps explained above. This involves detecting tissue present in the WSI, creating a skeleton of the shape of the detected tissue, selecting a skeleton path as the longest continuous path through the tissue, determining tissue pathologies of the detected tissue and generating an image.

[0075] The image is displayed on a display device 102. It includes the original WSI with the addition of a line representing the skeleton path with different line representations along the line representing different tissue pathologies. A pathology summary for the overall line is also displayed.

[0076] A first machine learning algorithm MLA1 is for example used for the tissue detection step. Such algorithms are known.

[0077] By way of example, reference is made to the article “tissueloc: Whole slide digital pathology image tissue localization” of Pingjun Chen et. al., Department of Biomedical Engineering, University of Florida DOI: 10.21105/joss.01148.

[0078] One or more second machine learning algorithms MLA2 are for example used for detecting tissue pathologies.

[0079] By way of example, reference is made to the article “Clinical-grade computational pathology using weakly supervised deep learning on whole slide images” of Gabriele Campanella et. al., Nature Medicine volume 25, pages 1301-1309(2019).

[0080] Further reference is made to the article “Automated deep-learning system for Gleason grading of prostate cancer using biopsies: a diagnostic study”, of Wouter Bulten et. al., DOI:https://doi.org/10.1016/S1470-2045(19)30739-9.

[0081] Algorithms are also well known for skeleton identification from shapes in images. EP 2 772 882 for example discloses the creation of a skeleton from a biopsy image as well as a length determination from the skeleton.

[0082] FIG. 11 shows a computer-implemented method for processing a whole slide image, WSI, of biopsy. The method comprises:

[0083] in step 110, detecting tissue present in the WSI relating to one biopsy;

[0084] in step 112, creating a skeleton of the shape of the detected tissue;

[0085] in step 114, selecting a skeleton path as the longest continuous path through the tissue;

[0086] in step 116, determining tissue pathologies of the detected tissue; and

[0087] in step 118, generating an image for display.

[0088] The image is generated by creating in step 120 a line representing the skeleton path with different line representations along the line representing different tissue pathologies and in step 122 creating a pathology summary for the overall line.

[0089] As discussed above, the system makes use of a processor to perform the data and image processing. The processor can be implemented in numerous ways, with software and/or hardware, to perform the various functions required. The processor typically employs one or more microprocessors that may be programmed using software (e.g., microcode) to perform the required functions. The processor may be implemented as a combination of dedicated hardware to perform some functions and one or more programmed microprocessors and associated circuitry to perform other functions.

[0090] Examples of circuitry that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0091] In various implementations, the processor may be associated with one or more storage media such as volatile and non-volatile computer memory such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and/or controllers, perform the required functions. Various storage media may be fixed within a processor or controller or may be transportable, such that the one or more programs stored thereon can be loaded into a processor.

[0092] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality.

[0093] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0094] A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. (optional)

[0095] If the term “adapted to” is used in the claims or description, it is noted the term “adapted to” is intended to be equivalent to the term “configured to”.

[0096] Any reference signs in the claims should not be construed as limiting the scope.