A SYSTEM AND METHOD FOR CLASSIFYING IMAGES OF RETINA OF EYES OF SUBJECTS

Abstract

The invention relates to a computing system and a computer-implemented method for classifying images of retina of eyes of subjects. A captured image of a retina is processed to obtain a plurality of different segmented images each having different selected portions of the captured image using different selection rules. The multiple segmented images are provided to respective dedicated machine learning models to output an image classification based on the respective segmented images provided as input. An ensemble classification is determined based on the multiple classifications obtained by means of the multiple trained machine learning models.

Claims

1. A computing system configured to classify images of retina of eyes of subjects, the system including one or more hardware computer processors, and one or more storage devices configured to store software instructions configured for execution by the one or more hardware computer processors in order to cause the system to: receiving an initial image of a retina captured by means of an imaging unit; processing the initial image of the retina in order to obtain at least a first segmented image and a second segmented image different from the first segmented image, wherein the first segmented image only includes a first selected portion of the captured image of the retina by employing a first selection rule, and wherein the second segmented image only includes a second selected portion of the captured image of the retina by employing a second selection rule, the first and second selection rules being different with respect to each other; providing at least a first trained machine learning model and a second trained machine learning model each configured to output an image classification based on an input image, the first and second machine learning models being different with respect to each other, and wherein the first machine learning model is trained using segmented images with only selected portions obtained by employing the first selection rule, and wherein the second machine learning model is trained using segmented images with only selected portions obtained by employing the second selection rule; providing the first segmented image with the first selected portion to the first machine learning model as input image in order to obtain a first classification, and providing the second segmented image with the second selection portion as input image to the second machine learning model as input image in order to obtain a second classification; and determining an ensemble classification based on at least the first classification and the second classification.

2. System according to claim 1, wherein the first and second selection rules are configured such that the first and second selected portions of the captured image have no overlap.

3. System according to claim 1 or 2, wherein the second selection rule is configured to provide at least partially inverted selection with respect to the first selection rule.

4. System according to claim 1, 2 or 3, wherein the first selection rule is based on identification of an optic nerve head of the eye, and wherein the second selection rule is based on an identification of blood vessels of the eye.

5. System according to any one of the preceding claims, wherein the first selection rule is configured to selectively exclude a region covering an identified optic nerve head of the eye in the captured initial image, wherein the second selection rule is configured to selectively include a region covering the identified optic nerve head of the eye in the captured initial image.

6. System according to any one of the preceding claims 1-4, wherein the first selection rule is configured to selectively exclude a region covering identified blood vessels of the eye in the captured initial image, wherein the second selection rule is configured to selectively include a region covering the identified blood vessels of the eye in the captured initial image.

7. System according to any one of the preceding claims, wherein the initial image of the retina is processed in order to obtain one or more further segmented images, wherein each of the one or more further segmented images only includes a further selected portion of the captured image of the retina by employing a further selection rule, the further selection rule being different with respect to other selection rules being employed; wherein one or more further trained machine learning models are provided configured to output an image classification based on an input image, each of the one or more further trained machine learning models being different with respect to other trained machine learning models being provided, wherein each of the one or more further machine learning models is trained using further segmented images with only further selected portions obtained by employing the further selection rule; wherein the one or more further segmented images are provided to the respective further machine learning models as input image in order to obtain one or more respective further classifications; wherein the ensemble classification is further based on the one or more further classifications.

8. System according to any one of the preceding claims, wherein at least four different trained machine learning models are employed each configured to receive a respective different segmented image obtained by employing different respective selection rules based on one or more eye features in the captured image.

9. System according to claim 7 or 8, wherein at least two selection rules are employed based on at least two of: an exclusion of region covering an identified optic disc in the captured initial image; an exclusion of region covering identified blood vessels in the captured initial image; an inclusion of region only covering an identified optic disc in the captured initial image; and an inclusion of region only covering identified blood vessels in the captured initial image.

10. System according to any one of the preceding claims, wherein segmentation in segmented images is performed by removing image data within at least one segment area of the captured image of the eye, the at least one segment area covering parts of the eye to be excluded.

11. System according to any one of the preceding claims, wherein removing of image data is performed by applying a patch over the at least one segment area.

12. System according to claim 10 or 11, wherein the image data within the segment area is removed by setting at least one of a normal distribution, a random distribution, or a uniform distribution of pixel values within the at least one segment area of the captured image.

13. System according to any one of the preceding claims, wherein the unsegmented captured image is further provided as input to the one or more trained machine learning models for classifying the eye.

14. The system according to any one of the preceding claims, wherein the classification is usable to infer or further analyze a condition of the subjects.

15. A computer-implemented method of classifying images of retina of eyes of subjects, the method comprising operating one or more hardware processors to: receiving an initial image of a retina captured by means of an imaging unit; processing the initial image of the retina in order to obtain at least a first segmented image and a second segmented image different from the first segmented image, wherein the first segmented image only includes a first selected portion of the captured image of the retina by employing a first selection rule, and wherein the second segmented image only includes a second selected portion of the captured image of the retina by employing a second selection rule, the first and second selection rules being different with respect to each other; providing at least a first trained machine learning model and a second trained machine learning model each configured to output an image classification based on an input image, the first and second machine learning models being different with respect to each other, and wherein the first machine learning model is trained using segmented images with only selected portions obtained by employing the first selection rule, and wherein the second machine learning model is trained using segmented images with only selected portions obtained by employing the second selection rule; providing the first segmented image with the first selected portion to the first machine learning model as input image in order to obtain a first classification, and providing the second segmented image with the second selection portion as input image to the second machine learning model as input image in order to obtain a second classification; and determining an ensemble classification based on at least the first classification and the second classification.

Description

BRIEF DESCRIPTION OF THE DRAWING

[0099] The invention will further be elucidated on the basis of exemplary embodiments which are represented in a drawing. The exemplary embodiments are given by way of non-limitative illustration. It is noted that the figures are only schematic representations of embodiments of the invention that are given by way of non-limiting example.

[0100] In the drawing:

[0101] FIG. 1 shows a captured image of a retina after contrast enhancement;

[0102] FIGS. 2A, 2B, and 2C show segmented images based on a captured image of a retina;

[0103] FIGS. 3A and 3B shows an unsegmented and a segmented image of a retina, respectively;

[0104] FIG. 4 shows a preprocessed captured image of a retina;

[0105] FIG. 5 shows a schematic diagram of an embodiment of a method;

[0106] FIG. 6 shows a schematic diagram of an embodiment of a method;

[0107] FIG. 7 shows a schematic diagram of an embodiment of a method;

[0108] FIG. 8 shows a schematic diagram of an embodiment of a method; and

[0109] FIG. 9 shows a schematic diagram of an embodiment of a method.

DETAILED DESCRIPTION

[0110] FIG. 1 shows a retina image 1 of an eye (after contrast enhancement). This image can be used by the computing system configured to classify images of retina of eyes of subjects. The system can be configured to receiving the image 1 of the retina captured by means of an imaging unit. Different types of hardware components can be used as the imaging unit. Furthermore, the system can be configured to segmenting the captured image 1 of the retina in order to obtain at least one segmented image. The at least one segmented image can be provided as input to one or more trained machine learning models configured to classify the eye. For example, the classification can be usable to infer or further analyze a condition of subjects.

[0111] In the captured retina image 1, an optic nerve head 3 and blood vessels 5 are distinguishable. These features can be seen as landmarks of the retina. Segmentation of the retina image 1 can be performed based on these landmarks, for instance including or excluding these landmarks. It is also envisaged that other landmarks are used for segmentation of the retina image 1. In some examples, some regions of the retina image are segmented. The segmentation can be performed for instance by neutralizing relevant information within segment areas, for instance by applying a uniform pixel color, random pixel values, etc.

[0112] A detection algorithm may be employed configured to recognize the segment to be excluded (cf. clipping). The segment can be excluded by neutralizing image data within a segment area. By excluding the segment of the retina image, the remaining information in the image can be used for making the prediction. For instance, the optic nerve may be excluded in a segmented image. The prediction of the disease/condition can then be performed using one or more machine learning models with the segmented image and the unsegmented image. Additionally or alternatively, this can be performed for the blood vessels, and/or other landmarks/features which can be identified in the captured retina image.

[0113] Optionally, information covering at least a part of one landmark is neutralized (e.g. deleted) or replaced in the segmented image. The landmark may form a structure or characteristic of the retina which is recognizable by processing of the captured image. The landmark may for instance be an optic nerve, blood vessel structure, etc. The landmark may be covered (for neutralization of data) by using different shapes, for example a circle, rectangular, polygon, ellipse, irregular shape, etc.

[0114] FIGS. 2A, 2B, and 2C show segmented images 1′. In these examples, the captured image 1 as shown in FIG. 1 is segmented. In these examples, the segmented images are segmented to exclude an optic nerve head 3 of the retina in the captured image. However, it is also possible to perform segmentation based on other landmarks identified in the captured image of the retina of the eye. In this example, segmentation in the segmented images is performed by neutralizing (cf. removing) image data within at least one segment area 7 of the captured image 1 of the retina/eye, the at least one segment area 7 covering parts of the eye to be excluded as relevant input for the one or more trained neural networks. This can be performed in various ways. For instance, in FIG. 2A, a segment area 7 substantially covers the optic nerve head with a uniform pixel color value (e.g. grey). The segment area 7 has a rectangular shape in FIG. 2A. The segment area 7 in FIG. 2B has a circular shape. Other shapes are also possible. It is also possible to use other pixel color values for neutralizing relevant image data within the part of the captured image covered by the segment area 7. For example, the pixel color values within the segment area 7 are set to black in FIG. 2B. Instead of employing uniform pixel color distributions, it is also possible to use color gradients, random colors (noise), patterns, or the like. For example, the segment area 7 in FIG. 2C has a regular pattern.

[0115] There can be separate information in the optical disc and the vessel structure in the retina image. The separate information in different landmarks/features of the retina can be separately assessed by trained machine learning models, enabling a better prediction. For example, in the captured retina image 1, the vessels may be segmented away (information in the regions covering the vessel structure may be removed by for instance employing a uniform color). The vessel structure can also be fed separately to the machine learning model. It will be appreciated that segmenting a part of the retina image out can be performed in different ways. Additionally or alternatively, the complete unsegmented image of the retina may also be fed to a machine learning model for classification. This allows a distinction to be made between changes to the vessels or optic nerve, which can be useful information for example for further analysis.

[0116] For example, a vascular dysfunction can be detected by analyzing the images with focus on the blood vessel structure in the captured image of the retina (optionally preprocessed). The system can also look at the optic nerve of the retina. For example, if there is false positive prediction by mainly taking into account the optic nerve in the image of the retina, a better overall prediction may be obtained by considering other parts of the image of the retina (for example the optic nerve segmented away).

[0117] FIG. 3A shows an unsegmented captured image 1 of the retina, and FIG. 3B shows a segmented image 1′ of the retina obtained based on the captured image. In this shown example, the captured image 1 in FIG. 3A is processed for determining the blood vessels of the retina as landmarks. The segmented image 1′ is obtained only including the identified blood vessels with differentiation between arteries (in red) and veins (in blue). It will be appreciated that it is also envisaged that other landmarks are used (e.g. optic nerve head, a sub-group of blood vessels, etc.). Instead of exclusively including data related to the identified blood vessels in the segmented image 1′, it is also possible to exclude this data from the captured image. In such a case, it is possible to perform an analysis which does not take into account data in the blood vessels, forcing the trained neural network to use other regions of the captured image to classify the image of the retina.

[0118] FIG. 4 shows a preprocessed captured image of a retina of an eye. The captured image has been processed such as to more easily identify the blood vessels. The preprocessing may for instance involve image preprocessing in which contrast, sharpness, and other imaging parameters are adjusted for obtaining an enhanced image allowing more easy identification of landmarks features of the retina. It is also possible to use filters or the like. In some examples, additionally or alternatively, a machine learning model is employed for obtaining a preprocessed image.

[0119] FIG. 5 shows a schematic diagram of an embodiment of a method 50. In the shown exemplary method, a convolutional neural network 51 (CNN) is used for image segmentation. The convolutional neural network 51 includes a plurality of layers 51i. The captured image of the retina 53 is provided as input 55 to the convolutional neural network 51. For example, each pixel value of the captured image may be provided as node input to nodes of a first layer of the CNN 51. The CNN 51 can be configured to provide as output 57 an identification of the blood vessels 5 in the captured image of the retina 53. In this example the CNN is able to distinguish arteries (depicted in red) from veins (depicted in blue). In some examples, the output provides an image with identified blood vessels 5. The output of the CNN 51 can be seen as a prediction. This output can be compared with a ground truth image 59″ for training of the CNN 51. Based on the comparison with the ground truth image 59″, backpropagation 61 may be employed for altering weights of the CNN 51. The ground truth image 59″ may be obtained for instance by means of manual labeling.

[0120] The resulting segmented image 1′ can be used as input to a machine learning model for classifying the image of the retina (not shown in this figure). In this example, the segmentation is performed based on the blood vessels, however, it is also envisaged to perform segmentation based on other retina landmarks, for example the optic nerve head, sub-groups of blood vessels, one or more predetermined regions of the retina, etc. By providing segmented images to the machine learning model, data used for classifying the retina image can be more easily controlled. For instance, the machine learning model for classifying the captured image can be forced to look only to the blood vessels of the retina, for example for inferring and/or further analyzing a condition of the subjects.

[0121] FIG. 6 shows a schematic diagram of an embodiment of a computer-implemented method 100 for classifying images of retina of eyes of subjects. One or more hardware processors are operated for performing the steps of the method 100. In a first step 101, an image of an eye captured by means of an imaging unit is received. In a second step 102, the captured image of the eye is segmented in order to obtain at least one segmented image. In a third step 103, the at least one segmented image is provided as input to one or more trained machine learning models configured to classify the eye.

[0122] FIG. 7 shows a schematic diagram of an embodiment of a computer-implemented method 200. The method 200 may be carried out on a computing system configured to classify images of retina of eyes of subjects. The system may include one or more hardware computer processors, and one or more storage devices configured to store software instructions configured for execution by the one or more hardware computer processors in order to cause the system to perform method steps according to the invention. In a first step 201, an initial image of a retina captured by means of an imaging unit is received. In a second step 202, the initial image of the retina is processed in order to obtain at least a first segmented image and a second segmented image different from the first segmented image, wherein the first segmented image only includes a first selected portion of the captured image of the retina by employing a first selection rule, and wherein the second segmented image only includes a second selected portion of the captured image of the retina by employing a second selection rule, the first and second selection rules being different with respect to each other. In a third step 203, at least a first trained machine learning model and a second trained machine learning model is provided, each configured to output an image classification based on an input image, the first and second machine learning models being different with respect to each other, and wherein the first machine learning model is trained using segmented images with only selected portions obtained by employing the first selection rule, and wherein the second machine learning model is trained using segmented images with only selected portions obtained by employing the second selection rule. In a fourth step 204, the first segmented image with the first selected portion is provided to the first machine learning model as input image in order to obtain a first classification, and the second segmented image with the second selection portion is provided as input image to the second machine learning model as input image in order to obtain a second classification. In a fifth step 205, an ensemble classification is determined based on at least the first classification and the second classification.

[0123] FIG. 8 shows a schematic diagram of an embodiment of a method 300. A captured initial retina image 1 of an eye (optionally preprocessed) is first processed to obtain at least two segmented images. At least first segmented image 1a and a second segmented image 1b different from the first segmented image. The segmented images 1a, 1b may have the same or different resolution with respect to the initial captured image. The first segmented image 1a only includes a first selected portion of the captured image of the retina by employing a first selection rule. In this way, some parts of the initially captured image are excluded or removed in the first segmented image 1a. Similarly, the second segmented image only includes a second selected portion of the captured image of the retina by employing a second selection rule different with respect to other selection rules employed (in this example, different with respect to at least the first selection rule). Furthermore, at least a first trained machine learning model 51a and a second trained machine learning model 51b each configured to output an image classification based on an input image are provided. The first and second machine learning models 51a, 51b are different with respect to each other and separately trained. The first machine learning model 51a is trained using images on which the first selection rule is applied, and the second machine learning model 51b is trained using images on which the second selection rule is applied. Hence, these machine learning models are particularly trained to take into account information in regions remaining in the segmented images by employing the respective selection rule. The first segmented image 1a is provided as input to the first machine learning model 51a, and the second segmented image 1b is provided as input to the second machine learning model 51b. This may be performed in parallel as shown in the figure. As a result, a first classification 57a and a second classification 57b is obtained, respectively. An ensemble classification 57′ is determined based on at least the first classification 57a and the second classification 57b.

[0124] It will be appreciated that a larger number of segmented images may be generated following unique selection rules and dedicated machine learning models. The ensemble classification may be based on the larger number of classifications obtained in this way.

[0125] Each machine learning model being employed may be trained to predict the same subject property or condition (e.g. a disease, age, health, etc.). The different machine learning model predictions and their confidence levels can be combined to come to a final prediction. Furthermore, this may give insight of where the changes to the retina have happened. Advantageously, the method enables a more accurate prediction/classification. Additionally, the explainability can be improved and more interpretable overall model is obtained. The machine learning method can be made more trustworthy.

[0126] FIG. 9 shows a schematic diagram of an embodiment of a method 400. In this example, the captured initial retina image 1 is processed to obtain four different segmented images 1a, 1b, 1c, 1d. Each of the segmented images is obtained by applying a predefined (unique) selection rule. Similarly as shown in the exemplary embodiment of FIG. 8, each segmented image 1a, 1b, 1c, 1d is provided as input to a different trained machine learning models 51a, 51b, 51c, 51d. These machine learning models 51a, 51b, 51c, 51d are trained by means of training images which are correspondingly generated by similarly applying predefined (unique) selection rules (same selection rule as the respective segmented image 1a-1d provided as input to the machine learning model 51a-51d). Additionally, in this example, also the captured initial retina image 1, 1e is provided to a separate machine learning model 51e. The machine learning model 51e is trained using unsegmented training images of the retina (similar to the captured initial retina image 1). The plurality of machine learning models 51a, 51b, 51c, 51d, 51e may generate respective classifications 57a, 57b, 57c, 57d, 57e as output. An ensemble classification 57′ is determined based on the plurality of classifications 57a, 57b, 57c, 57d, 57e.

[0127] In this example, the machine learning models are convolutional neural networks (CNN). It will be appreciated that other types of machine learning models may also be used.

[0128] The invention provides for an improved explainability in the decision in condition classification by the deep learning model using fundus images as input. Instead of using a technique in which parts of the test images are perturbed (standard occlusion technique), and the effect on performance recorded, different segmented images are used with obtained by means of different selection rules, each segmented image provided to differently trained machine learning models. One major downside of occlusion testing is the violation of having a similar distribution in train and test sets. When training on a complete image, and evaluating on a perturbed image, it is impossible to assess whether the change in prediction is due to the perturbation or because the omitted information was truly (un)informative.

[0129] The invention exploits the importance of other regions next to the regions typically known to have the most relevant data (e.g. ONH). The regions beyond the ONH can also be used and taken into account. An objective explainability in deep learning applications for classification of retina images (e.g. for glaucoma detection) can be provided.

[0130] It will be appreciated that the method may include computer implemented steps. All above mentioned steps can be computer implemented steps. Embodiments may comprise computer apparatus, wherein processes performed in computer apparatus. The invention also extends to computer programs, particularly computer programs on or in a carrier, adapted for putting the invention into practice. The program may be in the form of source or object code or in any other form suitable for use in the implementation of the processes according to the invention. The carrier may be any entity or device capable of carrying the program. For example, the carrier may comprise a storage medium, such as a ROM, for example a semiconductor ROM or hard disk. Further, the carrier may be a transmissible carrier such as an electrical or optical signal which may be conveyed via electrical or optical cable or by radio or other means, e.g. via the internet or cloud.

[0131] Some embodiments may be implemented, for example, using a machine or tangible computer-readable medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and/or operations in accordance with the embodiments.

[0132] Various embodiments may be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements may include processors, microprocessors, circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, microchips, chip sets, et cetera. Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, mobile apps, middleware, firmware, software modules, routines, subroutines, functions, computer implemented methods, procedures, software interfaces, application program interfaces (API), methods, instruction sets, computing code, computer code, et cetera.

[0133] Herein, the invention is described with reference to specific examples of embodiments of the invention. It will, however, be evident that various modifications, variations, alternatives and changes may be made therein, without departing from the essence of the invention. For the purpose of clarity and a concise description features are described herein as part of the same or separate embodiments, however, alternative embodiments having combinations of all or some of the features described in these separate embodiments are also envisaged and understood to fall within the framework of the invention as outlined by the claims. The specifications, figures and examples are, accordingly, to be regarded in an illustrative sense rather than in a restrictive sense. The invention is intended to embrace all alternatives, modifications and variations which fall within the spirit and scope of the appended claims. Further, many of the elements that are described are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, in any suitable combination and location.

[0134] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word ‘comprising’ does not exclude the presence of other features or steps than those listed in a claim. Furthermore, the words ‘a’ and ‘an’ shall not be construed as limited to ‘only one’, but instead are used to mean ‘at least one’, and do not exclude a plurality. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to an advantage.