Methods and systems for assessing retinal images, and obtaining information from retinal images

Abstract

A method of assessing the quality of an retinal image (such as a fundus image) includes selecting at least one region of interest within a retinal image corresponding to a particular structure of the eye (e.g. the optic disc or the macula), and a quality score is calculated in respect of the, or each, region-of-interest. Each region of interest is typically one associated with pathology, as the optic disc and the macula are. Optionally, a quality score may be calculated also in respect of the eye as a whole (i.e. over the entire image, if the entire image corresponds to the retina).

Claims

1. A computer-implemented method of assessing the quality of at least one retinal image of an eye of a subject, the method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI; (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes.

2. A method according to claim 1 in which there is a ROI for at least one of (i) the optic disk of the eye, and (ii) the macula.

3. A method according to claim 1, wherein the ROI is identified according to a predetermined order of ROIs in the retinal image.

4. A method according to claim 1 in which the extracted features include one or more of: a contrast measure; a blur measure; an entropy measure; and a blood vessel density measure.

5. A method of assessing an image comprising: determining whether the image is a retinal image; if the determination is positive, assessing the quality of the image by a computer-implemented method of assessing the quality of at least one retinal image of an eye of a subject, the method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI; (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes.

6. A method according to claim 5 in which the determination is performed using at least one of: a process of extracting bag-of-words (BoW) features of the image, and classifying the image using a bag-of-words classifier; and a structural similarity index (SSIM) classification performed by obtaining a numerical measure of the similarity of a plurality of characteristic numerical values of the image and characteristic numerical values of previously-obtained images of other eyes.

7. A method according to claim 6 in which the determination is performed by: making an initial classification using said bag-of-word classifier; and using said SSIM classification to re-evaluate images which were classified as not being retinal images by the bag-of-word classifier.

8. A method of obtaining information about an eye using a retinal image of the eye, comprising: assessing the quality of the image by a method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI, (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes; determining if the quality score(s) meet the predefined criteria; and if the determination is positive, obtaining the information about the eye by analysing the image.

9. A method of treating an eye comprising: obtaining information about an eye using a method of assessing an image comprising: determining whether the image is a retinal image; if the determination is positive, assessing the quality of the image by a computer-implemented method of assessing the quality of at least one retinal image of an eye of a subject, the method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI; (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes; wherein the determination is performed using at least one of: a process of extracting bag-of-words (BoW) features of the image, and classifying the image using a bag-of-words classifier; and a structural similarity index (SSIM) classification performed by obtaining a numerical measure of the similarity of a plurality of characteristic numerical values of the image and characteristic numerical values of previously-obtained images of other eyes; wherein the determination is performed by: making an initial classification using said bag-of-word classifier; and using said SSIM classification to re-evaluate images which were classified as not being retinal images by the bag-of-word classifier; performing a diagnostic step using the information; according to the diagnostic step, selecting a treatment procedure; and performing the treatment procedure.

10. A computer system comprising a processor and a data storage device storing computer program instructions operative, upon performance of the instructions by the processor, to cause the processor to perform a computer-implemented method of assessing the quality of at least one retinal image of an eye of a subject, the method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI; (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes.

11. A non-transitory computer program product, storing computer program instructions operative, upon performance of the instructions by a processor, to cause the processor to perform a computer-implemented method of assessing the quality of at least one retinal image of an eye of a subject, the method including: (a) generating at least one quality score for the retinal image by: (i) identifying at least one region of interest (ROI) in the retinal image, the ROI being a region image of a predefined structural portion of the eye; (ii) extracting one or more region image features of the ROI; (iii) generating a region quality score for the ROI, using the region image features extracted for the ROI and an adaptive model; (b) generating a full quality score for the full retinal image by: (i) extracting one or more full image features of the full retinal image; and (ii) generating the full image quality score for the full retinal image, using the full image features extracted for the full retinal image and the adaptive model; and (c) assessing the quality of the retinal image by comparing the region quality score and full quality score against predefined criteria, wherein the adaptive model is obtained using previously obtained region image features and full image features of retinal images of other eyes.

12. A method according to claim 1 wherein the predefined criteria comprise predefined thresholds of the region quality score and full quality score.

13. A method according to claim 1 wherein the region image features and full image features are based on common image quality measures.

14. A method according to claim 8 wherein the region image features and full image features are based on common image quality measures.

Description

BRIEF DESCRIPTION OF THE FIGURES

(1) A non-limiting embodiment of the invention will now be described for the sake of example only, with reference to the following figures in which:

(2) FIG. 1 is a typical fundus image of the eye with the optic disc and macular marked;

(3) FIG. 2, which is composed of FIGS. 2(a) and 2(b), shows schematically the vision loss associated with (a) glaucoma and (b) age-related macular degeneration;

(4) FIG. 3, which is composed of FIGS. 3(a)-(c) shows the portion of a fundus image representing the optic disk, in the case of (a) a normal eye, (b) an optic disc with a high cup-disk-ration, and (c) an optic disc with PPA;

(5) FIG. 4, which is composed of FIGS. 4(a)-(c) shows the portion of a fundus image representing the macula, in the case of (a) a normal macula, (b) a macula with drusen, and (c) a macula with a macula hole;

(6) FIG. 5 is composed of FIGS. 5(a) and 5(b) which are typical poor quality images;

(7) FIG. 6 is a flowchart of a method which is an embodiment of the invention;

(8) FIG. 7 shows the sub-steps of a retinal image identification step of the method of FIG. 6;

(9) FIG. 8 is a schematic version of FIG. 7;

(10) FIG. 9 shows the sub-steps of a confirmation step of the method of FIG. 6;

(11) FIG. 10 is composed of FIG. 10(a), which is a reference retinal image, FIG. 10(b) which is not a fundus image, and FIG. 10(c) which is a fundus image with some artefacts;

(12) FIG. 11 shows the sub-steps of a quality assessment step of the method of FIG. 6;

(13) FIG. 12 which is composed of FIGS. 12(a)-(f), shows a process of detecting a ROI for an optic disc;

(14) FIG. 13 is composed of FIGS. 13(a)-(d), which show optic disc ROIs from two different fundus images, and their respective vascular structures;

(15) FIG. 14 shows the technical architecture of a computer system for performing the method;

(16) FIG. 15 shows the ROC curve for a full image using an embodiment of the present invention and a prior art system;

(17) FIG. 16, which is composed of FIGS. 16(a)-(c), shows the optic disc regions of three different fundus images, which are given different quality scores by the method of FIG. 6;

(18) FIG. 17 shows the ROC curve for an optic disk ROI using an embodiment of the present invention and a prior art system; and

(19) FIG. 18 shows the ROC curve for a macula ROI obtained using an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENT

(20) Referring firstly to FIG. 6, a flow diagram is shown of a method 100 which is an embodiment of the method. The embodiment is referred to as ARIES (an Automated Retinal Interest Estimator System). ARIES automatically assesses the quality of input images as a pre-processing step before passing the processed images for subsequent analysis. In this way, ARIES will control the quality of input for subsequent analysis.

(21) A key feature of ARIES is that it analyzes at least one of the focal regions of interest. A specific example here is the optic disk. Since imaging artefacts can be local, ARIES will help to ensure that the usage of the input image is maximized by assessing the quality of the detected focal region of interest, rather than rejecting the entire image based on global characteristics. If the initial region of interest is not suitable, another ROI will be extracted and re-assessed. This is repeated until a suitable ROI is found, or when all possible ROI are exhausted. ARIES assesses the quality of an input image in three steps: a retinal image identification step 1 in which an initial assessment is made of whether an input image is a retinal image; if the initial assessment is negative, a step 2 of re-evaluation of the image for confirmation 2; if the initial assessment was positive, or if the confirmation step 2 indicates the image was after all a retinal image (i.e. the initial assessment step 1 reached the wrong conclusion), a quality assessment step 3, to generate a full image quality score and a focal region (optic disc or macula) quality score will be given. These can be used to filter out low quality images or provide a warning of low confidence for further steps of processing.

1. Retinal Image Identification Step

(22) The retinal and non-retinal image identification step 1 is important in filtering out irrelevant images in the computer aided diagnosis (CAD) of ocular diseases. A robust CAD system should be able to identify and isolate images of unknown types, and only process those with the correct image type.

(23) FIG. 7 shows the sub-steps of a possible implementation of step 1, which utilizes a weakly-supervised learning approach. In this approach, we can decide whether an image exhibits certain characteristics with no segmentation of objects or manual selection of features. In the first sub-step 11, the training images are represented using a bag of visual words representation, to generated a plurality of extracted features. In sub-step 12, the extracted features are used by a Support Vector Machine (SVM) classifier to generate a classification result.

(24) The SVM was obtained by training using a training set comprising a plurality of each images, and for each image the extracted features obtained by the bag of visual words representation, and a respective label indicating whether the images was a retinal image. Once this is done, the trained SVM classifier can be applied to test on new images.

(25) In one case, step 2 may be used to identify specifically whether the input image is a fundus image. In this case, the labels indicate whether each of the images in the training set is a fundus image or a non-fundus image. The latter classification is used even for input images which are another type of retinal image, such as Retcam images, OCT images and slit-lamp images. This process is shown schematically in FIG. 8. Although input images 14, 15 and 16 are all ocular images (input image 13 is not), only input image 14 is a fundus image, and is classified as such.

2. Confirmation Step

(26) Artifacts in fundus images can affect the accuracy of the image classification in the previous step. Artefacts, especially overexposure, often lead to wrong classification of fundus images into non-fundus types. Thus, a confirmation step 2 is desirable to distinguish between real non-fundus images and fundus images with artefacts which led to misclassification in step 1. The identified fundus image with artefacts can be classified as poor quality fundus image directly or be fed to ARIES quality assessment step to assess focal regions.

(27) In order to reduce the misclassification rate, we introduce a confirmation method based on the structural similarity index (SSIM). The sub-steps of the confirmation step 2 are shown in FIG. 9. SSIM is a method for measuring the similarity between two images. Certain prior art techniques use a full reference metric designed to improve on traditional metrics such as PSNR and MSE by considering image degradation as perceived change in structural information. The SSIM between two windows x and y is:

(28) $SSIM (x, y) = \frac{(2 u_{x} u_{y} + c_{1}) (2_{xy} + c_{2})}{(u_{x}^{2} + u_{y}^{2} + c_{1}) (_{x}^{2} +_{y}^{2} + c_{2})}$
where u.sub.x and u.sub.y are the average, .sub.x.sup.2 and .sub.y.sup.2 are the variance, .sub.xy is the covariance of x and y respectively.

(29) However, SSIM is used in a different way in step 2. Instead of the full reference approach, a reference image is generated by averaging a set of high quality fundus images. The input images are compared one-by-one to this reference image, to generate respective SSIM values (sub-step 21). The SSIM values may be compared with a predefined threshold value, thereby obtaining a decision of whether the images are fundus images or not (sub-step 22).

(30) Intuitively, non-fundus images such as scenery images and other ocular images should have low SSIMs. By contrast, fundus images with artefacts such as overexposed images should have high SSIMs. Therefore, we can separate overexposed fundus images from non-fundus images. FIG. 10 shows how this happens. FIG. 10(a) is the mean fundus image FIG. 10(b) show a non-fundus image (in fact, an image of scenery), having an SSIM of 0.42. FIG. 10(c) shows a poor quality fundus image, which nevertheless has an SSIM of 0.73. Thus, FIG. 10(c) is recognised as a fundus image if the threshold is below this value.

(31) Note that in the process of FIG. 6, SSIM is used in the confirmation step, rather than in the initial retinal image identification step. This is because we have found that SSIM is of lower accuracy in image identification than the BoW process described, but it is accurate in identifying misclassified retinal images and non-retinal images. If the confirmation step is omitted, some very low quality fundus images may be misclassified as non-retinal images.

3. Image Quality Assessment of Retinal Images

(32) The image quality assessment step 3 distinguishes retinal images with high quality from those with poor quality. As described previously, ARIES assesses focal regions of the image as well as the whole image. The process is illustrated in FIG. 11, and includes a sub-step of ROI quality assessment 31 and a sub-step 32 of full image quality assessment. Sub-step 32 may be performed more than once, each time using different structural regions of the eye to be the ROI (e.g. once with the ROI corresponding to the optic disc, and once with the ROI corresponding to the macula).

(33) It is to be appreciated that sub-steps 31 and 32 are independent, and thus may be performed in the opposite order or even in parallel.

(34) Thus, the first sub-sub-step of the focal region image quality assessment 31 is to detect a focal region of interest (ROI) 311. In fact, the embodiment tries to detect a plurality of ROIs in a pre-determined order. If the first ROI according to this pre-determined order cannot be detected, another ROI will be extracted and assessed. This is repeated for a pre-determined sequence of possible ROIs until a suitable ROI is found, or when all possible ROI are exhausted. In other words, when a certain ROI has been detected with high accuracy, the embodiment does not test the remaining ROIs on the predetermined list. The success rate of initial ROI detection for high quality images is very high. Thus, only images that fail the initial quality test will go through the process repeatedly.

(35) Subsequently, high level image quality measures (HIQM) are extracted from the ROI image 312 to form the feature space. Finally, a SVM classification is performed 313, to generate a ROI image quality score. The full image quality assessment process 32 follows a similar flow except that the full image rather than ROI image is used to extract the HIQM features. Thus, sub-step 32 includes a sub-sub-step 321 of HIQM feature extraction, and a sub-step 322 of classification and regression, which can be performed by a SVM classification, giving a full image quality score.

(36) The ROI image quality score(s) and full image quality score are general quality scores, and may be used in various ways according to the application. One option is to compare any one or more of them to respective thresholds, and then to reject them (at least for certain purposes) if the score(s) are below the threshold. For example, if the quality score for a first of the ROIs is below the threshold but the quality score for a second ROI is above the threshold, then the image may be rejected for use in obtaining information about the first ROI, but it may still be used to obtain information about the first ROI. However, this may be conditional on the full image quality score being above a respective threshold.

(37) In the following text, we will illustrate the process using the optic disc image.

(38) a) Sub-Sub-Step 311: Optic Disc ROI Detection

(39) In optic disc (OD) localization, we first find a pixel that belongs to the OD. The region-of-interest (ROI) is a cropped sub-image from the original image that contains the OD. The OD is normally brighter than other regions of the fundus image. However, due to uneven illumination or an out-of-focus image, the fringe of the eyeball can also be very bright. In order to detect the OD centre accurately based on intensity values, we identified bright fringes and removed them. The fringe was extracted by locating a circle slightly smaller than the eyeball in the greyscale image and thresholded for high intensity pixels outside the circle. The fringe-removed image can be obtained by subtracting the bright fringe from the greyscale image. This image is then thresholded to obtain the top 0.5% of pixels in intensity. The centre of the OD is approximated by the centroid of the remaining bright pixels. The ROI is then defined as an image that is about twice the diameter of the normal OD.

(40) An example of the OD localization and ROI detection is shown in FIG. 12. FIG. 12(a) is the original image (shown as black-and-white, but in fact the original image is in colour; FIG. 12(b) is a greyscale image obtained from the original image; FIG. 12(c) shows the extracted high intensity fringe; FIG. 12(d) is the image of FIG. 12(b) with the high intensity fringe removed; FIG. 12(e) shows the thresholded high intensity pixels; and FIG. 12(f) shows by a white rectangle the portion of the image of FIG. 12(b) which is extracted as the ROI.

(41) b) Sub-Sub-Step 312 HIQM Feature Extraction

(42) Instead of using low level image features that have extremely high dimensions, we propose to use high level image quality measures (HIQM) for their high relevance to the problem the embodiment addresses, and their low dimensionality and hence fast speed. There are three categories of HIQM features and they are summarized as follows.

(43) Contrast and Blur Features

(44) High level contrast and blurriness measures include the contrast ratio measurements, the blur measures, the intensity ranges, and saturation metrics.

(45) Contrast ratio is calculated as

(46) ${CR}_{j} = \frac{{\overline{p}}_{j}}{s_{j}}$
where p.sub.j=mean(I.sub.j) is the mean intensity of all the pixels in channel j of the image I in RGB color space, s.sub.j=std(I.sub.j) is the standard deviation of all the pixel intensities, and j can be red channel r, green channel g, blue channel b and greyscale gs.

(47) Higher contrast ratios correspond to higher blurriness. It is intuitively true as blur images usually have small variance in intensity, which leads to a high contrast ratio. Similar to contrast ratio, we also use the local contrast ratio on non-overlapping sub-windows of the image, which is defined as

(48) $LCR = \frac{{.Math.}_{i = 1}^{n} \frac{{\overline{p}}_{w, i}}{s_{w, i}}}{n}$
where w is an NN pixel window, and n is the total number of sub-windows.

(49) Blur metric (BM) is based on the discrimination between different levels of blur perceptible on the same picture. It requires no reference to compute and has a low cost implementation. This measure is robust in measuring focal blur and motion blur. Mathematically, it is obtained by comparing the intensity variations of the original image and its blurred version by a low-pass filter.

(50) Intensity ranges, including full intensity range (R), relative intensity range and interquartile range (IQR), are important metrics to measure the greyscale spread of images. Image with high quality or contrast usually has a larger intensity range compared to one with low quality.

(51) $R = \max (I) - \min (I)$ $RR = \frac{\max (I) - \min (I)}{mean (I)}$ $IQR = Q_{3} (I) - Q_{1} (I)$
where I is the array of all pixel intensities of a greyscale image, Q.sub.1 and Q.sub.3 are the 1.sup.st and 3.sup.rd quartile values.

(52) Saturation metrics include percentage of maximal (Pmax) and percentage of minimal (Pmin), measuring the proportions of pixels at the highest intensity and the lowest intensity respectively. The former is useful to identify overexposed images; while the latter is able to identify underexposed images.

(53) Entropy Features

(54) Entropy of an image can be used to represent the amount of information in it. It is calculated as:

(55) $E = - \underset{i}{.Math.} p_{i} \log_{2} (p_{i})$
where p.sub.i is the probability that the difference between two adjacent pixels is equal to i.

(56) If an image has been perfectly histogram equalized, the spread of greyscale values is at maximum. Thus, this image has maximum entropy. On the other hand, the entropy for a binary image is very low as it has only two states. If an image has flat-value pixels, the entropy is zero.

(57) For optic disc region of interest, a high quality image should contain clear structure of optic disc, optic cup and blood vessels, which corresponds to high image entropy. On the other hand, a poor quality image does not contain fine features of those structures and thus has low image entropy.

(58) Image Structure Features

(59) The optic disc region has a high density of blood vessels. Thus, blood vessel density (BVD) can be used as an important feature to distinguish between low quality and high quality optic disc images.

(60) FIG. 13 illustrates the difference of detected vasculatures of two different images, shown as 13(a) (a low-quality image) and 13(c) (a high quality image). The corresponding detected blood vessels are shown in FIGS. 13(b) and 13(d).

(61) Another important structure feature is the maximum edge length or edge spread (ES). The edges of blood vessels and optic disc boundary are usually continuous for a high quality image, as is the maximum edge spread. To compute BVD and ES, we first detect blood vessels in the image using a fast bottom-hat filtering method. The bottom-hat filter is applied to the histogram equalized green channel image (g) to obtain {tilde over (g)}. Subsequently, the blood vessel map M is determined by

(62) $M (i, j) = {\begin{matrix} 1 & \tilde{g} (i, j) > T \\ 0 & otherwise \end{matrix}$

(63) After obtaining the vessel map, BVD can be obtained through

(64) $BVD = \frac{{.Math.}_{i = 1, j = 1}^{m, n} M (i, j)}{m n}$
where m and n represent the width and height of the image respectively.

(65) The edge spread is calculated as the maximum major axis length of all connected components in M divided by the diagonal length of the image.

(66) $ES = \frac{\max (l_{1}, l_{2}, .Math., l_{k})}{\sqrt{m^{2} + n^{2}}}$
where l represents the major axis length of each connected component in M. Therefore, combining all three categories of features, HIQM contains 38 high level image quality measures. These features will be used in the next step.

(67) c) Sub-Sub-Step 313 Image Quality Classification and Assessment

(68) In the sub-sub-step 313, a support vector machine (SVM) is used to train the classifier. The SVM-based classification can be formulated as a quadratic programming problem with linear constraint:

(69) $\min_{w, b,} \frac{1}{2}^{T} + C {.Math.}_{i = 1}^{N}_{i}$ $subject to y_{i} (^{T} (x_{i}) + b) 1 -_{i}$ $_{i} 0, i = 1, .Math., N$
where x.sub.i is the training vector, y.sub.i is the training label, is the normal vector, b is the offset, .sub.i is the slack variable to implement soft margin, C is the penalty term and is the kernel function.

(70) Instead of using binary classification results only from SVM, the outputs of the SVM decision function (decision value) are also used. We normalized the decision values to produce an retinal image quality score (RQS):
RQS=wd+b
where d represents the decision value, w and b are constant coefficients. RQS have values from 0 to 1, with higher value represents better image quality.

(71) Sub-sub-steps 321 and 322 are performed in the same way as sub-sub-steps 312 and 313 respectively to generate a ROI image quality score for the input image as a whole.

(72) FIG. 14 shows a technical architecture of computer system 200 which can be used to implement the embodiment of FIG. 6. The technical architecture includes a processor 222 (which may be referred to as a central processor unit or CPU) that is in communication with memory devices including secondary storage 224 (such as disk drives), read only memory (ROM) 226, random access memory (RAM) 228. The processor 222 may be implemented as one or more CPU chips. The technical architecture may further comprise input/output (I/O) devices 230, and network connectivity devices 232.

(73) The secondary storage 224 is typically comprised of one or more disk drives or tape drives and is used for non-volatile storage of data and as an over-flow data storage device if RAM 228 is not large enough to hold all working data. Secondary storage 224 may be used to store programs which are loaded into RAM 228 when such programs are selected for execution.

(74) In this embodiment, the secondary storage 224 has an order processing component 224a comprising non-transitory instructions operative by the processor 222 to perform various operations of the method of the present disclosure. The ROM 226 is used to store instructions and perhaps data which are read during program execution. The secondary storage 224, the RAM 228, and/or the ROM 226 may be referred to in some contexts as computer readable storage media and/or non-transitory computer readable media.

(75) I/O devices 230 may include printers, video monitors, liquid crystal displays (LCDs), plasma displays, touch screen displays, keyboards, keypads, switches, dials, mice, track balls, voice recognizers, card readers, paper tape readers, or other well-known input devices.

(76) The network connectivity devices 232 may take the form of modems, modem banks, Ethernet cards, universal serial bus (USB) interface cards, serial interfaces, token ring cards, fiber distributed data interface (FDDI) cards, wireless local area network (WLAN) cards, radio transceiver cards that promote radio communications using protocols such as code division multiple access (CDMA), global system for mobile communications (GSM), long-term evolution (LTE), worldwide interoperability for microwave access (WiMAX), near field communications (NFC), radio frequency identity (RFID), and/or other air interface protocol radio transceiver cards, and other well-known network devices. These network connectivity devices 232 may enable the processor 222 to communicate with the Internet or one or more intranets. With such a network connection, it is contemplated that the processor 222 might receive information from the network, or might output information to the network in the course of performing the above-described method operations. Such information, which is often represented as a sequence of instructions to be executed using processor 222, may be received from and outputted to the network, for example, in the form of a computer data signal embodied in a carrier wave.

(77) The processor 222 executes instructions, codes, computer programs, scripts which it accesses from hard disk, floppy disk, optical disk (these various disk based systems may all be considered secondary storage 224), flash drive, ROM 226, RAM 228, or the network connectivity devices 232. While only one processor 222 is shown, multiple processors may be present. Thus, while instructions may be discussed as executed by a processor, the instructions may be executed simultaneously, serially, or otherwise executed by one or multiple processors.

(78) Although the technical architecture is described with reference to a computer, it should be appreciated that the technical architecture may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and/or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and/or parallel processing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the technical architecture 220 to provide the functionality of a number of servers that is not directly bound to the number of computers in the technical architecture 220. In an embodiment, the functionality disclosed above may be provided by executing the application and/or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and/or may be hired on an as-needed basis from a third party provider.

(79) It is understood that by programming and/or loading executable instructions onto the technical architecture, at least one of the CPU 222, the RAM 228, and the ROM 226 are changed, transforming the technical architecture in part into a specific purpose machine or apparatus having the novel functionality taught by the present disclosure. It is fundamental to the electrical engineering and software engineering arts that functionality that can be implemented by loading executable software into a computer can be converted to a hardware implementation by well-known design rules.

3. Experimental Results

(80) The proposed system is studied on images from a range of fundus image databases such as the Singapore Malay Eye Study (SiMES), Singapore Chinese Eye Study (SCES) and Blue Mountains Eye Study (BMES). A number of non-fundus image databases such as slit-lamp images for, OCT images, Retcam images and scenery images are also used to test the retinal image identification. A summary of the databases used is shown in Table I. Images for subsequent quality assessments are from SiMES database. A detailed description is given below.

(81) TABLE-US-00001 TABLE I Database summary Database Name Image Type Number of Images SiMES Fundus 5928 SCES Fundus 1676 BMES Fundus 6222 Other Fundus Fundus 12314 ACHIKO-NC Slit-lamp 5530 AGAR Database OCT 1664 AGATE Retcam 3699 Database Scenery Scenery 4509 Total 41542

(82) The fundus image identification algorithm was trained on 6200 images, including 2700 fundus images and 3500 non-fundus images. It was tested on a batch of 35342 images, consisting of 23441 fundus images and 11902 non-funds images. The system achieves 99.54% accuracy in fundus and non-fundus image classification in the testing set.

(83) In the training stage, 2700 fundus images, 500 OCT images, 500 Retcam images, 500 slit-lamp images and 2000 scenery images are randomly chosen as the training dataset. The rest of the images are used as the testing dataset. The summary of training of testing datasets and experiment results is shown in Table II.

(84) TABLE-US-00002 TABLE II Summary of experiment results for fundus image identification Accuracy # on # Accuracy Database Image Training Training Testing on Testing Name Type Images Set Images Set SiMES Fundus 500 100% 5428 98.80% SCES Fundus 500 100% 1176 100% BMES Fundus 500 100% 5722 99.76% Other Fundus Fundus 1200 100% 11114 99.26% ACHIKO-NC Slit-lamp 500 100% 5030 100% AGAR OCT 500 100% 1164 100% Database AGATE Retcam 500 100% 3199 100% Database Scenery Scenery 2000 100% 2509 100% Total 6200 100% 35342 99.54%

(85) We now describe the results of the Fundus Image Quality Assessment (step 3). As mentioned above, depending upon the application, the steps 31 and 32 can be performed in the opposite order, and the experimental results below are from an application in which step 32 is performed before step 31.

(86) a) Full Image Assessment (Sub-Step 32)

(87) The algorithm was trained and tested on a database of 740 images, in which 600 images of good quality and 140 images are of poor quality. 123 of the 140 poor quality images are due to existence of cataract, which accounts for about 88%. Others are mainly due to uneven illumination and overexposure during taking. All images are of the dimension of 30722048 pixels. In the experiment sefting, half of images are used for training (300 good-quality images and 70 poor-quality images) and the other half are used for testing.

(88) Images were classified as high or low quality according to whether the fullRQS for the image was respectively above or below a threshold. The threshold used was set by trial and error. In a commercial application of the embodiment using a given CAD system, the threshold may be set according to the properties of the system as each system can process images with different quality standards.

(89) The algorithm can achieve an accuracy of 94.3% and 91.6% of classification for the training set and testing set respectively. In terms of full image quality score (full RQS), the area under curve (AUC) of the receiver operating characteristic curve achieves 0.967 and 0.958 respectively.

(90) For comparison purpose, we implemented a no-reference quality metric (Q.sub.V) based on blood vessel trees in the fundus image. Tested on the same data set of 740 images, the Q.sub.V metric achieves an AUC of 0.796. The ROC curves from both methods are shown in FIG. 15. The ROC curve of the full RQS is shown dark, and the ROC curve of the Q.sub.V metric is shown light.

(91) b) Optic Disc ROI Assessment (Sub-Step 31 in the Case that the ROI is Based on the Optic Disc)

(92) For optic disc ROI quality assessment, the ROI images are extracted with a dimension of 800800 pixels using method explained above. The dataset used for this part is optic ROI images from the full image dataset defined above, and images are labelled as high quality by applying respective thresholds to the full image quality score and the optic disc quality score. The logic for this is that a high (low) quality full image typically has a high (low) quality optic disc region. Table III shows the result of 10-fold cross validation for the fundus image quality classification.

(93) TABLE-US-00003 TABLE III Accuracy of fundus image quality classification Testing Fold Training Set Set 1 0.962 0.959 2 0.968 0.959 3 0.957 0.943 4 0.962 0.943 5 0.957 0.954 6 0.962 0.957 7 0.949 0.959 8 0.962 0.949 9 0.962 0.951 10 0.954 0.959 Average 0.960 0.954 total

(94) The ARIES system also produces an OD quality score (odRQS) in the range of 0 to 1. A score that is close to 1 represents a high quality image. On the other hand, a score that is close to 0 indicates a very low quality image. Images can be classified as high or low quality according to whether odRQS is respectively above or below a threshold. In a commercial application of the embodiment using a given CAD system, the threshold may be set according to the properties of the system as each system can process images with different quality standards.

(95) FIG. 16 shows images with different levels of image quality score. FIG. 16(a) has a quality score of 0.30, FIG. 16(b) has a quality score of 0.66, and FIG. 16(c) has a quality score of 0.92.

(96) The Q.sub.V metric is also implemented on the optic disc ROI images, which achieves an AUC of 0.532. However, the proposed optic disc ROI RQS can do a much better job by achieving 0.987 of AUC. The ROC curves are shown in FIG. 17, where the optic disc RQS is shown by the dark line, and the Q.sub.V metric is shown by the light line.

(97) c) Macula ROI Assessment (Sub-Step 31 in the Case that the ROI is Based on the Macula)

(98) The last experiment was performed on macula ROI to test whether an image is good enough to diagnose AMD. Similar to full image and optic disc ROI, the system generates a macula ROI quality score (maculaRQS). This module is trained and tested on a database of 472 images, in which 350 are with good quality macula ROI and 122 are with poor quality macula ROI. The macula detection is achieved using an automatic method. Then, macula ROI is extracted as a 12201220 image centred at the macula.

(99) Using half of the dataset for training and the other half for testing, the average classification accuracy can achieve 89% and 86% percent for training and testing datasets in a 10-folder cross validation. Images can be classified as high or low quality according to whether maculaRQS is respectively above or below a threshold. In a commercial application of the embodiment using a given CAD system, the threshold may be set according to the properties of the system as each system can process images with different quality standards.

(100) The AUC for the maculaRQS metric can be as high as 0.92 for the testing set as shown in FIG. 18, which shows the ROC curve of macula RQS.

(101) Note that each of fullRQS, odRQS and maculaRQS provides an independent means of classifying images as high or low quality. odRQS and maculaRQS are targeted at specific respective ROIs. An image with a high fullRQS score may correspond to a poor quality macula region and high quality OD region, or a high quality macula region and poor quality OD region, or some other combination. Thus, according to the application of the embodiment, and in particular which region(s) of an image which is classified as high quality will be employed in the CAD, a different selection of the RQS values may be selected for use in the classification.

Methods and systems for assessing retinal images, and obtaining information from retinal images

Assignee

Inventors

Cpc classification

Classification Explorer

A61B2576/02

HUMAN NECESSITIES

Classification Explorer

G06V40/193

PHYSICS

Classification Explorer

G06V10/993

PHYSICS

Classification Explorer

A61B3/0025

HUMAN NECESSITIES

Classification Explorer

G06T7/0014

PHYSICS

Classification Explorer

G06T2207/30041

PHYSICS

Classification Explorer

G06V10/464

PHYSICS

Classification Explorer

G06V10/764

PHYSICS

Classification Explorer

A61B3/14

HUMAN NECESSITIES

Classification Explorer

G06T7/0012

PHYSICS

Classification Explorer

A61B3/12

HUMAN NECESSITIES

Classification Explorer

G06V40/14

PHYSICS

Classification Explorer

G06T2207/30168

PHYSICS

Classification Explorer

G06T2207/20081

PHYSICS

Classification Explorer

G06V2201/03

PHYSICS

Classification Explorer

G06F18/2411

PHYSICS

International classification

Classification Explorer

G06K9/46

PHYSICS

Classification Explorer

A61B3/14

HUMAN NECESSITIES

Classification Explorer

G06T7/00

PHYSICS

Classification Explorer

A61B3/12

HUMAN NECESSITIES

Classification Explorer

G06K9/03

PHYSICS

Classification Explorer

G06K9/00

PHYSICS

Classification Explorer

G06K9/62

PHYSICS

Classification Explorer

A61B3/00

HUMAN NECESSITIES

Abstract

Claims

Description