SYSTEM AND METHOD FOR IMAGING, SEGMENTATION, TEMPORAL AND SPATIAL TRACKING, AND ANALYSIS OF VISIBLE AND INFRARED IMAGES OF OCULAR SURFACE AND EYE ADNEXA

Abstract

An automatic system and method for non-invasive imaging and identification of specific ocular structures of the eye and adnexa tissues by synchronous segmentation of visual and infrared images; that can produce spatial temperature profiles within each segmented area of the eye and adnexa; that can track eye and head movement and eye-blinks during the period of measurement to remove artefacts and maintain synchronicity; that can track ocular surface and eye adnexa temperature profiles over time; that can assist in diagnosis of eye disease; that can produce diagnostic indicators for ocular disease diagnosis and study of the eye. The system comprises infrared and visible light cameras for imaging the ocular structures, and a digital signal processing unit for processing the acquired infrared and visible images to output segmentations of the images for identification of different areas of the eye surface, including pupil, cornea, conjunctiva, and eyelids. The system further captures synchronous infrared and visible images from each segmented area of the ocular surface over the time of measurement. A digital signal processing unit processes and analyzes the infrared and visible images to generate descriptive outputs on temporal and spatial changes in the infrared and visible images over the time of measurement, as well as produce diagnostic indicators for ocular disease diagnosis and study of the eye.

Claims

1. A system for measuring ocular surface temperature, comprising: one or more cameras adapted to capture an infrared (IR) thermal image and a visible light image of an ocular surface; a camera positioning controller for controlling the one or more cameras to automatically capture synchronous IR and visible light images of multiple segmented areas of the ocular surface; and one or more digital processing modules adapted to: process the captured IR and visible light images to measure the ocular surface temperature (OST) of each segmented area over time; monitor eye tracking and eye blinking during OST measurement; and identify and remove artefacts in the visible image frame and the corresponding IR thermal image frame to maintain synchronicity of the images and obtain a more accurate OST measurement.

2. The system of claim 1, wherein the one or more digital processing modules is further adapted to segment the images produced to identify specific ocular structures and track OST temperatures precisely for those identified ocular structures.

3. The system of claim 2, where the temporal and spatial changes in the IR and visible images in a specific identified ocular structure is tracked in real-time.

4. The system of claim 3, wherein the temporal and special changes of the specific identified ocular structure and tracked OST temperatures are utilized as diagnostic indicators for ocular disease diagnosis and progression, regression, or remission.

5. The system of claim 3, wherein the specific identified ocular structures include pupil, iris, conjunctiva, and eyelids.

6. The system of claim 5, wherein the temporal and structural changes of the pupil, iris, conjunctiva, and eyelids are tracked over a period of time.

7. The system of claim 1, wherein the one or more digital processing modules are further adapted to measure tear-film dynamic assessment and instability of the eye.

8. The system of claim 1, wherein the one or more digital processing modules are further adapted to non-invasively measure tear-film break-up time (TBUT).

9. The system of claim 1, wherein the one or more digital processing modules are adapted to non-invasively measure and diagnose eye inflammation.

10. The system of claim 1, wherein the one or more digital processing modules are adapted to non-invasively measure and diagnose dry eye.

11. A method of measuring ocular surface temperature, comprising: providing one or more cameras adapted to capture an infrared (IR) thermal image and a visible light image of an ocular surface; providing a camera positioning controller for controlling the one or more cameras to automatically capture synchronous IR and visible light images of multiple segmented areas of the ocular surface; and utilizing one or more digital processing modules: processing the captured IR and visible light images to measure the ocular surface temperature (OST) of each segmented area over time; monitoring eye tracking and eye blinking during OST measurement; and identifying and removing artefacts in the visible image frame and the corresponding IR thermal image frame to maintain synchronicity of the images and obtain a more accurate OST measurement.

12. The method of claim 11, further comprising segmenting the images produced to identify specific ocular structures and track OST temperatures precisely for those identified ocular structures.

13. The method of claim 12, further comprising tracking the temporal and spatial changes in the IR and visible images in a specific identified ocular structure in real-time.

14. The method of claim 13, further comprising identifying the temporal and special changes of the specific identified ocular structure and utilizing the tracked OST temperatures as diagnostic indicators for ocular disease diagnosis and progression, regression, or remission.

15. The method of claim 13, wherein the specific identified ocular structures include pupil, iris, conjunctiva, and eyelids.

16. The method of claim 15, wherein the temporal and structural changes of the pupil, iris, conjunctiva, and eyelids are tracked over a period of time.

17. The method of claim 11, further comprising utilizing one or more digital processing modules to measure tear-film dynamic assessment and instability of the eye.

18. The system of claim 11, further comprising utilizing one or more digital processing modules to measure tear-film break-up time (TBUT).

19. The system of claim 11, further comprising utilizing one or more digital processing modules to measure and diagnose eye inflammation.

20. The system of claim 11, further comprising utilizing one or more digital processing modules to measure and diagnose dry eye.

Description

BRIEF DESCRIPTION OF DRAWING

[0033] FIG. 1 shows a schematic view of the whole system in accordance with an illustrative embodiment.

[0034] FIG. 2 shows an illustrative frontal view of a subject's eye surface and adnexa that can be localized in the system for temperature profile.

[0035] FIG. 3 shows an example of ocular surface segmentation in accordance with an illustrative embodiment.

[0036] FIG. 4 demonstrates the function of each DSP module or unit and the output of each unit with sample video files recorded by the system in accordance with an illustrative embodiment.

[0037] FIG. 5 shows a schematic diagram of a computer system which may provide an operating environment for one or more embodiments of the present system and method.

DETAILED DESCRIPTION

[0038] As noted above, the present disclosure relates to a system and method for imaging and analysis of ocular disease and studies of the eye. More generally, the present system and method provides an improved system and method for automatically and non-invasively imaging the ocular surface and adnexa tissues using infrared (IR) thermal cameras and visible light cameras synchronously. In an embodiment, the present system and method segments the images produced to identify specific ocular structures and measures ocular surface temperature (OST) within segmented areas by tracking the OST precisely, including by monitoring eye tracking and eye blinking during measurement to remove artefacts and maintain synchronicity. Temporal and spatial changes in the IR and visible images are tracked over time, and diagnostic indicators for ocular disease diagnosis and study of the eye are produced.

[0039] A key requirement for ocular surface thermography is the ability to locate the corneal area in the thermogram. However, conduction of heat within the eye ensures that the thermal profile imaged by an IR thermal camera of the ocular surface describes an unfocused thermogram that does not match the underlying anatomical features. This makes it difficult to precisely locate an area of interest on the ocular surface using only the IR image. The majority of existing camera systems described in the background lack a method for detecting the corneal boundary, corneal centre, and conjunctiva, and all existing methods for identifying the point or area of interest in the eye require the input from the operator to manually select the point or area of interest.

[0040] A second requirement is the ability to consistently measure from the same location on the ocular surface. Current methods for IR imaging of the eye incorporate a fixation target for the subject to view. However, during steady fixation by the eye on a point of interest, small eye and head movements still occur. These movements cause relative movements in the areas of interest on the eye and ocular surface during the period of measurement which degrade the accuracy of measurement over the period of measurement. No current methods for IR imaging of the eye incorporate an eye-tracking ability to counteract the effects of eye and head movement.

[0041] A third requirement is the ability to track changes in the ocular surface temperature over a period of time without the effect of artefacts from eyelid blinking. The eyelid covers the area of interest on the ocular surface and introduces an artefact in the temporal temperature profile. All current systems are able to record temporal changes for the selected point or area of interest, but in the prior art data collected must be manually screened for blinking artefacts. No current systems for IR imaging of the eye incorporate an automatic method for removing eyelid blinking artefacts.

[0042] A fourth requirement is to be able to collect and analyse temperature data from all pixel points across the ocular surface within the image frame over the period of measurement. Current methods for IR imaging of the eye collect data from all pixels for image display, but select only a single pixel data point, multiple single pixel data points, or a described area of the surface for image analysis. No current system for IR imaging of the eye is able to report from all pixel points concurrently for data analysis.

[0043] A final requirement is that all of the four previously listed requirements should be completed automatically. No current system for IR imaging of the eye is able to automatically complete any of the four requirements.

[0044] The present system and method addresses at least some of these limitations.

[0045] In various aspects, the present system and method locates specific eye points of areas of interest in the thermogram; removes artefacts produced by eye and head movements; removes artefacts produced by eyelid blinking; gathers and analyses all data pixel points within the area of interest; outputs segmentations of the images for identification of different areas of the eye surface, including pupil, cornea, conjunctiva, and eyelids; generates descriptive outputs on temporal and spatial changes in the IR and visible images over the time of measurement; and produces diagnostic indicators for ocular disease diagnosis and study of the eye.

[0046] In an illustrative embodiment, the present system and method comprises one or more IR thermal cameras and one or more visible light cameras installed on a camera mount in close proximity to each other. The camera mount is fixed to a second mount installed on a movable base. The movable base can be moved in the x/y/z planes by the operator. This arrangement enables the camera mount to be moved to align the cameras with the subject's eye and to focus the image plans of the cameras on the subject's eye and adnexa tissues.

[0047] In an embodiment, the system and method further comprises a separate head-rest and chin-rest positioned in front of the camera mount on which the subject rests their head during measurements.

[0048] In another embodiment, the system and method further comprises a digital signal processing unit that registers and synchronises imaging data from IR and visible light camera image sequences.

[0049] In another embodiment, the system and method further comprises a digital signal processing unit that generates segmented areas of interest within the IR and visible light camera images.

[0050] In another embodiment, the system and method further comprises a digital signal processing unit that processes IR and visible images and produces segmentations of pupil, cornea, conjunctiva, eyelids, and areas within these regions, in the visible images of the subject's eye.

[0051] In another embodiment, a digital signal processing unit that processes IR and visible images and registers the IR and visible image sequences for precise localisation of the segmented pupil, cornea, conjunctiva, eyelids, and areas within these regions, from the visible images to the IR images of the subject's eye.

[0052] In another embodiment, a digital signal processing unit that processes IR and visible images and detects subject eye movements and tracks the area of interest in the visible images over the period of measurement.

[0053] In another embodiment, a digital signal processing unit that processes IR and visible images and detects subject eyelid blinks and removes any artefacts affecting the area of interest in the images over the period of measurement.

[0054] In another embodiment, a digital signal processing unit that processes and analyses the IR and visible light camera data streams per pixel for temperature, texture, and colour components.

[0055] In another embodiment, a digital signal processing unit that processes and analyses the IR and visible light camera data streams and outputs spatial temperature profiles across the area of interest of the subject's eye to produce three-dimensional plots of temperature change.

[0056] In another embodiment, a digital signal processing unit that processes and analyses the IR and visible light camera data streams and outputs temporal temperature profiles across the area of interest of the subject's eye and over the period of measurement to produce plots of temperature changes.

[0057] In another embodiment, a digital signal processing unit that processes and analyses the IR and visible light camera data streams and outputs texture and colour change profiles across the area of interest of the subject's eye over the period of measurement.

[0058] In another embodiment, a digital signal processing unit that processes and analyses the IR and visible light camera data streams and outputs descriptors of temperature change, texture change and colour change associated with ocular surface evaporation, ocular surface esthesiometry, tear break-up time, contact lens wear, computer vision syndrome, infection and disease of the eye and ocular adnexa.

[0059] Various illustrative embodiments of the present system and method will now be described in more detail with reference to the figures.

[0060] Now referring to FIG. 1, shown is a system in accordance with an illustrative embodiment. The system of FIG. 1 includes an IR thermal camera 101 to record thermal sequences from the subject's eye surface, a visible light camera 102 to record visible sequences, a camera mount 103 for camera installation, a vertical pillar 104 attached to a moveable base 105, and adjustment handles 106 for moving the camera system in front of the patient eyes, a head-rest and chin-rest 107 positioned in front of the camera mount on which the subject rests their head, a chin-rest height adjuster 108, and a digital signal processing (DSP) module or module or unit 109 for camera management and data analysis. One or more DSP modules or units 109 may be embodied, for example, by one or more computing devices as shown in FIG. 5 and described further below.

[0061] Still referring to FIG. 1, the system can capture both IR and visible image sequences from the surface of the subject's eye synchronously. The two cameras 101, 102 are aligned in such a way as to have the same field of view, and the DSP module or unit 109 registers the separate images from each camera together. As noted above, one or more IR thermal cameras and one or more visible light cameras may be installed on a camera mount in close proximity to each other. However, in alternative embodiments, the two cameras may be replaced by a single camera with one or more sensors, or replaced by a plurality of additional cameras to capture additional points of view. In this illustrative example, the camera mount 103 is positioned on top of the vertical pillar 104 that is attached to the movable base 105. The camera mount 103 is designed in a way that adjusts the relative position of the cameras forwards/backwards, up/down, left/right, and turning of the cameras in different angles of photography. The movable base 105 is designed to move forwards/backwards, up/down, and left/right to align the IR 101 and visible 102 cameras in front of the subject's eye.

[0062] In an embodiment, the cameras 102 and 103 are selected to capture images of a subject's eye or eyes at a sufficiently high resolution and at sufficiently high frame rates so as to capture clear, sharp images for processing. For example, camera image sensor resolutions of about 2MP or higher may be captured at high frame rates, or video images captured at 720p, 1080p, 4K or even higher resolutions at various frame rates may be utilized as may be required.

[0063] The DSP module or unit 109 is able to record both thermal and visible sequences synchronously. The recorded video files are time-coded and saved in a local disk for further analysis with the DSP module or unit 109. With a sufficiently high level of quality and sufficiently high frame rates for the captured images, the present system and method is able to process the images virtually in real-time.

[0064] In an embodiment, the DSP module or unit 109 processes the IR and visible sequence data to perform image processing and image analysis. In an initial step, the videos are overlaid using image registration techniques in the DSP module or unit 109 for further processing. In a subsequent step, the visible images are used to localize and segment specific parts of the eye including pupil, cornea, conjunctiva, and eyelids in the images, using image segmentation algorithms.

[0065] Now referring to FIG. 2, shown is an illustrative frontal view of a subject's eye surface and adnexa that can be localized in the system for temperature profile. The ocular surface parts and adnexa including pupil 201, iris 202, conjunctiva 203, and eyelid margin 204.

[0066] Now referring to FIG. 3, shown is an example of ocular surface segmentation in accordance with an illustrative embodiment. More specifically, ocular surface segmentation may include a central cornea sector 301, an inferior cornea sector 302, a conjunctiva sector 303, and an inferior eyelid margin sector 304.

[0067] In an embodiment, the segmented areas in the visible images are identified in the IR images.

[0068] In another embodiment, eye movements are detected in the synchronized images, and movement in the segmented areas identified and tracked. The dual camera system is synchronized by hardware triggering of the visible and thermal cameras under digital signal processing unit control. A further digital signal processing unit synchronizes the IR image files with the visible image files.

[0069] In another embodiment, eyelid blinking is identified in the synchronized images, and resulting artefacts removed from the image sequences. Semantic segmentation is used for corneal segmentation under digital signal processing unit control. The presence of a blink artefact is determined by monitoring the presence of the cornea in each visible light camera frame. The absence of a corneal segmentation indicates the presence of the eyelid, and the frame is detected as a containing a blink and removed from analysis.

[0070] In another embodiment, pixel characteristics from the IR and visible light camera images are analyzed over time to produce temperature, texture and colour profiles and rates of change across the area of interest of the subject's eye. Thermal data from the IR camera images and red/blue/green and grayscale data from the visible camera images for each frame of the recorded sequence is extracted for each pixel contained with the segmented area under observation. The video sequences for the IR and visible light camera images are recorded for storage. In an embodiment, a digital signal processing unit analyzes each frame from each video sequence to identify regions or profiles of thermal or texture change and to identify temporal and spatial changes in these regions or profiles over time. Presentation of this analysis is provided to the user in the form of statistical analyses that describe the thermal or textural characteristics of the segmented area. In an embodiment, analysis is completed after data collection, but may also be performed in real-time as the images are captured.

[0071] FIG. 4 demonstrates the function of each DSP module or unit and the output of each unit with sample video files recorded by the system in accordance with an illustrative embodiment using a dual camera setup for segmentation, tracking, and extracting temperature data of the cornea. As shown, the function of each DSP module or unit 109 and the output of each unit with sample video files recorded by the dual camera system is described in the figure. The IR and visible image sequences can be used as the input of the system 401. The image normalization unit 402 removes lens distortion from the image sequences. The undistorted image sequences are used as an input for the control point selection unit 403. The corresponding points on the first frames are selected. The corresponding points are localized on all subsequent frames of each camera's image sequence using an optical flow algorithm. The selected points and the normalized image sequences are used as an input for the video registration unit 404. The video registration unit registers the video files frame by frame using the control points. The visible video output file is used in the corneal segmentation unit 405. The cornea is segmented from the visible light camera image sequence using a semantic segmentation method. The corresponding corneal area in the IR image sequence is identified in the corneal segmentation unit. The blink frames are recognized from the image sequences and removed from the video files 406. The segmented IR image is mapped onto the visible image using the temperature mapping unit 407. The temperature of the corneal segment is tracked on the IR image sequence and extracted from each whole frame. Data analysis methods are used on the segmented IR data in the temperature mapping unit 408 and reported as the system output 409.

[0072] Advantageously, the output of the present system and method provides data on the localization of eye parts as the area of interest, and descriptive outputs data on temporal and spatial changes in ocular surface temperature (OST) over the area of interest.

[0073] Now referring to FIG. 5, the present system and method may be practiced in various embodiments. A suitably configured computer device, and associated communications networks, devices, software and firmware may provide a platform for enabling one or more embodiments as described above. By way of example, FIG. 5 shows a computer device 500 that may include a central processing unit (“CPU”) 502 connected to a storage unit 504 and to a random-access memory 506. The CPU 502 may process an operating system 501, application program 503, and data 523. The operating system 501, application program 503, and data 523 may be stored in storage unit 504 and loaded into memory 506, as may be required. Computer device 500 may further include a graphics processing unit (GPU) 522 which is operatively connected to CPU 502 and to memory 506 to offload intensive image processing calculations from CPU 502 and run these calculations in parallel with CPU 502. An operator 507 may interact with the computer device 500 using a video display 508 connected by a video interface 505, and various input/output devices such as a keyboard 510, pointer 512, and storage 514 connected by an I/O interface 509. In known manner, the pointer 512 may be configured to control movement of a cursor or pointer icon in the video display 508, and to operate various graphical user interface (GUI) controls appearing in the video display 508. The computer device 500 may form part of a network via a network interface 511, allowing the computer device 500 to communicate with other suitably configured data processing systems (not shown). It will be appreciated that computer device 500 may also be implemented in any number of different configurations, including as dedicated application-specific integrated circuits (ASIC) or chips integrated into the system.

[0074] Thus, in an aspect, there is disclosed a system for measuring ocular surface temperature, comprising: one or more cameras adapted to capture an infrared (IR) thermal image and a visible light image of an ocular surface; a camera positioning controller for controlling the one or more cameras to automatically capture synchronous IR and visible light images of multiple segmented areas of the ocular surface; and one or more digital processing modules adapted to: process the captured IR and visible light images to measure the ocular surface temperature (OST) of each segmented area over time; monitor at least one of head movement, eye tracking and eye blinking during OST measurement; and identify and remove artefacts in the visible image frame and the corresponding IR thermal image frame to maintain synchronicity of the images and obtain a more accurate OST measurement.

[0075] In an embodiment, the one or more digital processing modules is further adapted to segment the images produced to identify specific ocular structures and track OST temperatures precisely for those identified ocular structures.

[0076] In another embodiment, the temporal and spatial changes in the IR and visible images in a specific identified ocular structure is tracked in real-time.

[0077] In another embodiment, the temporal and special changes of the specific identified ocular structure and tracked OST temperatures are utilized as diagnostic indicators for ocular disease diagnosis and progression, regression, or remission.

[0078] In another embodiment, the specific identified ocular structures include pupil, iris, conjunctiva, and eyelids.

[0079] In another embodiment, the temporal and structural changes of the pupil, iris, conjunctiva, and eyelids are tracked over a period of time.

[0080] In another embodiment, the one or more digital processing modules are further adapted to measure tear-film dynamic assessment and instability of the eye.

[0081] In another embodiment, the one or more digital processing modules are further adapted to non-invasively measure tear-film break-up time (TBUT).

[0082] In another embodiment, the one or more digital processing modules are adapted to non-invasively measure and diagnose eye inflammation.

[0083] In another embodiment, the one or more digital processing modules are adapted to non-invasively measure and diagnose dry eye.

[0084] In another aspect, there is provided a method of measuring ocular surface temperature, comprising: providing one or more cameras adapted to capture an infrared (IR) thermal image and a visible light image of an ocular surface; providing a camera positioning controller for controlling the one or more cameras to automatically capture synchronous IR and visible light images of multiple segmented areas of the ocular surface; utilizing one or more digital processing modules: processing the captured IR and visible light images to measure the ocular surface temperature (OST) of each segmented area over time; monitoring at least one of head movement, eye tracking and eye blinking during OST measurement; and identifying and removing artefacts in the visible image frame and the corresponding IR thermal image frame to maintain synchronicity of the images and obtain a more accurate OST measurement.

[0085] In an embodiment, the method further comprises segmenting the images produced to identify specific ocular structures and track OST temperatures precisely for those identified ocular structures.

[0086] In another embodiment, the method further comprises tracking the temporal and spatial changes in the IR and visible images in a specific identified ocular structure in real-time.

[0087] In another embodiment, the method further comprises identifying the temporal and special changes of the specific identified ocular structure and utilizing the tracked OST temperatures as diagnostic indicators for ocular disease diagnosis and progression, regression, or remission.

[0088] In another embodiment, the specific identified ocular structures include pupil, iris, conjunctiva, and eyelids.

[0089] In another embodiment, the temporal and structural changes of the pupil, iris, conjunctiva, and eyelids are tracked over a period of time.

[0090] In another embodiment, the method further comprises utilizing one or more digital processing modules to measure tear-film dynamic assessment and instability of the eye.

[0091] In another embodiment, the method further comprises utilizing one or more digital processing modules to measure tear-film break-up time (TBUT).

[0092] In another embodiment, the method further comprises utilizing one or more digital processing modules to measure and diagnose eye inflammation.

[0093] In another embodiment, the method further comprises utilizing one or more digital processing modules to measure and diagnose dry eye.

[0094] While illustrative embodiments have been described, the scope of the invention is defined by the following claims.