SYSTEM AND METHOD FOR DETERMINING AN EYE MOVEMENT

Abstract

Computer-implemented method for determining an eye movement, the method comprising: recording a series of images of an eye of a person with a camera; generating a first signal indicative of eye openness depending on time; and for each point in time, determining whether the first signal indicates an open eye, a closed eye, or is inconclusive as to whether the eye is open or closed, depending on whether the first signal exceeds a predetermined first threshold; characterized in that the determination further depends on one or more predetermined criteria, wherein the predetermined criteria comprise minimum and maximum values for one or more of a blink duration, a number of blinks per minute, a time between blinks, and a maximum blink velocity during a movement of the eyelid when the eye is being opened and/or closed.

Claims

1. A computer-implemented method for determining a movement associated with an eye of a person, the method comprising: recording a series of images of the eye; based on the recorded series of images, generating a first signal representative of a movement associated with the eye; deriving one or more movement parameters from the first signal; for one or more points in time, determining whether the first signal indicates that the eye is in a first state, or indicates that the eye is in a second state, or is inconclusive as to whether the eye is in the first state or the second state, based on whether at least one of the one or more movement parameters exceeds a predetermined threshold; and selecting a portion of the first signal based on the determination.

2. The method of claim 1, wherein the one or more movement parameters indicate an amplitude of the movement, a duration of the movement, a velocity of the movement, or a frequency of the movement.

3. The method of claim 1, wherein the first signal is indicative of eye openness; wherein the first state is associated with an open eye; wherein the second state is associated with a closed eye; and wherein the one or more movement parameters comprise: an eye openness; and one or more of: a blink duration; a number of blinks per minute; a time between blinks; and a maximum blink velocity during a movement of an eyelid when the eye is being opened, being closed, or both.

4. The method of claim 1, wherein the first signal is indicative of eye gaze; wherein the first state is associated with a saccade; wherein the second state is associated with a fixation; and wherein the one or more movement parameters comprise: an eye movement velocity; and one or more of: a saccade amplitude; a saccade velocity; a saccade duration; and a fixation duration.

5. The method of claim 1, wherein the portion corresponds to one or more periods in time when the eye is open, and the method further comprises: based on the recorded series of images corresponding to the selected portion of the first signal, generating a second signal indicative of eye gaze; deriving one or more second movement parameters indicative of an eye movement velocity from the second signal; for one or more points in time, determining whether the second signal indicates a saccade, indicates a fixation, or is inconclusive as to whether it indicates a saccade or a fixation, based on whether at least one of the one or more second movement parameters exceeds a predetermined threshold.

6. The method of claim 5, further comprising: deriving a third movement parameter indicative of one or more of a saccade amplitude, a saccade velocity, a saccade duration, and a fixation duration.

7. The method of claim 1, wherein the determination comprises generating an eye movement classification comprising: generating a first movement parameter; generating a preliminary eye movement classification of the first signal as indicative of the first state if the first movement parameter exceeds a first threshold, and as indicative of the second state if the first movement parameter does not exceed the first threshold; generating a second movement parameter; and for one or more points in time, modifying the preliminary eye movement classification as inconclusive if the second movement parameter exceeds a second threshold, thereby generating the eye movement classification.

8. The method of claim 4, further comprising: determining one or more fixation periods, and determining, for each fixation period, an averaged eye gaze signal.

9. The method of claim 1, further comprising: filtering measurement noise from the first signal.

10. A system for determining an eye movement associated with an eye of a person, the system comprising: a camera operable to record a series of images of the eye; and a computing device operable to execute the steps of: based on the recorded series of images, generate a first signal representative of a movement associated with the eye; derive one or more movement parameters from the first signal, for one or more points in time, determine whether the first signal indicates that the eye is in a first state, or indicates that the eye is in a second state, or is inconclusive as to whether the eye is in the first state or the second state, based on whether at least one of the one or more movement parameters exceeds a predetermined threshold; and select a portion of the first signal based on the determination.

11. The system of claim 10, wherein the one or more movement parameters indicates an amplitude of the movement, a duration of the movement, a velocity of the movement, or a frequency of the movement.

12. The system of claim 10, wherein the first signal is indicative of eye openness, wherein the first state is associated with an open eye; wherein the second state is associated with a closed eye; and wherein the one or more movement parameters comprise: an eye openness; and one or more of: a blink duration; a number of blinks per minute; a time between blinks; and a maximum blink velocity during a movement of an eyelid when the eye is being opened, being closed, or both.

13. The system of claim 10, wherein the first signal is indicative of eye gaze; wherein the first state is associated with a saccade; wherein the second state is associated with a fixation; and wherein the one or more movement parameters comprise: an eye movement velocity; and one or more of: a saccade amplitude; a saccade velocity; a saccade duration; and a fixation duration.

14. The system of claim 10, wherein the portion corresponds to one or more periods in time when the eye is open, and the computing device is further operable to execute the steps of: based on the recorded series of images corresponding to the selected portion of the first signal, generate a second signal indicative of eye gaze; derive one or more second movement parameters indicative of an eye movement velocity from the second signal; for one or more points in time, determining whether the second signal indicates a saccade, indicates a fixation, or is inconclusive as to whether it indicates a saccade or a fixation, based on whether at least one of the one or more second movement parameters exceeds a predetermined threshold.

15. The system of claim 14, wherein the computing device is further operable to execute the steps of: derive a third movement parameter indicative of one or more of a saccade amplitude, a saccade velocity, a saccade duration, and a fixation duration.

16. The system of claim 10, wherein the determination comprises generating an eye movement classification comprising: generating a first movement parameter; generating a preliminary eye movement classification of the first signal as indicative of the first state if the first movement parameter exceeds a first threshold, and as indicative of the second state if the first movement parameter does not exceed the first threshold; generating a second movement parameter; and for one or more points in time, modifying the preliminary eye movement classification as inconclusive if the second movement parameter exceeds a second threshold, thereby generating the eye movement classification.

17. The system of claim 13, wherein the computing device is further operable to execute the steps of: determine one or more fixation periods, and determine, for each fixation period, an averaged eye gaze signal.

18. The system of claim 10, wherein the computing device is further operable to execute the steps of: filtering measurement noise from the signal.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference numerals refer to similar elements.

[0067] FIG. 1 depicts a block diagram of a system according to an embodiment;

[0068] FIG. 2 depicts a flow chart representing a method for determining eye openness according to an embodiment; and

[0069] FIG. 3 depicts a flow chart representing a method for determining eye movement according to an embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0070] FIG. 1 depicts a block diagram of a system 100 according to an embodiment. The system comprises a camera 102 to record a series of images of a human eye. An optional illumination unit 104 may provide artificial lighting. The camera may be a visible light or an infrared camera. By using infrared imaging and infrared illumination, the system may be used in the dark, e. g. if the system is attached to or comprised in a vehicle and determining eye movement of a driver at night. However, the present disclosure is not limited to vehicles and may be used in other environments. The input unit 106 is configured to receive one or more predetermined criteria for eye openness and eye movement. The criteria for eye openness may comprise minimum and maximum values for one or more of a blink duration, a number of blinks per minute, a time between blinks, and a maximum blink velocity during a movement of the eyelid when the eye is being opened and/or closed. The eye movement criteria may comprise one or more of a saccade amplitude, saccade velocity, saccade duration, and fixation duration. The criteria may be entered into the system and stored in a storage unit 110, which may comprise a non-volatile memory, in the computing device 108.

[0071] Upon operation, the camera 102 records a series of images. The images are first analyzed by an eye openness analysis unit 112, which determines a continuous signal indicative of an eye openness. The signal may relate to a maximum distance between upper and lower eyelid, and it may be subject to noise and measurement errors, such as an erroneous value due to the person turning the head away from the camera. The preliminary classifier 116 determines whether the signal exceeds a first threshold, and classifies the signal at that point in time as indicative of an open eye or a closed eye. Comparator 118 receives the output of the preliminary classifier 116 and determines whether the signal complies with the criteria. If, for example, two blinks are determined at a temporal delay that is below a threshold defined by physiological limits, the system may determine that the signal is not compliant with the criteria. In response to a determination that the signal is not compliant with the criteria, the classification modifier 120 changes the classification in a way that avoids an erroneous signal. For example, the classification may be determined as inconclusive for the period in time in which it is not compliant with the criteria. However, one or more additional or alternative classifications may be generated, that are indicative of an open eye or a closed eye under certain conditions. If, for example, the signal of only one image is indicative of a closed eye, whereas the signal indicates an open eye for the preceding and subsequent images, the signal may be considered indicative of an open eye for the purpose of determination of a blink period.

[0072] The image generated by the camera and the output of the eye openness analysis unit 112 are sent into an eye gaze analysis unit 122. The eye gaze analysis unit 122 is configured to analyze the images that have been determined to represent an open eye. Analysis of images depicting closed eyes or images where the first signal is inconclusive is thus avoided, and therefore, a first source of error is mitigated. The corresponding periods in time are marked as inconclusive of the eye gaze direction and movement. For the images representing an open eye, a signal generator 124 generates a signal indicative of the eye gaze, which may be expressed as either a pair of Euler angles, or as positions of the pupil relative to a center of the eye. Furthermore, a numerical derivative is calculated to determine a velocity of the eye movement. The preliminary classifier 126 classifies periods in time where the velocity exceeds a second threshold as saccades, and periods in time where the velocity is below the second threshold as fixations. Comparator 128 then verifies for each point in time if the classification complies with the eye movement criteria. If this is not the case, for example if a velocity exceeds a threshold indicative of a physiologically possible eye movement velocity, the classification modifier 130 may set the classification as inconclusive as to whether a fixation or a saccade is present. Thereby, every point in the time during which the images are taken is unambiguously classified as blink, saccade, fixation, or inconclusive.

[0073] FIG. 2 depicts a flow chart representing a method for determining eye movement according to an embodiment. The method begins, 202, when operation of the system 100 is initiated. If the system 100 is comprised in a vehicle, this may begin, e. g., once the driver is seated in the vehicle. A series of images is recorded, 204. For a practical eye tracker, the camera may run continuously, and the computing device performs a live analysis of the last seconds of video, i. e. the last recorded images. The system may thus analyze a sliding window in time. The images may be subject to noise and measurement errors. If for example, the camera or the head of the person is moving, the eye openness value may be wrong. Other sources of error may comprise electronic noise in case of a low signal in a dark environment, or a saturated camera in case of exposure to sunlight. A first signal of the openness of the eyes, e. g. a maximum distance between upper and lower eyelid, is generated, 206. The signal corresponding to a point in time may either correspond to a single frame or to a plurality of subsequent frames over which the signal is averaged. This defines a smallest time unit to which a point in time refers. Furthermore, filtering techniques as known in the art may be combined with the method of the present disclosure. The eye openness is then preliminarily classified, 208, for example by determining if the signal exceeds a threshold. Thereby, a binary signal is generated. This signal may, however, be subject to errors in case of the aforementioned effects. If, for example, the first signal is fluctuating and close to the threshold, the preliminary classification may create the impression of a fast fluctuation of closed and open eyes, which may physiologically not be possible. The result is then compared, 210, to the predetermined criteria as stored in the data storage unit 110, which may then, for example lead to the determination that the time between blinks as determined is lower than a minimum time between blinks as given in the criteria. In another example, this step may further include determining an experimental blink velocity by calculating a derivative of the first signal and comparing this to a physiological minimum or maximum. For points in time for which the signal does not comply with the criteria (blink duration, blinks per minute, time between blinks, blink velocity), the classification is then modified, 212, to indicate that the signal is inconclusive as to whether the eye is open or closed. Thereby, the system is prevented from generating an indication of an open or closed eye in case the signal is unclear. Thereby, the reliability of the eye openness classification is increased. The eye openness classification may then be used directly for one or more applications, such as detecting driver drowsiness. Furthermore, the classification may be used for determining eye gaze and/or eye movement features as described with respect to FIG. 3.

[0074] FIG. 3 depicts a flow chart representing a method 300 for determining eye movement according to an embodiment. The method receives, 202, the camera images and an eye openness classification generated in method 200. A second signal, indicative of eye movement, such as a pair of Euler angles, is generated, 304. The velocity is determined, 306, e. g. by calculating a derivative. A preliminary eye gaze classification is created based on the velocity. Thereby, for each point in time, the velocity is compared to a second threshold, and the data are classified as a saccade in case the velocity exceeds the second threshold, or a fixation in case the velocity is below the second threshold. Hereby, a point in time may refer to a time window of a small size, for example 3 frames, which may be recorded in an interval of 100 milliseconds. The classification is further compared, 310, to the predetermined criteria. For example, the amplitude of the saccade may be determined to a physiologically unrealistic value such as a 100 degrees angle, or a saccade velocity may be determined to be higher than realistic for a human eye. In these cases the corresponding points in time are reclassified as inconclusive. Furthermore, a check is performed if the saccades and fixations form contiguous periods in time that comply with predetermined criteria for minimum and maximum durations of saccades and fixations. Periods in time that do not comply with the criteria are reclassified as inconclusive, 312. Thereby, the reliability of the eye movement determination method is increased. Furthermore, filtering the first and second signals is optionally possible, but it is not necessary. Thereby, possible distortion of the signal is avoided, and all features of the signal are preserved.

[0075] In the end, 314, the classification of the eye gaze is provided for further analysis. For example, for the fixation periods, an averaged direction may be determined. Thereby, it can be determined at which object the person is looking. This information may be used for an augmented reality system. The data may also be used for determining a cognitive demand.

REFERENCE SIGNS

[0076] 100 System [0077] 102 Camera [0078] 104 Illumination [0079] 106 Input unit [0080] 108 Computing device [0081] 110 Storage unit [0082] 112 Eye openness analysis unit [0083] 114 Signal generator [0084] 116 Preliminary classifier [0085] 118 Comparator [0086] 120 Classification modifier [0087] 122 Eye gaze analysis unit [0088] 124 Signal generator [0089] 126 Preliminary classifier [0090] 128 Comparator [0091] 130 Classification modifier [0092] 200 Method for determining eye openness [0093] 202-214 Steps of method 200 [0094] 300 Method for determining eye movement [0095] 302-314 Steps of method 300