Wearable eye tracking system

Abstract

There is provided a method and wearable eye-tracking device for determining a fatigue level of a user, the method comprising the steps of acquiring two channels of an observed EEG (electro-encephalogram) signal using a plurality of silver chloride (AgCl) electrodes positioned in contact with and around the user's ear, obtaining user's inputs for a plurality of psychological questions and calculating an evaluation metric, decomposing the observed EEG signal using filter and blind signal separation techniques into a plurality of features, classifying and converting the plurality of features in combination with the calculated evaluation metric to a fatigue level using a classification algorithm and fuzzy logic and outputting the obtained fatigue level along with customized prompts to the user through visual and audio signals for preventing an accident.

Claims

1. A method of preventing drowsy driving, low concentration and bad decision making of a user using an eye-tracking device positioned around the user's ears, the method comprising the steps of: obtaining a temporal EEG (electro-encephalogram) signal using two EEG electrodes, separated by 1 cm and configured to be in contact with the user's ear, wherein a surface of each electrode comprises a plurality of solid gold balls for increasing conductivity of the two EEG electrodes; processing the obtained temporal EEG signal to generate information, classifying the information using an adaptive deep learning classifier; applying a fuzzy logic classifier to a combination of the classified information and the user's inputs from a plurality of psychological questions to obtain final results, wherein the final results comprise a calculated fatigue scale, decision-making score and concentration level; and determining a condition of the user from the final results and sending a customized visual or audio notification signal to the user depending on particular user profiles including workers, drivers and students, wherein the customized visual or audio notification comprises instructions including instructions for stopping a vehicle, washing face, having a coffee or for taking a power nap.

2. The method of claim 1, wherein processing the obtained temporal EEG signal comprises: decomposing the obtained EEG signal using filter and blind signal separation techniques into a plurality of features.

3. The method of claim 2, wherein the plurality of psychological questions are customized based on a profile of the user, thereby enabling the customized visual or audio notification signal to be provided to the user depending on the determined condition of the user.

4. The method of claim 2, wherein the features are the generated information, and they are subsequently combined with a calculated evaluation metric based on the user's inputs as input to the fuzzy logic classifier.

5. The method of claim 2, wherein an output of the fuzzy logic classifier is a percentage of the user's fatigue level.

6. The method of claim 1, wherein the condition of the user is a fatigue level, a drowsiness level, or a physiological status.

7. The method of claim 1, wherein the obtained EEG signal comprises left and right electro-oculography (EOG) signals.

8. The method of claim 1, wherein the processing of the EEG signal comprises decomposing the EEG signal into features comprising alpha and beta frequency bands, slow eye movements, blinking amplitudes and patterns and eletromyography (EMG) amplitudes.

9. The method of claim 1, wherein the eye-tracking device is operatively connected with a mobile application installed on a mobile device, and wherein the processing of the EEG signal is conducted using a mobile application running on the mobile device.

10. The method of claim 1, wherein the electrodes are selected from the group of silver chloride (AgCl) electrodes and AgCl electrode plates.

11. The method of claim 1, wherein the user is able to view the final results through a mobile application.

12. A wearable eye-tracking device for preventing drowsy driving, low concentration and bad decision making of a user, the eye-tracking device comprising: two EEG (electro-encephalogram) electrodes separated by 1 cm and configured to be in contact with the user's ear for obtaining a temporal EEG signal from the user, a microprocessor for processing the obtained temporal EEG signal to generate information, classifying the information using an adaptive deep learning classifier, and determining a condition of the user by applying a fuzzy logic classifier to the classified information in combination with the user's inputs for a plurality of psychological questions to obtain final results, wherein the finals results comprise a calculated fatigue scale, decision-making score and concentration level, and a communication unit in electrical communication with the microprocessor for sending a customized visual or audio notification signal to the user depending on particular user profiles including workers, drivers and students, wherein the customized visual or audio notification comprises instructions including instructions for stopping a vehicle, washing face, having a coffee or for taking a power nap, and wherein a surface of each electrode comprises a plurality of solid gold balls for increasing conductivity of the two EEG electrodes of the eye-tracking device.

13. The wearable eye-tracking device of claim 12, wherein processing the EEG signal comprises processing two channels of eye movements or left and right electro-oculography (EOG) signals.

14. The wearable eye-tracking device of claim 13, wherein processing the EEG signal comprises: decomposing the obtained EEG signal using filter and blind signal separation techniques into a plurality of features.

15. The wearable eye-tracking device of claim 12 wherein the communication unit comprises a Bluetooth communication system and at least one of a tri-colored LED indicator, a speaker and a vibrator.

16. The wearable eye-tracking device of claim 12, wherein the two electrodes are selected from the group of AgCl electrodes and AgCl electrode plates.

17. The wearable eye-tracking device of claim 12, wherein the device is adapted to be worn by the user such that it is positioned away from a visual field of the user, and wherein the device further comprises a support for enabling the device to be fixed around the user's ear during operation.

18. The wearable eye-tracking device of claim 12, wherein the processing of the EEG signal comprises decomposing the EEG signal into features comprising alpha and beta frequency bands, slow eye movements, blinking amplitudes and pattern and electromyography (EMG) amplitudes.

19. The wearable eye-tracking device of claim 12, wherein the user is able to view the final results through a mobile application.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The subject matter that is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other aspects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which:

(2) FIG. 1 illustrates a basic design of the eye tracking device, in accordance with the present invention.

(3) FIG. 2 depicts a flow chart of the eye tracking system, in accordance with the present invention.

(4) FIG. 3 (a) is a block diagram representing the various electronic components used for recording a user's brain waves and eye movements, and FIG. 3 (b) is the electrical circuit for filter and amplifier used in accordance with the present invention.

(5) FIG. 4 shows the algorithm flow chart, in accordance with the present invention.

DETAILED DESCRIPTION OF THE INVENTION

(6) The aspects of the device or system for tracking eye movements device and thereby detecting signs of fatigue and drowsiness in a user, according to the present invention will be described in conjunction with FIGS. 1-4. In the Detailed Description, reference is made to the accompanying figures, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. It is to be understood that other embodiments may be utilized and logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

(7) The proposed solution aims at designing a device for recording brain activity and thereby tracking various movements of a user's eyes. This primarily involves obtaining electrical signals using a compact radio device worn around the user's ears, which may additionally be used to detect frequency of blinking, yawning and direction of view of the user. Detection of these electrical signals leads to monitoring user or driver alertness, the degree of tiredness and exhaustion associated with the user or driver (along with other features such as the duration of travel, number of hours the user previously obtained sleep, and whether the user is driving at night or during the day). Upon detection by the device that the user is tired or drowsy, alerts are sent to a mobile application installed on an electronic device or wireless device, such as tips for the user to increase his attentiveness or vigilence.

(8) The electronic device includes, but is not limited to a smart phone, a mobile phone, a personal digital assistant (PDA), an e-book reader, a notebook computer, GPS receivers and other devices that include appropriate hardware and software components for processing information. Wireless device in general includes but is not limited to, wireless cell phones, computers with wireless WAN connections, computers with wireless LAN connections, or other electronic devices capable of connecting to wireless networks.

(9) In an embodiment of the present inevntion, the alerts or tips sent to the mobile application include instructions to stop the vehicle, wash face, have a coffee or to take a power nap. Further, at any point of operation of the eye tracking device, the user is able to view his or her alertness level through the mobile application (in percentage form e.g., 10% or 98% alertness).

(10) EEG recording is an advantageous alternative to measure eye movements, considering that EEG signals include EOG artifacts, and is appropriate for use in applications such as driving safety systems because the sensors used do not reduce the field of vision. Another advantage of this method is the inclusion of established positions for EEG sensors around the user's ears.

(11) FIG. 1 illustrates a design of the proposed eye-tracking device to record EOG and EEG signals. The device is designed such that the electrodes 101 are positioned around the user's ears, as this position is most suitable for distinguishing between four cardinal eye directions (or cardinal positions of gaze) of the user, namely upwards, downwards, right and left directions (which are divided into horizontal and vertical gaze).

(12) The proposed eye-tracking device further includes a support 102 for holding the device on the user's ears, a USB or battery slot 103 with ground terminal, GND 104. Also included is a tri-colored LED indicator 104, a speaker 105 and a power button 106. In an embodiment, the device is used bya driver behind a steering wheel 107, wherein the device is linked to an electronic device or mobile phone 109 via BLUETOOTH connection depicted as 108. As shown in the flow chart of FIG. 2, the proposed eye-tracking system includes a hardware section 201, an algorithm 202 and a mobile application 203 for monitoring and detecting fatigue scale, decision-making capabilities and concentration levels of the user, and in order to take necessary preventive actions. The eye-tracking device (hardware section) 201 and the mobile application 203 are linked via BLUETOOTH technology. The eye-tracking device 201 conducts functions such as acquisition of EOG and EEG signals, transmits and inputs these acquired signals to the algorithm section 202, whose functionalities are to classify the acquired signals and detect a level of fatigue or drowsiness. This classification and detected level (fatigue scale) are then transsmitted to the mobile application 203 in a human readable form, along with necessary preventive measures (or tips) that need to be taken by the user to avoid any mishappenings.

(13) Considering the hardware of the proposed eye-tracking system, FIG. 3 depicts each electronic component used to record brain activity and eye movements (using a low cost device). FIG. 3 (a) is a block diagram representing the various electronic components used for recording a user's brain waves and eye movements, and FIG. 3 (b) is the electrical circuit for filter and amplifier used in accordance with the present invention. The relevant components include a new design for the EEG-EOG electrodes using AgCl 302—each of 1 cm length in contact with the user's skin. The shape of the proposed electrode is very important in order to record a high quality of the EEG signal around the user's ears. Two separate plates or electrodes of AgCl, placed with a spacing of 1 cm between them, are used to play the role of electrical conductors that have an electric potential difference surrounded by an insulator. The surface of each electrode has a rough texture with numerous small solid balls (these balls may be made of gold to achieve high conductivity).

(14) Further components include two buffers 304—for example LMP7708, a differential amplifier 306 for exapmle TL074, filters (including amplifier using a potentiometer) 307—for example TL072CP, an analog to digital converter (ADC) 308—for example MICROCHIP PIC17F876 controller, a microcontroller 310—for example AT90S4433 or ATmega8, or Programmable System-on-Chip (PSoC) CYBL10X6X, a power system 312—such as a small 3V or 9V battery, an LED and speaker 314 and a wireless transmitter (BLUETOOTH v2.1) 316. The filters used in the circuit may be a high-pass filter (0.5 Hz), low-pass filter (30 Hz) along with an amplifier using a potentiometer. The low cost wearable hardware in accordance with the present invention is able to detect eye movements (via EOG signals), blinking patterns, detection of whether eyes are open or closed, temporal EEG signals, yawning patterns and so on.

(15) In another embodiment of the present inevtion, the proposed design has a multi-function capability based on an algorithm for detecting warning signs of fatigue and drowsiness further linked with a mobile application for measuring fatigue scale, monitoring associated decision making capabilities and concentration levels to take preventive actions for its users (which may include, but is not limited to drivers, insurance companies, workers or students). In a preferred embodiment of the present invention, the eye tracking device functions based on an algorithm which deploys feature extraction and analysis, the features comprising amplitudes, peak velocities and durations, frequency bands, etc. Various criteria (neuroscience facts) used for the calibration of the proposed algorithm includes that people blink more when they are tired (the average person blinks 15-20 times per minute), more the person is motivated and active, more his or her signal amplitude will be, slow eye movement (SEM) is an effective indicator for predicting delayed responses to take preventive actions and when a person is driving, his or her eyes must not stop moving for a more than 10 seconds at a time.

(16) The following equation summarizes all signals recorded using the proposed eye-tracking device:
EEG(t).sub.observed=EEG(t).sub.source+EOG(t)+EMG(t)+Artifacts
Wherein, EEG(t).sub.observed includes raw electrical signals recorded from around the user's ears, EEG(t).sub.source includes brain activity only to detect frequency bands (alpha or beta), EOG(t) includes detected gaze directions and eye movements related to eyeball, EMG(t) includes head movements and yawning patterns observed and the artifacts include any other minor signs of drowsiness such as excess blinking patterns, etc.

(17) The primary focus is on monitoring and observing brain activity (EEG.sub.source), user eye movements (EOG) and muscle activity such as yawning frequency, head movements and any other relevant artifacts such as user blinking patterns. The algorithm in accordance with the present invention as depicted as a flow chart in FIG. 4, deals with the classification of warning signs such as fatigue and drowsiness and linking the observations with a mobile application for viewing fatigue scale, decision-making and concentration levels in order to take necessary preventive actions. The algorithm is as follows—INPUT: User's answers of the psychological questions+User history+EEG.sub.observed and OUTPUT: Fatigue scale, Decision-making score, Concentration levels, associated and necessary preventive actions Initialize Questionnaire evaluation metric extracted from analyzing the user's answers OR user's history in case user is not a first-time user of the device Decomposing the observed raw signal using filter and blind signal separation method to EOG, brain source signal, and strong and weak EMG signals Feature extraction for building inputs vector from EEG observed: Alpha and Beta frequencies of brain activity (EEG), slow eye movements (SEM), blinking frequency and two channels of eye movements (right and left EOG signals), EMG amplitude, and number of blinks along with its amplitude, etc. In calibration phase, the classifier (adaptive deep learning) is trained to formulate a model every time a user uses the device, and every model formulated is saved in a small memory.

(18) Loop for each new cycle: Each new input measured is used to test the model-based machine learning. The model results and questionnaire evaluation results are converted to fatigue scale, decision-making and concentration levels using fussy logic in order to take preventive actions based on a choice of the user. A final decision is sent as a complete result list (based on the answers of the psychological questions+EEG classification) to a smartphone application to be converted to commands and further to control the LED and speaker. End for

(19) In another embodiment of the present invention, the eye-tracking device is wirelessly connected to an electronic device, or a mobile application. The proposed smartphone application uses the calculated degree of drowsiness in order to communicate results of the proposed algorithm with the user and provide instructions or alerts to take preventive actions. In another embodiment, the smartphone application allows the user to choose between various profiles such as worker, student or driver, so that the user may obtain a customized set of psychological questions on the basis of the calculated score of drowsiness or concentration and decision making levels. This customizable feature is also to ensure safety of the user and those around him. For example, a worker or laborer who works long shifts in dangerous environments is required to remain active and stay vigilant (thereby also reducing losses borne by insurance companies owing to traffic or work accidents).

(20) In an embodiment, only the results of the classification algorithm, subsequent to applying adaptive learning and fuzzy logic models (which are implemented via a software within the mobile application) are transmitted to the eye-tracking device (the hardware, particularly to the microcontroller, which contains one or more CPUs (processor cores) along with a memory and programmable input/output peripherals). In a situation where there is a break of communication between the smartphone and the proposed eye-tracking device—the processing is done within the hardware itself (it works like a backup). However, in normal cases, in order to preserve battery power, all the processing is done by the mobile application.

(21) The following table explains a sample case of calculating the degree of drowsiness of a user. In addition, the degree of user drowsiness or fatigue is proportional to concentration and decision-making levels. Concentration level is proportional to Beta band power since Beta band is associated with focused concentration and best defined in central and frontal brain areas. However, thinking of something peaceful with eyes closed results in an increase of alpha activity.

(22) TABLE-US-00001 USER PROFILES Preventive Student Worker Degree of Scale Action or Driver Concentration Decision- Drowsiness rating Feedback Fatigue scale scale making scale Functioning at 10% Green light Best time to Best time to Best time to peak pick your study and make an destination practice important decision Functioning at high 20% Green light Best time to Best time to Best time to level, but not at pick your study and make an peak; Able to destination practice important concentrate decision Feeling active, vital 30% Green light Best time to Best time to Best time to and alert pick your study and make an destination practice important decision Awake, but 40% Orange light + Best time to Best time to Best time to relaxed; Coffee pick your study and make an Responsive but not break destination practice important fully alert - Level 1 decision Awake, but 50% Red light + You can still You can still It is not relaxed; 1 beep + drive but be study if you recommended Responsive but not tips carful have exams but to take fully alert - Level 2 it might be important better to take decisions some rest Somewhat foggy; 60% Red light + You can still You can still It is not slowing down 5 beeps + drive but be study if you recommended tips carful have exams but to take it might be important better to take decisions some rest Fighting 70% 5 beeps + It might be You can still It is not drowsiness - Level strong better to stop study if you recommended 1 vibration driving have exams but to take it might be important better to take decisions some rest Fighting 80% 10 beeps + Stop driving, You are wasting Don't take an drowsiness - Level strong your life is in your time. Your important 2 vibration dangerous brain needs decision some rest Foggy and losing 90% 10 beeps + Stop driving, You are wasting Don't take an interest in staying 5 strong your life is in your time. Your important awake; slow down vibrations dangerous brain needs decision some rest Closed eyes, 100% Red light + Stop driving, You are wasting Don't take an drowsing, Fighting Non-stop your life is in your time. Your important sleep Asleep with strong beeps dangerous brain needs decision open eyes, looking and some rest but not seeing vibrations

(23) The displayed contents or results are based on the answers input by the user for the psychological questions along with using a classification algorithm on the observed or recorded EEG signals. In another embodiment of the present invention, a classification algorithm is developed and used for distinguishing between brain activity (using Alpha and Beta frequency bands for sleeping time and awake time, concentration intensity and decision making capabilities) and eye movements (based on detection of blinking patterns and gaze directions). The proposed algorithm (which is based on answers to the psychilogical questions and the detected EEG signals) converts the signals obtained from the user into a a degree of drowsiness, wherein Alpha bands appears when the user is drowsy or relaxed and Beta bands appears when the user is awake or active.

(24) In another preferred embosiment of the present invention, the proposed eye-tracking device is linked with a mobile application installed on an electronic device. The feature which differentiates the proposed design from traditionally used devices is that the proposed eye-tracking device is compact and suitable for simply fixing around the user's ears, as a result of which no obstructions are created to the field of vision of the user. Further, the proposed device is a low-cost hardware. This invention is maily intended to solve drowsy driving issues and to prevent drivers from falling asleep momentarily, and thereby to avoid car accidents using an affordable or low-cost device. The proposed device also is able to check fatigue and concentration levels of workers or students to improve their decision-making capabilities.

(25) The proposed eye-tracking system monitors and records features such as brain activity or brain waves, eye movements, winking and blinking patterns, head movements and also history data of the user (combination of physiological signals and user's answers from a questionnaire). Each user possesses different fatigue scales using the same algorithm. For example, if the user is a student, the mobile phone application presents a scale of fatigue or concentration along with some advice for students. If the user is a driver, the application presents fatigue scale with some advice for drivers. In addition, if the user is a shift worker, the application will present fatigue or decision-making scale along with additional advice related to shift workers. The proposed algorithm uses only two EEG electrodes around the user's ear to record numerous relevant signals that are classified into patterns to be used for deep learning method. Subsequently, the combination of the results from deep learning and the questionnaire evaluation results are used as input for fussy logic classifier to output to the user a percentage of his or her fatigue.

(26) This proposed design of a low-cost wearable eye tracking device is further aesthetically suitable for both men and women. Even a woman who wears a veil (Hijab) or a man who wears traditional clothes can utilize this good looking device hidden around their ears (preferably two similar devices around each ear to have accurate results). The device is beneficial for drivers, insurance companies, workers, and students to improve their daily-life performances, and is useful in various fields and suitable for daily life applications. One of its important applications is for detecting the warning signs of fatigued and drowsy driving with mobile phone application. The users are not required to physically look at the electronic device during use, since the mobile application provides the user with timely auditory, visual and haptic warnings or alert messages.

(27) Many changes, modifications, variations and other uses and applications of the subject invention will become apparent to those skilled in the art after considering this specification and the accompanying drawings, which disclose the preferred embodiments thereof. All such changes, modifications, variations and other uses and applications, which do not depart from the spirit and scope of the invention, are deemed to be covered by the invention, which is to be limited only by the claims which follow.