Non-contact identification of sleep and wake periods for elderly care

Abstract

Determining sleep patterns of a user includes detecting a plurality of point clouds, each corresponding to a different position of the user at different times, forming a plurality of bounding boxes, each corresponding to coordinates of captured points of one of the point clouds, creating a wake/sleep classifier based on features of the point clouds, determining sleep positions of the user as a function of time based on the bounding boxes, and determining sleep patterns of the user based on the sleep positions of the user and on results of the sleep/wake classifier. Detecting a plurality of point clouds may include using a tracking device to capture movements of the user. The features of the point clouds may include intermediate data that is determined using scalar velocities of points in the point clouds, absolute velocities of points in the point clouds, and/or counts of points in the point clouds.

Claims

1. A method of determining sleep patterns of a user, comprising: detecting a plurality of point clouds, each corresponding to a different position of the user at different times; forming a plurality of bounding boxes, each corresponding to coordinates of captured points of one of the point clouds; creating a wake/sleep classifier based on features of the point clouds; determining sleep positions of the user as a function of time based on the bounding boxes; and determining sleep patterns of the user based on the sleep positions of the user and on results of the sleep/wake classifier.

2. A method, according to claim 1, wherein detecting a plurality of point clouds includes using a tracking device to capture movements of the user.

3. A method, according to claim 2, wherein the tracking device uses radar.

4. A method, according to claim 1, wherein the features of the point clouds include intermediate data that is determined using at least one of: scalar velocities of points in the point clouds, absolute velocities of points in the point clouds, and counts of points in the point clouds.

5. A method, according to claim 4, wherein at least some of the features are filtered according to distance from a tracking device that is used to detect the plurality of point clouds.

6. A method, according to claim 5, wherein the intermediate data includes at least one of: a bag of point counts corresponding to a set of point counts at a series of sequential time frames, a bag of velocities corresponding to a set of point velocities at a series of sequential time frames, and a bag of absolute velocities corresponding to a set of absolute velocities at a series of sequential time frames.

7. A method, according to claim 6, wherein a set of aggregating, scaling and filtering functions are applied to the intermediate data to provide short-term feature aggregation values and mid-term feature aggregation values.

8. A method, according to claim 7, wherein the short-term feature aggregation values are determined based on time slots corresponding to a number of sequential time frames.

9. A method, according to claim 8, wherein the feature aggregation values include at least one of: mean values, median values, sum of values, minimum values and maximum values, and scaling function include logarithmic scaling function values.

10. A method, according to claim 8, wherein the mid-term feature aggregation values are derived from the short-term feature aggregation values.

11. A method, according to claim 10, wherein the mid-term feature aggregation values are determined based on epochs that represent contiguous collections of time slots.

12. A method, according to claim 1, wherein the features of the point clouds are used with truth information as training data for machine learning to provide an assessment of relative feature importance.

13. A method, according to claim 12, wherein the assessment of relative feature importance is determined using random forest machine learning.

14. A method, according to claim 1, wherein determining sleep positions includes determining if a breathing direction of the user is vertical or horizontal.

15. A method, according to claim 14, wherein, if the breathing direction is vertical, the sleep position is determined to be that the user is lying on the back of the user in response to a heart area of the user being detected on a left side of the user and the sleep position is determined to be that the user is lying on the stomach of the user in response to the heart area of the user being detected on a right side of the user.

16. A method, according to claim 14, wherein, if the breathing direction is horizontal, the sleep position is determined to be that the user is lying on a left of the user in response to a heart area of the user being detected in a relatively lower disposition and the sleep position is determined to be that the user is lying on a right side of the user in response to the heart area of the user being detected in a relatively upper disposition.

17. A method, according to claim 1, wherein sleep patterns of the user are determined based on correspondence of the sleep positions of the user with the results of the sleep/wake classifier as a function of time.

18. A method, according to claim 17, further comprising: tracking daily sleep patterns for the user.

19. A method, according to claim 18, further comprising: detecting a significant deviation from the daily sleep patterns.

20. A method, according to claim 19, further comprising: providing an alarm in response to detecting the significant deviation from the daily sleep patterns.

21. A non-transitory computer readable medium containing software that determines sleep patterns of a user, the software comprising: executable code that detects a plurality of point clouds, each corresponding to a different position of the user at different times; executable code that forms a plurality of bounding boxes, each corresponding to coordinates of captured points of one of the point clouds; executable code that creates a wake/sleep classifier based on features of the point clouds; executable code that determines sleep positions of the user as a function of time based on the bounding boxes; and executable code that determines sleep patterns of the user based on the sleep positions of the user and on results of the sleep/wake classifier, wherein the features of the point clouds include intermediate data that is determined using at least one of: scalar velocities of points in the point clouds, absolute velocities of points in the point clouds, and counts of points in the point clouds.

22. A non-transitory computer readable medium containing software that determines sleep patterns of a user, the software comprising: executable code that detects a plurality of point clouds, each corresponding to a different position of the user at different times; executable code that forms a plurality of bounding boxes, each corresponding to coordinates of captured points of one of the point clouds; executable code that creates a wake/sleep classifier based on features of the point clouds; executable code that determines sleep positions of the user as a function of time based on the bounding boxes; and executable code that determines sleep patterns of the user based on the sleep positions of the user and on results of the sleep/wake classifier, wherein determining sleep positions includes determining if a breathing direction of the user is vertical or horizontal.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the system described herein will now be explained in more detail in accordance with the figures of the drawings, which are briefly described as follows.

(2) FIG. 1 is a schematic illustration of a room, furniture, non-contact tracking device and point clouds for various user states, according to an embodiment of the system described herein.

(3) FIG. 2 is a schematic illustration of an intermediate data assembly from raw data for short-term feature extraction, according to an embodiment of the system described herein.

(4) FIG. 3 is a schematic illustration of feature construction, according to an embodiment of the system described herein.

(5) FIG. 4 is a schematic illustration of machine learning and feature ranking, according to an embodiment of the system described herein.

(6) FIGS. 5A-5D are schematic illustrations of identifying sleeping positions of a user, according to an embodiment of the system described herein.

(7) FIG. 6 is a schematic illustration of determining patterns of sleeping position and turning during sleep, according to an embodiment of the system described herein.

(8) FIG. 7 is a system flow diagram illustrating system functioning in connection with sleep/wake classification, according to an embodiment of the system described herein.

(9) FIG. 8 is a system flow diagram illustrating system functioning in connection with identifying sleeping positions and turning patterns, according to an embodiment of the system described herein.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

(10) The system described herein provides a mechanism for continuous non-contact identification of sleep and wake periods of a user along with sleeping positions and patterns of turning in bed based on classifiers acquired through machine learning and other algorithms and utilizing velocity, coordinate and directional data collected from point clouds, obtained by an always-on tracking device, embedded into a room or other facility where the user resides.

(11) FIG. 1 is a schematic illustration 100 of a room 110, furniture, non-contact tracking device and point clouds for various user states. A user resides in the room 110 that has a tracking device 120 plugged into an AC outlet on a wall where a radar signal 125 is used to track object movement in the room. The room 110 has a door 130a, a window 130b and is furnished with a bed 140a, a table 140b, and a couple of chairs 140c, 140d. FIG. 1 illustrates a first point cloud 150 tracking the user walking to the window 130b. The first point cloud has a denser point cloud and a larger size bounding box compared with a second point cloud 160, corresponding to the user sitting on the chair 140c, and a third point cloud 170 showing the user lying on the bed 140a.

(12) FIG. 2 is a schematic illustration 200 of intermediate data assembly from raw data for short-term feature extraction. The tracking device 120 emitting the radar signal 125 produces point clouds 210a-210c with bounding boxes 220a-220c for frames 230a-230c. Each point 240 of the point clouds 210a-210c may be characterized by coordinates and radial scalar velocity, p={{right arrow over (x)}, v} ({right arrow over (x)} is a three-dimensional coordinate vector, V is a velocity having a sign that depends on a radial direction to/from the tracking device).

(13) A time slot 250, τ={t.sub.1, . . . t.sub.n}, includes n frames t.sub.i with m.sub.i points in the point cloud corresponding to the i-th frame, so that a first point cloud 260 has m.sub.1 points and a last, n-th point cloud 265 has m.sub.n points. A bag of velocities 270 for the slot τ includes all point velocities for the point clouds in the slot, .sub.τ={v.sub.1.sup.1, v.sub.1.sup.m.sup.1, . . . v.sub.n.sup.1, . . . v.sub.n.sup.m.sup.n}, while a bag of point counts 280 includes all point counts in the slot, .sub.τ={m.sub.1, . . . m.sub.n}.

(14) FIG. 3 is a schematic illustration 300 of feature construction for training purposes. Intermediate data, corresponding to a time slot T and captured by the tracking device, includes the bag of velocities 270 (.sub.τ), the bag of point counts 280 (.sub.τ) and a bag of absolute velocities 275 (.sub.τ.sup.+), which are unsigned values corresponding to the bag of velocities 270 (.sub.τ), as explained elsewhere herein. One or multiple aggregating, scaling and filtering functions 310 may be applied to the intermediate data, resulting in a set of short-term (slot-related) features 320 (.sub.τ) used as a training set in machine learning. Subsequently, short-term features are aggregated into mid-term, epoch-related features as follows: (i) an ordinary epoch 330 (ε) combines several adjacent slots 250; ordinary epochs form a sequence of time intervals of equal lengths from a start of data collection session; (ii) a centered epoch 340 (ε.sub.c) surrounding a particular slot 255 placed in a center of the slot 255, similarly to a configuration of sliding averages; (iii) another set of aggregating, scaling and filtering functions 350 is applied to all short-term features within an ordinary or a centered epoch to produce mid-term, epoch related features 360 (.sub.ε), 370 (.sub.ε.sub.c). Mid-term features are also added to the training set.

(15) Tables 380, 390 illustrate short-term and mid-term feature aggregation and are similar to Tables 1, 2, described above. In the table 380, the intermediate data .sub.τ, .sub.τ, .sub.τ.sup.+ are aggregated into six short-term features .sub.τ.sup.1−.sub.τ.sup.6 using four aggregation and scaling functions: two of the functions use aggregation through mean and median values without scaling, while two other ones of the functions add logarithmic scaling. The table 390 illustrates aggregation of two short-term features from the table 380, .sub.τ.sup.5 and .sub.τ.sup.6, into twelve mid-term features, six for ordinary epochs and six for centered epochs, using four different aggregation and filtering functions: two of the filtering functions use aggregation through sum and maximum values without filtering, while two other ones of the filtering functions add filtering by distance from the tracking device 120 of FIG. 1, as explained elsewhere herein.

(16) FIG. 4 is a schematic illustration 400 of machine learning and feature ranking. At a training phase, a machine learning module 410 processes a training set in a multi-dimensional feature space 420. The training set includes points 430, one point per slot τ; coordinates of each of the points 430 combine short-term features .sub.τ for the slot τ, mid-term features .sub.ε for the only ordinary epoch E that contains the slot τ, and mid-term features .sub.ε.sub.c for the only centered epoch ε.sub.c that has the slot τ as a center of the centered epoch ε.sub.c (see FIGS. 2, 3 and the accompanying texts for details). Each point of the training set corresponds to a sleep state 435 (illustrated with unfilled shapes) or a wake state 437 (filled shapes).

(17) The machine learning module builds a sleep/wake state classifier 440. Machine learning may employ a random forest method 450, whereby decision trees 460, corresponding to sleep outcomes 470 and wake outcomes 480 are created for the classification purpose. The random forest method also allows for feature ranking, which is illustrated by a table 490, which includes a top ten most important features from the table 390 in FIG. 3 (the table 490 is similar to the Table 3, discussed above).

(18) FIGS. 5A-5D are schematic illustrations of identifying sleeping positions of a user.

(19) FIG. 5A illustrates identification of a sleeping position of the user when the user is lying on a bed 510 and is sleeping on the back of the user as shown by a pictogram 520. In this case, the chest of the user, represented by an upper face 560c of a bounding box of a point cloud, is moving up and down between an upper position 590i (full inhale) and a lower position 590e (full exhale), as shown by a vertical line 580ud. The vertical movement is captured by the tracking device 120. A heart area 570 of the user, also identified by the tracking device 120 that detects heart beats, is shifted to the left side of the body of the user, represented by a left face 5601 of the bounding box. Accordingly, the combination of two features: the up-and-down oscillation illustrated by the vertical line 580ud of the bounding box of the point cloud and location of the heart area 570 closer to the left face 560l of the bounding box of the point cloud may serve as indicators of the sleeping position of FIG. 5A.

(20) FIG. 5B illustrates identification of a sleeping position of the user on the stomach of the user as shown by a pictogram 530. In this case, the back of the user represented by an upper face 560b of a bounding box of a point cloud, is moving up and down as shown by the vertical line 580ud, as explained in conjunction with FIG. 5A, while the heart area 570 is shifted further from a right face 560r of the bounding box (which corresponds to the right side of the body of the user). So, the combination of up-and-down oscillation illustrated by the vertical line 580ud of the bounding box of the point cloud and location of the heart area 570 further from the right face 560r of the bounding box of the point cloud may serve as indicators of the sleeping position of FIG. 5B.

(21) FIG. 5C illustrates identification of a sleeping position of a user on the right side as shown by a pictogram 540. In the sleeping position of FIG. 5C, the chest and back of the user move left-and-right, as shown by a horizontal line 580lr. Two extreme coordinates for a face 560b of a bounding box, corresponding to the back of the user, correspond to full inhale 595i and full exhale 595e. The heart area 570 is closer to an upper face 560l of a bounding box, corresponding to the left side of the body of the user. Therefore, the combination of the left-and-right oscillation illustrated by the horizontal line 580lr of the bounding box and the location of the heart area 570 closer to the upper side 5601 of the bounding box may be indicators of the sleeping position of FIG. 5C.

(22) FIG. 5D illustrates identification of a sleeping position of the user on the left side as shown by a pictogram 550. In the sleeping position of FIG. 5D, the chest and back of the user move left-and-right, as shown by the horizontal line 580lr and as explained in conjunction with FIG. 5C. The heart area 570 is shifted further from an upper face 560r of the bounding box corresponding to the right side of the body of the user. Thus, the combination of left-and-right oscillation illustrated by the horizontal line 580lr of the bounding box and the location of the heart area 570 further from the upper side 560r of the bounding box represent two indicators of sleeping position of FIG. 5D.

(23) FIG. 6 is a schematic illustration 600 of determining patterns of sleeping position and turning during sleep. Intervals 610 of sleeping in each of the positions 540, 520, 550, 530, as well as wake intervals 620, may be recorded from a time the user goes to bed until a wake-up time 630. Based on data regarding sleep intervals and sleeping positions, collected through a sufficiently long period of time and processed statistically, the system may generate analytic representations of sleep patterns, such as a time distribution function 640 of sleeping positions 645 by average time 647 in each sleeping position and a probability distribution function 650 of sleeping intervals 655 between turning in bed by probability 657 thereof.

(24) The system may subsequently track and process field data 660 corresponding to daily user behavior 665 and build analogous distribution functions 670, 680 for each daily sleep period. If the field distributions significantly deviate from long-term patterns 640, 650 (in the example in FIG. 6, both of the functions 670, 680 demonstrate dramatic difference from the patterns 640, 650), the system may generate an alarm 690 and alert care personnel about potential sleep problems encountered by the user.

(25) Referring to FIG. 7, a system flow diagram 700 illustrates system functioning in connection with the sleep/wake classification. Processing begins at a step 710, where key timing parameters are defined, such as a frame rate per second, as well as slot and epoch duration. After the step 710, processing proceeds to a step 715, where number of users and number of sessions per user for collecting training data set are defined. After the step 715, processing proceeds to a step 720, where users are chosen and device installations are performed. After the step 720, processing proceeds to a step 725, where a first user is selected.

(26) After the step 725, processing proceeds to a step 730, where a first data collection session for the current user is selected. After the step 730, processing proceeds to a step 735, where the system uses a non-contact device to track the user and record session data, including point velocities. After the step 735, processing proceeds to a step 740, where the system obtains and records truth info: the factual user sleep-wake state for each frame (for machine learning purpose), as explained elsewhere herein. After the step 740, processing proceeds to a step 745, where point clouds for various frames are cleaned up from noise and normalized. After the step 745, processing proceeds to a step 750, where frames are grouped into slots, as explained elsewhere herein (see, for example, FIG. 2 and the accompanying text). After the step 750, processing proceeds to a step 755, where slots are grouped into epochs (see, for instance, FIG. 3 and the accompanying text). After the step 755, processing proceeds to a step 760, where the system optionally applies distance related corrections or filtering, as explained elsewhere herein (see table 390 in FIG. 3 and the accompanying explanations). After the step 760, processing proceeds to a step 765, where the shot-term, slot-related features are calculated for all slots in the session. After the step 765, processing proceeds to a step 770, where ordinary and centered mid-term, epoch related features are calculated. After the step 770, processing proceeds to a step 775, where new points are added to the training data set in the feature space, as explained elsewhere herein (including FIG. 4 and the accompanying text).

(27) After the step 775, processing proceeds to a test step 780, where it is determined whether the selected data collection session is the last session for the current user. If not, processing proceeds to a step 782, where the next data collection session for the current user is selected. After the step 782, processing proceeds back to the step 735, described above, which may be independently reached from the step 730. If it is determined at the test step 780 that the selected data collection session is the last session for the current user, processing proceeds to a test step 785, where it is determined whether the current user is the last user. If not, processing proceeds to a step 787, where the next user is selected. After the step 787, processing proceeds back to the step 730, described above, which may be independently reached from the step 725. If it is determined at the test step 785 that the current user is the last user, processing proceeds to a step 790, where the system uses machine learning for the accumulated training set. After the step 790, processing proceeds to a step 792, where an optimal sleep-wake classifier is determined as the result of machine learning. After the step 792, processing proceeds to a step 795, where feature ranking is obtained, as explained elsewhere herein (see, for example, FIG. 4 and the accompanying text). After the step 795, processing is complete.

(28) Referring to FIG. 8, a system flow diagram 800 illustrates system functioning in connection with identifying sleeping positions and turning patterns. Processing begins at a step 810, where the number of user sessions for a training set is determined. After the step 810, processing proceeds to a step 815, where a first training session for the user is selected. After the step 815, processing proceeds to a step 820, where the system starts a training session. After the step 820, processing proceeds to a step 825, where a non-contact tracking device monitors the user. After the step 825, processing proceeds to a step 830, where the system records session data (point clouds) with point velocities, breathing direction and heart position. After the step 830, processing proceeds to a step 835, where features are calculated for applying user sleep/wake classifier (see details, in particular, in conjunction with FIG. 7 and the accompanying text; this step in the flow diagram assumes that the sleep/wake classifier is already available).

(29) After the step 835, processing proceeds to a step 840, where the system applies the sleep-wake classifier to features calculated at the step 835. After the step 840, processing proceeds to a test step 845, where it is determined whether the user is asleep. If not, processing proceeds to the step 825, described above, which may be independently reached from the step 820; otherwise, processing proceeds to a step 850, where a sleeping position of the user is identified utilizing breathing direction and location of the heart area, as explained elsewhere herein (see FIGS. 5A-5D and the accompanying text). After the step 850, processing proceeds to a test step 855, where it is determined whether the current sleeping position is the first captured sleeping position in the current training session. If so, processing proceeds to a step 862, where a duration of the current sleeping position is incrementally increased. After the step 862, processing proceeds back to the step 825, described above, which may be independently reached from the steps 820, 845.

(30) If it is determined at the test step 855 that the identified sleeping position is not the first sleeping position during the current session, processing proceeds to a test step 860, where it is determined whether a sleeping position of the user has changed (i.e. the identified sleeping position is different from the previously registered sleeping position). If not, then processing proceeds to the step 862, described above, which may be independently reached from the step 855; otherwise, processing proceeds to a step 865, where the system records full duration of the previous sleeping position and interval between turns in the bed. After the step 865, processing proceeds to a step 870, where the system updates statistics of sleeping positions and intervals between turns. After the step 870, processing proceeds to a test step 875, where it is determined whether the current session has reached the end. If not, processing proceeds to the step 825, which may be independently reached from the steps 820, 845, 862; otherwise, processing proceeds to a step 880, where the statistics of sleeping positions and intervals between turns for the completed training session are added to the training set. After the step 880, processing proceeds to a test step 885, where it is determined whether the current session is the last session. If not, processing proceeds to a step 890 where the next session is selected. After the step 890, processing proceeds back to the step 820, described above, which may be independently reached from the step 815. If it is determined at the test step 885 that the current session is the last session, processing proceeds to a step 895 where the system uses machine learning for the constructed training set to identify patterns of sleeping position and turning in the bed. After the step 895, processing is complete.

(31) Various embodiments discussed herein may be combined with each other in appropriate combinations in connection with the system described herein. Additionally, in some instances, the order of steps in the flowcharts, flow diagrams and/or described flow processing may be modified, where appropriate. Subsequently, system configurations and functions may vary from the illustrations presented herein. Further, various aspects of the system described herein may be implemented using various applications and may be deployed on various devices, including, but not limited to smartphones, tablets and other mobile computers. Smartphones and tablets may use operating system(s) selected from the group consisting of: iOS, Android OS, Windows Phone OS, Blackberry OS and mobile versions of Linux OS. Mobile computers and tablets may use operating system selected from the group consisting of Mac OS, Windows OS, Linux OS, Chrome OS.

(32) Software implementations of the system described herein may include executable code that is stored in a computer readable medium and executed by one or more processors. The computer readable medium may be non-transitory and include a computer hard drive, ROM, RAM, flash memory, portable computer storage media such as a CD-ROM, a DVD-ROM, a flash drive, an SD card and/or other drive with, for example, a universal serial bus (USB) interface, and/or any other appropriate tangible or non-transitory computer readable medium or computer memory on which executable code may be stored and executed by a processor. The software may be bundled (pre-loaded), installed from an app store or downloaded from a location of a network operator. The system described herein may be used in connection with any appropriate operating system.

(33) Other embodiments of the invention will be apparent to those skilled in the art from a consideration of the specification or practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.