DEVICE AND METHOD FOR DETECTING VITAL SIGN INFORMATION OF A SUBJECT

Abstract

The present invention relates to a device and method for determining vital sign information of a subject (2). The proposed device comprises an input unit (11) for receiving image data of the subject, said image data including a sequence of images over time, an ROI selection unit (12) for selecting an initial region of interest, ROI, within an image, a feature selection unit (13) for selecting one or more spatial features of a body part within the initial ROI, a motion signal extraction unit (14) for extracting from said image data within the initial ROI the direction and/or amplitude of motion of a subject's body part related to a desired vital sign of the subject, a detection unit (15) for detecting one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject, a tracking unit (16) for tracking the initial ROI based on changes of the position of said one or more detected spatial features within the initial ROI, to obtain a final ROI, and a vital signs extraction unit (17) for extracting the desired vital sign from the final ROI.

Claims

1. Device for detecting vital sign information of a subject, comprising: an input unit for receiving image data of the subject, said image data including a sequence of images over time, an ROI selection unit for selecting an initial region of interest, ROI, within an image of the sequence of images, said initial ROI comprising a body part of the subject, a feature selection unit for selecting one or more spatial features within the initial ROI, a motion signal extraction unit for extracting from said image data the direction and/or amplitude of motion of the body part related to a desired vital sign of the subject, a detection unit for detecting one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject, based on the extracted direction and/or amplitude of motion of the body part, a tracking unit for tracking the initial ROI by detecting changes of the position of said one or more detected spatial features and by shifting the position of the initial ROI based on the detected changes to obtain a final ROI, and a vital signs extraction unit for extracting the desired vital sign from the final ROI within images of said sequence of images.

2. Device as claimed in claim 1, further comprising an ROI re-initialization unit for re-initializing the final ROI as new initial ROI.

3. Device as claimed in claim 1, wherein said tracking unit is configured to use as final ROI the ROI obtained from tracking of the initial ROI after the position of the one or more spatial feature has become stable over several image frames.

4. Device as claimed in claim 1, wherein said feature selection unit is configured to select one or more spatial features within the initial ROI, which are invariant, preferably most invariant, to motion of the body part related to a desired vital sign of the subject.

5. Device as claimed in claim 1, wherein the motion of the body part related to a desired vital sign of the subject is breathing motion and wherein the vital sign information is respiratory information.

6. Device as claimed in claim 1, wherein the motion of the body part related to a desired vital sign of the subject is heartbeat motion and wherein the vital sign information is pulse rate information.

7. Device as claimed in claim 5, wherein said feature selection unit is configured to select one or more spatial features within the initial ROI, which are invariant, in particular most invariant, to breathing motion or heartbeat motion, in particular spatial features showing edges or lines in an image within the initial ROI arranged along the main direction of said breathing motion or heartbeat motion within the initial ROI.

8. Device as claimed in claim 1, wherein said detection unit is configured to detect one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject, by detecting changes of the position of said selected spatial features within the initial ROI and the extracted direction and/or amplitude of motion of the body part related to a desired vital sign of the subject.

9. Device as claimed in claim 1, wherein said detection unit is configured to detect one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject, by analyzing changes of distances between selected spatial features within the initial ROI, the direction of changes of the position of selected spatial features within the initial ROI and/or the deviation of changes of the position of selected spatial features within the initial ROI from the motion of the body part related to a desired vital sign of the subject.

10. Device as claimed in claim 1, wherein said ROI selection unit is configured to select the initial ROI within an image by dividing the image into spatial blocks, extracting a vital sign from a plurality of said spatial blocks, clustering of two or more adjacent spatial blocks, from which the strongest vital signs are extracted, and selecting said clustered spatial blocks as initial ROI.

11. Device as claimed in claim 1, wherein said ROI selection unit is configured to select the initial ROI from a previously obtained final ROI by extracting a vital sign from a plurality of spatial blocks adjacent to the obtained final ROI, clustering of two or more adjacent spatial blocks, from which the strongest vital signs are extracted, and selecting said clustered spatial blocks as new initial ROI.

12. Device as claimed in claim 11, wherein said ROI selection unit is configured to select the initial ROI from a previously obtained final ROI if the reliability or quality of the vital sign from the final ROI drops below a predetermined threshold or by predetermined amount.

13. Device as claimed in claim 1, wherein said motion detection unit is configured to detect if the initial ROI is at least partly occluded and wherein said vital signs extraction unit is configured to extracting the desired vital sign from non-occluded parts of the initial ROI.

14. Method for detecting vital sign information of a subject comprising: receiving image data of the subject, said image data including a sequence of images over time, selecting an initial region of interest, ROI, within an image, said initial ROI comprising a body part of the subject, selecting one or more spatial features within the initial ROI, extracting from said image data the direction and/or amplitude of motion of the body part related to a desired vital sign of the subject, detecting one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject, based on the extracted direction and/or amplitude of motion of the body part, tracking the initial ROI by detecting changes of the position of said one or more detected spatial features and by shifting the position of the initial ROI based on the detected changes to obtain a final ROI, and extracting the desired vital sign from the final ROI within images of said sequence of images.

15. Computer program comprising program code means for causing a computer to carry out the steps of the method as claimed in claim 14 when said computer program is carried out on the computer.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. In the following drawings

[0038] FIG. 1 shows a schematic illustration of a general layout of a system and device for determining vital sign information of a subject according to the present invention,

[0039] FIG. 2 shows a schematic illustration of a subject's motion indicative of exemplary vital sign information,

[0040] FIG. 3 shows a timing diagram of an alternating signal derived from the subject corresponding to the vital sign information,

[0041] FIG. 4 shows a flow chart of an embodiment of the method according to the present invention, and

[0042] FIGS. 5a, 5b, and 5c show a schematic image sequence for illustrating the steps of the method according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0043] FIG. 1 shows a schematic diagram of system 1 and a device 10 for determining vital sign information of a subject 100. The subject 2 in this example is a patient lying in a bed 3, e.g. in a hospital or other healthcare facility, but may also be a neonate or premature infant, e.g. lying in an incubator, or person at home or in a different environment. Besides the device 10 the system 1 generally comprises an imaging unit 20, such as a camera, for acquiring image data of the subject 2, said image data including a sequence of images over time.

[0044] The image frames captured by the imaging unit 20 may particularly correspond to a video sequence captured by means of an analog or digital photosensor, e.g. in a (digital) camera. Such a camera usually includes a photosensor, such as a CMOS or CCD sensor, which may also operate in a specific spectral range (visible, IR) or provide information for different spectral ranges. The camera may provide an analog or digital signal.

[0045] The device 10 comprises an input unit 11 for receiving the image data of the subject 2, either directly from the imaging unit 20 or from a storage, in which image data acquired earlier have been stored or buffered.

[0046] In an ROI selection unit 12 an initial region of interest (ROI) is selected within an image included in said image data. This can be done in different ways, e.g. manually (via an interface) by the user or automatically, e.g. based on one of the known methods mentioned above, such as the method disclosed in WO 2014/131850 A1.

[0047] A feature selection unit 13 selects one or more spatial features of a body part within the initial ROI. A spatial feature is generally a combination of neighboring pixels, which is distinctive from other combinations of neighboring pixels in the vicinity, and which stays distinctive even after scaling or motion of a pixel area. The spatial features are preferably easily identifiable features or feature points shown within the initial ROI, which can be tracked through the images of the image data. Examples of spatial features are corners, edges or lines (e.g. of a pattern printed onto the subject's clothes or the blanket) or anatomic features such as eyes or nose.

[0048] A motion signal extraction unit 14 is provided for extracting from said image data within the initial ROI the direction and/or amplitude (or amplitude range) of motion of a subject's body part related to a desired vital sign of the subject. For instance, if breathing motion of the chest area 4 of the subject 2 shall be monitored, the direction and/or amplitude of this breathing motion is determined, e.g. by use of pattern recognition or other image processing means. In case of breathing motion mostly vertical and horizontal breathing motion components may be determined. In other embodiments the heartbeat-related motion of the chest, carotid or head may be determined. In case a person is sitting in front of a camera, heartbeat would mostly be associated with motion of a head along vertical axis. That means that any (combination of) spatial feature which is invariant to vertical motion can be selected. For instance, a distance between eyes will not change during vertical motion of a head, but will change during head rotation.

[0049] Further, a detection unit 15 is provided for detecting one or more spatial features of said selected spatial features, whose motion is not related to the desired vital sign of the subject. Hence, it is distinguished between motion of spatial features which is related to the desired vital sign (e.g. movements of the chest caused by breathing or heartbeat, such as lifting and lowering of the chest wall and the abdominal wall due to breathing) and which is not related to the desired vital sign (e.g. movements of the chest e.g. in lateral or rotational direction due to motion of the patient, such as rolling or shifting of the whole body). Preferred embodiments for implementing the function of the detection unit will be explained below.

[0050] A tracking unit 16 is provided for tracking the initial ROI based on changes of the position of said one or more detected spatial features within the initial ROI, to obtain a final ROI. In other words, the spatial features with are unrelated with the desired vital sign and whose position is not influenced by the motion related with the desired vital sign (such as heartbeat motion or breathing motion) are used to determine if over time a motion of the body part (or even the whole body) of the subject appears which is not caused by a motion related with the desired vital sign. This information is then used to adapt the position of the initial ROI accordingly, i.e. to shift the initial ROI according to the detected motion (which is not related to the desired vital sign) to a new position, said shifted ROI representing the final ROI.

[0051] Finally, in a vital signs extraction unit 17 the desired vital sign, e.g. respiratory information, such as respiration rate, and/or heartbeat information, such as pulse rate, is extracted from the final ROI. The vital sign is particularly derived from the motion of the body part within the final ROI (or within the ROI as continuously tracked in the above described way), e.g. by determining the frequency of lifting and lowering of the chest wall and/or the abdominal wall caused by breathing or by determining the frequency of tiny head movements (up and down) caused by heartbeat.

[0052] The extracted vital sign can then either be further processed or issued, e.g. on a display of a patient monitor, a central monitoring station (e.g.in a nurse room), a handheld monitoring device (e.g. a smartphone of a nurse or doctor), etc.

[0053] Preferably, the device 10 may further comprise an ROI re-initialization unit 18, as indicated with broken lines in FIG. 1, for re-initializing the final ROI as new initial ROI. Hence, the monitoring and tracking of the ROI as explained above may be iteratively carried out to further improve the accuracy and reliability of the extracted vital sign(s). Hence, after re-initialization of the initial ROI the processing preferably continues with the selection of one or more spatial features of a body part within the new initial ROI as performed by the feature selection unit 13. In an embodiment the same spatial features as used in one or more previous iterations are used again so that the processing may also continue with the step carried out by the motion signal extraction unit 14 or the detection unit 15.

[0054] It shall be noted that one or more of the elements of the device 1 can be implemented by dedicated hardware, software or a combination thereof. For instance, in an embodiment the elements are implemented by processing means, such as a processor or computer, which are programmed accordingly.

[0055] FIG. 2 shows a schematic illustration of the subject for illustrating the remote measurement of respiratory information, such as the respiration rate, of the subject 2. The subject 2 undergoes a characteristic motion of an indicative portion 4 (such as the chest) due to the respiration. When breathing, expansion and contraction of the lungs causes slight motion of characteristic portions of living beings, e.g. lifting and lowering of the chest 4. Also, abdominal breathing can cause characteristic motion of respective parts of the subject's body. At least partially periodic motion patterns included by physiological processes can occur in many living beings, particularly in human beings or animals.

[0056] Over time, as indicated by an arrow 6, the indicative portion 4 is moved between a contracted position, indicated by reference numerals 4a, 4c, and an extracted portion, indicated by 4b. Essentially, based on this motion pattern, for instance the respiration rate or respiration rate variability can be assessed by means of pattern or edge detection in a captured image sequence. While the indicative portion 4 is pulsating over time, other non-indicative portions such as the head 5 remain substantially motionless (unless otherwise moved by the subject 2), i.e. the non-indicative portions also undergo diverse motion over time. However, these motions do not correspond to the periodic pulsation of the chest 4, i.e. are not related to the desired vital sign (respiration information in this example) and shall be distinguished there from.

[0057] FIG. 3 shows a time diagram of an alternating signal S over time derived from the movement pattern and/or from motion vectors of the ROI which can be for example determined on the basis of a frame or an edge detection. The alternating signal S in this particular example corresponds to the movement of the chest 4 of the subject 2 derived from a ROI with the sequence of images of the image data. The alternating signal S shows a characteristic variation corresponding to the movement of the chest 4 i.e. the breathing rate of the subject 2. The alternating signal S also shows a high-frequency noise superimposed to the breathing rate. By use of a frequency analysis the frequency of the breathing motion, i.e. the breathing rate, can be determined.

[0058] FIG. 4 shows a flow chart of a preferred embodiment of the method 100 according to the present invention. FIGS. 5a, 5b, and 5c show a schematic image sequence for illustrating the steps of this embodiment of the method 100.

[0059] In a first step S10 an image 30 is divided into a plurality of spatial blocks 32 as shown in FIG. 5a. In a second step S12 a vital sign signal (e.g. respiratory motion) is extracted from a plurality or even each of the blocks 32. In a third step S14 adjacent spatial blocks with the strongest extracted signal (according to predefined criteria such as amplitude, SNR, frequency, or a shape of a signal) are clustered as shown in FIG. 5b, in which the clustered spatial blocks 32a-32d are indicated representing an initial ROI 34.

[0060] In a fourth step S16 strong spatial features are selected inside the initial ROI 34, which will be used for tracking. Such strong spatial features may be edges of the blanket in the direction of breathing motion, dynamic range of pixels, orientation of the edge of a blanket, any other features which are not changed by respiratory motion. In case of the edge of a blanket: the edge can move with each respiratory motion, while its orientation would not be changed by a respiratory motion, but by any other body motion (e.g. rotation of a body).

[0061] In a fifth step S18 one or more vital signs are extracted from the entire initial ROI 34, e.g. by detecting motion of the chest caused by breathing of the subject 2 as explained above. Based on the extracted vital sign(s) the main direction (e.g. mostly vertical or horizontal breathing motion component) and the amplitude range of the extracted vital sign signal are determined in a sixth step S20. That can be done by analyzing the amplitude of motion induced by a vital sign (respiration or pulse) along all possible directions, and select the one direction, which correlates the most with the vital signs motion.

[0062] In a seventh step S22 it is controlled whether the initial ROI 34 is moving, e.g. by analyzing the changes of distances between spatial features (or feature points) inside the initial ROI 34, the direction of changes of spatial features and/or the deviation of motion of spatial features from the main characteristics of the extracted vital sign signal.

[0063] In case a motion of the initial ROI 34 as detected in step S22, a motion estimation algorithm is started and the spatial features detected inside the initial ROI 34 are tracked in an eighth step S24. If the new position of the spatial features becomes stable over several frames, extraction of the vital signs signal, e.g. the respiratory signal, from this new (final) ROI is started in a ninth step S26. The position of the new (final) ROI 34′ is shown in FIG. 5c, illustrating that the patient has moved a bit to the right side with respect to the bed so that also the initial ROI is shifted by the same amount and to the same side.

[0064] Finally, in a tenth step S28 the spatial blocks around the new location of the ROI are analyzed, e.g. using the same approach as used in the third step S14, and the new (final) ROI is re-initialized as new initial ROI. The procedure may then be carried out iteratively, with step S16 being the next step. Further, in a tenth step S30 the desired vital sign information can be derived from the new (final) ROI.

[0065] A vital sign signal may be derived in the second steps S12 as follows. First, movement patterns are derived from a plurality or each of the image blocks 32 of the image frame 30 and the alternating signals S are determined from motion vectors determined from the movement pattern of each of the image blocks 32 as described above. The motion vectors may be determined by pattern detection or edge detection within the different image blocks 32. On the basis of the frequency analysis performed e.g. by a frequency analysis unit, it is determined whether the movement pattern of the different image blocks 32 corresponds to vital sign information or whether the movement patterns are disturbance signals or noise. The determination whether the movement patterns include vital sign information or not may be performed on the basis of the spectral parameter and/or the spectral energy and whether the spectral energy in a predefined frequency band is larger than a predefined percentage of the entire spectral energy of the respective alternating signal.

[0066] In the fourth step S16 spatial image features may be selected, which are used for detection of non-vital-sign-related body motion and reliable tracking of the ROI. Therefore, those image features should be the most (temporally) stable during vital-sign-related motion, such as respiratory motion in case of extracting respiratory information as vital sign information. For instance, if a monitored subject is sitting in front of a camera, and a respiratory motion has a strong vertical motion component, the image features selected inside the ROI should be least affected (ideally invariant) to vertical motion, e.g. vertical edges.

[0067] Generally, the vital sign signal extracted at step S26 might have a lower quality than a vital sign signal extracted at step S30. At the same time, this approach allows restart of extraction of vital signs immediately after the motion of a subject is completed, without the need to trigger a complete ROI detection algorithm (as performed in this embodiment in steps S12 and S14), which would reduce the gaps in monitoring of vital signs during motion and eventually improve the responsiveness of the entire device and method. An additional advantage of this approach is that the vital sign signal is extracted only from the stable ROI, identified during steps S12 and S14. Thus, any disturbances, or other subjects with the similar respiratory motion would not influence measurements.

[0068] In another embodiment of the present invention, step S28 (re-selection of spatial areas around the previous ROI) can be initialized every time when the reliability or quality of the extracted signals drops below a certain threshold.

[0069] In yet another embodiment of the present invention, the steps S20 and S22 can include a method for detection of (partial) occlusion of ROI, which would trigger the step S26 for extraction of the vital sign signal from non-occluded parts within the initial ROI. Among other methods, the occlusion detection can be done by analyzing the visibility of spatial features, identified in the previous steps.

[0070] In summary, the present invention applies two different modalities (i.e. vital signs based and spatial features based) in a particular combination for an advantageous selection and tracking of the ROI. The detection of non-respiratory (in general non-vital-sign-related) body motion is based on an analysis of image features inside the ROI, rather than on an analysis of the extracted vital sign signal. Further, tracking of the ROI used for vital sign monitoring is based on motion analysis of image features inside the ROI, which are least affected by the vital-sign-related motion, such as the respiratory motion.

[0071] The present invention thus uses (re-)initialization and tracking of ROI, based on the combination of temporal features and spatial features. The spatial features (e.g. the direction of edges, pixel patterns) are detected only within an area, initialized by strong vital-sign-related motion (e.g. strong breathing motion), and only those features (e.g. angle of an edge), which are not deformed by the vital-sign-related motion are used for maintaining or tracking of ROI, even in the absence of the vital sign signal, e.g. in the absence of respiration or a pulse signal.

[0072] Further, the decision to re-initialize ROI is based on analysis of displacement of spatial features, rather than based only on changes in a breathing temporal pattern. Therefore, the ROI will not be re-initialized if a subject stops breathing. At the same time, motion of a subject will be detected and ROI will be tracked (based on tracking of spatial features) even if a temporal breathing pattern does not change, but the spatial features are deformed. The spatial features should be invariant to the vital-sign-related motion (e.g. direction of an edge in ROI, which does not change due to respiratory motion). Hence, any change in those spatial features would indicate non-vital-sign-related motion, e.g. body motion, of the ROI. An example of such features is that a vertical edge of pixels in the presence of vertical breathing will not be affected by breathing, but will be displaced by even slight rotation of a body in a horizontal direction.

[0073] The present invention may primarily be applied in respiration monitoring devices and method within hospitals or in home care and sports.

[0074] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0075] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0076] A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0077] Any reference signs in the claims should not be construed as limiting the scope.