Abstract
A computer-implemented method for generating an annotated photoplethysmography signal, includes: recording an electrocardiogram or ECG signal; recording a photoplethysmography or PPG signal, semi-synchronously with the recording of the ECG signal; annotating segments in the ECG signal either algorithm-based or expert-based; time-aligning the PPG signal and the ECG signal; detecting ECG beats in the ECG signal; detecting PPG beats in the PPG signal; pairing the ECG beats onto the PPG beats; deriving annotations for PPG signal segments based on the nature of the ECG segment annotations and on how the ECG beats can be paired with the PPG beats; and annotating the PPG signal segments using the annotations, thereby generating the annotated PPG signal.
Claims
1.-15. (canceled)
16. A computer-implemented method for generating an annotated photoplethysmography signal, abbreviated annotated PPG signal, said method comprising: recording an electrocardiogram signal, abbreviated ECG signal, using an ECG monitoring system; recording a photoplethysmography signal, abbreviated PPG signal, semi-synchronously with said recording of said ECG signal, using a contact PPG sensor; annotating segments in said ECG signal either algorithm-based or expert-based, resulting in ECG segment annotations; time-aligning said PPG signal and said ECG signal; detecting cardiac beats in said ECG signal, named ECG beats; detecting cardiac beats in said PPG signal, named PPG beats; pairing said ECG beats with said PPG beats; deriving annotations for PPG signal segments based on the nature of said ECG segment annotations and on how said ECG beats can be paired with said PPG beats; and annotating said PPG signal segments using said annotations, thereby generating said annotated PPG signal.
17. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein deriving annotations for PPG signal segments comprises: copying an annotation of an ECG segment onto a time-corresponding PPG segment when an ECG beat can be paired with a PPG beat for said segment.
18. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein deriving annotations for PPG signal segments comprises: marking a PPG segment as insufficient quality segment when no ECG beat can be paired with a PPG beat for said segment or when no PPG beat can be paired with an ECG beat for said segment.
19. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein deriving annotations for PPG signal segments comprises: copying an annotation of an ECG segment onto a time-corresponding PPG segment when an ECG beat cannot be paired with a PPG beat for said segment or attributing a new annotation to a time-corresponding PPG segment when an ECG beat cannot be paired with a PPG beat for said segment.
20. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein deriving annotations for PPG signal segments comprises: marking a PPG segment as undefined segment when no ECG beat can be paired with a PPG beat for said segment because a time-corresponding ECG segment has insufficient quality.
21. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein pairing said ECG beats with said PPG beats comprises: selecting an ECG beat; determining an ECG signal portion comprising said ECG beat and no other ECG beats; determining a PPG signal portion time-aligned with said ECG signal portion; identifying a PPG beat lying within said PPG signal portion; associating said PPG beat with said ECG beat.
22. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein detecting said PPG beats comprises detecting local maxima in said PPG signal.
23. The computer-implemented method for generating an annotated PPG signal according to claim 21, further comprising: verifying if said PPG beat represents the highest local maximum within a time interval ranging from an ECG beat preceding said ECG beat in said ECG signal to an ECG beat following said ECG beat in said ECG signal; and associating said PPG beat with said ECG beat only when said PPG beat represents the highest local maximum.
24. The computer-implemented method for generating an annotated PPG signal according to claim 16, further comprising: storing said annotated PPG signal in a database of training data used for training in machine learning.
25. The computer-implemented method for generating an annotated PPG signal according to claim 20, further comprising: training a neural network with said annotated PPG signal.
26. The computer-implemented method for generating an annotated PPG signal according to claim 16, further comprising: recording a PPG signal, similarly to recording said PPG signal; pre-processing said PPG signal differently to obtain plural PPG signal versions; annotating said plural PPG signal versions, similarly to annotating said PPG signal, thereby generating plural annotated PPG signals; comparing said plural annotated PPG signals and deriving an optimal pre-processing for PPG signals based on the amount of insufficient quality annotations in said annotated PPG signals.
27. The computer-implemented method for generating an annotated PPG signal according to claim 16, wherein said annotations comprise one or more of the following: regular sine rhythm; insufficient quality; undefined; irregular rhythm annotations comprising: atrial fibrillation; atrial flutter; supraventricular tachycardia; ventricular tachycardia; sinus tachycardia; bradycardia; ventricular fibrillation; premature atrial contraction; and premature contraction; other non-rhythm clinical annotations comprising: sleep apnea.
28. A controller comprising at least one processor and at least one memory including computer program code, the at least one memory and computer program code being configured to, with the at least one processor, cause the controller to perform: recording an electrocardiogram signal, abbreviated ECG signal; recording a photoplethysmography signal, abbreviated PPG signal, semi-synchronously with said recording of said ECG signal; annotating segments in said ECG signal either algorithm-based or expert-based, resulting in ECG segment annotations; time-aligning said PPG signal and said ECG signal; detecting cardiac beats in said ECG signal, named ECG beats; detecting cardiac beats in said PPG signal, named PPG beats; pairing said ECG beats with said PPG beats; deriving annotations for PPG signal segments based on the nature of said ECG segment annotations and on how said ECG beats can be paired with said PPG beats; and annotating said PPG signal segments using said annotations, thereby generating said annotated PPG signal.
29. A computer readable storage medium comprising computer-executable instructions of a program for performing the following steps when the program is run on a computer: recording an electrocardiogram signal, abbreviated ECG signal; recording a photoplethysmography signal, abbreviated PPG signal, semi-synchronously with said recording of said ECG signal; annotating segments in said ECG signal either algorithm-based or expert-based, resulting in ECG segment annotations; time-aligning said PPG signal and said ECG signal; detecting cardiac beats in said ECG signal, named ECG beats; detecting cardiac beats in said PPG signal, named PPG beats; pairing said ECG beats with said PPG beats; deriving annotations for PPG signal segments based on the nature of said ECG segment annotations and on how said ECG beats can be paired with said PPG beats; and annotating said PPG signal segments using said annotations, thereby generating said annotated PPG signal.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0117] FIG. 1 is a flow scheme illustrating an embodiment of the method for generating an annotated PPG signal according to the present invention;
[0118] FIGS. 2A and 2B illustrate the steps 101-103 of recording an ECG signal, recording a PPG signal and annotating an ECG signal in an embodiment of the method for generating an annotated PPG signal according to the present invention;
[0119] FIGS. 3A and 3B illustrate the steps 105-106 of detecting ECG beats and detecting PPG beats in an embodiment of the method for generating an annotated PPG signal according to the present invention;
[0120] FIGS. 4A and 4B illustrate the steps 107-109 of mapping ECG beats onto PPG beats, deriving PPG annotations and annotating the PPG signal in an embodiment of the method for generating an annotated PPG signal according to the present invention;
[0121] FIGS. 5A and 5B illustrate the steps 107-109 of mapping ECG beats onto PPG beats, deriving PPG annotations and annotating the PPG signal in a variant embodiment of the method for generating an annotated PPG signal according to the present invention;
[0122] FIGS. 6A and 6B illustrate the steps 107-109 of mapping ECG beats onto PPG beats, deriving PPG annotations and annotating the PPG signal in another variant embodiment of the method for generating an annotated PPG signal according to the present invention;
[0123] FIGS. 7A and 7B illustrate the steps 107-109 of mapping ECG beats onto PPG beats, deriving PPG annotations and annotating the PPG signal in yet another variant embodiment of the method for generating an annotated PPG signal according to the present invention; and
[0124] FIG. 8 illustrates a suitable computing system 800 for realizing embodiments of the method for PPG signal annotation in accordance with the present invention.
DETAILED DESCRIPTION OF EMBODIMENT(S)
[0125] FIG. 1 shows the steps executed in an embodiment of the computer-implemented method for generating an annotated PPG signal 402 according to the present invention. In the steps 101 and 102, an ECG signal 201 and a PPG signal 202 are semi-synchronously recorded using different devices on a single person or animal. The ECG signal 201 is obtained by an ECG monitoring system, for example a Holter monitoring system. A 60 seconds long portion of the ECG signal 201 is shown in FIG. 2A. The PPG signal 202 is obtained by a PPG sensor, for example a pulse oximeter or smartphone camera. A 60 seconds long portion of the PPG signal 202 is shown in FIG. 2B. In step 103, the ECG signal 201 is annotated either algorithm-based or expert-based by one or plural people like for instance nurses, medical technicians, data scientists, cardiologists, etc. The annotating of the ECG signal 201 results in a first part of the 60 seconds portion shown in FIG. 2A being annotated as “Undefined” or “UD”, a second part being annotated as “Atrial Fibrillation” or “AF”, a third part being annotated as “Regular Sine Rhythm” or “SR”, a fourth part being annotated as “Premature Atrial Contraction” or “PAC”, a fifth part being annotated as “Regular Sine Rhythm” or “SR”, and a sixth part being annotated as “Undefined” or “UD”. The ECG signal 201 and PPG signal 202 that were semi-synchronously recorded are time-aligned in step 104. This means the ECG signal 201 and PPG signal 202 are synchronised in time with milliseconds accuracy by compensating for the offset in between the start times of the recordings of the ECG and PPG signals in steps 101 and 102, and by compensating for the drift between the clocks of the ECG monitoring system used in step 101 and the PPG sensor device used in step 102. It is noticed that the 60 seconds portions of the ECG signal 201 and PPG signal 202 shown in FIGS. 2A and 2B are time-corresponding portions. Thus, it is assumed in FIG. 2A and FIG. 2B that the time-alignment of step 104 has already been executed.
[0126] In steps 105-107, individual ECG beats are paired with PPG beats. In step 105, cardiac beats are identified in the ECG signal 201. The cardiac beats are shown in FIG. 3A wherein ECG signal 301 corresponds to ECG signal 201 and the detected cardiac beats are identified through the dots. It is noticed that the detection of cardiac beats in the ECG signal 201 may have taken place before or after the ECG signal was expert-based or algorithm based annotated in step 103, or may have taken place as part of this step 103. If for instance the ECG annotation in step 103 is executed on beat-to-beat segments, i.e. segments of variable length that range from one cardiac beat to the next cardiac beat, the cardiac beats must have been detected before the annotation in step 103 took place. If however the annotation in step 103 was done on fixed length segments of for instance 1 second, the cardiac beat detection for the ECG signal may be executed after the annotation of this signal in step 103. In step 106, for every ECG beat detected in step 105, a corresponding PPG beat is searched in the aligned PPG signal by looking for a local maximum in the PPG signal. This results in PPG signal 302 which corresponds to PPG signal 202 with detected beats identified by the dots. In an example embodiment of the invention, local maxima are detected in small, aligned time windows, in the PPG signal 302. As there might subsist some small misalignment between an ECG beat in ECG signal 301 and the corresponding PPG beat in PPG signal 302, the local maximum is searched for in a small window, for instance having a total length of 100 milliseconds (50 ms left and 50 ms right of the ECG beat). A PPG beat is detected if it aligns with the ECG beat in the specified small time window, and if the PPG beat is the highest local maximum in the time interval between the previous ECG beat and the next ECG beat, i.e. the interval [beat−1; beat+1] in the ECG signal 301. In such case, the detected PPG beat is mapped onto the time-corresponding ECG beat in step 107 or—in other words—the detected PPG beat is paired with the time-corresponding ECG beat. In case another local maximum is detected in that interval with higher amplitude, for instance as a result of an artefact in the PPG signal measurement, no PPG beat is detected and no pairing of the time-corresponding ECG beat is possible in step 107.
[0127] Once the ECG beats are paired with the PPG beats, the ECG annotations from step 103 (more precisely their nature) and information on how the ECG beats can be paired with the PPG beats in step 107 are used in step 108 to derive annotations for PPG signal segments and to annotate the PPG signal accordingly in step 109. PPG signal portions for which no ECG beat can be paired with a PPG beat in step 107, are assumed to be low quality portions in an embodiment of the invention illustrated by FIG. 4A and FIG. 4B. For such PPG signal portions, the annotation of the time-corresponding ECG signal portion is not copied. Instead, the PPG signal portion receives the annotation “Insufficient Quality” or “IQ”. In FIG. 4B, two portions of the PPG signal 402 are annotated “IQ” because it is not possible for segments that form part of these PPG signal portions to pair an ECG beat with a PPG beat. For PPG signal portions wherein it is possible in step 107 to pair an ECG beat with a PPG beat, the annotation of the time-corresponding ECG segments is copied onto the PPG segments. In FIG. 4B, a first portion of the PPG signal 402 receives the annotation “UD” copied from the time-corresponding segments of the ECG signal 401, a third portion of the PPG signal 402 receives the annotation “AF” copied from the time-corresponding segments of the ECG signal 401, a fifth portion of the PPG signal 402 receives the annotation “AF” copied from the time-corresponding portion of the ECG signal 401, a sixth portion of the PPG signal 402 receives the annotation “SR” copied from the time-corresponding segments of the ECG signal 401, a seventh portion of the PPG signal 402 receives the annotation “PAC” copied from the time-corresponding segments of the ECG signal 401, an eight portion of the PPG signal 402 receives the annotation “SR” copied from the time-corresponding segments of the ECG signal 401, and a ninth portion of the PPG signal 402 receives the annotation “UD” copied from the time-corresponding segments of the ECG signal 401, because it was possible in all of these signal portions to pair each ECG beat with a PPG beat. It is noticed that the ECG signal 401 corresponds to the ECG signal 301 of FIG. 3A and ECG signal 201 of FIG. 2A.
[0128] The so obtained annotated PPG signal 402 with derived insufficient quality portions can be used as training data for a machine learning/deep learning tool that is used as PPG signal classifier. The machine learning tool trained this way shall be able to reliably detect various heart rhythms in a PPG signal—outcome of the PPG signal classifier is a medical diagnosis—and to identify insufficient quality portions in a PPG signal—outcome of the PPG signal classifier is “Insufficient Quality”. Consequently, the machine learning tool obtained through such embodiment of the present invention can inform that another measurement is needed before a reliable medical diagnosis can be made.
[0129] FIGS. 5A and 5B illustrate a variant embodiment of the method for generating an annotated PPG signal according to the present invention. If in step 107, ECG beats cannot be paired with PPG beats, step 108 does not derive the annotation “Insufficient Quality” for the time-corresponding PPG sections but instead preserves the annotation of the time-corresponding ECG segments. Consequently, the annotation “UD” of a first portion of ECG signal 501 is copied onto a time-corresponding portion of PPG signal 502, the annotation “SR” of a second portion of ECG signal 501 is copied onto a time-corresponding portion of PPG signal 502, the annotation “PAC” of a third portion of ECG signal 501 is copied onto a time-corresponding portion of PPG signal 502, the annotation “SR” of a fourth portion of ECG signal 501 is copied onto a time-corresponding portion of PPG signal 502, and the annotation “UD” of a fifth portion of ECG signal 501 is copied onto a time-corresponding portion of PPG signal 502. Such an embodiment is desirable, for instance in situations wherein an ECG beat remains invisible in the PPG signal, for instance in case of a premature cardiac contraction which leads to low blood ejection from the heart. Rather than annotating the time-corresponding PPG section as “Insufficient Quality”, it is desirable in such case to preserve the time-corresponding ECG segment annotation(s).
[0130] FIGS. 6A and 6B illustrate another variant embodiment of the method for generating an annotated PPG signal according to the present invention. If in step 107, ECG beats cannot be paired with PPG beats, step 108 does not derive the annotation “Insufficient Quality” for the time-corresponding PPG sections but instead sets a new annotation for the time-corresponding ECG segments. Consequently, the annotation “UD” of a first portion of ECG signal 601 is copied onto a time-corresponding portion of PPG signal 602, the annotation “SR” of a second portion of ECG signal 601 is copied onto a time-corresponding portion of PPG signal 602, the new annotation “Premature Contraction” or “PC” is set for a third portion of the PPG signal 602 because ECG beats cannot be mapped onto PPG beats for the time-corresponding third portion of the ECG signal 601 with annotation “PAC”, the annotation “SR” of a fourth portion of ECG signal 601 is copied onto a time-corresponding portion of PPG signal 602, and the annotation “UD” of a fifth portion of ECG signal 601 is copied onto a time-corresponding portion of PPG signal 602. Such an embodiment is desirable, for instance in situations wherein an ECG beat remains invisible in the PPG signal, for instance in case of a premature cardiac contraction which leads to low blood ejection from the heart. Rather than annotating the time-corresponding PPG section as “Insufficient Quality”, it may be desirable in such case to attribute a new annotation, for example “Premature Contraction”, derived from the nature of the time-corresponding ECG segment annotation.
[0131] FIGS. 7A and 7B illustrate yet another variant embodiment of the method for generating an annotated PPG signal according to the present invention. If in step 107, a section of the ECG signal 701 has insufficient quality, the time-corresponding PPG section is assigned the annotation “Undefined” or “UD” in step 108, because the PPG signal not necessarily has insufficient quality when the ECG signal has insufficient quality. Consequently, the annotation “UD” of a first portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “SR” of a second portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “UD” is assigned to a third portion of the PPG signal 702 because the time-corresponding portion of the ECG signal 701 has insufficient quality not allowing to pair ECG beats with PPG beats, the annotation “SR” of a fourth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “PAC” of a fifth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “SR” of a sixth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “UD” is assigned to a seventh portion of the PPG signal 702 because the time-corresponding portion of the ECG signal 701 has insufficient quality not allowing to pair ECG beats with PPG beats, the annotation “SR” of a fourth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “SR” of an eight portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “PAC” of a ninth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “SR” of a tenth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “PAC” of an eleventh portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, the annotation “SR” of a twelfth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702, and the annotation “UD” of a thirteenth portion of ECG signal 701 is copied onto a time-corresponding portion of PPG signal 702.
[0132] The skilled person will appreciate that several of the rules for deriving annotations for PPG signal segments based on the nature of time-corresponding ECG signal annotations and on how the pairing of ECG beats onto PPG beats can be done as described here above for the embodiments illustrated by FIG. 4A-4B, FIG. 5A-5B, FIG. 6A-6B and/or FIG. 7A-7B can be combined with each other to generate further variant embodiments of the method for PPG signal annotation according to the present invention.
[0133] FIG. 8 shows a suitable computing system 800 according to an embodiment of the invention. Computing system 800 is suitable for implementing embodiments of the method for PPG signal annotation in line with the present invention. Computing system 800 may in general be formed as a suitable general-purpose computer and comprise a bus 810, a processor 802, a local memory 804, one or more optional input interfaces 814, one or more optional output interfaces 816, a communication interface 812, a storage element interface 806 and one or more storage element 808. Bus 810 may comprise one or more conductors that permit communication among the components of the computing system 800. Processor 802 may include any type of conventional processor or microprocessor that interprets and executes programming instructions. Local memory 804 may include a random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 802 and/or a read only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor 802. Input interface 814 may comprise one or more conventional mechanism that permit an operator or user to input information to the computing device 800, such as a keyboard 820, a mouse 830, a pen, voice recognition and/or biometric mechanisms, a camera, etc. Output interface 816 may comprise one or more conventional mechanisms that output information to the operator or user, such as a display 840, etc. Communication interface 812 may comprise any transceiver-like mechanism such as for example one or more Ethernet interfaces that enables computing system 800 to communicate with other devices and/or systems, for example with other computing devices 881, 882, 883. The communication interface 812 of computing system 800 may be connected to such another computing system by means of a local area network (LAN) or a wide area network (WAN) such as for example the internet. Storage element interface 806 may comprise a storage interface such as for example a Serial Advanced Technology Attachment (SATA) interface or a Small Computer System Interface (SCSI) for connecting bus 810 to one or more storage elements 808, such as one or more local disks, for example SATA disk drives, and control the reading and writing of data to and/or from these storage elements 808. Although the storage elements 808 above is described as a local disk, in general any other suitable computer-readable media such as a removable magnetic disk, optical storage media such as a CD or DVD, -ROM disk, solid state drives, flash memory cards, . . . could be used. It is noticed that the entire method according to the present invention can be executed centralized, e.g. on a server in a management centre or in a cloud system, or it can be partially executed on a remote electronic device, e.g. worn by the user, and partially on a central server. Computing system 800 could thus correspond to the processing system available centrally or the processing system available in the electronic device.
[0134] Although the present invention has been illustrated by reference to specific embodiments, it will be apparent to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied with various changes and modifications without departing from the scope thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. In other words, it is contemplated to cover any and all modifications, variations or equivalents that fall within the scope of the basic underlying principles and whose essential attributes are claimed in this patent application. It will furthermore be understood by the reader of this patent application that the words “comprising” or “comprise” do not exclude other elements or steps, that the words “a” or “an” do not exclude a plurality, and that a single element, such as a computer system, a processor, or another integrated unit may fulfil the functions of several means recited in the claims. Any reference signs in the claims shall not be construed as limiting the respective claims concerned. The terms “first”, “second”, third”, “a”, “b”, “c”, and the like, when used in the description or in the claims are introduced to distinguish between similar elements or steps and are not necessarily describing a sequential or chronological order. Similarly, the terms “top”, “bottom”, “over”, “under”, and the like are introduced for descriptive purposes and not necessarily to denote relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances and embodiments of the invention are capable of operating according to the present invention in other sequences, or in orientations different from the one(s) described or illustrated above.
