Abstract
Haemodynamic parameters such as the amplitude and phase of a pulse wave passing through a region of interest can be obtained from a video image of the exposed skin of a patient by processing of the reflectance photoplethysmographic signal using signal averaging. The region of interest is defined and a reflectance photoplethysmographic signal obtained by finding the mean pixel intensity across the region of interest for each video frame. Signal averaging is performed on the resulting pulsatile waveform by detecting peaks in the waveform, selecting those parts of the waveform which lie within a window centred on the peaks, and summing the selected parts of the waveform to find an average pulse waveform. The region of interest is then divided into sub-regions and an average pulse waveform for the video sequence is found for each of the sub-regions in the same way. The amplitudes of the average pulse waveforms for the sub-regions can be measured and displayed, for example as a spatial map across the region of interest. The phase of the average pulse waveforms in the sub-regions in the sub-regions relative to the average pulse waveform for the whole region of interest can be measured and displayed, again as a spatial map. The phase and amplitude maps give an indication of the quality of perfusion across the region of interest.
Claims
1. A method of measuring and displaying a haemodynamic parameter comprising the steps of: acquiring a video image of a region of interest on the skin of a subject; processing the video image to obtain a first remote reflectance photoplethysmogram signal for the region of interest; performing signal averaging on the first remote reflectance photoplethysmogram signal for the region of interest to obtain a reference average pulse waveform; splitting the region of interest into plural subsidiary regions of interest and processing the video signal to obtain a subsidiary remote reflectance photoplethysmogram signal for each subsidiary region of interest, performing signal averaging on each subsidiary remote reflectance photoplethysmogram signal to obtain a subsidiary average pulse waveform for each subsidiary region of interest; determining the phase difference between each subsidiary average pule waveform and the reference average pulse waveform; and displaying as said haemodynamic parameter the phase difference for each subsidiary region of interest.
2. The method according to claim 1 wherein the signal averaging is performed on each subsidiary remote reflectance photoplethysmogram signal using reference time points defined for the signal averaging used to obtain the reference average pulse waveform.
3. The method according to claim 2 wherein the reference time points are the times of occurrence of repeated signal features, such as peaks, in the first remote reflectance photoplethysmogram signal.
4. The method according to claim 2 wherein the signal averaging comprises defining respective windows positioned with respect to each the reference time points and summing the signals in the windows.
5. The method according to claim 4 wherein the size of the window is set in dependence upon the expected pulse rate of the subject.
6. The method according to claim 1 wherein the remote reflectance photoplethysmogram signal is resampled before signal averaging is performed.
7. The method according to claim 1 wherein the remote reflectance photoplethysmogram signal is bandpass filtered before signal averaging is performed.
8. The method according to claim 7 wherein the passband for the bandpass filtering is set in dependence upon the expected pulse rate of the subject.
9. The method according to claim 1 wherein the remote reflectance photoplethysmogram signal is obtained by averaging pixel intensities of the video image over the region of interest.
10. The method according to claim 1 further comprising measuring the amplitude of each subsidiary average pulse wave and displaying the amplitude as a second haemodynamic parameter.
11. The method according to claim 1 wherein the haemodynamic parameter or parameters for each subsidiary region are displayed as a map.
12. A computer program comprising program code means for causing a programmed computer to execute on an acquired video signal the method of claim 1.
13. An apparatus comprising a video camera for acquiring a video image of a region of interest on the skin of a subject, an image processor for executing on the acquired image the method of claim 1, and a display for displaying the haemodynamic parameter.
Description
[0015] The invention will be further described by way of example with reference to the accompanying drawings in which:
[0016] FIG. 1A schematically illustrates a series of image frames of a subject forming a video image;
[0017] FIG. 1B illustrates a region of interest on the skin of a subject;
[0018] FIG. 1C illustrates a raw rPPG signal obtained from the region of interest illustrated in FIG. 1B;
[0019] FIG. 2A illustrates a raw rPPG signal,
[0020] FIG. 2B illustrates peak detection in a band-pass filtered version of the rPPG signal of FIG. 2A;
[0021] FIG. 2C illustrates the setting of windows around each peak in the filtered rPPG signal;
[0022] FIG. 2D illustrates the response of the band-pass filter used in the embodiment of FIG. 2;
[0023] FIG. 2E illustrates an average pulse waveform obtained by signal averaging the rPPG signal of FIG. 2A;
[0024] FIG. 3A illustrates the division of the region of interest into subsidiary regions of interest according to one embodiment of the invention;
[0025] FIG. 3B illustrates the reference average pulse waveform for the whole region of interest and a subsidiary average pulse waveform for one of the subsidiary regions of interest of FIG. 3A;
[0026] FIG. 3C illustrates the calculation of the phase difference using cross-correlation;
[0027] FIGS. 4A-C illustrate maps of local rPPG phase for the three colour channels measured with one embodiment of the invention;
[0028] FIGS. 4D-F illustrate the local rPPG amplitude for the 3 colour channels of an rPPG signal;
[0029] FIG. 5A illustrates a region of interest in an image of an adult patient and FIGS. 5B and 5C illustrate maps of the local rPPG phase and amplitude respectively for the adult patient;
[0030] FIG. 6 is a flow diagram explaining the processing in accordance with one embodiment of the invention; and
[0031] FIG. 7 schematically illustrates an apparatus for executing the method of one embodiment of the invention.
[0032] An embodiment of the invention will now be described which consists of three main steps: the derivation of the remote reflectance photoplethysmography (rPPG) signal from video, the processing of the rPPG signal using signal averaging, and spatial mapping of the derived haemodynamic parameters.
[0033] As explained above the rPPG signal measures colour changes in the skin caused by changes in blood volume in the skin capillaries. The rPPG signal can be obtained by measuring the amount of ambient light reflected from a subject's skin using a video camera placed up to two meters away. FIG. 1 schematically illustrates the process and FIG. 7 the apparatus. FIG. 1A represents a video signal as a plurality of image frames 10 of a subject 1, in this case a neonatal patient. As illustrated in FIG. 1B a region of interest 3 on the skin of the patient 1 is defined. The region of interest 3 may be manually defined, as indicated in step 101 of FIG. 6, and maintained in the same place from frame to frame, or it may track movement of the patient 1 using conventional motion tracking techniques. Alternatively the region of interest 3 may be automatically selected by recognising skin areas in the image using known image recognition techniques. For each frame 10 of the video sequence the mean pixel intensity in the region of interest 3 is calculated, as indicated in step 102. Each of the three RGB colour channels is treated separately. These mean pixel intensities are then formed into a time-series, as indicated in step 103, which represents the rPPG signal for the region of interest, as illustrated in FIG. 1C (for one colour channel).
[0034] FIG. 2A illustrates 10 seconds of a raw rPPG signal and the periodic pulse waveform can easily be seen. This periodicity can be exploited by using a technique based on signal averaging to increase the signal-to-noise ratio and thus reduce effects such as low-frequency artefacts caused by subject movement or changes in ambient lighting, as well as higher frequency noise due to the flickering of artificial lighting and electronic noise on the camera's image sensor. In general terms signal averaging involves adding each of the individual pulse waveforms in the rPPG signal over a certain time period to produce an average pulse waveform for that period. In more detail, in step 104 the raw rPPG signal is filtered using a finite impulse response (FIR) band-pass filter, e.g. with the filter response illustrated in FIG. 2D. In this embodiment an FIR filter with an order 100 was selected and the cut-off frequencies are chosen according to the expected pulse rate of the patient. For neonatal patients the filter can have cut-off frequencies of 1.5 Hz and 5 Hz (corresponding to heart rates of 90 to 300 BPM) whereas for adult patients the filter can have cut-off frequencies of 0.6 Hz and 2.2 Hz corresponding to heart rates of 36 to 132 BPM. Rather than pre-setting the cut-off frequencies, these may be varied based on a detected pulse rate.
[0035] Then in step 105 the raw rPPG signal is resampled, in this case to 1 kHz using cubic spline interpolation and then in step 106 peak detection is performed on the resampled signal, in this embodiment using a decaying threshold to allow for sudden changes in amplitude. FIG. 2B illustrates the resampled signal with detected peaks indicated by circles. Resampling with cubic spline interpolation allows the peak and trough locations to be estimated more accurately (when assumptions about the rPPG signal are taken into account, namely that it is smooth and approximately sinusoidal).
[0036] Then in step 107, for each peak found in step 105 the location of the peak is taken as a reference time point 11 and a section of the re-sampled raw rPPG signal around the reference time point 11 is selected. These sections can be regarded as windows 12 centred on the reference time points 11 as illustrated in FIG. 2C. The length of the window 12 depends on the pulse rate and can be, for example, 0.4 seconds for neonates and 1 second for adults, though different window lengths are possible. Typically, the window length is chosen to be approximately the time taken for one cardiac cycle (e.g. 1 second for a typical pulse rate of 60 BPM). It is also possible to adjust the window length in dependence upon the detected pulse rate.
[0037] In step 108 the selected sections of the re-sampled rPPG signal for this section of video are summed, sample by sample, after synchronisation with respect to the salient point at the centre of the window (i.e. the peak as in FIG. 2) and scaled to produce a single average waveform as illustrated in FIG. 2E. Thus in the example of FIG. 2, each 10-second section of video would produce three average waveforms, one for each of the three colour channels. The process can then be repeated for the next 10-second section of video. The next section can follow contiguously, or can overlap the previous section, e.g. 10-second windows which are slid by a step of, for example, 1 second each time.
[0038] The average waveform 14 illustrated in FIG. 2E is effectively a reference average pulse waveform for the whole of the region of interest 3. To obtain haemodynamic parameters for the region of interest 3, it is then, in step 109, divided into plural subsidiary regions of interest 30, for example in a rectangular grid as illustrated in FIG. 3A. Signal averaging is then repeated for each of the subsidiary regions of interest 30 again separately on each of the three colour channels. Thus in step 110 the mean pixel intensity in each subsidiary region of interest for each frame in the video sequence is obtained to construct a local rPPG signal for that subsidiary region of interest. In step 111 using the reference time points 11 from the peak detection process of step 106, and using corresponding windows 12 centred on each reference time point, the local rPPG signal within each window 12 are summed and scaled to produce a subsidiary average pulse waveform for that subsidiary region of interest 30. Each subsidiary region of interest 30 will have its own subsidiary average pulse waveform for each colour channel and each (10-second in this example) section of video signal.
[0039] Then the phase and amplitude of the local pulse waveforms 140 for the subsidiary regions of interest 30 can be calculated. The amplitude is calculated in step 112 by subtracting the minimum of the subsidiary average pulse waveform 140 from its maximum and the amplitude for each subsidiary region of interest 30 is intensity coded and displayed in a map for each colour channel as illustrated in FIGS. 4D, E and F.
[0040] To find the phase of the subsidiary average pulse waveform 140 in each subsidiary region of interest 30, each subsidiary average waveform 140 and the reference average waveform 14 for the whole region of interest are firstly scaled to take values between 0.0 and 1.0 by subtracting their minimum and dividing by their range. Other ways of normalising the waveforms to have approximately equal (peak−trough) amplitude can be used instead, such as, for each waveform, subtracting its mean value and dividing by its standard deviation.
[0041] FIG. 3B illustrates one of the scaled subsidiary average pulse waveforms 140 and the scaled reference average waveform 14. Then the cross-correlation of each of the subsidiary average pulse waveforms 140 with the reference average pulse waveform 14 is calculated for a variety of phase shifts and the phase difference is taken to be that phase shift at which the cross-correlation has its maximum value. FIG. 3C illustrates the cross-correlation between the two pulse waveforms of FIG. 3B for a range of phase shifts up to one pulse width either side, from −400 to +400 samples (i.e. from −0.4 s to +0.4 s, given the 1 kHz resampling frequency in this embodiment) in the illustrated example. It can be seen that the cross-correlation has a maximum value at a phase shift of minus 14 samples, which corresponds to a time lag of plus 14 ms. The phase difference in each of the subsidiary regions of interest 30 can be intensity coded and displayed as a map for each of the three RGB colour channels as shown in FIGS. 4A, B and C. Rather than intensity coding, colour coding can be used. To reduce noise in the maps, the maps can be spatially filtered, for example using a three-by-three pixel Gaussian filter. The maps in FIGS. 4A-F are based on averaging pulse waveforms for a one-minute section of video (rather than 10 seconds illustrated in FIGS. 2A to C). However different lengths of video may be averaged.
[0042] FIG. 5 illustrates a single frame from a video sequence of an adult patient with the region of interest 50 outlined. FIGS. 5B and C show phase and amplitude maps respectively for one minute of video in this case for a monochrome camera with only one channel. Referring to FIG. 5B it can be seen that advanced phase (darker) areas are towards the bottom right of the region of interest and generally retarded phase (lighter areas) towards the top and left hand side. This shows that the pulse wave in this case generally travels from bottom to top. FIG. 7 schematically illustrates the apparatus required for the invention. The video signal may be acquired by a standard monochrome or colour video camera 70 and the output provided to a general purpose computer 72 programmed to execute the signal averaging processes and amplitude and phase difference calculations explained above and illustrated with reference to FIGS. 1 to 6. The amplitude and phase differences may be displayed on a display 74 either as spatial maps as shown in FIGS. 4A-F and FIGS. 5B and C, or as charts or values.
