A SYSTEM AND METHOD FOR TISSUE ANALYSIS USING REMOTE PPG

Abstract

A tissue analysis system and method process images to derive a remote PPG perfusion map from PPG amplitude levels obtained from the images as well as a PPG delay map from PPG relative delays between the PPG signals for each image region (pixel) with respect to a reference signal. The images are segmented into one or more tissue regions based on the PPG delay map (or segmentation information identifying tissue regions derived from the PPG delay map are received as input). The different tissue regions have distinct PPG delay characteristics. A level of perfusion can then be determined separately for each tissue region. Thus PPG delay information is used to enable different tissue types to be identified.

Claims

1. A system for tissue analysis, comprising: a processor adapted to receive images captured by an image sensor and to process the images to: derive a PPG perfusion map from PPG amplitude levels obtained from the images; derive a PPG delay map from PPG relative delays between different image regions; segment the images into one or more tissue regions based on the PPG delay map or receive segmentation information identifying tissue regions derived from the PPG delay map, wherein the different tissue regions have distinct PPG delay characteristics; and determine a level of perfusion separately for each tissue region.

2. The system of claim 1, wherein the processor is adapted to segment the images based on regions with distinct ranges of PPG delay.

3. The system of claim 2, wherein the processor is adapted to segment the images using a machine learning algorithm.

4. The system of claim 1, wherein the processor is adapted to control a display to display the PPG perfusion map and the PPG delay map, and receive the segmentation information from a user to whom the PPG perfusion map and PPG delay map have been displayed.

5. The system of claim 1, wherein the processor is adapted to normalize the level of perfusion for each tissue region relative to a maximum level of perfusion within each tissue region.

6. The system of claim 1, wherein said relative delays comprise a delay relative to a reference PPG signal which is an average delay time period for all image regions for a frame of image data.

7. The system of claim 6, wherein the reference PPG signal is an average PPG signal for a global region of interest including all of the tissue regions.

8. The system of claim 1, wherein the tissue regions comprise tissue regions supplied by different arterial branches.

9. The system of claim 1, further comprising an image sensor for capturing the images.

10. A computer-implemented tissue analysis method, comprising: receiving image sensor images; deriving a PPG perfusion map from PPG amplitude levels obtained from the images; deriving a PPG delay map from PPG relative delays between PPG signals of each image region with respect to a reference signal; segmenting the images into one or more tissue regions based on the PPG delay map or receiving segmentation information identifying tissue regions derived from the PPG delay map, wherein the different tissue regions have distinct PPG delay characteristics; and determining a level of perfusion separately for each tissue region.

11. The method of claim 9, comprising segmenting the images based on regions with distinct range of PPG delay.

12. The method of claim 10, comprising normalizing the level of perfusion for each tissue region relative to a maximum level of perfusion within each tissue region.

13. The method of claim 10, wherein said relative delays comprise a delay relative to a reference PPG signal which is a spatial average of all the image regions in the image sensor image.

14. A computer program comprising computer program code means which is adapted, when said program is run on a computer, to implement the method of claim 10.

15. A processor which is programed with the computer program of claim 14.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0047] For a better understanding of the invention, and to show more clearly how it may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0048] FIG. 1 shows a representation of a frame of a video acquired of the intestine and a global PPG signal derived from the video;

[0049] FIG. 2 shows separate PPG signals for separate regions of interest;

[0050] FIG. 3 shows a PPG perfusion map;

[0051] FIG. 4 shows a zoom-in of the PPG signals extracted from the two regions of interest from FIG. 2 and a PPG delay map;

[0052] FIG. 5 shows an image of an intestinal area after open bowl resection, the corresponding amplitude map and the corresponding delay map;

[0053] FIG. 6 shows an average PPG amplitude for each area;

[0054] FIG. 7 shows the delay map for both areas;

[0055] FIG. 8 shows a segmented version of the delay map;.

[0056] FIG. 9 shows the perfusion map for the first area;

[0057] FIG. 10 shows the perfusion map for the second area;

[0058] FIG. 11 shows system for tissue analysis;

[0059] FIG. 12 shows a method for tissue analysis; and

[0060] FIG. 13 shows a hypercube of a hyperspectral camera;

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0061] The invention will be described with reference to the Figures.

[0062] The detailed description and specific examples, while indicating exemplary embodiments of the apparatus, systems and methods, are intended for purposes of illustration only and are not intended to limit the scope of the claims. These and other features, aspects, and advantages of the apparatus, systems and methods of the present invention will become better understood from the following description, appended claims, and accompanying drawings. The Figures are merely schematic and are not drawn to scale. The same reference numerals are used throughout the Figures to indicate the same or similar parts.

[0063] The disclosure provides a tissue analysis system and method, which process images to derive a remote PPG perfusion map from PPG amplitude levels obtained from the images as well as a PPG delay map from PPG relative delays between the PPG signals for each image region (pixel) with respect to a reference signal. The images are segmented into one or more tissue regions based on the PPG delay map (or segmentation information identifying tissue regions derived from the PPG delay map is received as input). The different tissue regions have distinct PPG delay characteristics. A level of perfusion can then be determined separately for each tissue region. Thus PPG delay information is used to enable different tissue types to be identified.

[0064] Before describing the system and method of the invention, the known operation of a remote PPG imaging system, and the known image processing methods, will first be described.

[0065] Remote PPG imaging enables a determination of tissue perfusion from images captured of the tissue of interest, e.g. the skin or even tissue beneath the skin. Remote PPG typically uses ambient light, functioning as broad band white light source, and the diffuse and specular reflections are analyzed for different color components. Remote PPG imaging may be used to construct a PPG amplitude map and a PPG delay map.

[0066] For this purpose, a camera or a series of cameras (at one or multiple wavelengths) captures video of the tissue area, e.g. skin, at a distance. The measurements derived from the video are remote photoplethysmography (PPG) images, which provide non-contact measurement of a pulse signal by analyzing subtle changes of skin color (or organ color) i.e. at different wavelengths of the light.

[0067] It has been proposed to use remote PPG for inflammation detection. It has also been proposed in European Patent Application No. 20214955.5, which is incorporated by reference herein, to measure perfusion based both on a PPG amplitude level and also information about the distribution of the PPG phases, such as a standard deviation or interquartile range of a phase map.

[0068] By extracting pulse signals at each individual location of the skin region (corresponding to each pixel of the cameras) a spatial map of the pulse signal can be derived, showing both amplitude and delay. This perfusion map thus represents the amplitude and delay of the PPG signal per pixel and hence per location of the skin.

[0069] FIG. 1 (a) shows a representation of a frame of a video acquired of the intestine. By spatial averaging the pixel values in the region of interest (ROI) 10, a signal can be derived from the video, modulated at the heart rate. This signal is shown in FIG. 1 (b) as a normalized intensity of the PPG signal versus time.

[0070] The PPG signal of FIG. 1 (b) represents an average for the whole region of interest.

[0071] However, separate signals may also be captured from smaller regions of interest, shown as region 1 and region 2 in FIG. 1 (a).

[0072] FIG. 2 shows separate PPG signals, as a first signal 20 for the first region of interest and a second signal 22 for the second region of interest. The two PPG signals are modulated at the same heart rate frequency, but show a different amplitude. By extracting the amplitude from the PPG signal of each separate region of tissue, i.e. from each corresponding pixel of the captured video, a PPG perfusion map may be obtained, as shown in FIG. 3.

[0073] FIG. 3 is a black and white representation. However, the PPG perfusion may be color-coded, for example assigning a more red color to areas with higher perfusion and a more blue color to areas with lower perfusion. The PPG amplitude map thus represents the amplitude of the PPG signal per pixel and hence per location of the skin or other tissue being analyzed.

[0074] Additional valuable information may be obtained by analyzing the PPG signals.

[0075] FIG. 4 (a) shows a zoom-in of the PPG signals extracted from the two regions of interest from FIG. 2. Even though the signals are extracted simultaneously from the same organ, the pulsation arrives slightly before in one area than in the other. There is a small delay in the blood pulsation arrival, due to small differences in the microcirculatory bed, such as the resistance and elasticity of the vessels, as well as different artery branches that supply the recorded tissues.

[0076] The delays of the PPG signals of each pixel with respect to a reference signal may be extracted and used for building a delay map. The delay map is shown in FIG. 4.

[0077] Since the delay between the signals is not always constant but varies during the acquisition, an average delay is used between signals per pixel.

[0078] The average delay represents the average time delay between the PPG signal of each pixel and a reference PPG signal. In this way a signal in time is built. The length of the PPG signal should include at least a heartbeat cycle, so it is subject dependent.

[0079] The average delay is a value of delay assigned to each pixel. From the plot of FIG. 4, it can be seen that there is a difference in time arrival between the peaks of the PPG signals. Therefore, for each pixel, the average of these delay (with respect to a reference signal) is assigned. At the end, the delay map is built, where each pixel contains the average delay between the PPG signal of that pixel and the reference PPG signal.

[0080] By acquiring a video stream, a series of images is obtained over time, for example with a frame rate of 20 frames per second. To compute the global PPG signal over time, the pixel values of the frame 1 are spatially averaged, so that the 2D set of pixels yields one value. The pixels of frame 2 are then averaged, and so on.

[0081] Eventually, a PPG signal is obtained over time (with the same length as the video that has been acquired), where each value of the signal is a spatial average of one corresponding frame of the video acquired. The image frames for example comprise 968728 pixels, by way of example.

[0082] The PPG signal of each pixel is thus compared with the global PPG signal being used as a reference. The value assigned to each pixel n of the delay map thus represents the average delay between the PPG signal in the pixel n and the global PPG signal being used as a reference. Thus, the value of the delay for each pixel in the PPG delay map represents the average delay (in terms of average time shift) between the PPG signal of that pixel and the global PPG signal. The delays are for example computed by using the lock-in amplification algorithm.

[0083] The delay map thereby provides a measure of the average time delay between the PPG signal wave of each pixel and the reference PPG signal for at least one heartbeat cycle. The reference PPG signal is the global PPG signal of FIG. 1 (b) extracted from the entire region on interest of the intestine.

[0084] Since the reference signal is extracted from the entire region of interest, a PPG signal from a given location of the image is likely to be in phase with the reference.

[0085] Similar to the map of the amplitude, FIG. 4 (b) is a black and white representation. However, the PPG delay map may be color-coded. The Hue, Saturation, Value (HSV) color system is for example employed, since it works well with the periodicity of the PPG signal.

[0086] For extracting the amplitude maps and delay maps, a lock-in amplification method may be used.

[0087] Details on how to calculate the PPG maps using the lock-in amplification method can be found in, both of which are incorporated by reference herein: [0088] (i) Lai M, Shan C, Ciuhu-Pijlman C, Izamis ML. Perfusion monitoring by contactless photoplethysmography imaging, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019) 2019 Apr. 8 (pp. 1778-1782). IEEE; and [0089] (ii) Lai M, Dicorato CS, de Wild M, Verbakel F, Shulepov S, Groen J, Notten M, Lucassen G, van Sambeek MR, Hendriks BH. Evaluation of a non-contact Photo-Plethysmographic Imaging (iPPG) system for peripheral arterial disease assessment, Medical Imaging 2021: Biomedical Applications in Molecular, Structural, and Functional Imaging 2021 Feb. 15 (Vol. 11600, p. 116000F). International Society for Optics and Photonics.

[0090] Because of the great advantages that the remote PPG technology has shown for remotely and non-invasively assessing skin-level perfusion, PPG imaging can be translated to organ perfusion assessment for detecting the perfusion of the microvasculature tissue bed beneath the organ surface, without any modification to the current acquisition setup.

[0091] FIG. 5(a) shows an image of an intestinal area after open bowl resection. FIG. 5(b) shows the corresponding amplitude map (normalized with respect to the maximum value) and FIG. 5(c) shows the corresponding delay map.

[0092] The delay map most clearly shows that there are two areas which each have a relatively uniform delay value but they are different to each other (in color, the two images are predominantly different colors, such as blue and green).

[0093] The areas are supplied by different arterial branches and there is a relative difference in the PPG time arrival between the areas. This can be assessed by comparing the PPG signals extracted from the two areas.

[0094] FIG. 6 shows the average PPG amplitude for each area (with PPG signal 1 and PPG signal 2). The PPG amplitude level is different, but it can also be seen that one PPG signal arrives slightly before the other. The relative delay is 174 ms.

[0095] In the case of FIG. 1, the delay between different areas is relatively small and is mainly due to slight changes in vascular resistance and compliance. However, the delay seen in FIG. 6 is dependent on the PPG arrival time in the two areas.

[0096] The two areas will each have a spread of different delay times. However, each area will have a different average and an associated spread. Thus, ranges of PPG delay times, from the PPG delay map, may be used for classification.

[0097] For example, two areas of a PPG delay map may be considered separate if the difference of their delays is larger than a threshold such as 20 ms, preferably more than 50 ms and even more preferred larger than 100 ms.

[0098] The approach of the invention is explained with referenced to FIGS. 7 to 10.

[0099] FIG. 7 shows the delay map, corresponding to FIG. 5 (c).

[0100] The delay map is analyzed to identify uniform areas with different PPG delay arrival time, by identifying regions with distinct ranges of PPG delay as explained above. These regions have a delay relative to each other greater than a first threshold and also a spread of delays within the region below a second threshold. This spread of delays is for example a standard deviation or an interquartile range, or any other statistical measure of spread.

[0101] This allows the field of view to be segmented into the different regions as shown in FIG. 8.

[0102] These areas of uniform but distinct delay could be identified by an operator by simply looking at the image and then providing an input to the system, or else an automatic or semi-automatic algorithm may be used for image classification (such as a computer vision algorithm or a deep learning algorithms).

[0103] Once the separate areas are identified on the delay map, the same areas are overlaid and selected on the PPG amplitude map.

[0104] The level of perfusion is then assessed within each selected amplitude map area, for example normalized with respect to their respective maxima.

[0105] FIG. 9 shows the perfusion map for the first area and FIG. 10 shows the perfusion map for the second area. Separate perfusion measurements may thus be provided for the different tissue areas. Thus, a global average perfusion measure is made for each segmented area.

[0106] The systems, methods and programs disclosed and claimed herein may be used for comparing or assessing the delay in PPG arrival time of organs supplied by different branches. For example, if the field of view of the camera is large enough, several organs could be imaged all at once. It could also be used for assessing the delay of PPG arrival outside the body, for example in videos where both hands and feet are recorded simultaneously in the same field of view. In oncological surgery, parts of the tissue containing tumor tissue have to be resected as explained above. Knowing which area is perfused by the same blood vessel is important for the resection i.e. where to cut for the resection especially when also blood vessel have to resected when embedded in the tumor.

[0107] FIG. 11 shows system 110 for tissue analysis. The system comprises one or more imaging cameras 112 and/or image repositories 112 and a processor 114 adapted to receive the images and process the images 96.

[0108] The image processing segments the tissue into different regions as explained above and perfusion measures P1, P2, based on average PPG amplitude levels for the segmented areas, are obtained for each tissue area from a PPG perfusion map.

[0109] The system has a display 116 for displaying the output, for example as a perfusion map and a delay map and/or as PPG signals over time.

[0110] The tissue classification may use machine learning algorithms, such as Support Vector Machines (SVMs) and Convolutional Neural Networks (CNNs).

[0111] The field of view of the camera could include all of the organs inside the abdomen or only a part of the organs inside. By classifying the tissue types present in the field of view, a comparison of perfusion is prevented between tissue which can have different levels of perfusion due to their different functions and they can even be supplied by different artery branches.

[0112] FIG. 12 shows a computer-implemented tissue analysis method 120. The method 120 comprises receiving image sensor images in step 122.

[0113] A PPG perfusion map is obtained from PPG amplitude levels obtained from the images in step 124.;

[0114] In step 126, a PPG delay map is derived from PPG relative delays between different image regions.

[0115] Images are then segmented in step 128 into one or more tissue regions based on the PPG delay map (or after receiving segmentation information identifying tissue regions derived from the PPG delay map). The different tissue regions have distinct PPG delay characteristics.

[0116] A level of perfusion is then determined separately for each tissue region.

[0117] In FIG. 12, the PPG perfusion and PPG delay map are shown sequentially, but in practice the processing is preferably performed simultaneously.

[0118] The remote PPG sensing used in the system and method described above may be performed using broad band illumination, such as using ambient light and visible light cameras.

[0119] However, there is also the option of using hyperspectral imaging. Hyperspectral imaging (HSI) is an emerging imaging modality for medical applications that offers great potential for non-invasive disease diagnosis and surgical guidance. The objective of hyperspectral imaging is to collect a three-dimensional dataset of spatial and spectral information, known as hypercube.

[0120] As shown in FIG. 13, the hypercube is three-dimensional dataset 50 comprising two-dimensional images 52 at each of a set of wavelengths. FIG. 13 also shows a reflectance curve (i.e. the spectral signature) of a pixel in each image.

[0121] The use of hyperspectral images allows additional image processing to be performed, for example it enables a level of oxygenation of tissue to be obtained. A contact PPG probe contains two LEDs and photodetectors at two different wavelengths. By combining the readings at these two wavelengths, an estimation of the oxygen level is possible. For non-contact PPG, RGB cameras have red, green, and blue channels but they are sensitive to much broader wavelength ranges, so extracting oxygenation from a normal RGB camera is not possible. The HSI camera acquires images at specific and narrow wavelengths, so the images can be combined to extract oxygenation values.

[0122] The imaging sensor or camero 112 of the system may for example comprise a charged coupled device (CCD) to record images and output the images in analog or digital electronic format or signal. The imaging sensor can comprise or be a camera. Such imaging sensors or cameras are known in the art and available from several vendors.

[0123] The system, processor or processor circuit may have an input interface 200 communicatively coupled to the processor, or processor circuit where the input interface is configured to receive images 96 from the imaging sensors to be used by the processor. For example, the images may be received in the analog or digital electronic format or signal.

[0124] Functions and methods relating to image processing, data processing and/or output or image generation for output to a user as described herein can be implemented in a general-purpose computer, a processor, processor circuit. Suitable processors or processor circuits include, by way of example processor 203 comprising one or more of, a general-purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine. Such processors may be made with semiconductor technology as is known in the art.

[0125] The system or processor (processor circuit) preferably includes a memory 204 accessible by the processor.

[0126] The system or processor (processor circuit) includes an output interface 208 in communicatively coupled to the processor (processor circuit) and configured to output result data and/or images 210 to a user interface or other device 116 for further processing of the result. Exemplifying user interfaces include a display device having an input for coupling to the output interface and receiving the result data to generate a display of images or the result visible to a user.

[0127] Functions and methods related to e.g., image and/or data processing and/or data or image generation described herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium and/or downloadable from a communications network for execution by a general-purpose computer, processor or processor circuit.

[0128] A computer program 206, software or firmware may thus be stored/distributed on a suitable medium an optical storage medium or a solid-state medium supplied together with or as part of other hardware. Examples of non-transitory computer-readable storage media include a read only memory (ROM,) such as electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM or a random access memory (RAM) such as Dynamic RAM (DRAM) or static RAM (SRAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magnetooptical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). The medium may be a memory of the system accessible by the processor or processor circuit of the system.

[0129] Alternatively, or additionally a computer program, software or firmware may be stored and distributed and downloadable from communications network such as wide area network (WAN), local area network (LAN) or Wireless LAN (WLAN), such as the Internet or via other wired or wireless telecommunication systems such as for example 3G, 4G, or 5G networks. If the term adapted to is used in the claims or description, it is noted the term adapted to is intended to be equivalent to the term configured to. If the term arrangement is used in the claims or description, it is noted the term arrangement is intended to be equivalent to the term system, and vice versa.

[0130] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure and the appended claims. In the claims, the word comprising does not exclude other elements or steps, and the indefinite article a or an does not exclude a plurality.

[0131] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0132] Any reference signs in the claims should not be construed as limiting the scope.