IMAGE PROCESSING METHOD AND APPARATUS USING ELASTIC MAPPING OF VASCULAR PLEXUS STRUCTURES

Abstract

The invention relates to an image processing method and a medical observation device (1) such as a microscope (2) or endoscope. The device and method are used for displaying output image data (54) of soft biological tissue (12). In image-guided surgery, pre-operative three-dimensional image data (6) of the soft biological tissue (12) are elastically matched to interoperative image data (14) which are acquired during surgery. By displaying the elastically matched pre-operative three-dimensional image data (6) together with the interoperative image data (14), the surgeon may be made aware of the consistence of the soft biological tissue (12) below the visible surface layer. Existing systems for image-guided surgery need to be manually readjusted if surgery is done on soft biological tissue (12), which may deform and shift. To avoid this, the device and method according to the invention perform an elastic matching of the pre-operative three-dimensional image data (6) based on the interoperative image data (14) of the soft biological tissue (12). At least one vascular plexus structure (48) is first identified in the interoperative image data (14) and then the same vascular plexus structure (48) is identified in the pre-operative three-dimensional image data (6). The vascular plexus structure (48) in the pre-operative three-dimensional image data (6) is then elastically matched to the vascular plexus structure (48) in the interoperative image data (14). Output image data (54) are generated combining the elastically matched pre-operative three-dimensional image data (6) to the interoperative image data (14). Preferably, the at least one vascular plexus structure (48) is identified using fluorescent light from a fluorophore (22) which has been injected into the soft biological tissue (12).

Claims

1. An image processing method for displaying output image data (54) of soft biological tissue (12) in real time during surgery, comprising the steps of: providing pre-operative three-dimensional image data (6) of the soft biological tissue (12); acquiring interoperative image data (14) of the soft biological tissue (12) in at least one of the visible-light spectrum and the near-infrared spectrum; automatically identifying at least one vascular plexus structure (48) in the interoperative image data (14); identifying the at least one identified vascular plexus structure (48) in the pre-operative three-dimensional image data (6); elastically matching the pre-operative three-dimensional image data (6) to the interoperative image data (14) based on the at least one identified vascular plexus structure (48); and forming the output image data (54) from the elastically matched pre-operative three-dimensional image data (6) and interoperative image data (14).

2. The image processing method according to claim 1, wherein the interoperative image data (14) in which the vascular plexus structure (48) is automatically identified is acquired using fluorescent light from a fluorophore (22).

3. The image processing method according to claim 1, wherein the identified vascular plexus structure (48) in the interoperative image data (14) has been recorded using light in the visible spectrum.

4. The image processing method according to claim 1, further comprising the step of displaying the output image data (54).

5. The image processing method according to claim 1, wherein at least one optical filter arrangement (26) for reducing specular reflections is used for acquiring the interoperative image data (14).

6. The image processing method according to claim 1, wherein a blood flow direction (64) is computed at at least one location (66) in the identified vascular plexus structure (48).

7. The image processing method according to claim 4, further comprising the step of acquiring position data (60) representative of a position of a field of view (10) of an optical lens system (62) used for acquiring the interoperative image data (14) at the time of acquiring the interoperative image data (14), and wherein the position data (60) are used in at least one of identifying the vascular plexus structure (48) in the pre-operative three-dimensional image data (6), elastically matching the pre-operative three-dimensional image data (6) to the interoperative image data (14), and displaying the output image data (54).

8. The image processing method according to claim 7, wherein the position data comprise at least one of a focal length of the optical lens system (62), a field depth of the optical lens system (62), and a distance setting of the optical lens system (62).

9. The image processing method according to claim 7, wherein the position data comprise a size, a dimension and an orientation of the field of view (10) of the optical lens system (62).

10. A medical observation device (1) for generation of output image data (54) of soft biological tissue (12) during surgery, the medical observation device (1) comprising: a memory assembly (4) comprising pre-operative three-dimensional image data (6) of the soft biological tissue (12); a camera assembly (8) for acquiring interoperative image data (14) of the soft biological tissue (12) in at least one of the visible-light spectrum and the near-infrared spectrum; an image processor assembly (40) which comprises a pattern-matching module (46) configured to identify at least one vascular plexus structure (48) in the interoperative image data (14) and to identify the at least one identified vascular plexus structure (48) in the pre-operative three-dimensional image data (6), a matching module (50) configured to elastically match the pre-operative three-dimensional image data (6) to the interoperative image data (14) based on the at least one identified vascular plexus structure (48), and an image-forming module (52) configured to combine at least part of the elastically matched pre-operative three-dimensional image data (6) with the interoperative image data (14) to form the output image data (54); and an output interface (55) configured to output the output image data (54).

11. The medical observation device (1) according to claim 10, further comprising a display assembly (56) for displaying the output image data (54).

12. The medical observation device (1) according to claim 10, wherein the camera assembly (8) comprises at least one filter arrangement (26) for reducing specular reflections, the filter arrangement (26) comprising at least one pair of cross-polarizers.

13. The medical observation device (1) according to claim 10, wherein the camera assembly (8) comprises at least one filter arrangement (28) having a pass band matched to a fluorescence spectrum of at least one fluorophore (22).

14. The medical observation device (1) according to claim 10, wherein the camera assembly (8) comprises an optical lens system (62) and at least one position sensor (58) for acquiring position (60), the position data (60) being representative of at least one of a focal length, a field depth and a distance setting of the optical lens system (62), or a size, dimension and orientation of the field of view (10).

15. The medical observation device (1) according to claim 10, wherein the image processor assembly (40) is configured to compute a blood flow direction (64) in the identified vascular plexus structure (48) and wherein the image-forming module (52) is adapted to combine the identified vascular plexus structure (48) in the output image data (54) with time-varying marker data which are representative of the blood flow direction (64).

16. The medical observation device (1) according to claim 10, wherein the medical observation device (1) is a microscope (2).

17. The medical observation device (1) according to claim 10, wherein the medical observation device (1) is an endoscope.

18. A non-transitory computer-readable medium storing a programme causing a computer to execute the image processing method according to claim 1.

Description

BRIEF DESCRIPTION OF THE DRAWING VIEW

[0030] FIG. 1 shows an exemplary embodiment of the method and device according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0031] The configuration and function of an optical observation device 1 for observing live tissue, in particular during surgery, is explained. The medical observation device is shown to be a microscope 2 just for the purposes of explanation. The medical observation device 1 may also be an endoscope (not shown).

[0032] The medical observation device 1 comprises a memory assembly 4, in which pre-operative three-dimensional image data 6 are stored. The memory assembly 4 may comprise standard computer memory.

[0033] The medical observation device 1 further comprises a camera assembly 8, which has a field of view 10. During surgery, soft biological tissue 12, such as brain tissue, muscle tissue, lymph tissue or tissue of an internal organ or of other soft body parts, may be arranged in the field of view 10. During surgery, the camera assembly 8 acquires interoperative image data 14, which may be structured as a single input frame 16 or a time series 18 of input frames 16. The interoperative image data 14 may comprise or consist of pixels 20. The interoperative image data 14 may be two-dimensional, i.e. representing a plane in the field of view 10, three-dimensional, i.e. representing a volume in the field of view 10, or multi-dimensional image data which may e.g. comprises three-dimensional data in the field of view 10 at different spectral wavelengths.

[0034] The camera assembly 8 may comprise at least one of an RGB camera, a multi-spectral camera and a hyper-spectral camera.

[0035] The interoperative image data comprise or consist of fluorescent-light image data. Such fluorescent-light image data may be obtained when a fluorophore 22, such as indocyanine green, is injected into the tissue 12 and illuminated at wavelengths which trigger the fluorescence.

[0036] The camera assembly 8 may comprise one or more filter assemblies 24, which are only schematically shown in FIG. 1. The filter assembly 24 may comprise a filter arrangement 26 for blocking specular reflections. Examples of such a filter arrangement are described in European patent application EP 16 204 933.2, which is herewith incorporated by reference in its entirety.

[0037] The filter assembly 24 may also comprise a band-pass filter arrangement 28 for restricting the light in the interoperative image data 14 to the fluorescence wavelengths of the at least one fluorophore 22. Such a filter arrangement is shown in European patent application EP 17 179 019.8, which is herewith incorporated in its entirety by reference.

[0038] The medical observation device 1 may further include an illumination assembly 32 for generating illumination light 34 having an illumination spectrum. The illumination spectrum may be restricted to or include wavelengths that trigger fluorescence of the at least one fluorophore 22. The illumination light 34 may further comprise or be restricted to wavelengths that match the reflectance spectrum of arterial blood. The illumination light 34 may be restricted to or comprise wavelengths that are matched to the reflectance spectrum of venous blood. Restricting the illumination spectrum of the illumination light 34 to a single or to preferably separate wavelengths reduced cross-talk between the various frequency bands. This facilitates an automatic analysis of the interoperative image data 14. Subsequent input frames 16 may have been acquired at different illumination spectra. Alternatively, the interoperative image data 14 contain information preferably in at least one of the visible-light spectra, e.g. in at least one of the reflective spectrum of arterial blood and venous blood, and the near-infrared spectrum, e.g. in the fluorescence wavelengths of the at least one fluorophore 22.

[0039] The medical observation device 1 further includes an image processor assembly 40, which only by way of example is shown as an integrated circuit in FIG. 1. The image processor 40 and its constituents may be software-implemented, hardware-implemented or be implemented as a combination of hardware and software. The memory assembly 4 may be part of the image processor 40. The image processor assembly 40 may comprise several modules which may be differentiated functionally and/or structurally. The image processor assembly 40 is connected to the camera assembly 8 by a data connection 42, which may be wired and/or wireless. An input interface 44 of the image processor assembly 40 may be adapted to acquire interoperative image data 14 from at least one camera assembly 8 and/or a storage where the interoperative image data 14 are stored or buffered, e.g. after pre-processing, such as memory assembly 4.

[0040] The image processor assembly 40 may comprise a pattern-matching module 46 for identifying at least one vascular plexus structure 48 in the interoperative image data 14 and for identifying the at least one identified vascular plexus structure 48 in the pre-operative three-dimensional image data 6.

[0041] The at least one vascular plexus structure 48 may be identified e.g. in interoperative image data 14a which are restricted to the fluorescence spectrum of the fluorophore 22. In addition or alternatively, the at least one vascular plexus structure 48 may be identified in interoperative image data 14b which have been recorded in the visible-light spectrum and may in particular be restricted to at least one of the reflectance spectrum of arterial blood and venous blood. Algorithms for identifying a vascular plexus structure 48 in the interoperative image data 14 and then identifying this structure in the pre-operative three-dimensional image data 6 are, for example, given in Rouchdy, Y.; Cohen, L.: A Geodesic Voting Method of Tubular Tree and Centrelines, DOI: 10.1109/ISBI.2011.5872566, pp. 979-983, and Suri, J. S.; Laxminarayan, S. (eds): Angiography and Plaque Imaging: Advanced Segmentation Techniques, CRC Press, pp. 501-518. Further, a method of identifying vascular plexus structures by using a bolus of at least one fluorophore is described in EP 17 174 017.5, which is included in its entirety by reference.

[0042] The image processor assembly 40 may further comprise an elastic-matching module 50 for elastically matching the pre-operative three-dimensional image data 6 to the interoperative image data 14 based on the at least one identified vascular plexus structure 48. Again, an algorithm for performing such elastic matching is described in IRCS Technical Reports Series, 192, Elastically Deforming a Three-Dimensional Atlas to Match Anatomical Brain Images as given above. As a result of the elastic matching, the pre-operative three-dimensional image data 6 are shifted, rotated and/or distorted so that the identified vascular plexus structure 48 in both data coincides geometrically. For this, the step of elastic matching may also define a section through the pre-operative three-dimensional image data 6 which results in the field of view 10 represented in the interoperative image data 14. Further, if the camera assembly 8 comprises at least one of a multispectral camera and a hyperspectral camera for acquiring the interoperative image data 14, the blood vessel structure 48 and its type may be determined using the apparatus and method described in Opt. Express 17, 15670-15678, Ultra-fast Multispectral Optical Imaging of Cortical Oxigenation, Blood Flow, and Intracellular Calcium Dynamics, as given above.

[0043] The image processor assembly 40 may further comprise an image-forming module 52 for combining the elastically matched pre-operative three-dimensional image data 6 or a section thereof with the interoperative image data 14 into output image data 54. The image processor assembly may further comprise an output interface 55 for outputting the output image data 54. The output interface 55 may comprise at least one standard connector, such as an HDMI, DVI, RGB or any other suitable type of connector, and/or a wireless connection, including the matching data transmission protocol.

[0044] For displaying the output image data 54, the medical observation device 1 may comprise a display assembly 56, which may include stereoscopic display, for example an eyepiece of a microscope or endoscope and/or a monitor. The display assembly 56 may be connected to the output interface 55 by wire or wireless.

[0045] The computational burden for locating and/or identifying the identified vascular plexus structure 48 of the interoperative image data 14 in the pre-operative three-dimensional image data 6 may be high. This burden can be reduced if position information is provided as to where the field of view 10, the vascular plexus structure 48, and/or the interoperative image data 14 are located within the soft biological tissue 12. This information is useful if, for example, an elastic-matching process has already been successfully carried out for one input frame 16 of a time series 18 and then has to be repeated for subsequent input frame 16 of the time series 18 later during surgery. For this, the medical observation device 1 may comprise at least one position sensor 58 for acquiring and/or providing position data 60 representative of at least one of focal length, field depth and distance setting of an optical lens system 62 of the medical observation device 1 and size, distance from the optical lens system 62 and the image processor assembly 40, such as a dimension of the field of view 10.

[0046] The position data 60 may be input into at least one of the pattern-matching modules 46, the elastic-matching module 50 and the image-forming module 52 and be used in identifying the at least one identified vascular plexus structure 48 in the pre-operative three-dimensional image data 6, the elastic matching of the pre-operative three-dimensional image data 6 to the interoperative image data 14 and for forming the output image data 54 from the elastically matched pre-operative three-dimensional image data 6 and the interoperative image data 14.

[0047] The image processor assembly 40 may be adapted to compute the blood flow direction 64 at a location 66 in the at least one identified vascular plexus structure 48 as is described in European patent application EP 17 210 909.2 which is included by reference in their entirety. A location may be a pixel 20 or a preferably coherent array of pixels in the interoperative image data 14. The tissue type of the identified vascular plexus structure 48, i.e. a differentiation between venous, arterial and capillary tissue and/or vessels may be carried out using the method and system described in EP 17 174 047.5.

[0048] As described, the above-described system and method rely solely on information that is automatically acquired when the interoperative image data 14 are recorded. The system and method may further comprise positioning data 60 that result from the settings of the medical observation device 1 for acquiring the interoperative image data 14.

REFERENCE NUMERALS

[0049] 1 medical observation device [0050] 2 microscope [0051] 4 memory assembly [0052] 6 pre-operative three-dimensional image data [0053] 8 camera assembly [0054] 10 field of view [0055] 12 soft biological tissue [0056] 14 interoperative image data [0057] 14a fluorescent-light interoperative image data [0058] 14b visible-light interoperative image data [0059] 16 input frame [0060] 18 time series of input frames [0061] 20 pixel [0062] 22 fluorophore [0063] 24 filter assembly [0064] 26 filter arrangement for blocking specular reflections [0065] 28 band-pass filter arrangement for fluorescent light [0066] 32 illumination assembly [0067] 34 illumination light [0068] 40 image processor assembly [0069] 42 data connection [0070] 44 input interface [0071] 46 pattern-matching module [0072] 48 vascular plexus structure [0073] 50 elastic-matching module [0074] 52 image-forming module [0075] 54 output image data [0076] 55 output interface [0077] 56 display assembly [0078] 58 position sensor [0079] 60 positioning data [0080] 62 optical lens system [0081] 64 blood flow direction [0082] 66 location