SYSTEM AND METHOD FOR IMAGE REGISTRATION

Abstract

Embodiments of the invention provide systems and methods for providing augmented reality to a surgeon with the steps of acquiring at least one preoperative medical scan image of a region of interest in which the surgery is to be performed and introducing a fiducial device non-invasively or minimal invasively into a body lumen present in the region of interest of a patient. The fiducial device will be in particular introduced into a lumen, which can be clearly identified in the medical scan image. The fiducial device is adapted to emit infrared light either by using infrared light source or by using fluorescent material placed in or in the fiducial device ad to be excited by illumination light or exciting light. After acquiring at least one imaging light live image and one infrared live image the tissue structure of the body lumen with the fiducial device placed therein is clearly detectable within the live image and can reliably registered and matched to the scan image for presenting the overlaid images and to the surgeon.

Claims

1. An augmented reality system comprising: a scan apparatus adapted to acquire at least one medical scan at least one pre-operative medical scan image of a region of interest can be derived from, in which region a surgery is to be performed, a fiducial device adapted to be introduced into a body lumen present in the region of interest of a patient, the fiducial device being adapted to emit infrared light and/or visible light, a light source adapted to illuminate the surgery site with illumination light, where the surgery is to be performed, a live imaging device adapted to acquire at least one illumination light live image and an infrared light live image, an image registering system adapted to register or re-register the at least one pre-operative medical scan image and the at least one infrared light live image together with the illumination light live image, and an image reproduction device adapted to reproduce the visible light image registered with the medical scan image.

2. The augmented reality system according to claim 1, wherein the fiducial device comprises a body comprising at least one area with infrared, in particular near-infrared fluorescent material.

3. The augmented reality system according to claim 2, wherein the fiducial device at least two areas with infrared, in particular near-infrared fluorescent material placed in a distance one from another.

4. The augmented reality system according to claim 1, wherein the body of the fiducial device is a stiff or flexible tube or rod-like element.

5. The augmented reality system according to claim 1, wherein the image registering system comprises a contour detector adapted to identify anatomic structures in the scan image.

6. The augmented reality system according to claim 5, wherein the contour detector is adapted to detect the contour of the body lumen adapted to receive the fiducial device.

7. The augmented reality system according to claim 1, wherein the imaging light is visible light.

8. The augmented reality system according to claim 1, wherein the light source is adapted to emit excitation light adapted to cause the fluorescent material emit infrared or near-infrared light.

9. The augmented reality system according to claim 1, wherein the image registering system is adapted to determine the image scale of the imaging light live image and enhance or reduce the medical scan image to the same scale.

10. The augmented reality system according to claim 3, wherein the scale is defined by the distance the at least two areas with infrared, in particular near-infrared fluorescent material are placed one from another.

11. The augmented reality system according to claim 1, wherein the image registering system is adapted to determine deformation of the imaging light live image versus the medical scan image by comparing the shape and position of the lumen in the scan image with the shape and position of the fiducial device in the infrared image.

12. The augmented reality system according to claim 11, wherein the image registering system is adapted to deform the medical scan image in accordance with the deformation of the infrared image.

13. The augmented reality system according to claim 1, wherein the live imaging device is connected to a position tracking system.

14. The augmented reality system according to claim 1, wherein the fiducial device is connected to a position tracking system.

15. A method for providing augmented reality to a surgeon, comprising the steps of: acquiring at least one pre-operative medical scan image of a region of interest, in which a surgery is to be performed, introducing a fiducial device into a body lumen present in the region of interest of a patient, the fiducial device being adapted to emit infrared light, illuminating the surgery site, where the surgery is to be performed, with illumination light and exiting light, acquiring at least one imaging light live image and an infrared light live image, registering the at least one pre-operative medical scan image and the at least one infrared light live image together with the visible light live image, and reproducing the visible light image registered with the medical scan image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details and advantages can be taken from the drawing and the embodiments of description and from claims as well. Embodiments of the invention are illustrated in the drawings in which

[0020] FIG. 1 illustrates the inventive system during medical scan of a patient an during surgery in a highly abstract way,

[0021] FIG. 2 an illustration of a laparoscopic camera and tissue of the patient during surgery,

[0022] FIG. 3 medical scan images taken during medical imaging,

[0023] FIG. 4 an illumination light live image,

[0024] FIG. 5 an infrared light live image,

[0025] FIG. 6 a medical scan image of the same region of interest as illustrated in FIGS. 4 and 5,

[0026] FIG. 7 an overlay image consisting of the illumination light image f FIG. 4 and the medical scan image of FIG. 6 registered along the infrared light image of FIG. 5,

[0027] FIG. 8 an illumination light live image similar to FIG. 4,

[0028] FIG. 9 an infrared light image of the operation site with a fiducial device introduced into a hollow organ present in the region of interest,

[0029] FIG. 10 a medical scan image of the region of interest,

[0030] FIG. 11 a distortion map obtained by comparing the infrared light image of FIG. 9 and the scan image of FIG. 10 and

[0031] FIG. 12 an overlay image presented to the surgeon obtained by overlaying the visible light image of FIG. 8 with the scan image of FIG. 10 corrected along the distortion indicated in FIG. 11.

DETAILED DESCRIPTION

[0032] An augmented reality system 14 can be taken from FIG. 1, which illustrates a patient 15 resting on a table 16 during medical imaging. As a scan apparatus 17 for acquiring CT scans a C-arm is illustrated for illustration purposes only. It should be understood that any medical imaging system and any medical imaging method and modality may be used for acquiring preoperative medical scan images and supplying them to a processing system 18.

[0033] The processing system 18 may control any movement and activation or deactivation of the C-arm or any other scan apparatus 17. The processing system may be any type of computer or computer network or hard and/or software adapted to produce scan images from a medical scan.

[0034] FIG. 3 illustrates the operation of the scan apparatus 17 together with the processing system 18. The scan apparatus 17 provides a number of scans e.g. CT scans 19. The processing unit 18 may determine a volume model 20 of the body portion of interest. Furthermore, the processing system 18 may provide one or several medical scan images 21 obtained by intersecting the volume model 20 in any desired plane. The medical scan image 21 may be displayed on an image reproduction device 21 (FIG. 1), which may be any type of screen, flat screen, stereoscopic screen, virtual reality glasses or the like.

[0035] The augmented reality system 14 further comprises an operation site 23 where the patient 15 can be placed on a suitable support 24 like a table, chair or any other type of support for having the patient 15 rest in a suitable position. The augmented reality system 14 further provides a preferably non-invasive fiducial device 25 adapted to be introduced into the patient's 15 body, preferably into a body lumen as there is the ureter. A tracking system 26 may allow for locating the fiducial device 25 relative to the patient 15 e.g. by triangulation. At least two position markers 27, 28 may be provided preferably at the proximal end of fiducial device 25 which are visible by the locating system 26. Cameras 29, 30 may locate the markers 27, 28 and hence the location and orientation of the fiducial device 25 by triangulation. It is noted that any other type of tracking systems may be used which is suited for determining location and orientation of the device 25 relative to the patient 15 and/or the support 24.

[0036] Furthermore, the augmented reality system 14 may comprise a live imaging device 31, which may be a laparoscopic camera, an endoscopic camera or any other camera system for providing live images from any region of interest 32 of the patient. The region of interest 32 is symbolized by a dotted line circle in FIG. 1 which is the region where the surgery is to be performed. Exactly that region has been imaged during the preoperative scan performed by the scan apparatus 17. The volume model 20 typically includes the region of interest 32. Normally the pre-operative scan is taken in a non-inflated state of the patient's body which will later be somewhat inflated with CO2 for laparoscopic surgery.

[0037] The laparoscopic camera or any other type of live imaging device 31 may be connected to the locating system 26 so that location and orientation of the live imaging device can be obtained e.g. by triangulation. The live imaging device 31 may be connected to at least two locating marks 33, 34, 35, which can be located by the cameras 29 and 30. However, it is noted that any other type of location system may equally work.

[0038] The locations and orientations of the fiducial device 25 and the live imaging device 31 determined by the locating system 26 will be supplied to the processing system 18 as well as will be the live images taken by the live imaging device 31.

[0039] FIG. 2 illustrates the distal end 31 in a functional representation. The live imaging device 31 comprises a camera 37 for capturing a live image 38 (FIG. 4) generated with illumination light 39. A light source 40 may be provided for producing illumination light 39 and illuminating at least that portion of the region of interest 32 which is seen by the camera 37. The illumination light 39 may be visible light, e.g. white light. The light is in particular directed to a location 41 at which the surgery is to be performed and which is seen by the camera 37.

[0040] The surgery site 41 may include a body lumen, e.g. an ureter 42 in which the fiducial device 25 is placed. The fiducial device 25 may be a stiff or flexible rod or tube-like member having at least one preferably two or more marks 43 to 45, which are adapted to emit light capable of permeating biological tissue 46, in particular soft tissue like the wall of a tubular structure (e.g. the ureter) or a vessel, fat, muscles, fascia or the like. The at least one mark 43 (44, 45) may be punctiform, axially linear, ring shaped or the like.

[0041] The marks 43 to 45 are in particular adapted to emit infrared in particular near infrared light IR. The marks 43 to 45 may be illuminated by the fibre optics or LED members. Preferably, however, the marks 43 to 45 are passive light sources formed by fluorescent material producing infrared light. The infrared light preferably is near infrared light 47 emitted when excited by exciting light 48 sent out by a light source 49. The light source 48 may be identical with the light source 40 so that the exciting light 48 and the illumination light 39 are identical. However, it is likewise possible to use an exciting light source 49 different from the illumination light source 40. While the illumination light 39 may have a broad spectrum the exciting light source 49 may or may not be a narrow band source producing infrared light, visible light, or ultraviolet light depending on the type of fluorescent material present in the marks 43, 44, 45.

[0042] The camera 37 may acquire the live image 38 shown in FIG. 4 and an infrared light live image 50 illustrated in FIG. 5. The life images 38 and 50 may be captured together in one single image or separately.

[0043] The augmented reality system 14 comprises and image registering system 51, which may be part of the processing system 18. The registering system 51 may be or include a contour detector for detecting at least on soft tissue contour. Such a contour may be a vessel or any other hollow organ or tubular structure (e.g. the ureter) or the surface of an organ.

[0044] The image registering system is adapted to register at least one preoperative medical scan image 21 and the illumination light live image 38. For doing so, the image registering system 51 uses the infrared light live image 50 and in particular the at least two marks 43, 44, which are placed within the ureter 42 illustrated in FIG. 5 in dotted lines. While the ureter 42 is a tissue structure clearly identifiable and clearly identified in the medical scan image 21, the ureter may not be so visible in the illumination light image 38. However, by placing the fiducial 25 in the ureter 42, the marks 43 and 44 will indicate where the ureter is in the live image 38 so that the image registering system 51 can now reliably and precisely register the medical scan image 21 to the illumination light image 38 (and the infrared light image 50 as well). After registering the illumination light live image 38 together with the scan image 21 (and if desired the infrared light image 59 may be displayed on the image reproduction device 22 as an overlay as illustrated in FIG. 7.

[0045] In doing so, the locating system 26 will always determine location and orientation of both, the fiducial device 25 and the live imaging device 31 which provides the illumination light image 38 and the infrared light image 50 together or separately. The processing system 18 will use the volume model 20 and find the scan image 21 at a plane, which is defined by the orientation of the live imaging device 31. The scan image 21 may include colours or graphics enhancing tissue structures for making visible different tissues like parenchyma tissue or tubular structures (e.g. the ureter) or vessels or the like. The system is in particular helpful if surgery in a distance of solid body portions like bones is performed. Organs of the patient may move due to relocation of the patient, due to relaxation, or due to inflation of the surgery site during laparoscopy. Using hollow organs for preferably non-invasively placing the fiducial device 25 therein, will help to avoid misalignment of the scan image 21 and the live image 38, which otherwise, could occur when only relying on bones or other body structures which soft tissue may move relative to.

[0046] Embodiments of the invention pave the way for further improvements:

[0047] As mentioned above soft tissue of the patient may undergo heavy deformations between medical imaging and surgery. Even the introduction of the fiducial devise 25 into the ureter 42 or any other body lumen as well as inflating the internal body cavities for laparoscopic surgery may deform tissue structures. Embodiments allow for adapting the medical scan image 21 similar or even the same way as the real body was deformed so that the deformed scan image can better be registered with the real live image. This process is to be performed by the processing system 18 and illustrated in FIGS. 8 to 12. The illumination light live image 38 is illustrated in FIG. 8. So is the infrared light image 50 illustrated in FIG. 9. As can be seen, the ureter 42 is straight so that the three marks 43 to 45 emitting infrared light are arranged in a straight line. Obviously, the patient's body is deformed since the medical scan image 21 includes the ureter 42 curved. Furthermore, the scan image 21 includes a solid body structure like a bone 52, which of course may not be deformed at all. So clearly there will always be some degree of misalignment no matter how one tries to register the illumination light live image 38 and the scan image 21. The processing system in particular, the image registering system 51 (which includes a contour detector) may now distort the medical scan image 21 the same way as the live image 38 is distorted relative to the (non-distorted) medical scan image 21. For doing so, the image registering system 51 now identifies the tissue structure the fiducial device 25 is placed in e.g. the ureter 42. In addition, it may identify natural solid body structures as bone 52. While the bone 52 can be seen as being non-distorted it can be seen that distortion of the scan image is necessary for bringing the scan image ureter 42 in registration with the three marks 43, 44, 45 of the infrared image 50 as illustrated in FIG. 11. Small arrows symbolize the degree of distortion of the scan image, which is necessary for bringing the medical scan image 21 into registration with the live images 38 and 50. The result can be seen from FIG. 12 which illustrates the distorted medical scan image 21′ in registration with the live images 38, 50.

[0048] Embodiments of the invention overcome formerly present difficulties and reliability issues of patent registration in laparoscopic surgeries and allows for augmented reality assistance. Embodiments provide a novel method of patient 15 registration using an infrared camera and a fiducial device 25 to be introduced in a body lumen like the ureter or another body orifice. The fiducial device 25 comprise fiducial markers 43 to 45. An embodiment of the invention reduces the number and complexity of planning work flow steps and also reduces or even eliminates the need of additional invasive procedures. Further the method allows for changing the patient's positions during surgery with minimal or no hindrances to the surgical procedures. The method does not require additional radiological imaging procedures included into the work flow.

[0049] The inventive method in particular allows for registration of preoperative scans taken with a “not deformed” patient body with a “heavily deformed” patient body as for example after insulation with CO2 gas for allowing laparoscopic surgery. Because of the deformation a preoperative scan image is no longer identical with an intraoperative medical scan image. Embodiments allow for compensating these deformations and removes the necessity of intraoperative scan imaging. This contributes to the patient's safety dramatically reduces the operation time and to some extent makes possible those surgery at all.

[0050] Embodiments provide systems and methods for providing augmented reality to a surgeon with the steps of acquiring at least one preoperative medical scan image of a region of interest in which the surgery is to be performed and introducing a fiducial device 25 non-invasively or minimal invasively into a body lumen present in the region of interest of a patient. The fiducial device will be in particular introduced into a lumen, which can be clearly identified in the medical scan image. The fiducial device 25 is adapted to emit infrared light either by using infrared light source or by using fluorescent material placed in or on the fiducial device and to be excited by illumination light 39 or exciting light 48. After acquiring at least one imaging light live image and one infrared live image the tissue structure of the body lumen with the fiducial device 25 placed therein is clearly detectable within the live image 38 and can reliably registered and matched to the scan image 21 for presenting the overlaid images 21 and 38 to the surgeon.