METHOD FOR OPERATING A MEDICAL NAVIGATION SYSTEM FOR AN IMAGE-GUIDED SURGICAL PROCEDURE

Abstract

A method for operating a medical navigation system for an image-guided surgical procedure including: training a first artificial neural network on a set of surgical image video data of a target object of a patient representing a video image, and identifying the structure of the target object; training a second artificial neural network on at least one virtual three-dimensional target object model of the target object and identifying the structure of the virtual three-dimensional target object model by the identified structure of the target object; training a third artificial neural network by the identified structure of the target object and the structure of the virtual three-dimensional target object model and aligning the identified structure of the target object, with the structure of the corresponding target object of the virtual three-dimensional target object model, which is identified by the second artificial neural network.

Claims

1. A method for operating a medical navigation system for an image-guided surgical procedure comprising: training of a first artificial neural network on a set of surgical image video data of a target object of a patient, which represents a video image, and identifying the structure of the target object of the patient; training of a second artificial neural network on at least one virtual three-dimensional target object model of the target object and identifying the structure of the virtual three-dimensional target object model by the structure of the target object of the patient, which is identified by the first artificial neural network; and training of a third artificial neural network by the identified structure of the target object of the patient and the structure of the virtual three-dimensional target object model and aligning the structure of the target object of the patient, which is identified by the first artificial neural network, with the structure of the corresponding target object of the virtual three-dimensional target object model, which is identified by the second artificial neural network.

2. The method according to claim 1, wherein the set of surgical image video data is captured by a video processor.

3. The method according to claim 1, further comprising displaying the set of surgical image video data on a monitor and displaying the structure of the virtual three-dimensional target object model, which is identified by the second artificial neural network on a monitor.

4. The method according to claim 2, further comprising displaying the set of surgical image video data in a predetermined orientation and displaying the structure of the virtual three-dimensional target object model, which is identified by the second artificial neural network in an orientation which corresponds or aligns with the predetermined orientation of the surgical image video data.

5. The method according to claim 3, further comprising detecting a landmark in the surgical image video data that triggers the second artificial neural network for training on the virtual three-dimensional model.

6. The method according to any of claim 5, wherein the second artificial neural network is configured to be trained on virtual three-dimensional models of the target object of the patient in order to learn and identify a corresponding landmark of the landmark, which is identified by the first artificial neural network, in the structure of the virtual three-dimensional target object model.

7. The method according to claim 6, wherein in case that the corresponding landmark in the structure of the virtual three-dimensional target object model cannot be displayed due to a hidden pose of the corresponding landmark in the structure of the virtual three-dimensional target object model, the method further comprising repositioning the structure of the virtual three-dimensional target object model such that the structure of the virtual three-dimensional target object model with the corresponding landmark is displayed on a monitor.

8. The method according to claim 1, wherein the third neural network is configured to be trained to match a pose of the landmark in the set of surgical image video data in order to align the structure of the target object of the patient, which is identified by the first artificial neural network, with the structure of the virtual three-dimensional target object model, which is identified by the second artificial neural network.

9. The method according to claim 1, wherein the first artificial neural network is an object recognition network, the object recognition network is configured to learn representations of key anatomical structures of the target object of the patient, and the representations are intraoperative representations.

10. The method according to claim 1, wherein the second artificial neural network is an object recognition network, the object recognition network is configured to learn representations of anatomical structures of the target object of the patient in different virtual three-dimensional models.

11. The method according to claim 1, wherein the third artificial neural network is configured to determine world coordinates of the different virtual three-dimensional model with respect to the orientation of the anatomical structures of the target object of the patient in the virtual three-dimensional model.

12. Non-transitory computer-readable storage medium storing instructions that cause a computer to: train of a first artificial neural network on a set of surgical image video data of a target object of a patient, which represents a video image, and identifying the structure of the target object of the patient; train of a second artificial neural network on at least one virtual three-dimensional target object model of the target object and identifying the structure of the virtual three-dimensional target object model by the structure of the target object of the patient, which is identified by the first artificial neural network; and train of a third artificial neural network by the identified structure of the target object of the patient and the structure of the virtual three-dimensional target object model and aligning the structure of the target object of the patient, which is identified by the first artificial neural network, with the structure of the corresponding target object of the virtual three-dimensional target object model, which is identified by the second artificial neural network.

13. A method for operating a medical navigation system for an image-guided surgical procedure comprising: training of a first artificial neural network on surgical image video data of a target object of a patient and identifying the structure of the target object of the is patient; acquiring a segmentation layer of a predetermined identified structure of the target object of the patient; and training of a further artificial neural network by the structure of the target object of the patient, identified by the first artificial neural network, and the predetermined identified structure of the target object of the patient, provided by the segmentation layer, and aligning the structure of the target object of the patient, identified by the first artificial neural network, with the predetermined structure of the corresponding target object, provided by the segmentation layer.

14. The method according to claim 13, wherein the set of surgical image video data is captured by a video processor.

15. The method according to claim 13, further comprising displaying the set of surgical image video data on a monitor and displaying the predetermined structure of the target object, provided by the segmentation layer on a monitor.

16. The method according to claim 14, further comprising displaying the set of surgical image video data in a predetermined orientation and displaying the predetermined structure of the target object in an orientation, which corresponds or aligns with the predetermined orientation of the surgical image video data.

17. The method according to claim 13, wherein the segmentation layer comprises meta data of a computed tomography image of the target object of the patient, wherein the meta data represent the predetermined identified structure of the target object of the patient.

18. The method according to claim 13, wherein in case that the predetermined identified structure of the target object of the patient cannot be displayed due to a hidden pose of the predetermined identified structure of the target object of the patient, the method further comprising repositioning the predetermined identified structure of the target object of the patient such that the predetermined identified structure of the target object of the patient is displayed on a monitor.

19. The method according to claim 13, wherein the further neural network is configured to be trained to match a pose of the landmark in the set of surgical image video data in order to align the structure of the target object of the patient, which is identified by the first artificial neural network, with the predetermined identified structure of the target object of the patient, provided by the segmentation layer.

20. The method according to claim 13, wherein the first artificial neural network is an object recognition network, wherein the object recognition network is configured to learn representations of key anatomical structures of the target object of the patient, wherein the representations are intraoperative representations.

21. The method according to claim 13, wherein the further artificial neural network is configured to determine world coordinates of the different virtual three-dimensional model with respect to the orientation of the anatomical structures of the target object of the patient in the virtual three-dimensional model.

22. Non-transitory computer-readable storage medium storing instructions that cause a computer to: train of a first artificial neural network on surgical image video data of a target object of a patient and identifying the structure of the target object of the patient; acquire a segmentation layer of a predetermined identified structure of the target object of the patient; and train of a further artificial neural network by the structure of the target object of the patient, identified by the first artificial neural network, and the predetermined identified structure of the target object of the patient, provided by the segmentation layer, and aligning the structure of the target object of the patient, identified by the first artificial neural network, with the predetermined structure of the corresponding target object, provided by the segmentation layer.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Further characteristics will become apparent from the description of the embodiments together with the claims and the included drawings. Embodiments can fulfill individual characteristics or a combination of several characteristics.

[0055] The embodiments are described below, without restricting the general intent of the invention, based on exemplary embodiments, wherein reference is made expressly to the drawings with regard to the disclosure of all details that are not explained in greater detail in the text. In the drawings:

[0056] FIG. 1 schematically illustrates three images of a laparoscopic surgery of a liver with a display of a virtual 3D model of the liver, and

[0057] FIG. 2 illustrates a schematic workflow of a method for operating a medical navigation system for an image-guided surgical (IGS) procedure according to one embodiment.

DETAILED DESCRIPTION

[0058] FIG. 1 shows schematically three examples, which are taken from a laparoscopic liver surgery. All images show an image of a liver (as a target object of a patent) during a surgery with a display of a virtual 3D model of the liver (as 3D target object model) in the upper right corners of a monitor, such as a surgical monitor.

[0059] In the upper image 1 of FIG. 1, a virtual 3D model is displayed on a surgical monitor. The virtual 3D model is displayed on the laparoscopic video screen encircled in FIG. 1 in the upper image 1.

[0060] By means of a first artificial neural network, an anatomical structure in the laparoscopic image of a real liver captured by a video sensor is detected. The detected anatomical structure is shown in the middle image in the video image. The anatomical structure is marked with a V-shaped mark and an arrow on the left side of the (middle) image 2 of FIG. 1.

[0061] A second artificial neural network is then used and/or triggered, wherein the second artificial neural network is configured to take the information about the anatomical structure that was detected in the video image as input and to detect the same structure on virtual 3D model (see small right arrow).

[0062] The middle image 2 in the middle indicates a match, where the notch of the liver at the position of the falciform ligament was detected by the first artificial neural network in the laparoscopic video image (highlighted and with an arrow on the left side) and the same anatomical landmark structure was detected by the second artificial neural network on the virtual 3D model as well (highlighted and with an arrow on the right side).

[0063] Afterwards a third artificial neural network is used and configured to align the virtual 3D model of the liver (as target object) with the position of the liver in the laparoscopic image, as shown in the bottom image 3.

[0064] The bottom image 3 of FIG. 1 shows the case in which the third artificial neural network estimates the parameters for repositioning of the virtual 3D model in a way that it aligns with the intraoperative pose (or orientation) of the liver.

[0065] FIG. 2 shows a schematic workflow of a method for operating a medical navigation system for an image-guided surgical (IGS) procedure according to one embodiment.

[0066] After capturing an image of a target object of a patient by a digital video sensor during a surgery, with e.g., a laparoscope, the set of the surgical image video data are analyzed by a first artificial neural network in step 101, wherein the anatomical structure of interest, like landmarks in the image, is detected and identified on the video image.

[0067] In the second step 102, the anatomical structure of interest is detected on the virtual 3D model of the target object by a second artificial neural network.

[0068] Then in the next step 103, the current position of a virtual camera in a 3D rendering scene is determined. Afterwards, in the following step 104 the decision has to be made whether the virtual camera is positioned in a way that the anatomical structure of interest can be seen by the surgeon. In case Yes, in the step 105 the anatomical structures of interest in video image as well as in virtual 3D model are highlighted on the screen.

[0069] In case that the anatomical structure of interest cannot be seen by the surgeon on the monitor in step 105, then the method carries out step 107 to reorient the virtual camera in the rendering scene so that the anatomical structure of interest will become visible to the surgeon on the monitor.

[0070] Finally, in step 106 a matching of the organ pose is performed by a third artificial neural network, wherein the matching is based on the anatomical is structure of interest in video image and the anatomical structure of interest virtual 3D model.

[0071] While there has been shown and described what is considered to be embodiments of the invention, it will, of course, be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. It is therefore intended that the invention be not limited to the exact forms described and illustrated, but should be constructed to cover all modifications that may fall within the scope of the appended claims.