Devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display are disclosed.

A surgical system includes an XR headset and an XR headset controller. The XR headset is configured to be worn by a user during a surgical procedure and includes a see-through display screen configured to display an XR image for viewing by the user. The XR headset controller is configured to receive a plurality of two-dimensional (“2D”) image data associated with an anatomical structure of a patient. The XR headset controller is further configured to generate a first 2D image from the plurality of 2D image data based on a pose of the XR headset. The XR headset controller is further configured to generate a second 2D image from the plurality of 2D image data based on the pose of the XR headset. The XR headset controller is further configured to generate the XR image by displaying the first 2D image in a field of view of a first eye of the user and displaying the second 2D image in a field of view of a second eye of the user.

Augmented reality (AR) systems and methods involve an interactive head-mounted device (HMD), an external display, and a medical image computer, which is in communication with the HMD and the external display. The external display displays one or more planes of a medical image or a 3D model provided by the medical image computer. A user wearing the HMD may manipulate a medical image or 3D model displayed on the external display by focusing the user's gaze on a control object and/or a portion of a medical image or 3D model displayed on a display of the interactive HMD.

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

A navigated surgery system includes at least one processor that is operative to obtain a 2D medical image slice of anatomical structure of a patient. The operations further obtain a 3D graphical model of anatomical structure. The operations determine a pose of a virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the 2D medical image slice. The operations control the XR headset to display the 2D medical image slice of the anatomical structure of the patient, display the 3D graphical model of the anatomical structure, and display a graphical object oriented with the pose relative to the 3D graphical model of the anatomical structure.

Disclosed are systems, devices, and methods for performing treatment along a lumen network, an exemplary method comprising receiving image data of a patient's lungs, mapping one or more luminal networks inside the patient's lungs based on the received image data, identifying a treatment target in the image data, determining a luminal pathway to the treatment target via at least one of the luminal networks, configuring treatment parameters for treatment of the treatment target and at least one of the luminal networks, navigating a tool inside at least one of the luminal networks to the treatment target, treating the treatment target with a primary treatment modality, and treating the luminal pathway of at least one of the luminal networks leading to or from the treatment target with a secondary treatment modality.

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

A mixed reality system includes a head-mounted device (HMD) having an HMD coordinate system and a camera to capture images. A localizer has a localizer coordinate system and position sensors supported by a housing to track a pose of a real object. Registration markers are configured to be recognized from the captured images of the camera of the HMD. The registration markers are disposed on the housing of the localizer in a predetermined spatial relationship with respect to the localizer coordinate system. At least one controller is configured to utilize the registration markers recognized from the captured images to register the localizer coordinate system and the HMD coordinate system.

Systems and methods for virtual reality or augmented reality (VR/AR) visualization of 3D medical images using a VR/AR visualization system are disclosed that includes a computing device operatively coupled to a VR/AR device, which includes a holographic display and at least one sensor. The holographic display is configured to display a holographic image to an operator. The computing device is configured to receive at least one stored 3D image of a subject's anatomy and at least one real-time 3D position of at least one surgical instrument, to register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one 3D image of the subject's anatomy, and to generate the holographic image comprising the at least one real-time position of the at least one surgical instrument overlaid on the at least one 3D image of the subject's anatomy.

Examples relate to a microscope system, such as surgical microscope system, and to a system, a method, and a computer program for a microscope system. The system is configured to obtain sensor data of at least one sensor of the microscope system, the sensor data representing a view of a sample being observed through a microscope of the microscope system. The system is configured to determine a three-dimensional representation of the sample based on the sensor data. The system is configured to determine information on a viewing angle of an observer. The system is configured to determine a computer-generated view of the sample from the viewing angle of the observer based on the three-dimensional representation of the sample. The system is configured to generate a display signal for a display device, the display signal comprising the computer-generated view of the sample.