Devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display are disclosed.

A projection system for projecting medical information onto the surface of the human body such that the medical information conforms and registers to the contour of the underlying shape of the region it is being projected upon. The projection system is in communication with one or more projectors and one or more sensors which capture the topography/area/volume of the region of interest and/or the view of the user, and alter the medical information such that the medical information remains accurate in dimensions when projected into the area of interest.

A method for actuating an X-ray device during robot-assisted navigation of at least two objects introduced into a body, such as a hollow organ, of a patient by at least one robotic system, includes providing a first selection criterion for selecting one of the objects, and providing a second selection criterion for selecting one of the objects. One of the at least two objects is selected based on the first selection criterion and the second selection criterion, and recording parameters of the X-ray device are automatically set, such that the selected object in an X-ray image to be recorded is highlighted compared to the at least one other object. The selected object may be highlighted with respect to image quality, image flavor, positioning on the X-ray image, and/or a collimator setting of a collimator of the X-ray device. An X-ray image is recorded with the set recording parameters.

Apparatus and methods are described including tracking a tool portion and a patient marker from a first line of sight, using a first tracking device disposed upon a first head-mounted device that includes a display. The tool portion and the patient marker are tracked from a second line of sight, using a second tracking device. When a portion of the patient marker and the tool portion are both within the first line of sight, an augmented reality image is generated upon the first display based upon data received from the first tracking device and without using data from the second tracking device. When at least the patient marker portion and the tool portion are not both within the first line of sight, a virtual image of the tool and anatomy of the patient is generated using data received from the second tracking device. Other applications are also described.

A virtual representation of an operating room is generated based on robot information and sensing of the OR with depth cameras. One of the depth cameras is integrated with a portable electronic device, operated by a local user in the operating room. The virtual representation of the OR is communicated to the virtual reality headset, with three-dimensional point cloud data. A virtual reality environment is rendered to a display of the virtual reality headset, operated by a remote user. A virtual representation of the remote user is rendered in augmented reality to a display of the portable electronic device.

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

A method of controlling a tool of a surgical robot with a processing unit such that the tool follows input of the input handle of a user console includes receiving movement of the input handle in a master frame of the user console, translating the movement of the input handle in the master frame to movement of the tool in a camera frame of a camera providing real-time images of a surgical site, translating the movement of the tool in the camera frame to movement of the tool in a world frame defined in a surgical theater, translating the movement of the tool in the world frame to movement of the tool in a tool frame which is defined by an arm supporting the tool, and transmitting control commands to a motor controlling the tool such that the tool follows movement of the input handle.

A method of calibrating a registration of an augmented reality device 2 comprised in a surgical navigation system 100 is provided. The method comprises obtaining a first transformation 38 between a coordinate system 40 of the augmented reality device 2 and a reference coordinate system 42 of the surgical navigation system 100, a second transformation 44 between a coordinate system of a reference object 24 and the reference coordinate system 42 and obtaining geometrical properties of the reference object 24. The method further comprises determining a visual representation 34 of the reference object 24 to be displayed by the augmented reality device 2 and obtaining at least one first viewing direction 56 of a user using the augmented reality device 2, the at least one first viewing direction 56 being associated with the reference coordinate system 42. The method comprises determining a calibrated first transformation, which is a transformation between the coordinate system 40 of the augmented reality device 2 and the reference coordinate system 42.

Disclosed are systems, devices, and methods for performing treatment along a lumen network, an exemplary method comprising receiving image data of a patient's lungs, mapping one or more luminal networks inside the patient's lungs based on the received image data, identifying a treatment target in the image data, determining a luminal pathway to the treatment target via at least one of the luminal networks, configuring treatment parameters for treatment of the treatment target and at least one of the luminal networks, navigating a tool inside at least one of the luminal networks to the treatment target, treating the treatment target with a primary treatment modality, and treating the luminal pathway of at least one of the luminal networks leading to or from the treatment target with a secondary treatment modality.

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.