This document describes devices used during surgical procedures for the treatment of heart conditions. For example, this document describes technology to monitor the operations of a heart-lung machine and then shows associated read outs on a head-worn display in order to provide an augmented-reality presentation. For example, various sensors on and around a heart-lung machine, patient, and/or extracorporeal circuit can monitor the operations of the procedure using the heart-lung machine.

A system is disclosed that includes an optical tracking device and a surgical computing device. The optical tracking device includes a structured light module and an optical module that includes an image sensor and is spaced from the structured light module at a known distance. The surgical computing device includes a display device, a non-transitory computer readable medium including instructions, and processor(s) configured to execute the instructions to generate a depth map from a first image captured by the image sensor during projection of a pattern into a surgical environment by the structured light module. The pattern is projected in a near-infrared (NIR) spectrum. The processor(s) are further configured to execute the stored instructions to reconstruct a 3D surface of anatomical structure(s) based on the generated depth map. Additionally, the processor(s) are configured to execute the stored instructions to output the reconstructed 3D surface to the display device.

A method includes tracking one or more of a plurality of bones of a patient, adjusting a relationship between the plurality of bones to a desired rotation between the plurality of bones, and creating an implant placement plan based on the relationship. The implant placement plan includes a placement of a plate across the plurality of bones. The method also includes robotically assisting preparation of the plurality of bones to receive the plate in accordance with the implant placement plan.

An augmented reality, AR, system (100) for use in a medical procedure is disclosed. The AR system (100) comprises an AR headset (2), and a processor (12). The AR headset (2) comprises a camera (6a, 6b), a near eye display (4a, 4b) and a depth sensor (10a, 10b). The processor (12) is configured to adjust the position of the image obtained by the camera (6a, 6b) on the display (4a, 4b) throughout the medical procedure based on changes in the distance measured by the depth sensor (10a, 10b).

An example method includes determining, by the one or more processors, an actual orientation of a surgical pin as installed in a bone of a patient; obtaining, by the one or more processors, a planned orientation of the surgical pin; determining, by the one or more processors and based on a comparison between the actual orientation of the surgical pin and the planned orientation of the surgical pin, whether the surgical pin was installed as planned; and responsive to determining that the surgical pin was not installed as planned, outputting, via a visualization device, virtual guidance to assist a surgeon in correcting the installation of the surgical pin.

A medical support arm includes: a support arm that supports an endoscope; an actuator that drives the support arm; a measurement unit that measures a load applied to the actuator; a generation unit that generates three-dimensional information in a body into which the endoscope is inserted; and a correction unit that corrects the three-dimensional information on a basis of the measured load.

Methods and systems for displaying an overlay superimposed with an intraoperative image of surgical field in a medical ophthalmic procedure, such that the overlay appears at a desired depth within the image are provided. Methods and system for displaying an overlay superimposed with a stereoscopic intraoperative image pair of a surgical field in a medical ophthalmic procedure are provided.

Various embodiments of an apparatus, methods, systems and computer program products described herein are directed to Field Visualization Engine. The Field Visualization Engine tracks one or more collimator poses relative to one or more Augmented Reality (AR) headset device poses. Each respective collimator pose and each respective headset device pose corresponds to a three-dimensional (3D) unified coordinate space (“3D space”). The Field Visualization Engine generates an AR representation of a beam emanating from the collimator based at least on a current collimator pose and a current headset device pose. The Field Visualization Engine further generates an AR visualization of emanation of the beam throughout an AR display of medical data.

An endoscope navigation system is provided that updates an anatomy model based on a live camera signal. As the endoscope advances within the patient, a camera signal, and corresponding position signal, of the endoscope, is provided and used to update the anatomy model based on identified divergences between the camera signal at a particular position and anatomy model.

A system and method is provided for planning a surgical procedure. The system includes a model of an anatomical region including simulated bone, a tracked tool, a tracking system, a display device, and a computer system configured to receive information from the tracking system, generate and display the model and the tracked tool, and store the tracking system information in a computer readable memory that is portable to a surgical navigation system. The method includes acquiring volumetric data of a patient anatomical region, extracting, providing and registering a three-dimensional model with the volumetric data, using a tracked tool to perform a simulated surgical procedure including removing and replacing a bone section on the model, viewing the model and the tracked tool on a display device, recording the procedure with a tracking system, storing the procedure in a computer readable memory, and porting the procedure into a surgical navigation system.