Systems and methods are provided for planning a procedure. A display device is configured to display a first virtual element. A controller device having a processor is configured to be in communication with the display device, and the controller device is further configured to direct the display device to display the first virtual element. A physical control element is in communication with the controller device, and is configured to correspond to the first virtual element such that an actual manipulation of the control element is displayed, via the processor of the controller device and on the display device, as a corresponding response of the first virtual element to the actual manipulation of the control element. Associated systems, methods, and computer program products are also provided.

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

A method to securely transfer data to aid in a medical procedure or to a data management system to be accessed in a medical care facility is provided that includes a set of data from a data storage system being downloaded via a network to a visible light communication (VLC) enabled device. The set of data from the VLC enabled device is then transferred to a VLC enabled device located in an operating room (OR) via point to point VLC or uploaded to a data management system. The transferred set of data is then used to aid in a medical procedure or be accessed by a medical care facility. Software executables are similarly transferred and run in an OR context.

A method comprises the steps of: imaging a patient anatomy; selecting an implant strategy for at least one bone fastener having a first member; registering the imaging of the patient anatomy with imaging of at least a portion of a robot; engaging the first member with tissue of the patient anatomy via robotic guidance according to the implant strategy; and subsequently, manipulating the patient anatomy. Systems, spinal constructs, implants and surgical instruments are disclosed.

Provided is an image processing device including a matching unit that performs matching processing between a predetermined pattern on a surface of a 3D model of a biological tissue including an operating site generated on the basis of a preoperative diagnosis image and a predetermined pattern on a surface of the biological tissue included in a captured image during surgery, a shift amount estimation unit that estimates an amount of deformation from a preoperative state of the biological tissue on the basis of a result of the matching processing and information regarding a three-dimensional position of a photographing region which is a region photographed during surgery on the surface of the biological tissue, and a 3D model update unit that updates the 3D model generated before surgery on the basis of the estimated amount of deformation of the biological tissue.

A system for executing a three-dimensional (3D) intraoperative scan of a patient is disclosed. A 3D scanner controller projects the object points included onto a first image plane and the object points onto a second image plane. The 3D scanner controller determines first epipolar lines associated with the first image plane and second epipolar lines associated with the second image plane based on an epipolar plane that triangulates the object points included in the first 2D intraoperative image to the object points included in the second 2D intraoperative image. Each epipolar lines provides a depth of each object as projected onto the first image plane and the second image plane. The 3D scanner controller converts the first 2D intraoperative image and the second 2D intraoperative image to the 3D intraoperative scan of the patient based on the depth of each object point provided by each corresponding epipolar line.

Disclosed are systems, methods, and techniques for registering a HMD coordinate system of a head-mounted display (HMD) and a localizer coordinate system of a surgical navigation localizer. A camera of the HMD captures at least one image of a registration device having a registration coordinate system and a plurality of registration markers. The registration markers are analyzed in the at least one image to determine a pose of the HMD coordinate system relative to the registration coordinate system. One or more position sensors comprised in the localizer detect a plurality of tracking markers comprised in the registration device to determine a pose of the registration coordinate system relative to the localizer coordinate system. The HMD coordinate system and the localizer coordinate system are registered using the registration device, wherein positions of the registration markers are known with respect to positions of the tracking markers in the registration coordinate system.

A method and system for hand tracking in a robotic system includes a hand tracking system and a controller coupled to the hand tracking system. The controller is configured to receive, from the hand tracking system, a plurality of locations of a hand; determine if the hand is in a first hand pose based on the plurality of locations; in response to determining that the hand is in the first hand pose, and switch the robotic system to a hand trajectory detection mode. While in the hand trajectory detection mode, the control unit is configured to detect, based on hand tracking information from the hand tracking system, that the hand has performed a first hand trajectory of a plurality of known hand trajectories; and in response to detecting the first hand trajectory, change a mode of operation of the robotic system.

A system and method for tracking completeness of co-registered medical image data is disclosed herein. The system and method tracks the position of an anatomical reference marker positionable on a patient and an ultrasound probe during an imaging session and co-registers medical images based on positional data received from the anatomical reference marker and the ultrasound probe. Using the co-registered image data, the system and method generates a surface contour of a region of interest (ROI) of the patient, such as a breast. The surface contour is defined to represent an interface between a chest wall structure and tissue of the ROI in a plurality of co-registered medical images. A completeness map of the image data within the defined surface contour during the imaging session is generated and overlaid on a graphic representation of the ROI.

Methods of immobilizing the head of a patient include heating a thermoplastic mask preform to a formable temperature, forming the thermoplastic mask over at least a mouth of the patient, and pushing a mouth receiving end of a bite piece into a portion of the thermoplastic mask preform that is positioned over the mouth of the patient, which moves the portion of the thermoplastic mask preform into the mouth of the patient. The methods also include allowing the thermoplastic mask preform to cool from the formable temperature, causing the portion of the thermoplastic mask moved into the mouth of the patient to harden.