A computer assisted system is disclosed that includes an optical tracking system and one or more computing devices. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, generate depth maps from the color images, reconstruct three-dimensional surfaces of structures based on the depth maps, and output a display comprising the reconstructed three-dimensional surface and one or more surgical objects that are associated with the tracked fiducials. The computer assisted system can further include a monitor or a head-mounted display (HMD) configured to present augmented reality (AR) images during a procedure.

A number of improvements are provided relating to computer aided surgery. The improvement relates to both the methods used during computer aided surgery and the devices used during such procedures. Some of the improvement relate to controlling the selection of which data to display during a procedure and/or how the data is displayed to aid the surgeon. Other improvements relate to the structure of the tools used during a procedure and how the tools can be controlled automatically to improve the efficiency of the procedure. Still other improvements relate to methods of providing feedback during a procedure to improve either the efficiency or quality, or both, for a procedure.

A system for aiding surgery on a patient is described including a display device and a storage device that stores an image of at least a portion of the anatomy of the patient, including one or more surgical navigation markers positioned on the patient, for display on the display device. An analyser is adapted to receive positional data of a probe based on positioning of the probe relative to the one or more markers on the patient. Based on the positional data, the analyser outputs correctional data to adjust an alignment of the image on the display device to match locations of said one or more markers.

Systems and methods are provided in which intraoperatively acquired surface data is employed to verify the correspondence of an intraoperatively selected spinal level with a spinal level that is pre-selected based on volumetric image data. Segmented surface data corresponding to the pre-selected spinal levels may be obtained from the volumetric image data, such that the segmented surface data corresponds to a spinal segment that is expected to be exposed and identified intraoperatively during the surgical procedure. The segmented surface data from the pre-selected spinal level, and adjacent segmented surface data from an adjacent spinal level that is adjacent to the pre-selected spinal level, is registered to the intraoperative surface data, and quality measures associated with the registration are obtained, thereby permitting an assessment or a determination of whether or not the pre-selected spinal surface (in the volumetric frame or reference) is likely to correspond to the intraoperatively selected spinal level.

A force sensed surface scanning system (20) employs a scanning robot (41) and a surface scanning controller (50). The scanning robot (41) includes a surface scanning end-effector (43) for generating force sensing data informative of a contact force applied by the surface scanning end-effector (43) to an anatomical organ. In operation, the surface scanning controller (50) controls a surface scanning of the anatomical organ by the surface scanning end-effector (43) including the surface scanning end-effector (43) generating the force sensing data, and further constructs an intraoperative volume model of the anatomical organ responsive to the force sensing data generated by the surface scanning end-effector (43) indicating a defined surface deformation offset of the anatomical organ.

A method and system for facilitating identification and marking of a target in a displayed Fluoroscopic Three-Dimensional Reconstruction (F3DR) of a body region of a patient. The system includes a display and a storage device storing instructions for receiving an initial selection of the target in the F3DR, fining the F3DR based on the initial selection of the target, displaying the fined F3DR on the display, and receiving a final selection of the target in the fined F3DR via a user selection. The system further includes at least one hardware processor configured to execute said instructions. The method and instructions may also include receiving a selection of a medical device in two two-dimensional fluoroscopic images, where the medical device is located in an area of the target, and initially fining the F3DR based on the selection of the medical device.

Systems and methods for generating patient-specific surgical guides comprising: capturing a first and second images of an orthopedic element in different reference frames using a radiographic imaging technique, detecting spatial data defining anatomical landmarks on or in the orthopedic element using a neural network, applying a mask to the orthopedic element defined by an anatomical landmark, projecting the spatial data from the first image and the second image to define volume data, applying the neural network to the volume data to generate a reconstructed three-dimensional (“3D”) model of the orthopedic element; and calculating dimensions for a patient-specific surgical guide configured to abut the orthopedic element.

A system for constructing fluoroscopic-based three-dimensional volumetric data of a target area within a patient from two-dimensional fluoroscopic images including a structure of markers, a fluoroscopic imaging device configured to acquire a sequence of images of the target area and of the structure of markers, and a computing device. The computing device is configured to estimate a pose of the fluoroscopic imaging device for at least a plurality of images of the sequence of images based on detection of a possible and most probable projection of the structure of markers as a whole on each image of the plurality of images. The computing device is further configured to construct fluoroscopic-based three-dimensional volumetric data of the target area based on the estimated poses of the fluoroscopic imaging device.

A method of modeling lungs of a patient includes acquiring computed tomography data of a patient's lungs, storing a software application within a memory associated with a computer, the computer having a processor configured to execute the software application, executing the software application to differentiate tissue located within the patient's lung using the acquired CT data, generate a 3-D model of the patient's lungs based on the acquired CT data and the differentiated tissue, apply a material property to each tissue of the differentiated tissue within the generated 3-D model, generate a mesh of the 3-D model of the patient's lungs, calculate a displacement of the patient's lungs in a collapsed state based on the material property applied to the differentiated tissue and the generated mesh of the generated 3-D model, and display a collapsed lung model of the patient's lungs based on the calculated displacement of the patient's lungs.

A method and system of correcting alignment of catheter relative to a target including receiving signals from an inertial measurement unit located at a distal end of a catheter, determining movement of the distal end of the catheter caused by physiological forces, receiving images depicting the distal end of the catheter and the target, identifying the distal end of the catheter and the target in the images, determining an orientation of the distal end of the catheter relative to the target and articulating the distal tip of the catheter in response to the detected movement to achieve and maintain an orientation towards the target such that a tool extended from an opening at the distal end of the catheter would intersect the target.