A computer assisted system is disclosed that includes an optical tracking system and one or more computing devices. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, generate depth maps from the color images, reconstruct three-dimensional surfaces of structures based on the depth maps, and output a display comprising the reconstructed three-dimensional surface and one or more surgical objects that are associated with the tracked fiducials. The computer assisted system can further include a monitor or a head-mounted display (HMD) configured to present augmented reality (AR) images during a procedure.

A system and a method are disclosed that allow for generation of a model or reconstruction of a model of a subject based upon acquired image data. The image data can be acquired in a substantially mobile system that can be moved relative to a subject to allow for image acquisition from a plurality of orientations relative to the subject. The plurality of orientations can include a first and final orientation and a predetermined path along which an image data collector or detector can move to acquire an appropriate image data set to allow for the model of construction.

Imaging marker embodiments that may be used for marking sites within a patient's body are discussed. Some imaging marker embodiments are particularly useful for imaging with ultrasound imaging modalities and some imaging marker embodiments may be suitable for imaging with multiple modes of imaging modalities. Method embodiments for making and using imaging markers are also discussed herein.

Disclosed biopsy markers are adapted to serve as localization markers during a surgical procedure. Adaptation includes incorporation of materials detectable under ultrasound during surgery, as well as features for co-registration with image guidance or other real-time imaging technologies during surgery. Such biopsy markers, when used as localization markers, improve patient comfort and reduce challenges in surgical coordination and surgery time. Additional disclosed biopsy markers are adapted to serve as monitoring and/or detection apparatuses, Localization of an implanted marker may be done with ultrasound technology. Ultrasound image data is analyzed to identify the implanted marker. A distance to the marker or a lesion may be determined and displayed. The determined distance may be a distance between the ultrasound probe and the marker or lesion, a distance between the marker or lesion and an incision instrument, and/or a distance between the ultrasound probe and the incision instrument.

Surgical systems for use in surgical procedures utilizing robotic devices. The surgical system having one or more components for housing a sensor or one or more tools for anchor or sensor delivery. The surgical system may include a surgical sensor anchor and/or a surgical sensor anchor delivery tool. A method of performing a robotically assisted surgical procedure, comprising using a surgical sensor anchor during a surgical procedure which utilizes a robot to track movement of at least one portion of a body structure undergoing a surgical procedure or to track movement of a body structure near a surgical site.

Aspects of the present disclosure relate to systems, devices and methods for performing a surgical step or surgical procedure with visual guidance using a head mounted display. A computer processor can be configured to adjust or select the focal plane and/or focal point of the display of the virtual data by the head mounted display using a measured or determined distance of the head mounted display to an anatomic structure or a physical surgical tool, instrument, implant and/or device.

A surgery assistive system includes an instrument having a tool and a manipulator connected to the tool, a spatial sensor system for detecting spatial information of the tool, and a computer system for manipulating a kinematic state of the manipulator. A method for obtaining surface information by the surgery assistive system includes the steps of: defining a target region and a plurality of reference points on a virtual anatomical model of a subject; prompting a user to generate sampling information by using the instrument to sample a plurality of sampling points on the subject corresponding to the reference points; and designating the sampling information as surface information of the sampling points. Each piece of the sampling information includes a coordinate of one of the sampling points, an angle of a contact of the tool at the sampling point, and parameters associated with the contact.

A computer assisted system is disclosed that includes an optical tracking system. The optical tracking system includes an RGB sensor and is configured to capture color images of an environment in the visible light spectrum and tracking images of fiducials in the environment in a near-infrared spectrum. The computer assisted surgical system is configured to generate a color image of the environment using the color images, identify fiducial locations using the tracking images, register pre-operative and/or intra-operative data to the color images using the fiducial locations, and generate an augmented reality (AR) image of the environment by overlaying the data over the color image. The computer assisted surgical system can further include a monitor or a head-mounted display (HMD) configured to present the AR image.

A method includes placing a set of electrodes on a body surface of a patient's body. The method also includes digitizing locations for the electrodes across the body surface based on one or more image frames using range imaging and/or monoscopic imaging. The method also includes estimating locations for hidden ones of the electrodes on the body surface not visible during the range imaging and/or monoscopic imaging. The method also includes registering the location for the electrodes on the body surface with predetermined geometry information that includes the body surface and an anatomical envelope within the patient's body. The method also includes storing geometry data in non-transitory memory based on the registration to define spatial relationships between the electrodes and the anatomical envelope.

Disclosed biopsy markers are adapted to serve as localization markers during a surgical procedure. Adaptation includes incorporation of materials detectable under ultrasound during surgery, as well as features for co-registration with image guidance or other real-time imaging technologies during surgery. Such biopsy markers, when used as localization markers, improve patient comfort and reduce challenges in surgical coordination and surgery time. Additional disclosed biopsy markers are adapted to serve as monitoring and/or detection apparatuses. Localization of an implanted marker may be done with ultrasound technology. Ultrasound image data is analyzed to identify the implanted marker. A distance to the marker or a lesion may be determined and displayed. The determined distance may be a distance between the ultrasound probe and the marker or lesion, a distance between the marker or lesion and an incision instrument, and/or a distance between the ultrasound probe and the incision instrument.