For three-dimensional segmentation from two-dimensional intracardiac echocardiography imaging, the three-dimension segmentation is output by a machine-learnt multi-task generator. Rather than the brute force approach of training the generator from 2D ICE images to output a 2D segmentation, the generator is trained from 3D information, such as a sparse ICE volume assembled from the 2D ICE images. Where sufficient ground truth data is not available, computed tomography or magnetic resonance data may be used as the ground truth for the sample sparse ICE volumes. The generator is trained to output both the 3D segmentation and a complete volume (i.e., more voxels represented than in the sparse ICE volume). The 3D segmentation may be further used to project to 2D as an input with an ICE image to another network trained to output a 2D segmentation for the ICE image. Display of the 3D segmentation and/or 2D segmentation may guide ablation of tissue in the patient.

Presented herein are methods, systems, devices, and computer-readable media for image guided surgery. The systems herein allow a physician to use multiple instruments for a surgery and simultaneously provide image-guidance data for those instruments. Various embodiments disclosed herein provide information to physicians about procedures they are performing, the devices (such as ablation needles, ultrasound wands or probes, scalpels, cauterizers, etc.) they are using during the procedure, the relative emplacements or poses of these devices, prediction information for those devices, and other information. Some embodiments provide useful information about 3D data sets. Additionally, some embodiments provide for quickly calibratable surgical instruments or attachments for surgical instruments.

An exemplary system includes a memory storing instructions and a processor communicatively coupled to the memory. The processor may be configured to execute the instructions to obtain one or more parameters of a non-robotic device in a surgical space, the non-robotic device engaged by a computer-assisted surgical system; generate, based on at least the one or more parameters of the non-robotic device, guidance content for use by the computer-assisted surgical system to facilitate guided teleoperation of the non-robotic device; and provide the guidance content to the computer-assisted surgical system.

A system to treat a patient comprises a user interface that allows a physician to view an image of tissue to be treated in order to develop a treatment plan to resect tissue with a predefined removal profile. The image may comprise a plurality of images, and the planned treatment is shown on the images. The treatment probe may comprise an anchor, and the image shown on the screen may have a reference image marker shown on the screen corresponding to the anchor. The planned tissue removal profile can be displayed and scaled to the image of the target tissue of an organ such as the prostate, and the physician can adjust the treatment profile based on the scaled images to provide a treatment profile in three dimensions. The images shown on the display may comprise segmented images of the patient with treatment plan overlaid on the images.

Various embodiments of the present disclosure encompass an endoluminal therapy system employing an endoluminal therapy device (21) and an endoluminal therapy monitoring controller (10). In support an endoluminal procedure, the endoluminal therapy device (21) is controlled to treat a site to be treated within a lumen. The controller (10) is operated to synchronize an activated dwell timing of the endoluminal therapy device (21) within the lumen to a tracked positioning of the endoluminal therapy device (21) contiguous with the site to be treated within the lumen. The controller (10) is further operated to monitor the site to be treated within the lumen induced by the endoluminal therapy device (21) during the synchronization by the controller (10) of the activated dwell timing of the endoluminal therapy device (21) within the lumen to the tracked positioning of the endoluminal therapy device (21) contiguous with the site to be treated within the lumen.

Embodiments of the invention provide systems and methods for providing augmented reality to a surgeon with the steps of acquiring at least one preoperative medical scan image of a region of interest in which the surgery is to be performed and introducing a fiducial device non-invasively or minimal invasively into a body lumen present in the region of interest of a patient. The fiducial device will be in particular introduced into a lumen, which can be clearly identified in the medical scan image. The fiducial device is adapted to emit infrared light either by using infrared light source or by using fluorescent material placed in or in the fiducial device ad to be excited by illumination light or exciting light. After acquiring at least one imaging light live image and one infrared live image the tissue structure of the body lumen with the fiducial device placed therein is clearly detectable within the live image and can reliably registered and matched to the scan image for presenting the overlaid images and to the surgeon.

Surgical systems are provided. The surgical system includes an energy applying surgical instrument configured to apply energy to a natural body lumen or organ. A first scope device is configured to transmit image data of a first scene within a field of view. A second scope device is configured to transmit image data of a second scene within a field of view. A controller is configured to receive the transmitted image data of the first and second scenes and to provide a merged image of first and second scenes, where the merged image facilitates coordination of a location of energy to be applied to an inner surface of a tissue wall at a surgical site relative to an intended interaction location of a second instrument on an outer surface of the tissue wall in a subsequent procedure step at the surgical site.

In general, devices, systems, and methods for multi-source imaging are provided.

A method comprises registering a robotic system with a patient based on first image data of the patient generated by a first modality to form a registered system, determining, within the registered system, a location for placing an electrode on the patient based on an area of interest for the patient, and causing the robotic system to place the electrode at the location on the patient.

A remote control apparatus to remotely control a patient-side system including medical equipment and an endoscope to capture an image of a surgery site according to an embodiment includes: an operation handle with which an operator controls the medical equipment; and an operation pedal section. The operation pedal section includes: plural pedals configured to be pressed down to execute functions concerning the medial equipment; and a base on which the plural pedals are arranged, the plural pedals being arranged at locations not overlapping each other in a planar view. The plural pedals include first height pedals with upper ends thereof located at a first height position and second height pedals with upper ends thereof located at a second height position, which is different from the first height position. The first height pedals are arranged alternately with the second height pedals.