A system, method, and apparatus for displaying a fused reconstructed image with a multidimensional image are disclosed. An example imaging system receives a selection corresponding to a portion of a displayed multidimensional visualization of a surgical site. At the selected portion of the multidimensional visualization, the imaging system displays a portion of a three-dimensional image which corresponds to the selected multidimensional visualization such that the displayed portion of the at least one of the three-dimensional image or model is fused with the displayed multidimensional visualization.

Computer-assisted surgical systems and methods provide intraoperative playback of and interaction with recorded video images and/or a 3D model while displaying current video images. The methods and related surgical systems involve capturing, by a two-dimensional (2D) or three-dimensional (3D) video camera, current video images and recording the captured video images. A user interface displays the current video images and the recorded video images and/or the 3D model and enables a user to interact with either or both of them.

The invention provides, in some aspects, a system for implementing a rule derived basis to display volumetric image sets. In various embodiments of the invention, the selection of the images to be displayed, the generation of the 3-D volumetric image from measured 2-D images including the rendering parameters and styles, the choice of viewing directions and 2-D projection images based on the viewing directions, the layout of the projection images, and the formation of a video can be determined using a rule derived basis. In an embodiment of the present invention, the user is presented with sequential images making up a video displayed based on their preferences without having to first manually adjust parameters. The present invention allows for novel ways of viewing such images to detect microcalcifications and obstructions when reviewing Digital Breast Tomosynthesis and other volumetric mammography images.

Systems and methods for improving endoscopy procedures are described that provide not only a conventional real time image of the view obtained by an endoscope, but in addition, a near real time 3D model and/or a 2D flattened image of an interior surface of an organ, which model and image may be processed using AI software to highlight potential tissue abnormalities for closer examination and/or biopsy during the procedure. A navigation module interacts with other system outputs to further assist the endoscopist with navigational indicia, e.g., landmarks and/or directional arrows, that enhance the endoscopists' spatial orientation, and/or may provide navigational guidance to the endoscopist to assist manipulation of the endoscope.

A method includes receiving data characterizing a target surface extending in three dimensions. The method also includes rendering in a graphical user interface display space, a first visual representation including a two-dimensional image of a first portion of the target surface, and a second visual representation including a three-dimensional representation of at least a subset of the first portion of the target surface included in the first visual representation. The method further includes receiving, based on a first user interaction with the three-dimensional representation via a cursor configured to move over the three-dimensional representation, a first user input indicative of selection of a first location of the target surface. The method also includes rendering a first graphical object at a first target position in the three-dimensional representation and a second graphical object at a second target position in the two-dimensional image. The first and the second target positions are indicative of the first location of the target surface.

An image processing device includes a UI controller and a rendering processor. The rendering processor obtains, from volume data constituted by multiple points, each point having a value, surface data representing the shape of a surface. The shape of the surface is defined by magnitude of each of the values. The UI controller sets a threshold for the values to determine whether to display the surface data. The rendering processor determines an intersecting point at which a straight line passing through a position set as a viewpoint and a point specified by a user intersects with the shape of the surface which is to be displayed in accordance with the threshold. Based on a plane passing through the intersecting point, the rendering processor controls the displaying of the shape of the surface represented by the surface data.

There is disclosed an augmented reality user interface including dual representation of a physical location including generating two views for viewing the augmented reality objects, a first view includes the video data of the view including the augmented reality objects superimposed thereover in augmented reality locations and a second view that includes data derived from the physical location to generate a map with the augmented reality objects from the first view visible as objects on the map in the augmented reality locations, combining the location, the motion data, the video data, and the augmented reality objects into an augmented reality video such that when the computing device is in a first position, the first view is visible and when the computing device is in a second position, the second view is visible, and displaying the augmented reality video on a display.

A method includes receiving a first image including color data and depth data, determining a viewpoint associated with an augmented reality (AR) and/or virtual reality (VR) display displaying a second image, receiving at least one calibration image including an object in the first image, the object being in a different pose as compared to a pose of the object in the first image, and generating the second image based on the first image, the viewpoint and the at least one calibration image.

Systems and method of manually and automatically generating 3D models and correcting 3D models by identifying vasculature withing image scan data and constructing a series of connected segments, each segment of the 3D model having a diameter of an oblique view of the vasculature associated with that segment.

A system and method for planning surgical procedure including a treatment zone setting view presenting at least one slice of a 3D reconstruction generated from CT image data including a target. The treatment zone setting view presenting a treatment zone marker defining a location and a size of a treatment zone and configured to adjust the treatment zone marker in response to a received user input. The system and method further including a volumetric view presenting a 3D volume derived from the 3D reconstruction and a 3D representation of the treatment zone marker relative to structures depicted in the 3D volume.