A system for use in surgery includes a central body, a visualization system operably connected to the central body, a video rendering system, a head-mounted display for displaying images from the video rendering system, a sensor system, and a robotic device operably connected to the central body. The visualization system includes at least one camera and a pan system and/or a tilt system. The sensor system tracks the position and/or orientation in space of the head-mounted display relative to a reference point. The pan system and/or the tilt system are configured to adjust the field of view of the camera in response to information from the sensor system about changes in at least one of position and orientation in space of the head-mounted display relative to the reference point.

An offsite device for remote augmented reality communication for guided surgery configured to receive, by an offsite device, a plurality of representations of a plurality of frames and a plurality of indices, wherein the plurality of indices includes at least a first frame index and at least a second frame index, display, by the offsite device and to an offsite surgeon, a second frame of the second frame index, input, by the offsite device, an illustration from the offsite surgeon, wherein the illustration is drawn on the second frame, register, by the offsite device, the illustration to the second frame, wherein registering produces registration data, and transmit, by the offsite device, the illustration and the registration data.

A surgical hub may be configured to receive an image from a laparoscopic scope and surgical information from at least one surgical instrument. The surgical hub may be operatively connected to multiple displays such as a primary display and a secondary display. The surgical hub may generate visualization data for the primary display. The surgical hub may obtain a visualization control mode based on a visualization control parameter and may determine whether to generate a different set of visualization data for a secondary display based on the visualization control mode. When the visualization control mode supports multiple display capabilities, the surgical hub may generate the visualization data specifically for the secondary display. When the visualization control mode does not support multiple display capabilities, the surgical hub may send the same the visualization data for display at the secondary display as the primary display.

The present disclosure involves object recognition as a method of registration, using a stereoscopic camera on Augmented Reality (AR) glasses or an endoscope as the image capture technology. Exemplary objects include surgical tools, anatomical components or features, such as bone or cartilage, etc. By detecting just a portion of the object in the image data of the surgical scene, the present disclosure may register and track a portion of the patient's anatomy, such as the pelvis, the knee, etc. The present disclosure also optionally displays information on the AR glasses themselves, such as the entire pelvis, the femur, the tibia, etc. The present disclosure may include combinations of the foregoing features, and may eliminate the need for electromagnetic, inertial, or infrared stereoscopic tracking as the tracking technology.

An example method includes registering, via a visualization device, a virtual model of a portion of an anatomy of an ankle of a patient to a corresponding portion of the anatomy of the ankle viewable via the visualization device, the virtual model obtained from a virtual surgical plan for an ankle arthroplasty procedure to attach a prosthetic to the anatomy. The example method also comprises displaying, via the visualization device and overlaid on the portion of the anatomy, a virtual guide that guides at least one of preparation of the anatomy for attachment of the prosthetic or attachment of the prosthetic to the anatomy.

A display system can include a head-mounted display configured to project light to an eye of a user to display augmented reality image content to the user. The display system can include one or more user sensors configured to sense the user and can include one or more environmental sensors configured to sense surroundings of the user. The display system can also include processing electronics in communication with the display, the one or more user sensors, and the one or more environmental sensors. The processing electronics can be configured to sense a situation involving user focus, determine user intent for the situation, and alter user perception of a real or virtual object within the vision field of the user based at least in part on the user intent and/or sensed situation involving user focus. The processing electronics can be configured to at least one of enhance or de-emphasize the user perception of the real or virtual object within the vision field of the user.

A method, a system, and a computer program for determining a visualization of content to be displayed to a surgeon on a display of a portable display device are provided. First pose data indicative a first pose of an anatomical feature of a patient in an operating room and second pose data indicative of a second pose of a portable display device in the operating room are obtained. Based on the first pose data and the second pose data, a third pose of a surgeon relative to the patient is estimated. Based on the estimated third pose, a visualization of content to be displayed to the surgeon on a display of the portable display device is determined.

A system for use in surgery includes a central body, a visualization system operably connected to the central body, a video rendering system, a head-mounted display for displaying images from the video rendering system, a sensor system, and a robotic device operably connected to the central body. The visualization system includes at least one camera and a pan system and/or a tilt system. The sensor system tracks the position and/or orientation in space of the head-mounted display relative to a reference point. The pan system and/or the tilt system are configured to adjust the field of view of the camera in response to information from the sensor system about changes in at least one of position and orientation in space of the head-mounted display relative to the reference point.

A system for use in surgery includes a central body, a visualization system operably connected to the central body, a video rendering system, a head-mounted display for displaying images from the video rendering system, a sensor system, and a robotic device operably connected to the central body. The visualization system includes at least one camera and a pan system and/or a tilt system. The sensor system tracks the position and/or orientation in space of the head-mounted display relative to a reference point. The pan system and/or the tilt system are configured to adjust the field of view of the camera in response to information from the sensor system about changes in at least one of position and orientation in space of the head-mounted display relative to the reference point.

A navigated surgery system includes at least one processor that is operative to obtain a 2D medical image slice of anatomical structure of a patient. The operations further obtain a 3D graphical model of anatomical structure. The operations determine a pose of a virtual cross-sectional plane extending through the 3D graphical model of the anatomical structure that corresponds to the anatomical structure of the 2D medical image slice. The operations control the XR headset to display the 2D medical image slice of the anatomical structure of the patient, display the 3D graphical model of the anatomical structure, and display a graphical object oriented with the pose relative to the 3D graphical model of the anatomical structure.