A handheld user interface device for controlling a robotic system may include a member, a housing at least partially disposed around the member and configured to be held in the hand of a user, and a tracking sensor system disposed on the member and configured to detect at least one of position and orientation of at least a portion of the device. At least one of the detected position of the portion of the device and detected orientation of the portion of the device is correlatable to a control of the robotic system.

A method facilitates touchless control of medical equipment devices in an OR. The method involves: providing a three-dimensional control menu, which comprises a plurality of menu items selectable by the practitioner by one or more gestures made in a volumetric spatial region corresponding to the menu item; displaying an interaction display unit (IDU) image corresponding to the three-dimensional control menu to provide indicia of any selected menu items; estimating a line of sight of a practitioner; and when the estimated line of sight is directed within a first spatial range around a first medical equipment device, determining that the practitioner is looking at the first medical equipment device. Then the method involves providing a first device-specific three-dimensional control menu displaying a first device-specific IDU image.

A system includes a mobile device having one or more cameras to take images; a sensor detecting reflected light from one or more lasers and a diffuser to detect object range or dimension; code for motion tracking, environmental understanding by detecting planes in an environment, and estimating light and dimensions of the surrounding based on the one or more lasers; code to estimate a three-dimensional (3D) volume of an object from multiple perspectives and from projected laser beams to measure size or scale and determine locations of points on the object's surface in a plane or a slice using time-of-flight, wherein positions and cross-sections for different slices are correlated to construct a 3D model of the object, including object position and shape; the device receiving user request to select a content from one or more augmented, virtual, or extended reality contents and rendering a reality view of the environment.

An apparatus comprises an electrically power surgical instrument having a handle assembly. The apparatus also comprises a communication device positioned within the handle assembly. The communication device is operable to communicate with at least a portion of the electrically powered surgical instrument. The apparatus further comprises an external device in wireless communication with the communication device. The external device is operable to receive information from the communication device and the external device is operable to provide an output viewable to the user.

A method of determining a focal length of a camera and/or of adjusting a viewing direction in a graphical representation of a pre-operative image is provided. The method includes providing uncalibrated camera data of a camera, specifying an initial value of a focal length of the camera, specifying a working distance value of a distance between the camera and the at least part of the tracking system, calculating a distance value of the distance between the camera and the at least part of the tracking system based on the uncalibrated camera data and based on the specified initial value of the focal length of the camera, calculating a change factor based on the specified working distance value and the calculated distance value, and calculating an adapted value of the focal length of the camera based on the initial value of the focal length and based on the change factor.

A virtual model a planned instrument attachment can be provided to ensure correct selection of a physical instrument attachment. An XR headset controller can generate a shape and a pose of the virtual model of the planned instrument attachment based on predetermined information associated with the planned instrument attachment and based on a pose of an instrument relative to the XR headset. An XR headset can display the virtual model on a see-through display screen of the XR headset that is configured to allow at least a portion of a real-world scene to pass therethrough.

Systems and methods for vision-based position and orientation determination for endovascular tools are disclosed. In one example, a method includes receiving a two-dimensional medical image including a view of at least a distal portion of a medical instrument, the distal portion of the medical instrument including one or more fiducials positioned thereon, the one or more fiducials being radio-opaque and visible in the medical image. The method also includes detecting, within the medical image, a two-dimensional appearance of the one or more fiducials, and based on the two-dimensional appearance of the one or more fiducials, determining at least one of a roll angle of the distal portion of the medical instrument, and an incline of the distal portion of the medical instrument.

Methods and systems that provide quantitative assessments of in vivo thermal treatments, such as ablations, during image-guided surgeries using a high-resolution pre-operative MRI image segmented with a shape constrained and deformable mesh representations of brain structures and generating 3-D visualizations of thermally treated volumes during the thermal treatment that can provide near real time visual and quantitative feedback to a clinician.

Systems and methods for vision-based position and orientation determination for endovascular tools are disclosed. In one example, a method includes receiving a two-dimensional medical image including a view of at least a distal portion of a medical instrument, the distal portion of the medical instrument including one or more fiducials positioned thereon, the one or more fiducials being radio-opaque and visible in the medical image. The method also includes detecting, within the medical image, a two-dimensional appearance of the one or more fiducials, and based on the two-dimensional appearance of the one or more fiducials, determining at least one of a roll angle of the distal portion of the medical instrument, and an incline of the distal portion of the medical instrument.

For a scalable filtering infrastructure, a library of filters each usable at different control rates is provided by defining filters in a continuous time mode despite eventual use for digital filtering. For implementation, a filter is selected and discretized for the desired control rate. The discretized filter is then deployed as a discrete time realization for convolution. In a distributed system with multiple control rates, the library may be used to more rapidly and conveniently generate the desired filters.