An apparatus for delivering virtual reality data portions to a client device, including a processing unit configured to perform the following in each one of a plurality of iterations: (1) receive from a network a current orientation data indicating a current orientation of a client device, (2) apply a rotation to a segment of a sphere defined in a virtual reality (VR) video file according to the current orientation, (3) crop from the rotated segment of the sphere in an equirectangular projection format an extended field of view (EFOV) frame in the equirectangular projection format according to the current orientation, and (4) instruct the network to transmit the EFOV frame to the client device.

An apparatus for delivering virtual reality data portions to a client device, including a processing unit configured to perform the following in each one of a plurality of iterations: (1) receive from a network a current orientation data indicating a current orientation of a client device, (2) apply a rotation to a segment of a sphere defined in a virtual reality (VR) video file according to the current orientation, (3) crop from the rotated segment of the sphere in an equirectangular projection format an extended field of view (EFOV) frame in the equirectangular projection format according to the current orientation, and (4) instruct the network to transmit the EFOV frame to the client device.

An information processing system includes a drawing portion that draws a display image for a terminal including one or more display media positioned in a virtual space; an acquisition portion that acquires input from a user; a state switching portion that switches, between a plurality of states, a state of the display medium; a position changing portion that changes the position of the display medium associated with one user in the virtual space, based on a first input from the one user acquired by the acquisition portion when the state of the display medium is a first state; and a viewpoint switching portion that switches a viewpoint of when the drawing portion draws the display image from a first viewpoint to a second viewpoint when the state of the display medium is switched from the first state to a second state by the state switching portion.

An information processing system includes a drawing portion that draws a display image for a terminal including one or more display media positioned in a virtual space; an acquisition portion that acquires input from a user; a state switching portion that switches, between a plurality of states, a state of the display medium; a position changing portion that changes the position of the display medium associated with one user in the virtual space, based on a first input from the one user acquired by the acquisition portion when the state of the display medium is a first state; and a viewpoint switching portion that switches a viewpoint of when the drawing portion draws the display image from a first viewpoint to a second viewpoint when the state of the display medium is switched from the first state to a second state by the state switching portion.

A system for modifying data representing a virtual environment includes: an environment navigation unit operable to control navigation within the virtual environment to generate one or more viewpoints within the virtual environment, an environment identification unit operable to identify one or more aspects of the geometry of the virtual environment visible in the one or more viewpoints, a geometry evaluation unit operable to evaluate the visibility of one or more aspects of the geometry based upon the identification for each of one or more viewpoints, and a data modification unit operable to modify one or more elements of data representing the virtual environment.

A system for modifying data representing a virtual environment includes: an environment navigation unit operable to control navigation within the virtual environment to generate one or more viewpoints within the virtual environment, an environment identification unit operable to identify one or more aspects of the geometry of the virtual environment visible in the one or more viewpoints, a geometry evaluation unit operable to evaluate the visibility of one or more aspects of the geometry based upon the identification for each of one or more viewpoints, and a data modification unit operable to modify one or more elements of data representing the virtual environment.

One disclosed example provides a method for displaying a hologram via a head-mounted display (HMD) device. The method comprises, via a camera system on the HMD device, acquiring image data capturing a surrounding environment by detecting illumination light output by a projector located on a case for the HMD device. A distance is determined from the HMD device to an object in the surrounding environment based upon the image data. The method further comprises displaying via the HMD device a hologram, the hologram comprising a left-eye image and a right-eye image each having a perspective based upon the distance determined.

A surgical virtual reality user interface generating system comprising a sensor and tracking unit for sensing and tracking a position a user and generating position data based on movement of the user, a computing unit for receiving the position data and processing the position data and generating control signals. The system also includes a surgical robot system for receiving the control signals and having a camera assembly for generating image data, and a virtual reality computing unit for generating a virtual reality world. The virtual reality computing unit includes a virtual reality rendering unit for generating an output rendering signal for rendering the image data for display, and a virtual reality object generating unit for generating virtual reality informational objects and for emplacing the informational objects in the virtual reality world. A display unit is provided for displaying the virtual reality world and the informational objects to the user.

Systems are configured for generating temporally corrected pass-through images. In some instances, the systems obtain depth maps of an environment at a first timepoint, generate a 3D representation of the environment by unprojecting the depth information represented in the depth map, and obtain one or more first images of the environment captured at a second timepoint. The systems may also be configured to perform a first intermediate projection to identify first texture information from the one or more first images, identify a display pose associated with the system, generate a display projection of the 3D representation, and creating a composite image based on the display projection and the first texture information.

A surgical system includes an XR headset and an XR headset controller. The XR headset is configured to be worn by a user during a surgical procedure and includes a see-through display screen configured to display an XR image for viewing by the user. The XR headset controller is configured to receive a plurality of two-dimensional (“2D”) image data associated with an anatomical structure of a patient. The XR headset controller is further configured to generate a first 2D image from the plurality of 2D image data based on a pose of the XR headset. The XR headset controller is further configured to generate a second 2D image from the plurality of 2D image data based on the pose of the XR headset. The XR headset controller is further configured to generate the XR image by displaying the first 2D image in a field of view of a first eye of the user and displaying the second 2D image in a field of view of a second eye of the user.