The technology disclosed herein is an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to function as: an obtaining unit configured to obtain information for specifying a position of an object included in multi-viewpoint image data obtained by image capturing using a plurality of imaging apparatuses; and a generation unit configured to generate a virtual viewpoint path data to generate virtual viewpoint image data by inputting the information obtained by the obtaining unit to an output unit which is a learned model learned from the virtual viewpoint path data to be training data and at least information for specifying a position of an object to be input data corresponding to the virtual viewpoint path data and is configured to output virtual viewpoint data by receiving input of information for specifying a position of an object.

The invention relates to creating and viewing stereo images, for example stereo video images, also called 3D video. At least three camera sources with overlapping fields of view are used to capture a scene so that an area of the scene is covered by at least three cameras. At the viewer, a camera pair is chosen from the multiple cameras to create a stereo camera pair that best matches the location of the eyes of the user if they were located at the place of the camera sources. That is, a camera pair is chosen so that the disparity created by the camera sources resembles the disparity that the user's eyes would have at that location. If the user tilts his head, or the view orientation is otherwise altered, a new pair can be formed, for example by switching the other camera. The viewer device then forms the images of the video frames for the left and right eyes by picking the best sources for each area of each image for realistic stereo disparity.

The invention relates to creating and viewing stereo images, for example stereo video images, also called 3D video. At least three camera sources with overlapping fields of view are used to capture a scene so that an area of the scene is covered by at least three cameras. At the viewer, a camera pair is chosen from the multiple cameras to create a stereo camera pair that best matches the location of the eyes of the user if they were located at the place of the camera sources. That is, a camera pair is chosen so that the disparity created by the camera sources resembles the disparity that the user's eyes would have at that location. If the user tilts his head, or the view orientation is otherwise altered, a new pair can be formed, for example by switching the other camera. The viewer device then forms the images of the video frames for the left and right eyes by picking the best sources for each area of each image for realistic stereo disparity.

Methods, apparatus and systems for geometric matching of virtual reality (VR) or augmented reality (AR) output contemporaneously with video output formatted for display on a 2D screen include a determination of value sets that when used in image processing cause an off-screen angular field of view of the at least one of the AR output object or the VR output object to have a fixed relationship to at least one of the angular field of view of the onscreen object or of the 2D screen. The AR/VR output object is outputted to an AR/VR display device and the user experience is improved by the geometric matching between objects observed on the AR/VR display device and corresponding objects appearing on the 2D screen.

A display assembly generates environmentally matched virtual content for an electronic display. The display assembly includes a display controller and a display. The display controller is configured to estimate environmental matching information for a target area within a local area based in part on light information received from a light sensor. The target area is a region for placement of a virtual object. The light information describes light values. The display controller generates display instructions for the target area based in part on a human vision model, the estimated environmental matching information, and rendering information associated with the virtual object. The display is configured to present the virtual object as part of artificial reality content in accordance with the display instructions. The color and brightness of the virtual object is environmentally matched to the portion of the local area surrounding the target area.

Examples of the disclosure describe systems and methods for recording augmented reality and mixed reality experiences. In an example method, an image of a real environment is received via a camera of a wearable head device. A pose of the wearable head device is estimated, and a first image of a virtual environment is generated based on the pose. A second image of the virtual environment is generated based on the pose, wherein the second image of the virtual environment comprises a larger field of view than a field of view of the first image of the virtual environment. A combined image is generated based on the second image of the virtual environment and the image of the real environment.

A surgical system includes an XR headset and an XR headset controller. The XR headset is configured to be worn by a user during a surgical procedure and includes a see-through display screen configured to display an XR image for viewing by the user. The XR headset controller is configured to receive a plurality of two-dimensional (“2D”) image data associated with an anatomical structure of a patient. The XR headset controller is further configured to generate a first 2D image from the plurality of 2D image data based on a pose of the XR headset. The XR headset controller is further configured to generate a second 2D image from the plurality of 2D image data based on the pose of the XR headset. The XR headset controller is further configured to generate the XR image by displaying the first 2D image in a field of view of a first eye of the user and displaying the second 2D image in a field of view of a second eye of the user.

A stereoscopic image acquisition method includesacquiring a first three-dimensional model corresponding to a two-dimensional image according to the two-dimensional image and a depth image; acquiring a target area and an area to be repaired corresponding to the first three-dimensional model; obtaining a second three-dimensional model by repairing a pixel in the area to be repaired according to a color value and a depth value of a pixel in the target area; and acquiring a stereoscopic image to be displayed according to a preset viewing angle change path and the second three-dimensional model.

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.

Configurations are disclosed for presenting virtual reality and augmented reality experiences to users. The system may comprise an image capturing device to capture one or more images, the one or more images corresponding to a field of the view of a user of a head-mounted augmented reality device, and a processor communicatively coupled to the image capturing device to extract a set of map points from the set of images, to identify a set of sparse points and a set of dense points from the extracted set of map points, and to perform a normalization on the set of map points.