Systems and methods for foreground/background separation and for studio production of a FVV. A method includes projecting, onto objects in a filming area within a studio, a predefined pattern including a large set of features; generating, based on signals reflected off of the objects and captured by a plurality of depth cameras deployed in proximity to the filming area, a local point cloud for each depth camera; separating, based on the local point clouds, between a background and a foreground of the filming area; creating, based on the local point clouds, a unified point cloud; meshing points in the unified point cloud to generate a 3D model of the objects; texturing the 3D model based on the separation and images captured by the depth cameras; and rendering the textured 3D model as a FVV including a series of video frames with respect to at least one viewpoint.

A device for image capture includes a memory configured to store a preview image, and one or more processors coupled to the memory and configured to receive a user selection to focus at a region of the preview image stored in the memory, determine whether there is image content at different depths in the region, based on the determination that there is image content at different depths in the region, output an alternative version of the image content in the region, receive a subsequent user selection within the alternative version, and control a focus level for a camera based on the subsequent user selection.

Methods and systems for generating free viewpoint videos (FVVs) based on images captured in a sports arena. A method includes projecting, onto objects within a filming area within the sports arena, a predefined pattern including a large set of features; generating, based on signals captured by each of a plurality of depth cameras, a point cloud for each depth camera, wherein the plurality of depth cameras is deployed in proximity to the filming area, wherein the captured signals are reflected off of the objects within the filming area; creating, based on the plurality of point clouds, a unified point cloud; meshing points in the unified point cloud to generate a three-dimensional (3D) model of the objects; texturing the 3D model based on images captured by the plurality of depth cameras; and rendering the textured 3D model as a FVV including a series of video frames with respect to a viewpoint.

The image processing device includes an imaging unit that images a subject and a distance map acquisition unit that acquires information regarding a distance distribution of the subject as map data. The distance map acquisition unit acquires map data with an image deviation amount or a defocused amount related to a captured image or distance map data in conformity with a TOF scheme or an imaging plane phase difference detection scheme of using a pupil division type image sensor. An image processing unit generates data of a texture image in which a low-frequency component of a captured image is inhibited and combines the data of the texture image and the map data acquired by the distance map acquisition unit to generate image data in which a distance distribution of a subject is expressed.

A system is disclosed that includes an optical tracking device and a surgical computing device. The optical tracking device includes a structured light module and an optical module that includes an image sensor and is spaced from the structured light module at a known distance. The surgical computing device includes a display device, a non-transitory computer readable medium including instructions, and processor(s) configured to execute the instructions to generate a depth map from a first image captured by the image sensor during projection of a pattern into a surgical environment by the structured light module. The pattern is projected in a near-infrared (NIR) spectrum. The processor(s) are further configured to execute the stored instructions to reconstruct a 3D surface of anatomical structure(s) based on the generated depth map. Additionally, the processor(s) are configured to execute the stored instructions to output the reconstructed 3D surface to the display device.

An image generating device, a method and a computer-readable recording medium are provided. The image generating device includes a passive sensor, an active sensor, a mask forming unit, a memory storing at least one instruction, and a processor configured to execute the at least one instruction to obtain a first depth map with respect to a first space by using the passive sensor, identify an unidentified region of the first depth map, obtain a second depth map with respect to the unidentified region by using the active sensor and the mask forming unit configured to form a mask based on the unidentified region, and obtain a third depth map with respect to the first space based on the first depth map and the second depth map.

A method for processing video data by a system, including a recording device configured to generate video data; a projection screen having a front side facing the recording device and a rear side facing away from the recording device, and a plurality of light sources is provided. The method includes: providing background data which indicate a video background with a first resolution, and determining image signals from the background data in such a way that the image signals indicate the video background with a second resolution that is lower than the first resolution; transmitting the image signals to the light sources and irradiating the rear side by the light sources; and recording first video data by the recording device, wherein the first video data indicate a video of the front side and of a foreground object arranged between the projection screen and the recording device. A system is also provided.

An apparatus includes an acquisition unit configured to acquire distance information indicating a distance to an object included in a divided area obtained by dividing an image capturing area, a calculation unit configured to calculate a distance distribution on the object included in the divided area based on a position of the divided area and the distance information, and a determination unit configured to determine a settable range of exposure conditions for the divided area based on the calculated distance distribution.

An image sensor according to an embodiment of the present invention includes: a pixel array in which a plurality of pixels are arrayed in a grid shape, and which converts reflection light signals reflected from an object into electrical signals; an image processor which converts the electrical signals to generate subframes, and extracts pieces of second depth information having a higher resolution than pieces of first depth information extracted from a plurality of the subframes; and a memory for storing the pieces of first depth information, wherein the reflection light signals are input to the pixel array through mutually different optical paths shifted in sub-pixel units of the pixel array, and the memory stores a plurality of the pieces of first depth information that correspond to the mutually different optical paths.

There is provided a depth information generating apparatus. A first generating unit generates first depth information on the basis of a plurality of viewpoint images which are obtained from first shooting and which have mutually-different viewpoints. A second generating unit generates second depth information for a captured image obtained from second shooting by correcting the first depth information so as to reflect a change in depth caused by a difference in a focal distance of the second shooting relative to the first shooting.