A system and method for tracking the movement of a recording device to form a composite image is provided. The system has a user device with a sensor array capturing motion data and velocity vector data of the recording device when the recording device is in motion, an attachment member for coupling the user device to the motion capturing device, and a server with program modules. The program modules described are a calibration module for calibrating a position of the user device relative to a position of a lens of the recording device, a recorder module for receiving the motion data and velocity vector data from the sensor array; and a conversion module for combining the position of the user device relative to the lens of the recording device, and the motion data and velocity vector data and transforming the data into a file that is usable by a compositing suite, a three-dimensional application, or both.

A system and method for tracking the movement of a recording device to form a composite image is provided. The system has a user device with a sensor array capturing motion data and velocity vector data of the recording device when the recording device is in motion, an attachment member for coupling the user device to the motion capturing device, and a server with program modules. The program modules described are a calibration module for calibrating a position of the user device relative to a position of a lens of the recording device, a recorder module for receiving the motion data and velocity vector data from the sensor array; and a conversion module for combining the position of the user device relative to the lens of the recording device, and the motion data and velocity vector data and transforming the data into a file that is usable by a compositing suite, a three-dimensional application, or both.

Various embodiments describe systems and processes for capturing and generating multi-view interactive digital media representations (MIDMRs). In one aspect, a method for automatically generating a MIDMR comprises obtaining a first MIDMR and a second MIDMR. The first MIDMR includes a convex or concave motion capture using a recording device and is a general object MIDMR. The second MIDMR is a specific feature MIDMR. The first and second MIDMRs may be obtained using different capture motions. A third MIDMR is generated from the first and second MIDMRs, and is a combined embedded MIDMR. The combined embedded MIDMR may comprise the second MIDMR being embedded in the first MIDMR, forming an embedded second MIDMR. The third MIDMR may include a general view in which the first MIDMR is displayed for interactive viewing by a user on a user device. The embedded second MIDMR may not be viewable in the general view.

Various embodiments describe systems and processes for capturing and generating multi-view interactive digital media representations (MIDMRs). In one aspect, a method for automatically generating a MIDMR comprises obtaining a first MIDMR and a second MIDMR. The first MIDMR includes a convex or concave motion capture using a recording device and is a general object MIDMR. The second MIDMR is a specific feature MIDMR. The first and second MIDMRs may be obtained using different capture motions. A third MIDMR is generated from the first and second MIDMRs, and is a combined embedded MIDMR. The combined embedded MIDMR may comprise the second MIDMR being embedded in the first MIDMR, forming an embedded second MIDMR. The third MIDMR may include a general view in which the first MIDMR is displayed for interactive viewing by a user on a user device. The embedded second MIDMR may not be viewable in the general view.

An electronic device includes a memory, a display and a processor configured to set at least one virtual plane in a three-dimensional (3D) virtual position based on at least a portion of at least one two-dimensional (2D) first image displayed on the display, set a 3D virtual space comprising the at least one virtual plane and the at least one 2D first image, insert a 3D object into the 3D virtual space, and generate at least one 2D second image by synthesizing the 3D object inserted into the 3D virtual space and the at least one 2D first image.

A captured scene captured of a live action scene while a display wall is positioned to be part of the live action scene may be processed. To perform the processing, stereoscopic image data of the live action scene is received, and display wall metadata of the precursor image is determined. Further, a first portion of the stereoscopic image data comprising the stage element in the live action scene is determined based on the stereoscopic image data and the display wall metadata. A second portion of the stereoscopic image data comprising the display wall in the live action scene with the display wall displaying the precursor image is also determined. Thereafter, an image matte for the stereoscopic image data is generated based on the first portion and the second portion.

A recording device includes: a memory; and a processor including hardware. The processor is configured to generate, based on temporal change in plural sets of image data that have been generated by an endoscope and arranged chronologically, biological information on a subject, associate the plural sets of image data with the biological information to record the plural sets of image data with the biological information into the memory, and select, based on the biological information, image data from the plural sets of image data that have been recorded in the memory to generate three-dimensional image data.

A recording device includes: a memory; and a processor including hardware. The processor is configured to generate, based on temporal change in plural sets of image data that have been generated by an endoscope and arranged chronologically, biological information on a subject, associate the plural sets of image data with the biological information to record the plural sets of image data with the biological information into the memory, and select, based on the biological information, image data from the plural sets of image data that have been recorded in the memory to generate three-dimensional image data.

Client configured for retrieving a video data representation of an immersive video scene streamed by a server using a streaming protocol, wherein the server is configured for providing a plurality of streams to the client, wherein each of the streams comprises a portion of the immersive video scene, the client comprising: a sending interface; a reception interface; a viewing direction receiving unit; and a stream selecting unit; wherein the sending interface is configured for transmitting a streaming request for streaming the one or more selected streams as the video data representation of the immersive video scene, wherein the sending interface receives from the stream selecting unit a selected stream information identifying the one or more selected streams, wherein the selected stream information is created by the stream selecting unit based on the viewing direction of the user of the client and based on the manifest.

A background image is also generated, e.g., by filling portions of a captured image where a foreground object was extracted and communicated to the playback device, Foreground objects are identified and point cloud representations of the foreground objects are generated and communicated to a playback device so that they can be used in generating images including the background which is communicated separately. In the case of a point cloud representation a number of points in an environment, e.g., 3D space, are communicated to the playback device along with color information. Thus in some embodiments a foreground object is represented as a set of points with corresponding color information on a per point basis. Foreground object information is communicated and processed in some embodiments at a different rate, e.g., faster rate, then the background textures. The playback device renders images which are sent to the display by first rendering a background layer using the communicated background information, e.g., background texture(s), UV map and environmental geometry, e.g., mesh, to which the background textures are applied.