At least two images are acquired. At least two crop images are acquired by cropping the at least two images for face-containing images. Triangular patch deformation is performed on two neighbour images, generating a first triangular patch deformation frame image sequence and a second triangular patch deformation frame image sequence. Similarity transformation is performed on each image sequence of the first triangular patch deformation frame image sequence, acquiring a first transform frame image sequence. Similarity transformation is performed on each image sequence of the second triangular patch deformation frame image sequence, acquiring a second transform frame image sequence. The first and the second transform frame image sequences are fused, acquiring a video frame sequence corresponding to the two neighbour images. Of the at least two images, a video frame sequence generated by all neighbour images is coded, acquiring a destined video.

Disclosed are systems and methods for portrait animation. An example method includes receiving, by a computing device, a scenario video, where the scenario video includes at least one input frame and the at least one input frame includes a first face, receiving, by the computing device, a target image, where the target image includes a second face, determining, by the computing device and based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face in the target image, where the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression of the first face, and applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.

A computer implemented method for warping virtual content includes generating warped virtual content by transforming source virtual content. The method also includes determining whether a memory location corresponding to an X, Y location of the warped virtual content in an output frame of reference is occupied by pre-existing virtual content. The method further includes storing the warped virtual content in the memory location if the memory location is not occupied. Moreover, the method includes comparing respective Z locations of the warped virtual content and the pre-existing virtual content to identify virtual content with a Z location closer to a viewing location if the memory location is occupied. The method also includes storing the warped virtual content in the memory location corresponding to the X, Y location if a Z location of warped virtual content is closer to the viewing location than a pre-existing Z location of pre-existing virtual content.

Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face of a first person. The method further includes receiving a target image with a second face of a second person. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face and a background in the target image. The 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying the 2D deformations to the target image to obtain at least one output frame of an output video.

In one aspect, a computerized method for implementing a multi-pass iterative closest point (ICP) registration in an automated facial reconstruction process includes the step of providing an automated facial reconstruction process. The method includes the step of detecting that a face tracking application programming interface (API) misalignment has occurred during the automated facial reconstruction process. The method includes the step of implementing a multi-pass ICP process on the face tracking API misalignment by the following steps. One step includes obtaining a first point cloud (PC1) of the automated facial reconstruction process. The method includes the step of obtaining a second point cloud (PC2) of the automated facial reconstruction process. The method includes the step of obtaining a first face mask (mask1) of the automated facial reconstruction process. The method includes the step of obtaining a second face mask (mask2) of the automated facial reconstruction process. The method includes the step of performing a first point cloud (PC1)-to-mask ICP operation by inputting and utilizing PC1 and mask1. A transformation output of the first PC-to-mask ICP operation is provided to a third PC-to-PC ICP and a fourth PC-to-PC ICP. The method includes the step of performing a second PC-to-mask ICP on PC2 and mask2. A transformation output of the second PC-to-mask ICP step is provided to a second PC-to-PC ICP and to the third PC-to-PC ICP. The method includes the step of performing a first PC-to-PC ICP on the PC1 and the PC2. The first PC-to-PC ICP outputs a first transformation of the first PC-to-PC ICP and a first error value of the first PC-to-PC ICP. The method includes the step of performing a second PC-to-PC ICP on the transformation output by the second PC-to-mask ICP and the PC1. The second PC-to-PC ICP outputs a second transformation of the second PC-to-PC ICP and a second error value of the second PC-to-PC ICP. The method includes the step of performing a third PC-to-PC ICP on the transformation output of the first PC-to-mask ICP and the transformation output of the second PC-to-mask ICP. The third PC-to-PC ICP outputs a third transformation of the third PC-to-PC ICP and a third error value of the third PC-to-PC ICP. The method includes the step of performing a fourth PC-to-PC ICP on the transformation output of the first PC-to-mask ICP and the PC2. The fourth PC-to-PC ICP outputs a fourth transformation of the fourth PC-to-PC ICP and a fourth error value of the fourth PC-to-PC ICP.

A first three-dimensional (3D) model of a target in a 3D space is acquired based on a first two-dimensional (2D) image including the target. A 3D morphing parameter is acquired. The first 3D model is transformed into a second 3D model based on the 3D morphing parameter. First 2D coordinates are acquired by mapping the first 3D model to a 2D space. Second 2D coordinates are acquired by mapping the second 3D model to the 2D space. A second 2D image including a morphed target is acquired by morphing the target in the first 2D image based on the first 2D coordinates and the second 2D coordinates.

A video processing method includes detecting, as a reference pose, a pose of an individual at a reference time point in an input video sequence; at a second, different, time point in the input video sequence, detecting a second pose of the individual; generating from one or more source images of the individual, a transitional video sequence representing a transition of the individual from the second pose to the reference pose; and associating the transitional video sequence with the input video sequence to generate an output video sequence including at least the transitional video sequence to implement a non-linear replay branch from the second time point to the reference time point.

Techniques related to capturing 3D faces using image and temporal tracking neural networks and modifying output video using the captured 3D faces are discussed. Such techniques include applying a first neural network to an input vector corresponding to a first video image having a representation of a human face to generate a morphable model parameter vector, applying a second neural network to an input vector corresponding to a first and second temporally subsequent to generate a morphable model parameter delta vector, generating a 3D face model of the human face using the morphable model parameter vector and the morphable model parameter delta vector, and generating output video using the 3D face model.

Techniques are disclosed for training and applying nonlinear face models. In embodiments, a nonlinear face model includes an identity encoder, an expression encoder, and a decoder. The identity encoder takes as input a representation of a facial identity, such as a neutral face mesh minus a reference mesh, and outputs a code associated with the facial identity. The expression encoder takes as input a representation of a target expression, such as a set of blendweight values, and outputs a code associated with the target expression. The codes associated with the facial identity and the facial expression can be concatenated and input into the decoder, which outputs a representation of a face having the facial identity and expression. The representation of the face can include vertex displacements for deforming the reference mesh.

Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face of a first person. The method further includes receiving a target image with a second face of a second person. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations of the second face and a background in the target image. The 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying the 2D deformations to the target image to obtain at least one output frame of an output video.