According to a general aspect, a method can include receiving a photo of a virtual conference participant, and a depth map based on the photo, and generating a plurality of synthesized images based on the photo. The plurality of synthesized images can have respective simulated gaze directions of the virtual conference participant. The method can also include receiving, during a virtual conference, an indication of a current gaze direction of the virtual conference participant. The method can further include animating, in a display of the virtual conference, an avatar corresponding with the virtual conference participant. The avatar can be based on the photo. Animating the avatar can be based on the photo, the depth map and at least one synthesized image of the plurality of synthesized images, the at least one synthesized image corresponding with the current gaze direction.

An exercise method and an exercise equipment are provided. The exercise method includes: determining an exercise guiding video according to a selected music input/audio signal, wherein the exercise guiding video includes a first exercise guiding video and/or a second exercise guiding video, the first exercise guiding video is a live video automatically generated according to the selected music input/audio signal, the second exercise guiding video is a video previously recorded according to the selected music input/audio signal; generating CGA and special-effect/animated feedbacks corresponding to the music information/audio signal and instruction/cuing in the exercise guiding video; playing the exercise guiding video, the CGA, the special-effect/animated feedbacks and the selected music input/audio signal on a display and computing device; receiving user performance data; displaying interactive feedback data on the display and computing device, according to a result obtained by matching the user performance data with music information/audio signal analyzed from selected music input/audio signal.

A method for providing an animated art experience to a user includes a user device receiving an image of an art piece selected by the user. The user device obtains information about the art piece. The user device presents a three-dimensional (3D) animated image that corresponds with the selected art image. Upon receiving an action by the user caused by a rotation or tilt of the user device, the user device provides a depth perspective view in correlation with the action and associated viewer angle of the art image such that further portions of the art image become visible. A background and a foreground of the image appear to move naturally as actions and associated viewer angles change.

The present invention is designed to allow easy synchronization of the movement of a computer graphics (CG) model with sound data. The data structure according to an embodiment of the present invention presents a data structure that relates to a computer graphics (CG) model, including first time-series information for designating the coordinates of the components of the CG model on a per beat basis, and the first time-series information is used on a computer to process the CG model.

A virtual environment platform may receive, from a user device, a request to access a virtual reality (VR) environment and may verify, based on the request, a user of the user device to allow the user device access to the VR environment. The virtual environment platform may receive, after verifying the user of the user device, user voice input and user handwritten input from the user device. The virtual environment platform may generate processed user speech by processing the user voice input, wherein a characteristic of the processed user speech and a corresponding characteristic of the user voice input are different and may generate formatted user text by processing the user handwritten input, wherein the formatted user text is machine-encoded text. The virtual environment platform may cause the processed user speech to be audibly presented and the formatted user text to be visually presented in the VR environment.

A method of providing an avatar service includes obtaining a user-uttered voice and a spatial information of a user-utterance space, transmitting the user-uttered voice and the spatial information to a server, receiving, from the server, a first avatar voice answer and an avatar facial expression sequence corresponding to the first avatar voice, which are determined based on the user-uttered voice and the spatial information, determining first avatar facial expression data, based on the first avatar voice answer and the avatar facial expression sequence, identifying a certain event during reproduction of a first avatar animation created based on the first avatar voice answer and the first avatar facial expression data, determining second avatar facial expression data or a second avatar voice answer, based on the certain event, and reproducing a second avatar animation created based on the second avatar facial expression data or the second avatar voice answer.

A computer-implemented method for generating a machine-learned model to generate facial position data based on audio data comprising training a conditional variational autoencoder having an encoder and decoder. The training comprises receiving a set of training data items, each training data item comprising a facial position descriptor and an audio descriptor; processing one or more of the training data items using the encoder to obtain distribution parameters; sampling a latent vector from a latent space distribution based on the distribution parameters; processing the latent vector and the audio descriptor using the decoder to obtain a facial position output; calculating a loss value based at least in part on a comparison of the facial position output and the facial position descriptor of at least one of the one or more training data items; and updating parameters of the conditional variational autoencoder based at least in part on the calculated loss value.

Provided is a method for controlling avatar motion, which is operated in a user terminal and includes receiving an input audio by an audio sensor, and controlling, by one and more processors, a motion of a first user avatar based on the input audio.

A computer-implemented method includes obtaining source video data comprising a plurality of image frames, and using a face tracker to detect one or more instances of faces within respective sequences of image frames of the source video data. For a first instance of a given face detected within a first sequence of image frames, the method includes determining a framewise location and size of the first instance of the given face in the first sequence of image frames, using a neural renderer to obtain replacement video data comprising a replacement instance of the given face, and using the determined framewise location and size to replace at least part of the first instance of the given face with at least part of the replacement instance of the given face.

A preprocessor for use with natural language processors for control of computerized avatars provides for an embedding of avatar control information in a speech response file of the natural language processor providing avatars with improved perception of emotional intelligence. Rapid avatar response is provided by independent end of speech detection and a response cache bypassing text-to-speech conversion times. The preprocessor may be shared among multiple websites to provide a shared analysis of query optimization.