Techniques for generating a visual response to a user input are described. A system may receive input data corresponding to a user input, determining a first skill component is to determine a response to the user input, and determine a second skill component is to determine supplemental content related to the user input. The system may also determine a template for presenting a visual response to the user input, where the template is configured for presenting the response and the supplemental content. The system may receive, from the first skill component, first image data corresponding to the first response. The system may also receive, from the second skill component, second image data corresponding to the first supplemental content. The system may send, to a device including a display, a command to present the first image data and the second image data using the template.

Techniques are disclosed that enable training a global model using gradients provided to a remote system by a set of client devices during a reporting window, where each client device randomly determines a reporting time in the reporting window to provide the gradient to the remote system. Various implementations include each client device determining a corresponding gradient by processing data using a local model stored locally at the client device, where the local model corresponds to the global model.

Provided are a video generation method and apparatus, a generation model training method and apparatus, a medium and a device. The method includes: acquiring target audio data to be synthesized; extracting an acoustic feature of the target audio data as a target acoustic feature; determining phonetic posteriorgrams (PPG) corresponding to the target audio data according to the target acoustic feature and generating an image sequence corresponding to the target audio data according to the PPG; and performing a video synthesis on the target audio data and the image sequence corresponding to the target audio data to obtain target video data.

A method includes receiving training data that includes unspoken text utterances, un-transcribed non-synthetic speech utterances, and transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. Each transcribed non-synthetic speech utterance is paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances, the un-transcribed non-synthetic speech utterances, and the transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.

A method may including obtaining a voice transcript corpus and a chat transcript corpus, extracting voice transcript sentences from the voice transcript corpus and chat transcript sentences from the chat transcript corpus, encoding, by a series of neural network layers, the voice transcript sentences to generate voice sentence vectors, encoding, by the series of neural network layers, the chat transcript sentences to generate chat sentence vectors, determining, for each voice sentence vector, a matching chat sentence vector to obtain matching voice-chat vector pairs, and adding, to a parallel corpus, matching voice-chat sentence pairs using the matching voice-chat vector pairs. Each of the matching voice-chat sentence pairs may include a voice transcript sentence and a matching chat transcript sentence. The method may further include training a disfluency remover model using the parallel corpus.

A method includes receiving training data that includes unspoken text utterances and un-transcribed non-synthetic speech utterances. Each unspoken text utterance is not paired with any corresponding spoken utterance of non-synthetic speech. Each un-transcribed non-synthetic speech utterance is not paired with a corresponding transcription. The method also includes generating a corresponding synthetic speech representation for each unspoken textual utterance of the received training data using a text-to-speech model. The method also includes pre-training an audio encoder on the synthetic speech representations generated for the unspoken textual utterances and the un-transcribed non-synthetic speech utterances to teach the audio encoder to jointly learn shared speech and text representations.

A system and method for generating a response to a user query. The method encompasses receiving, at a transceiver unit [102], the user query. The method thereafter leads to identifying, by an encoder unit [104], a user context associated with the user query based on one or more pre-stored datasets. Further the method encompasses predicting, by a prediction unit [106], one or more parameters corresponding the user query based on at least one of one or more offline-policies and one or more online-policies. The method thereafter comprises generating, by a decoder unit [108], the response to the user query based at least on the user context associated with the user query and the one or more parameters corresponding to the user query.

Methods and apparatus for improving speech recognition accuracy in media content searches are described. An advertisement for a media content item is analyzed to identify keywords that may describe the media content item. The identified keywords are associated with the media content item for use during a voice search to locate the media content item. A user may speak the one or more of the keywords as a search input and be provided with the media content item as a result of the search.

An audio-visual automated speech recognition model for transcribing speech from audio-visual data includes an encoder frontend and a decoder. The encoder includes an attention mechanism configured to receive an audio track of the audio-visual data and a video portion of the audio-visual data. The video portion of the audio-visual data includes a plurality of video face tracks each associated with a face of a respective person. For each video face track of the plurality of video face tracks, the attention mechanism is configured to determine a confidence score indicating a likelihood that the face of the respective person associated with the video face track includes a speaking face of the audio track. The decoder is configured to process the audio track and the video face track of the plurality of video face tracks associated with the highest confidence score to determine a speech recognition result of the audio track.

A method includes obtaining a plurality of training data sets each associated with a respective native language and includes a plurality of respective training data samples. For each respective training data sample of each training data set in the respective native language, the method includes transliterating the corresponding transcription in the respective native script into corresponding transliterated text representing the respective native language of the corresponding audio in a target script and associating the corresponding transliterated text in the target script with the corresponding audio in the respective native language to generate a respective normalized training data sample. The method also includes training, using the normalized training data samples, a multilingual end-to-end speech recognition model to predict speech recognition results in the target script for corresponding speech utterances spoken in any of the different native languages associated with the plurality of training data sets.