An embodiment includes tokenizing an input passage into an n-gram sequence of tokens. The embodiment also includes evaluating the input passage using a trained classification model that generates an output indicative of a classification of the input passage. The embodiment also includes generating a first token vector for a first token of the sequence of tokens and projecting the first token vector to a higher dimensional space, resulting in a first projected token vector. The embodiment also includes generating a first similarity score for the first projected token vector based on comparisons of the first projected token vector to each of a plurality of class representations. The embodiment also includes generating a ranked list of the tokens, wherein the generating of the ranked list includes ranking the first token among others of the tokens based on the first similarity score.

Embodiments provide a software framework for evaluating and troubleshooting real-world task-oriented bot systems. Specifically, the evaluation framework includes a generator that infers dialog acts and entities from bot definitions and generates test cases for the system via model-based paraphrasing. The framework may also include a simulator for task-oriented dialog user simulation that supports both regression testing and end-to-end evaluation. The framework may also include a remediator to analyze and visualize the simulation results, remedy some of the identified issues, and provide actionable suggestions for improving the task-oriented dialog system.

Embodiments provide a software framework for evaluating and troubleshooting real-world task-oriented bot systems. Specifically, the evaluation framework includes a generator that infers dialog acts and entities from bot definitions and generates test cases for the system via model-based paraphrasing. The framework may also include a simulator for task-oriented dialog user simulation that supports both regression testing and end-to-end evaluation. The framework may also include a remediator to analyze and visualize the simulation results, remedy some of the identified issues, and provide actionable suggestions for improving the task-oriented dialog system.

A system and method implemented on a computing device for incrementally discovering new intents and goals by collecting data from a first corpus of a plurality of newer digitally-recorded conversations, performing dimensionality reduction to prepare the extracted conversations for clustering, clustering the prepared conversations, identifying new intent and/or goal labels using a trained Artificial Intelligence (AI) engine, applying a model-based filter to remove new labels which overlap already-known labels in the first corpus, and outputting the newly-discovered labels in association with the extracted conversations into a computer-readable file.

A system and method implemented on a computing device for incrementally discovering new intents and goals by collecting data from a first corpus of a plurality of newer digitally-recorded conversations, performing dimensionality reduction to prepare the extracted conversations for clustering, clustering the prepared conversations, identifying new intent and/or goal labels using a trained Artificial Intelligence (AI) engine, applying a model-based filter to remove new labels which overlap already-known labels in the first corpus, and outputting the newly-discovered labels in association with the extracted conversations into a computer-readable file.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating and operating voice conversing virtual agents with pre-modeled and inherited human behavior across use cases and domains. One of the methods includes: using a first non-domain specific neural network based model to predict a non-domain specific conversational situation, the first neural network based model trained with labelled parts of conversations from more than one domain; forwarding the non-domain specific conversational situation to a second domain specific neural network based model; using the second domain specific neural network based model to predict a conversational situation and to provide a system intent, the second domain specific neural network based model trained with labelled parts of conversation from a specified domain; and generating a response based at least in part on the predicted conversational situation and system intent.

A chatbot capable of empathic engagement with a user is disclosed. An identified trend in a user's mood or goals between a first time and a second time can be associated with open input (e.g., open text string input) from the user. As the user's mood or goals continue to be tracked, a subsequent trend can be identified that is the same as, similar to, different from, or opposite to the first identified trend. The user can then be automatically engaged based on the open input associated with the first identified trend. In an example, a user may input thoughts or reasons why they have been having a positively trending mood over a duration of time. The chatbot can then repeat or otherwise use those same thoughts or reasons to engage the user empathically when the chatbot detects that the user is experiencing a negatively trending mood.

A chatbot capable of empathic engagement with a user is disclosed. An identified trend in a user's mood or goals between a first time and a second time can be associated with open input (e.g., open text string input) from the user. As the user's mood or goals continue to be tracked, a subsequent trend can be identified that is the same as, similar to, different from, or opposite to the first identified trend. The user can then be automatically engaged based on the open input associated with the first identified trend. In an example, a user may input thoughts or reasons why they have been having a positively trending mood over a duration of time. The chatbot can then repeat or otherwise use those same thoughts or reasons to engage the user empathically when the chatbot detects that the user is experiencing a negatively trending mood.

In a method for handling out-of-sequence caller dialog, an intelligent voice interface is configured to lead callers through pathways of an algorithmic dialog that includes available voice prompts for requesting different types of caller information. The method may include, during a voice communication with a caller via a caller device, receiving from the caller device caller input data indicative of a voice input of the caller, without having first provided to the caller device any voice prompt that requests a first type of caller information, and determining, by processing the caller input data, that the voice input includes caller information of the first type. The method also includes after determining that the voice input includes the caller information of the first type, bypassing one or more voice prompts, of the available voice prompts, that request the first type of caller information.

Systems and methods for providing dialog data, from an initially invoked automated assistant to a subsequently invoked automated assistant. A first automated assistant may be invoked by a user utterance, followed by a dialog with the user that is processed by the first automated assistant. During the dialog, a request to transfer dialog data to a second automated assistant is received. The request may originate with the user, by the first automated assistant, and/or by the second automated assistant. Once authorized, the first automated assistant provides the previous dialog data to the second automated assistant. The second automated assistant performs one or more actions based on the dialog data.