The disclosure relates to a method and system for monitoring the performance of a voice recognition (VR) assistance system in a data sensitive environment, wherein the VR assistance system comprises one or more client devices and a server, the server comprising a monitoring component. The method comprises determining, by at least one client device, client input data; processing, by the VR assistance system, the client input data; determining, by the monitoring component, one or more anonymized performance indicators of the VR assistance system; determining, by the monitoring component, one or more anonymized performance indicator values for the one or more anonymous performance indicators during the processing of the client input data; outputting and/or saving, by the monitoring component, the determined one or more anonymized performance indicator values.

A method of evaluating a language model using negative data may include accessing a first language model that is trained using a first training corpus, and accessing a second language model. The second language model may be configured to generate outputs that are less grammatical than outputs generated by the first language model. The method may also include training the second language model using a second training corpus, and generating output text from the second language model. The method may further include testing the first language model using the output text from the second language model.

A method of evaluating a language model using negative data may include accessing a first language model that is trained using a first training corpus, and accessing a second language model. The second language model may be configured to generate outputs that are less grammatical than outputs generated by the first language model. The method may also include training the second language model using a second training corpus, and generating output text from the second language model. The method may further include testing the first language model using the output text from the second language model.

Methods and systems are provided for assisting operation of a vehicle using speech recognition. One method involves analyzing a transcription of an audio communication with respect to the vehicle to characterize a nonstandard pattern within the transcription of the audio communication, obtaining a ground truth for the transcription of the audio communication, determining one or more performance metrics associated with the nonstandard pattern within the transcription based on a relationship between the transcription of the audio communication and the ground truth for the transcription, updating a speech recognition vocabulary for the vehicle to include the nonstandard pattern based at least in part on the one or more performance metrics and determining an updated speech recognition model for the vehicle using the updated speech recognition vocabulary and the audio communication.

Methods and systems are provided for assisting operation of a vehicle using speech recognition. One method involves analyzing a transcription of an audio communication with respect to the vehicle to characterize a nonstandard pattern within the transcription of the audio communication, obtaining a ground truth for the transcription of the audio communication, determining one or more performance metrics associated with the nonstandard pattern within the transcription based on a relationship between the transcription of the audio communication and the ground truth for the transcription, updating a speech recognition vocabulary for the vehicle to include the nonstandard pattern based at least in part on the one or more performance metrics and determining an updated speech recognition model for the vehicle using the updated speech recognition vocabulary and the audio communication.

Disclosed are a method and apparatus for testing a full-duplex speech interaction system. The method includes: determining a scene mixed corpus set by mixing a valid corpus set related to a test scene with an invalid corpus set unrelated to the test scene; playing each corpus audio in the scene mixed corpus set to a speech interaction device under test equipped with the full-duplex speech interaction system; acquiring a work log of the speech interaction device under test, the work log including at least a first log and a second log; and obtaining number of false responses by counting number of log entries which have false response result in the second log, and determining a false response rate based on the number of false responses and a total number of corpus audios played. End-to-end testing of the full-duplex speech interaction system is realized.

Disclosed are a method and apparatus for testing a full-duplex speech interaction system. The method includes: determining a scene mixed corpus set by mixing a valid corpus set related to a test scene with an invalid corpus set unrelated to the test scene; playing each corpus audio in the scene mixed corpus set to a speech interaction device under test equipped with the full-duplex speech interaction system; acquiring a work log of the speech interaction device under test, the work log including at least a first log and a second log; and obtaining number of false responses by counting number of log entries which have false response result in the second log, and determining a false response rate based on the number of false responses and a total number of corpus audios played. End-to-end testing of the full-duplex speech interaction system is realized.

A system may include processor(s), and memory in communication with the processor(s) and storing instructions configured to cause the system to correct ASR errors. The system may receive a transcription comprising transcribed word(s) and may determine whether the transcribed word(s) exceed associated predefined confidence level(s). Responsive to determining a transcribed word does not exceed a predefined confidence level, the system may generate a predicted word. The system may calculate a distance between numerical representations of the transcribed word and the predicted word and may determine whether the distance exceeds a predefined threshold. Responsive to determining the distance exceeds the predefined threshold, the system may determine whether at least one red flag word of a list of red flag words corresponds to a context of the transcription, and, responsive to making that determination, may classify the transcription as associated with a first category.

A system may include processor(s), and memory in communication with the processor(s) and storing instructions configured to cause the system to correct ASR errors. The system may receive a transcription comprising transcribed word(s) and may determine whether the transcribed word(s) exceed associated predefined confidence level(s). Responsive to determining a transcribed word does not exceed a predefined confidence level, the system may generate a predicted word. The system may calculate a distance between numerical representations of the transcribed word and the predicted word and may determine whether the distance exceeds a predefined threshold. Responsive to determining the distance exceeds the predefined threshold, the system may determine whether at least one red flag word of a list of red flag words corresponds to a context of the transcription, and, responsive to making that determination, may classify the transcription as associated with a first category.

The present application discloses a wakeup indicator monitoring method, apparatus and electronic device. The present disclosure includes: acquiring M pieces of audio data of a device to be monitored; determining a first wakeup confidence of each piece of M pieces of audio data, wherein the first wakeup confidence indicates a probability that the audio data contains a first wakeup word for waking up the device to be monitored; acquiring a first audio data with a first wakeup confidence in a target zone in M pieces of audio data, wherein the wakeup confidence in the target zone indicates that the audio data contains a wakeup word for waking up an audio device; and determining the ratio of the first audio data to M pieces of audio data as a wakeup rate of a device to be monitored, where the wakeup indicator of the device to be monitored includes the wakeup rate.