METHOD AND SYSTEM FOR MONITORING THE PERFORMANCE OF A VOICE RECOGNITION ASSISTANCE SYSTEM IN A DATA SENSITIVE ENVIRONMENT

Abstract

The disclosure relates to a method and system for monitoring the performance of a voice recognition (VR) assistance system in a data sensitive environment, wherein the VR assistance system comprises one or more client devices and a server, the server comprising a monitoring component. The method comprises determining, by at least one client device, client input data; processing, by the VR assistance system, the client input data; determining, by the monitoring component, one or more anonymized performance indicators of the VR assistance system; determining, by the monitoring component, one or more anonymized performance indicator values for the one or more anonymous performance indicators during the processing of the client input data; outputting and/or saving, by the monitoring component, the determined one or more anonymized performance indicator values.

Claims

1. A method for monitoring performance of a voice recognition (VR) assistance system in a data sensitive environment, wherein the VR assistance system comprises one or more client devices and a server, the server comprising a monitoring component, the method comprising: determining, by at least one client device of the one or more client devices, client input data; processing, by the VR assistance system, the client input data; determining, by the monitoring component, one or more anonymized performance indicators of the VR assistance system; determining, by the monitoring component, one or more anonymized performance indicator values for the one or more anonymous performance indicators during the processing of the client input data; and at least one of outputting or saving, by the monitoring component, the determined one or more anonymized performance indicator values.

2. The method of claim 1, wherein the one or more anonymous performance indicators are consistent with predetermined general data protection regulations.

3. The method of claim 1, wherein determining, by the monitoring component, one or more anonymized performance indicator values comprises increasing one or more performance indicator counters, in particular, a plurality of respective counters for a plurality of time intervals.

4. The method of claim 3, wherein the one or more performance indicator counters are indicative of a processing efficiency of the VR assistance system, processing time of the VR assistance system, reply performance of the VR assistance system, processing errors of the VR assistance system, a capacity of the one or more client devices, a power usage of the one or more client devices, a capacity of the server, or a power usage of the server.

5. The method of claim 3, wherein the one or more performance indicator counters are indicative of a usage behavior of the one or more client devices, a request intensity of the one or more client devices, one or more client input data types, software performance of the one or more client devices, or hardware performance or the one or more client devices.

6. The method of claim 4, wherein the one or more performance indicator counters are indicative of an occurrence of a language of the client input data.

7. The method of claim 1, further comprising: comparing, by the monitoring component, the one or more anonymized performance indicator values to one or more previously determined anonymized performance indicator values or one or more previously determined performance indicator threshold values.

8. The method of claim 1, further comprising: generating, by the VR assistance system, client output data based on the processed client input data; outputting, by the at least one client device, the client output data; and deleting, by the VR assistance system, the client input data and the client output data.

9. The method of claim 1, wherein the monitoring component is comprised in a separate docker container on the server.

10. The method of claim 1, wherein the VR assistance system is a multi-language VR assistance system.

11. A monitoring component for use in a voice recognition (VR) assistance system, wherein the monitoring component is configured to: determine one or more anonymized performance indicators of the VR assistance system; determine one or more anonymized performance indicator values for the one or more anonymous performance indicators during processing of client input data by the VR assistance system; and at least one of output or save the determined one or more anonymized performance indicator values.

12. The monitoring component of claim 11, wherein the one or more anonymous performance indicators are consistent with predetermined general data protection regulations.

13. The monitoring component of claim 11, wherein to determine one or more anonymized performance indicator values comprises to increase one or more performance indicator counters.

14. The monitoring component of claim 13, wherein to increase the one or more performance indicator counters comprises to increase a plurality of respective counters for a plurality of time intervals.

15. The monitoring component of claim 13, wherein the one or more performance indicator counters are indicative of a processing efficiency of the VR assistance system, a processing time of the VR assistance system, a reply performance of the VR assistance system, processing errors of the VR assistance system, a capacity of a client device, a power usage of a client device, a capacity of a server, or a power usage of the server.

16. The monitoring component of claim 13, wherein the one or more performance indicator counters are indicative of a usage behavior of a client device, a request intensity of the client device, one or more client input data types, software performance of the client device, or hardware performance or the client device.

17. The monitoring component of claim 11, wherein the monitoring component is further configured to compare the one or more performance indicator values to one or more previously determined performance indicator values or one or more previously determined performance indicator threshold values.

18. The monitoring component of claim 11, wherein the monitoring component is comprised in a separate docker container on a server.

19. A voice recognition (VR) assistance system, the VR assistance system comprising: one or more client devices; and a server, the server comprising a monitoring component; wherein the monitoring component is configured to perform a method comprising: processing client input data of at least one of the one or more client devices; determining one or more anonymized performance indicators of the VR assistance system; determining one or more anonymized performance indicator values for the one or more anonymous performance indicators during the processing of the client input data; and at least one of outputting or saving the determined one or more anonymized performance indicator values.

20. The VR assistance system of claim 19, wherein the one or more anonymous performance indicators are consistent with predetermined general data protection regulations.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The features, objects, and advantages of the present disclosure will become more apparent from the detailed description set forth below when taken in conjunction with the drawings in which like reference numerals refer to similar elements.

[0028] FIG. 1 depicts a flow chart of a method for monitoring the performance of a voice recognition assistant;

[0029] FIG. 2 depicts an example output of performance indicator values of a query recognition performance of the VR assistance system generated by the method;

[0030] FIG. 3 depicts an example output of performance indicator values of the processing time of the VR assistance system generated by the method; and

[0031] FIG. 4 depicts a block diagram of a system for monitoring the performance of a voice recognition assistance system.

DETAILED DESCRIPTION

[0032] FIG. 1 depicts a flow chart of a method 100 for monitoring the performance of a voice recognition assistance system in a data sensitive environment. In a preferred embodiment, the system is a multi-language VR assistance system. In step 102, client input data are determined by at least one client device. The client input data may comprise audio data containing a query by a user of the VR assistance system. The client input data are then processed by the VR assistance system in step 104. To this end, at least a part of the client input data may be transferred to a server of the VR recognition system. In order words: The client input data may be processed by the client device and/or the server. The input data processing may comprise speech-to-text transcription, natural language processing and text-to-speech transition. The client input data may comprise personal information about the user of the VR assistance system such as time, location, device ID as well as personal information contained in the audio data such as user voice and personal content. In a preferred embodiment, the VR assistance system generates a response, i.e. client output data and this response is output via the client device. For data security, the client input and output data are deleted after the output.

[0033] In step 106, the monitoring component of the VR assistance system determines one or more anonymized performance indicators of the VR assistance system. These performance indicators do not rely on personal information of the user. In a preferred embodiment, the performance indicators are in accordance with predetermined general data protection regulations. In step 108, the monitoring component determines one or more anonymized performance indicator values for the one or more anonymous performance indicators during the processing of the client input data. In a preferred embodiment, the determination of the performance indicator values comprises increasing one or more performance indicator counters. Such counters may be increased during a predetermined time period and/or using input data of several client devices.

[0034] In a preferred embodiment, the method further comprises comparing performance indicator values to previously determined one or predetermined threshold values in step 110.

[0035] The determined one or more anonymized performance indicator values are output or stored by the monitoring component in step 112. The output may comprise for example, performance indicator values, comparisons of performance indicator values determined during different time periods or trigger signals based on the performance indicator values or comparisons of performance indicator values. Example outputs are shown in FIGS. 2 and 3 and described below. The determined performance indicators may alternatively or additionally be stored in a database and access by the monitoring component later, e.g. for value comparisons.

[0036] Example anonymized performance indicators and values comprises categories regarding the VR performance efficiency, such as performance rates of reply, recognition error, technical issues, performance rate changes between two data sets, for example determined at different times. An example output of anonymized performance indicator values for these categories is shown in FIG. 2. Performance indicators may also comprise usage indicators such as number of activations within a predetermined time period, the activation type (wakeup word, button or proactive conversation) and/or activation language in case of a multi-language VR assistance system. For multi-language voice recognition system, in a preferred embodiment, performance indicator values may be counted for each language individually. Further, additional performance indicators may be determined compared to a single-language VR assistance system, such as performance efficiency across languages, number of device activations across languages, distribution of requests across languages. For example, determining of the performance indicator values may include increasing counters for the number of activations.

[0037] Further, the performance indicators may comprise VR system power indicators including total number of activations, number of activations per device, time of conversations, network traffic, server time and/or client devices. Indicators of the client device may include indicators of the number of client devices connected to the VR assistance system, the operating system of the client devices.

[0038] An example output of anonymized performance indicator values of the processing time of the VR assistance system is shown in FIG. 3. For example, determining of the performance indicator values may include increasing counters indicative of the conversation time or server processing time.

[0039] Further, the indicators may include the VR system knowledge via dialog flows, number of transitions within a dialog, most frequent questions and/or most frequent query topics. The indicators may also include hardware usage comprising load of CPU, memory, HDD, NetHDD and/or Swap.

[0040] The created output may be distributed using a subscription service. In particular, the monitoring component may generate an output containing performance indicator values as shown in FIGS. 2 and 3. This output may be converted to html or PDF format and send via an admin user interface to a predetermined list of subscribers via e-mail in direct copy, carbon copy or blind carbon copy. The distributed output may contain all or selected performance indicator values.

[0041] FIG. 4 depicts a block diagram of a system 400 for monitoring the performance of a voice recognition assistance system. The system 400 comprises one or more client devices 402 for capturing a user query and determining client input data. In a preferred embodiment, the one or more client devices may comprise mobile phones, VR devices or other devices with a suitable microphone and speaker. The system further comprises a server 404. The server comprises a monitoring component 408 which is access via a network interface 406. In a preferred embodiment, the monitoring component is a separate docker container. The server may further comprise containers for speech-to-text transcription, natural language processing and text-to-speech transition, respectively. The server may further comprise a controller. The containers may communicate through a private internal network. The system 400 is configured to execute the methods of all above embodiments.