PREDICTING A SET OF FITTED KNOWLEDGE ELEMENTS

Abstract

A computer-implemented method for predicting knowledge elements in answer to requests for information, and an associated system that processes the information request (1) made using an intermediate predictive model (5) and a knowledge element prediction fit model (8) to generate a set of fitted knowledge elements to prepare an answer, associated with their respective probabilities of use (12), as a suggestion for the preparation of an answer to an information request (15). The suggested knowledge elements are corrected and/or updated to prepare answers based on historical data, such as: data predicted by the intermediate predictive model (5), answers sent to requesters (15), contents of one or more knowledge elements used in the answers (17) and/or feedback data on the relevance of the answers (21) sent.

Claims

1. A computer-implemented method for predicting a set of fitted knowledge elements for preparation of answers to customer information requests, the method comprising: a) submission of an information request (1) by a customer, from the customer's computing device (2), to a predictive system of answer suggestions to information requests (4), via a communication channel (3); b) receipt of the information request (1) by the predictive system of answer suggestions to information requests (4); c) analysis of the information request (1) by an intermediate predictive model (5), which is configured to generate a set of answer categories that are related to the information request and associated with scores relating to their respective relevance probability (6); d) storing of the set of answer categories that are related to the information request and associated with scores related to their respective relevance probability (6), in computational memory (7); e) submission of the set of answer categories that are related to the information request and associated with scores related to their respective relevance probability (6), to a knowledge element prediction fit model (8); f) estimation by the knowledge element prediction fit model (8) of the probability of use of a knowledge element in preparing an answer to an information request (1), wherein: use of each knowledge element is predicted by considering the answer categories predicted by the intermediate predictive model (5) and recorded in the set of answer categories that are related to the information request and associated with scores related to their respective relevance probabilities (6); and the knowledge element prediction fit model (8) is configured to generate a set of fitted knowledge elements for preparing an answer associated with their respective probabilities of use (12); g) submission of the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12) to one or more answering agents (14), whereby an answering agent (14) is selected from a group consisting of a human agent and a conversational robot agent; h) submission of an answer to an information request (15) by one or more answering agents (14) from the computing device of a customer service unit (13) to the computing device of the customer (2), via a communication channel (3); i) submission of feedback relating to an answer to an information request (16) to the computational memory (7), whereby the referred feedback relating to the answer to an information request (16) consists of at least one element selected from the group consisting of the answer to an information request (15), the content of one or more knowledge elements used in answering an information request (17) and the external feedback of the relevance of the answer to the information request (21), which may be sent by the customer; and j) storage of the feedback relating to the answer to an information request (16) in the computational memory (7); whereby the knowledge element prediction fit model (8) is configured to process, in step f): the set of answer categories that are related with the information request and associated with scores relating to their respective relevance probabilities (6); a historical dataset predicted by the intermediate predictive model relating to the categories that are related with the information request and associated with their respective relevance probabilities (9); and at least one additional historical dataset selected from one or more of the group consisting of a historical dataset of answers sent to a customer (10), a historical dataset of knowledge elements used in the preparation of the answers sent to a customer (11) and a historical dataset of external feedback (22), in order to generate the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12).

2. The method of claim 1, wherein the computational memory (7) is configured to pair the set of response categories related to an information request and associated with scores related to their respective relevance probabilities (6) with the elements of the respective feedback set relating to the answer to an information request (16).

3. The method of claim 2, wherein the pairing between elements is performed with basis on an identification code of the referred information request (1).

4. The method of claim 1, wherein the knowledge element prediction fit model (8) processes in step f): at least one historical dataset predicted by the intermediate predictive model relating to categories related to an information request and associated with their respective relevance probabilities (9), a historical dataset of answers sent to a customer (10), a historical dataset of knowledge elements used in the preparation of answers to be sent to a customer (11), and a historical dataset of external feedback (22) related to a recent time interval of information requests (1).

5. The method of claim 1, wherein the knowledge element prediction fit model (8) processes, in step f): at least one historical dataset predicted by the intermediate predictive model relating to categories related to an information request and associated with their respective relevance probabilities (9), a historical dataset of answers sent to a customer (10), a historical dataset of knowledge elements used in the preparation of answers to be sent to a customer (11), and a historical dataset of external feedback (22) related to a series of previously defined time intervals of information requests (1).

6. The method of claim 1, wherein: the knowledge element prediction fit model (8) estimates at least one conditional probability, P(e.sub.j|c.sub.i), of the use of a knowledge base element, e(r.sub.k); the intermediate predictive model (5) classifies an information request r.sub.k as class c.sub.i; and the conditional probability, P(e.sub.j|c.sub.i), is estimated by considering a temporal sample of historical data relating to a set of information requests (1) indexed in the computational memory (7).

7. The method of claim 1, wherein: the knowledge element prediction fit model (8) estimates at least one conditional probability, P(e.sub.j|c.sub.i), of the use of a knowledge base element, e(r.sub.k); the intermediate predictive model (5) classifies an information request r.sub.k as class c.sub.i; and the conditional probability, P(e.sub.j|c.sub.i), is updated by considering samples of the historical data collected at regular time intervals.

8. The method of claim 7, wherein the knowledge element prediction fit model (8) estimates the conditioned probability of the use of a knowledge element in the preparation of an answer to an information request (1) in step f) by calculating a simple moving average, a weighted moving average or an exponential moving average.

9. The method of claim 1, wherein the feedback set relating to a response to an information request (16) consists of an answer to an information request (15) sent by a conversational robot agent acting as an answering agent (14).

10. The method of claim 1, wherein the human agent rectifies various sets of feedback related to the answer to a request for information (16).

11. The method of claim 1, wherein the intermediate predictive model (5) and/or the knowledge element prediction fit model (8) is configured to access the contents of available knowledge elements (18) to generate the set of answer categories that are related to an information request and associated with scores related to their respective probabilities of relevance (6) and the set of fitted knowledge elements for the preparation answers associated with their respective probabilities of use (12).

12. The method of claim 1, wherein a knowledge element extrapolation module (19) processes at least one answer to a request for information (15), and the content of the available knowledge elements (18), to generate, by extrapolation, the content of one or more knowledge elements used in an answer to a request for information (17).

13. The method of claim 1, wherein: the knowledge element prediction fit model (8) is configured to weight historical pairs of information requests (1) and answers to an information request (15) differently; and a greater weight is given to the referred historical pairs in at least one of the selected conditions of the group consisting of historical pairs that are more recent, historical pairs that have received external feedback as being a relevant answer to a favorable request for information (21), and historical pairs relating to answers to an information request (15) sent by a human answering agent (14).

14. A predictive system for the preparation of answers to information requests (4), the system comprising: an intermediate predictive model (5), which is configured to analyze an information request (1) sent by a customer from the customer's computing device (2) to the predictive system of answer suggestions to information requests (4) via a communication channel (3), wherein the intermediate predictive model (5) is additionally configured to analyze the information request and generate a set of answer categories related to an information request, and associated with scores related to their respective relevance probabilities (6); a computational memory (7), which is configured to receive the set of answer categories related with the request for information and associated with scores related to their respective relevance probabilities (6), from the intermediate predictive model (5), wherein the computational memory (7) is also configured to receive a feedback set relating to the answer to an information request (16), in which the referred feedback set relating to the answer to an information request (16) consists of at least one element selected from the group consisting of the answer to an information request (15), the content of one or more knowledge elements used in answering an information request (17) and the external feedback of the relevance of the answer to the information request (21), which may be sent by the customer; a knowledge element prediction fit model (8), which is configured to estimate the probability of use of a knowledge element in the preparation of an answer to an information request (1) based on the generation of a set of fitted knowledge elements for the preparation of an answer associated with its respective probabilities of use (12), wherein the knowledge element prediction fit model (8) is also configured to process: the set of answer categories that are related with an information request and associated with scores relating to their respective relevance probabilities (6); a historical dataset predicted by the intermediate predictive model relating to the categories that are related with the information request and associated with their respective relevance probabilities (9); and at least one additional historical dataset selected from the group consisting of a historical dataset of answers sent to a customer (10), a historical dataset of knowledge elements used in the preparation of the answers sent to a customer (11) and a historical dataset of external feedback (22), in order to generate the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12); and a computing device of a customer service unit (13), which is configured to send an answer to an information request (15) via one or more answer agents (14), whereby an answer agent (14) is selected from a group consisting of the human agent and the conversational robot agent, to the customer computing device (2), via a communication channel (3), wherein the computational device of a customer service unit (13) is additionally configured for sending feedback relating to the answer to an information request (16), to the computational memory (7).

15. The predictive system of claim 14, wherein the computing device of a customer service unit (13) comprises a processor, at least one memory, and at least one communication interface with a communication channel (3).

16. The predictive system of claim 15, wherein the communication channel (3) comprises at least one communication network selected from the group consisting of a public network, an interconnected set of public and/or private networks, and a private network.

17. The predictive system of claim 14, wherein the computational memory (7), the intermediate predictive model (5), and the knowledge element prediction fit model (8) are installed in a unit selected from the group consisting of one or more servers and one or more computing devices.

18. The predictive system of claim 14, wherein the intermediate predictive model (5) and the knowledge element prediction adjustment model (8) are installed in a storage unit selected from the group consisting of one or more servers, one or more computing devices, and one or more programmable integrated circuits.

19. The predictive system of claim 14, wherein the intermediate predictive model (5) and/or the knowledge element prediction fit model (8) are configured to access the contents of available knowledge elements (18) to generate the set of answer categories that are related to an information request and associated with scores related to their respective probabilities of relevance (6) and the set of fitted knowledge elements for the preparation answers associated with their respective probabilities of use (12).

20. The predictive system of claim 14, wherein the computational memory (7) is configured to pair the set of response categories related to an information request and associated with scores related to their respective relevance probabilities (6) with the elements of the respective feedback set relating to the answer to an information request (16).

21. The predictive system of claim 14, further comprising: a knowledge element extrapolation module (19) configured to process an answer to an information request (15) and the content of available knowledge elements (18), in order to generate, by extrapolation, the content of one or more knowledge elements used in an answer to an information request (17).

22. The predictive system of claim 21, wherein the knowledge element extrapolation module (19) is integrated into the knowledge element prediction fit model (8).

23. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method for predicting a set of fitted knowledge elements for preparation of answers to customer information requests, the method comprising: a) submission of an information request (1) by a customer, from the customer's computing device (2), to a predictive system of answer suggestions to information requests (4), via a communication channel (3); b) receipt of the information request (1) by the predictive system of answer suggestions to information requests (4); c) analysis of the information request (1) by an intermediate predictive model (5), which is configured to generate a set of answer categories that are related to the information request and associated with scores relating to their respective relevance probability (6); d) storing of the set of answer categories that are related to the information request and associated with scores related to their respective relevance probability (6), in computational memory (7); e) submission of the set of answer categories that are related to the information request and associated with scores related to their respective relevance probability (6), to a knowledge element prediction fit model (8); f) estimation by the knowledge element prediction fit model (8) of the probability of use of a knowledge element in preparing an answer to an information request (1), wherein: use of each knowledge element is predicted by considering the answer categories predicted by the intermediate predictive model (5) and recorded in the set of answer categories that are related to the information request and associated with scores related to their respective relevance probabilities (6); and the knowledge element prediction fit model (8) is configured to generate a set of fitted knowledge elements for preparing an answer associated with their respective probabilities of use (12); g) submission of the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12) to one or more answering agents (14), whereby an answering agent (14) is selected from a group consisting of a human agent and a conversational robot agent; h) submission of an answer to an information request (15) by one or more answering agents (14) from the computing device of a customer service unit (13) to the computing device of the customer (2), via a communication channel (3); i) submission of feedback relating to an answer to an information request (16) to the computational memory (7), whereby the referred feedback relating to the answer to an information request (16) consists of at least one element selected from the group consisting of the answer to an information request (15), the content of one or more knowledge elements used in answering an information request (17) and the external feedback of the relevance of the answer to the information request (21), which may be sent by the customer; and j) storage of the feedback relating to the answer to an information request (16) in the computational memory (7); whereby the knowledge element prediction fit model (8) is configured to process, in step f): the set of answer categories that are related with the information request and associated with scores relating to their respective relevance probabilities (6); a historical dataset predicted by the intermediate predictive model relating to the categories that are related with the information request and associated with their respective relevance probabilities (9); and at least one additional historical dataset selected from one or more of the group consisting of a historical dataset of answers sent to a customer (10), a historical dataset of knowledge elements used in the preparation of the answers sent to a customer (11) and a historical dataset of external feedback (22), in order to generate the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12).

Description

DESCRIPTION OF THE FIGURES

[0040] To promote an understanding of the principles involved with the modes of the disclosed embodiments, reference will be made to the modes shown in the figures and to the terminology used to describe them. In any event, it should be understood that there is no intention to limit the scope of the embodiments to the content of the figures. Any subsequent changes or modifications of the inventive features shown herein, as well as any additional applications of the principles and modes of the disclosed embodiments, which would normally occur to an expert in the field with access to this description, are considered within the scope.

[0041] FIG. 1 shows a first method of a predictive system of knowledge elements for preparing answers to customer requests for information of the prior art.

[0042] FIG. 2 shows a second method of a predictive system of knowledge elements for preparing answers to customer information requests of the prior art.

[0043] FIG. 3 shows a first method of the predictive system of a set of fitted knowledge elements for preparing answers to customer information requests according to some embodiments.

[0044] FIG. 4 shows a second method of the predictive system of a set of fitted knowledge elements for preparing answers to customer information requests according to some embodiments.

[0045] FIG. 5 shows a method of operation of a module for extrapolating the knowledge elements included in an answer, according to some embodiments.

DETAILED DESCRIPTION

[0046] The following description is presented to enable any person skilled in the art to make and use the disclosed embodiments, and is provided in the context of one or more particular applications and their requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the scope of those that are disclosed. Thus, the present invention or inventions are not intended to be limited to the embodiments shown, but rather are to be accorded the widest scope consistent with the disclosure.

[0047] As shown in FIGS. 3 and 4, a method used in the present embodiments takes into consideration the submission of an information request (1) by a customer, from the customer's computing device (2), to a predictive system of answer suggestions to information requests (4), via a communication channel (3).

[0048] The client computing device is equipped with some type of computing device configured for sending messages, such as, for example, a computer, a smartphone, a tablet or a smartwatch.

[0049] As will be understood by someone skilled in the art, a communication channel (3) may be, for example, a communication network, which includes at least one network selected from a group consisting of a public network, an interconnected set of public and/or private networks, such as the interne, and a private network. Additionally, the communication network may consist of cables or a means of wireless communication.

[0050] After the predictive system of answer suggestions to information requests (4) receives the information request (1), the content of the information request (1) is analyzed by an intermediate predictive model (5), which is configured to generate a set of answer categories that are related to the information request and associated with scores relating to their respective relevance probability (6).

[0051] By way of example, the intermediate predictive model (5) receives an information request, r.sub.k, and categorizes it into a set of possible subject categories with associated relevance scores or probabilities. The system operates with various types of subject categories generated by the intermediate predictive model (5), including subjects related to generic operations of company routines, e.g., “Exchange of contact e-mail”, “Problems with billing”, “Delayed product delivery”. Alternatively, the intermediate predictive model (5) can be configured to generate results related directly with elements that are intended to be extracted from the knowledge base, such as standard answers, text articles, pictures and/or audio-visual elements.

[0052] A set of answer categories that are related to an information request and associated with scores related to their respective relevance probability (6), predicted by intermediate predictive model (5), can be shown in Table 4.

TABLE-US-00004 TABLE 4 Probability or Category of the answer score (scale of 0 Category 11 0.71 Category 2 0.22 Category 7 0.03

[0053] If the scores of the set of categories of answers that are related to an information request and associated with scores related to their respective relevance probabilities (6), for example identified by s(c.sub.i,r.sub.k), have not already been standardized between 0 and 1, the scores can be standardized to indicate the probability of a given category c.sub.i being the correct categorization for an information request r.sub.k. The standardization for such an indication of probability can be obtained by applying equation (I):

[00001]$\begin{array}{cc}P\left(c_i.Math.r_k\right)=\frac{e^{s\left(c_i,r_k\right)}}{{.Math.}_j^c{e}^{s\left(c_j,r_k\right)}}& \left(I\right)\end{array}$

[0054] Where “i” and “j” are indices that index the categories predicted by the intermediate predictive model (5) when it classifies the information requests (1), where “c.sub.i” is the i.sup.th possible category, “c.sub.j” is the j.sup.th possible category, and C is the total number of categories that the intermediate predictive model (5) can predict.

[0055] In certain execution modes, namely when the intermediate predictive model (5) does not provide scores for some less relevant categories, the method and system of some embodiments are configured in such a way that the probability calculated by the equation (I) is zero.

[0056] It should be noted that an information request (1), in particular an information request r.sub.k, may consist of various types of elements, such as text, image, audio, temporal information, information about the user who sent the information request. Similarly, an answer to an information request (15), namely an answer to(r.sub.k), may also consist of various knowledge elements, many of which belong to the company's knowledge base, such as text, images, links to articles, audio, audio-visual elements, temporal information, and information relating to the agent preparing the answer.

[0057] After generating the set of answer categories that are related to the information request and associated with scores related to their respective relevance probability (6), it is sent to the computational memory (7) and to a knowledge element prediction fit model (8), as shown in FIGS. 3 and 4.

[0058] In the preferred modes of the present embodiments, the computational memory (7) stores information related to the set of answer categories that are related to an information request and associated with scores related to their respective relevance probabilities (6); the information related to the feedback set that is related to the answer to an information request (16); a historical dataset predicted by the intermediate predictive model relating to the categories that are related to an information request and associated with their respective relevance probabilities (9); a historical dataset of answers sent to a customer (10); a historical dataset of knowledge elements used in preparing the answers that are sent to a customer (11); and a historical dataset of external feedback (22).

[0059] Next, an estimate of the probability of a use of a knowledge element in the preparation of an answer to an information request (1) is performed by the knowledge element prediction fit model (8), which is configured to generate a set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12), as shown in Table 5.

TABLE-US-00005 TABLE 5 Probability or score Knowledge element (scale of 0 to 1) Element 3 0.71 Element 12 0.22 Element 9 0.03

[0060] In order to estimate the probability of use of a knowledge element in the preparation of an answer to an information request (1) and in order to generate the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12), the knowledge element prediction fit model (8) is essentially configured to process the following data:

[0061] the set of answer categories that are related to an information request and associated with scores related to their respective relevance probability (6), predicted by intermediate predictive model (5), for the information request (1) being processed; and

[0062] a historical dataset predicted by the intermediate predictive model relating to categories that are related to an information request and associated with their respective relevance probabilities (9), for example a historical dataset s(c.sub.i,r.sub.j).

[0063] In preferred modes of the present embodiments, the knowledge element prediction fit model (8) is also configured to process the following information in order to correct and adjust the results predicted by the intermediate predictive model (5): [0064] at least one additional dataset selected from the group consisting of a historical dataset of answers sent to a customer (10), i.e., a historical dataset of a(r.sub.k) for various r.sub.k requests; [0065] a historical dataset of knowledge elements used in the preparation of answers that are sent to a client (11), i.e., a historical dataset of e(r.sub.k) for various r.sub.k requests; and [0066] a historical dataset of external feedback (22).

[0067] Illustratively, for each information request (1), for example, a r.sub.k request, the knowledge element prediction fit model (8) will estimate the conditional probability, P(e.sub.j|c.sub.i), of the use of a knowledge base element, e(r.sub.k), given that the intermediate predictive model (5) classified the r.sub.k information request as class Estimation is performed by taking into account the historical data related with information request pairs (1) and answer to an information request (15) within a recent time interval, T.

[0068] In some embodiments, the temporal information for estimating the probability P.sub.T(e.sub.j|c.sub.i) is made by reference to the frequencies of use of the answer preparation agents during a given recent time period T, for example the last 2 months of operation of a customer service center. As such, the probability P.sub.T(e.sub.j|c.sub.i) can be calculated using equation (II).

[00002]$\begin{array}{cc}{P}_T\left(e_j.Math.c_i\right)=\frac{{\#}_T\left(e_j,c_i\right)}{{\#}_T\left(c_i\right)}& \left(\mathrm{II}\right)\end{array}$

[0069] whereby #T(e.sub.j,c.sub.i) is the number of information requests (1) answered during the period T that were classified with the category c.sub.i and whereby the agent preparing the answer used knowledge element e.sub.j from the knowledge base when replying to the information request (1), and #T(c.sub.i) is the number of information requests answered during the same time interval T that were classified with the category c.sub.i.

[0070] In some embodiments, the number of counts in equation (II) may be adjusted so as not to penalize elements that appear very few times. For example, a regularization technique (such as a smoothing technique) can be used in which extra R elements are added that count all combinations of element-category pairs (e.sub.j,c.sub.i), as indicated in equation (III):

[00003]$\begin{array}{cc}{P}_T\left(e_j.Math.c_i\right)=\frac{{\#}_T\left(e_j,c_i\right)+R}{{\#}_T\left(c_i\right)+\mathrm{ER}}& \left(\mathrm{III}\right)\end{array}$

[0071] Whereby E in equation (III) is the total number of knowledge elements. Depending on the value of R, the probability PT gets closer to the true frequency count of equation (II) (where R=0) or closer to the continuous uniform distribution (where R=infinite).

[0072] Therefore, the knowledge element prediction fit model (8) can suggest knowledge elements that can be used by the answer preparation agents in replying to an information request (1), based on the intermediate predictive model classification (5) and the probability estimate P.sub.T(e.sub.j|c.sub.i).

[0073] In some modes of the embodiments, the suggestion of knowledge elements may be made through hard links, wherein the most likely classification of a given ci category, predicted by the intermediate predictive model (5) is assumed to be 100% correct and the probability of using a knowledge base element e.sub.j for a given subject category is estimated by means of equation (IV):

P(e)=P.sub.T(e|c.sub.i)   (IV)

[0074] In other modes, the suggestion of knowledge elements can be made by soft linking, which takes advantage of the uncertainty of the intermediate predictive model (5) and the probability of using an e.sub.j element in the knowledge base for a given subject category is estimated.

[0075] Estimating the likelihood P.sub.T(e.sub.j|c.sub.i) requires accessing several elements in the knowledge base and doing a heavy count of the events. This process may become too time-consuming for a typical computational system. Therefore, in other modes, the probabilities P.sub.T(e.sub.j|c.sub.i) may be estimated by considering a temporal sample of historical data relating to all information requests (1) indexed in the computational memory (7).

[0076] Alternatively, the conditional probability, P(e.sub.j|c.sub.i) is updated taking into consideration samples of the historical data collected at regular intervals T.sub.P, such as daily or hourly. As such, the small difference that is obtained between an instantaneous or periodically scheduled probability is very small, especially if the aggregation time interval T is much higher when compared to the frequency of updating the probability T.sub.P.

[0077] Therefore, in the preferred modes of the present embodiments consisting of the use of the predictive system of answer suggestions to information requests (4) in a customer service center of a company or organization, as shown in FIGS. 3 and 4, the knowledge element prediction fit model (8) processes at least one historical dataset predicted by the intermediate predictive model relating to categories that are related to an information request and associated with their respective pertinence probabilities (9), a historical dataset of answers sent to a client (10), a historical dataset of knowledge elements used in preparing answers that will be sent to a client (11), and a historical dataset of external feedback (22), whereby the historical data may be related to a recent time interval of information requests (1) or related to a series of previously defined time intervals of records registered in the computational memory (7). For this purpose, the computational memory (7) is configured to store the respective time data of information requests (1) and of answers to information requests (15). As will be understood by a person skilled in the art, the time interval for acquiring historical data may be differentiated according to the types of input data of the knowledge element prediction fit model (8).

[0078] As shown in FIGS. 3 and 4, the intermediate predictive model (5) and the knowledge element prediction fit model (8) may be configured to access the contents of available knowledge elements (18) to generate respective predictions of the set of answer categories that are related to an information request and associated with scores related to their respective probabilities of relevance (6) and of the set of fitted knowledge elements for the preparation answers associated with their respective probabilities of use (12).

[0079] In some embodiments, the computational memory (7) is configured to pair the set of answer categories related to an information request and associated with scores related to their respective relevance probabilities (6) with the elements of the respective feedback set that is related to the answer to an information request (16), such as, for example, an answer to an information request (15), the content of one or more knowledge items used in an answer to an information request (17) and the external feedback on the relevance of the answer to an information request (21). Preferably, the referred pairing between elements is performed based on an identification code of the referred information request (1), which is stored in the computational memory (7).

[0080] The knowledge element prediction fit model (8) consists of a prediction fitting process based on the history of questions and their respective answers with knowledge elements. This prediction fitting process may be updated instantaneously in situations where the probabilities are fitted every time feedback is provided by the client and/or one or more of the answering agents (14). In other modes, the prediction fitting process is performed in a timed manner, in cases where the frequencies of use of knowledge elements are updated at predefined times

[0081] Preferably, the knowledge element prediction fit model (8) estimates the conditional probability of the use of a knowledge element in preparing an answer to an information request (1) in step f) of the method according to some embodiments by calculating a moving average.

[0082] The conditional probability P.sub.T(e.sub.j|c.sub.i), of the use of a knowledge base element, e(r.sub.k), given a subject category c.sub.i, can be estimated with variants of equation (IV). In particular, the knowledge element prediction fit model (8) may use some moving average where the points, namely historical pairs of information requests (1) and of answers to an information request (15) are weighted differently, giving greater importance to requests that are more recent.

[0083] In some embodiments, the knowledge element prediction fit model (8) may weight the historical pairs of information requests (1) and answers to an information request (15) based on external feedback of the relevance of the answer to an information request (21), namely according to the level of relevance of the answer given to the customer.

[0084] In other modes, the knowledge element prediction fit model (8) may weight historical pairs of information requests (1) and answers to an information request (15) based on the association between the answer to an information request (15) and the respective answering agent (14) that sent it. For this purpose, the answer to an information request (15), present in the feedback set relating to the answer to an information request (16), includes information relating to which answering agent (14) sent the answer to the customer. As such, the model (8) may be configured to weight the history of answers to an information request (15) according to which answering agent (14) generated the answer, for example, assigning a greater weight to answers from a human agent or disregarding answers prepared by a robot agent.

[0085] In other modes, the knowledge element prediction fit model (8) may weight historical pairs of information requests (1) and answers to an information request (15) based on the characteristics of the information request (1), such as on the size of the request; the type of channel used to send it (such as e-mail, Whatsapp, Facebook or chat); or the classification of the request into a particular set of categories (such as on language or intent).

[0086] In general, if a new knowledge element is used by one or more answering agents (14) and receives a satisfactory feedback set regarding the answer to an information request (16) by one or more answering agents (14), the new knowledge element is inserted into the historical loop and will be suggested by the knowledge element prediction fit model (8). As such, the new knowledge element may be forcibly used by one or more answering agents (14). Alternatively, the new knowledge element may be used by a human answering agent (14) after he performs a search of the knowledge element database without resorting to the use of suggestions.

[0087] When there is a need to prepare an answer to an information request (15) identifying, for example, personalized customer data in the message, a human answering agent (14) will prepare such an answer. As will be recognized by an expert in the art, the preparation of answers that contain personalized data for a customer may also be automated to some extent, and may be performed by a conversational robot agent. In the same way as an answer to a standardized information request (15), a personalized answer is also sent to the computational memory (7), and is likely to be included in the feedback set relating to the answer to an information request (16).

[0088] After estimating the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12), the answer is sent to one or more answering agents (14), whereby an answering agent (14) is selected from a group consisting of a human agent and a conversational robot agent. The human agent for the preparation of an answer is a human operator within the context of some embodiments. In illustrative implementations, various answering agents (14) prepare the answer to an information request (15) based on the set of fitted knowledge elements for the preparation of an answer associated with their respective probabilities of use (12).

[0089] In this manner, the answering agent (14) receives the set of fitted knowledge elements to prepare an answer associated with their respective probabilities of use (12) and may prepare the answer to an information request (15) based on the information resulting from the processing performed by the knowledge element prediction fit model (8).

[0090] Then, one or more answering agents (14) send the answer to an information request (15) from a computing device of a customer service unit (13) to the customer's computing device (2) via a communication channel (3). In some embodiments, sending an answer to an information request (15) by one or more answering agents (14) from a computing device of a customer service unit (13) to the customer computing device (2), is performed via a data and information delivery module (20) and a communication channel (3), as shown in FIGS. 3 and 4. At the same time, a computing device of a customer service unit (13) sends a feedback set relating to the answer to an information request (16) to the computational memory (7), whereby the referred feedback set relating to the answer to an information request (16) consists of at least one element selected from a group consisting of the answer to an information request (15) and the content of one or more knowledge elements used in an answer to an information request (17). As shown in FIG. 4, the feedback set relating to the answer to an information request (16) may also consist of a third information element, namely the external feedback on the relevance of the answer to an information request (21), which may be sent by the customer via a communication channel (3).

[0091] In other modes of implementation of the present embodiments, as shown in FIG. 4, at least one answer agent (14) is a conversational robot agent, which is installed in the computing device of a customer service unit (13), whereby the referred conversational robot agent, which prepares the answer based on the set of fitted knowledge elements for the preparation of an answer associated with their respective usage probabilities (12). In these modes, the referred feedback set relating to the answer to an information request (16) consists of at least one element selected from the group consisting of the answer to an information request (15) and the content of one or more knowledge elements used in answering an information request (17), and may be complemented by the external feedback of relevance of the answer to the information request (21). Alternatively, various sets of feedback relating to the answer to an information request (16) may be rectified by a human agent based on an analysis of the relevance of a historical dataset of answers to an information request (15), previously sent by the conversational robot agent, or based on a historical dataset of external feedback of the pertinence of the answer to an information request (21), sent by the customers. The feedback set relating to the answer to an information request (16) may be supplemented by a human agent based on the history of the content of one or more knowledge elements provided in an answer to an information request (17).

[0092] In terms of the synchronous and asynchronous operation of the present system, the knowledge element prediction fit model (8) receives the information generated by the intermediate predictive model (5), so both models work in sequence.

[0093] The historical dataset, namely the historical dataset predicted by the intermediate predictive model referring to categories related to an information request and associated with their respective pertinence probabilities (9), the historical dataset of answers sent to a customer (10), the historical dataset of knowledge elements used in the preparation of the answers sent to a client (11) and the historical dataset of external feedback (22) may be stored in the computational memory (7) and submitted for processing by the knowledge element prediction fit model (8) asynchronously, in other words, the information is stored in the computational memory (7) as it is made available. In this regard, when the information is sent to the knowledge element prediction fit model (8), the information that has been stored up to the time of use in the computational memory (7) may be used. In the preferred modes of some embodiments, the historical dataset (9, 10, 11, 20) is sent to the knowledge element prediction fit model (8) periodically, for example every hour, and does not need to be instantaneous, allowing for a time lag when being written to the computational memory (7) by the computing device of a customer service unit (13).

[0094] As shown in FIGS. 3 and 4, the step of submitting an answer to an information request (15) by one or more answering agents (14) from the computing device of a customer service unit (13) to the computing device of the customer (2), is made via a communication channel (3). In parallel and asynchronously, the feedback set relating to the answer to an information request (16) by one or more answer agents (14) is sent from the computing device of a customer service unit (13) to the computing memory (7), so that this information is available for further processing by the knowledge element prediction fit model (8).

[0095] A knowledge element prediction fit model (8) generates a set of fitted knowledge elements for the preparation of an answer associated with their respective usage probabilities (12), according to the needs of an answering agent (14), and the prediction may be programmed to run automatically whenever a request is opened in the e-mail box. The suggestion request can thus be made more than once for the same request for information (1), if the feedback set that is related to the answer to a request for information (16) indicates that the answer to a previous request for information (15) was not relevant.

[0096] In some embodiments, the computational memory (7) used by the predictive system for suggesting answers to information requests (4) consists of one or more databases, which will be selected from one or more of a group consisting of a flat database model, a tabular database model, a network database, a hierarchical database model, a relational database model, an object-oriented database model, and an object-relational database, and a graph-oriented database model.

[0097] In some of the implementation modes, if one or more knowledge elements used in an answer to an information request (17) that were used by one or more answer agents (14) are not explicitly accessible, they can be extrapolated through the content of the answer itself, a.sub.k(r.sub.i), in order to be subsequently used by the knowledge element prediction fit model (8) in the form of a historical knowledge elements dataset used in the preparation of the answers sent to a customer (11). As such, in these implementation modes, as illustrated in FIG. 5, a knowledge element extrapolation module (19) may also be configured to process only one answer to an information request (15), whereby the knowledge element extrapolation module (19) is configured to process one answer to an information request (15) and the content of available knowledge elements (18) in order to generate, by extrapolation, the content of one or more knowledge elements used in an answer to an information request (17). Therefore, the content of one or more knowledge elements used in an answer to an information request (17) is extrapolated from an answer to an information request (15) and is stored in the computational memory (7), and may be integrated into the historical dataset of knowledge elements used in the preparation of answers sent to a customer (11), eventually used by the knowledge element prediction fit model (8). The knowledge element extrapolation module (19) may be included in a stand-alone manner in the predictive system of answer suggestions to information requests (4) or it may be included in another processing module, for example in the knowledge element prediction fit model (8). When the knowledge element extrapolation module (19) is integrated in the knowledge element prediction fit model (8), there is no need to write to the computational memory (7) as it makes direct use of the content of one or more knowledge elements used in an answer to an information request (17). In illustrative implementations, the knowledge element extrapolation module (19) processes a historical dataset of answers sent to a client (10) in order to convert them into a historical dataset of knowledge elements used in the preparation of answers sent to a customer (11), before writing the information into the computational memory (7).

[0098] Therefore, the predictive system of answer suggestions to information requests (4) is particularly configured to implement the computer-implemented method for generating answers to customer information requests, in accordance with the first instance.

[0099] In some embodiments, the computing device of a customer service unit (13) consists of a processor, at least one memory and at least one communication interface with a communication channel (3).

[0100] In some embodiments, the computational memory (7), the intermediate predictive model (5) and the knowledge element prediction fit model (8) are installed in a unit selected from a group consisting of one or more servers and one or more computing devices. Even more preferably, the computational memory (7) is installed in a server or computing device.

[0101] In some embodiments, the computational memory (7) is non-volatile computational memory.

[0102] In some embodiments, the computational memory (7) is a collection of computational memories aggregated in the computational apparatus or distributed over one or more servers or one or more computing devices.

[0103] In some embodiments, the intermediate predictive model (5) is a model that operates based on rules and/or based on automated learning. In terms of automated learning, the model parameters may be learned based on the history of requests and their respective answers and/or knowledge elements use to prepare those answers, in which the intermediate predictive model (5) can be a supervised classification model, as described by Shervin Minaee et al. in “Deep learning based text classification: A comprehensive review” and in patent application US2017286972A1, by Anantharaman Arvind Kunday et al., published on 5 Oct. 2017; a search and sorting model based on fixed features, such as the example described by S. Robertson and H. Zaragoza. “The probabilistic relevance framework: bm25 and beyond”. Found. Trends Inf. Retr., 3(4):333389, April 2009; or learned based on message history, as described in J. Lin, R. Nogueira, and A. Yates. “Pretrained transformers for text ranking: Bert and beyond”, 2020; or the multi-step sequence of sorting and training models, described by Y. Q. Y. Ding et al. in “An optimized training approach to dense passage retrieval for open-domain question answering”, 2020. It should be noted that, given the variability of models described above, the intermediate model (5) may include, the intermediate model (5) may in certain cases have access during its execution to different pieces of information, such as an information request history (1).

[0104] In illustrative implementations, the intermediate predictive model (5) and the knowledge element prediction fit model (8) are installed in a storage unit selected from a group consisting of one or more servers, one or more computing devices and one or more programmable integrated circuits.

[0105] As used throughout this patent application, the term “or” is used in the inclusive sense rather than the exclusive sense, unless the exclusive sense is clearly defined in a particular situation. In this context, a sentence of the type “X uses A or B” should be interpreted as including all relevant inclusive combinations, for example “X uses A”, “X uses B” and “X uses A and B”.

[0106] As used throughout this patent application, the indefinite articles “a or an” should generally be interpreted as “one or more” unless the meaning the singular meaning is clearly defined in a specific situation.

[0107] As presented in this description, terms relating to examples should be interpreted for the purpose of illustrating an example of something and not to indicate a preference.

[0108] As used in this description, the term “substantially” means that the real value falls within about 10% of the desired value, variable or related limit, “particularly” within about 5% of the desired value, variable or related limit or “especially” within about 1% of the desired value, variable or related limit.

[0109] The subject matter described above is provided to illustrate examples of the disclosed embodiments and should not be construed to limit those embodiments. Also, the terminology used for the purpose of describing specific modes should not be construed to limit the embodiments. As used in the description, the definite and indefinite articles, in their singular form, are intended to be interpreted as also including the plural forms, unless the context of the description explicitly indicates otherwise. It will be understood that the terms “comprise” and “include” when used in this description specify the presence of the related characteristics, elements, components, steps and operations, but do not exclude the possibility that other characteristics, elements, components, steps and operations are also considered.

[0110] All modifications, provided that they do not modify the essential characteristics of the claims that follow, should be considered to fall within the scope of the protection of the disclosed embodiments.

LIST OF REFERENCES

[0111] 1. A request for information

[0112] 2. The computing device of a customer

[0113] 3. A communication channel

[0114] 4. A predictive system of answer suggestions to information requests

[0115] 5. An intermediate predictive model

[0116] 6. A set of answer categories that are related to an information request and associated with scores related to their respective relevance probability

[0117] 7. Computational memory

[0118] 8. A knowledge element prediction fit model

[0119] 9. A historical dataset predicted by the intermediate predictive model relating to categories that are related to an information request and associated with their respective relevance probabilities

[0120] 10. A historical dataset of answers sent to a customer

[0121] 11. A historical dataset of knowledge elements used in the preparation of answers sent to a customer

[0122] 12. A set of fitted knowledge elements used for preparing an answer associated to its respective probabilities of use

[0123] 13. A computing device of a customer service unit

[0124] 14. An answering agent

[0125] 15. An answer to a request for information

[0126] 16. The feedback set relating to an answer to an information request

[0127] 17. The content of one or more knowledge elements used in an answer to a request for information

[0128] 18. The content of the available knowledge elements

[0129] 19. A knowledge element extrapolation module

[0130] 20. A data and information delivery module

[0131] 21. The external feedback relating to the relevance of a request for information

[0132] 22. A historical dataset of external feedback

[0133] 23. An answer category predictive model

[0134] 24. An automatic task defined manually by a human

[0135] 25. A set of knowledge elements used for preparing an answer associated to its respective probabilities of use

[0136] 26. A knowledge element predictive model

[0137] An environment in which one or more embodiments described above are executed may incorporate a general-purpose computer or a special-purpose device such as a hand-held computer or communication device. Some details of such devices (e.g., processor, memory, data storage, display) may be omitted for the sake of clarity. A component such as a processor or memory to which one or more tasks or functions are attributed may be a general component temporarily configured to perform the specified task or function, or may be a specific component manufactured to perform the task or function. The term “processor” as used herein refers to one or more electronic circuits, devices, chips, processing cores and/or other components configured to process data and/or computer program code.

[0138] Data structures and program code described in this detailed description are typically stored on a non-transitory computer-readable storage medium, which may be any device or medium that can store code and/or data for use by a computer system. Non-transitory computer-readable storage media include, but are not limited to, volatile memory; non-volatile memory; electrical, magnetic, and optical storage devices such as disk drives, magnetic tape, CDs (compact discs) and DVDs (digital versatile discs or digital video discs), solid-state drives, and/or other non-transitory computer-readable media now known or later developed.

[0139] Methods and processes described in the detailed description can be embodied as code and/or data, which may be stored in a non-transitory computer-readable storage medium as described above. When a processor or computer system reads and executes the code and manipulates the data stored on the medium, the processor or computer system performs the methods and processes embodied as code and data structures and stored within the medium.

[0140] Furthermore, the methods and processes may be programmed into hardware modules such as, but not limited to, application-specific integrated circuit (ASIC) chips, field-programmable gate arrays (FPGAs), and other programmable-logic devices now known or hereafter developed. When such a hardware module is activated, it performs the methods and processes included within the module.

[0141] The foregoing embodiments have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit this disclosure to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. The scope is defined by the appended claims, not the preceding disclosure.