Abstract
The present invention relates generally to the field of providing a computer-implemented system and method that supports ad hoc multilingual Natural Language database questions that are transformed in realtime to Database queries using a suite of cascading Deep Neural Networks which are initially and continuously trained through schema information and question-query examples.
Claims
1. A system to facilitate online ad hoc multilingual natural language database questions that are transformed in realtime to database queries using a suite of cascading Deep Neural Networks which are initially and continuously trained through schema information and question-query examples, the system comprising: a processor; and a memory device coupled to the processor and storing executable program instructions therein, which, when executed by the processor, cause the system to perform operations comprising: receiving over a network from a client device a client communication including an ad hoc multilingual natural language question of a database; cascading the natural language query through at most three successive attempts by three different instances of Deep Neural Networks components in the system, each of which is activated if the previous Deep Neural Network component is unable to predict an output database query.
2. The system of claim 1, wherein the submitted natural language question is cascaded to the first instance of a Deep Neural Network component in the system, whose corresponding model is initially and continuously trained in auto label generated natural language questions and their corresponding queries based on the database schema. This first instance of a Deep Neural Network component in the system, using fuzzy logic, attempts to match the submitted natural language question against its model, and if successful returns the matching SQL query for execution.
3. The system of claim 1, wherein if a match is not successful using the method in claim 2, the system then cascades the submitted natural language question to the second instance of a Deep Neural Network component in the system, whose corresponding model is initially and continuously trained with previously submitted natural language questions and their corresponding queries, along with a set of inferred natural language questions based on previously submitted natural language questions. This second instance of a Deep Neural Network component in the system, attempts to resolve the linguistic ambiguity in the submitted natural language question using its model, and if successful returns the matching SQL query for execution.
4. The system of claim 1, wherein if a match is not successful using the method described in claim 3, the system then cascades the submitted natural language question to the third instance of a Deep Neural Network component in the system, whose corresponding model is initially and continuously trained with a vast range of hypothetical natural language questions and their corresponding queries based on the database schema. This third instance of a Deep Neural Network component in the system, attempts to match the submitted natural language question against its model and if successful returns an SQL query for execution.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0044] FIG. 1 is an overview block diagram of the major components and users of a computer-implemented client-server system and method that supports ad hoc Natural Language database questions that are transformed in realtime to SQL queries using a suite of Deep Neural Networks.
DETAILED DESCRIPTION
[0045] Embodiments of the present invention provide a computer-implemented system that transforms ad hoc Natural Language database questions into appropriately formatted queries.
[0046] In one embodiment of the present invention, users submit ad hoc Natural Language database questions through a conversational user interface to a system that is initially and continually configured using database schema information and example question-query pairs. The system uses a suite of Deep Neural Networks to transform the question into a SQL query targeted at databases that are based on a SQL query interface. The query is then executed on a pre-configured database server and results are returned to the user.
[0047] FIG. 1 is an overview block diagram of the major components and users of a computer-implemented client-server system and method that supports ad hoc Natural Language database questions that are transformed in realtime to SQL database queries using a suite of cascading Deep Neural Networks.
[0048] Referring to FIG. 1 and FIG. 2 those elements depicted within circles represent various users and administrators of the system that access the system via various client interfaces or devices including but not limited to voice recognition apps or devices, web browsers and mobile apps. All other elements depicted in the figure are the computer-implemented components of a client-server system according to one embodiment of the present invention.
[0049] The dotted horizontal line across FIG. 2 marks the Demarcation Zone (DMZ) 500 between the premises controlled by the business using the present invention, and the premises controlled by the operators of the present invention. All users, administrators and components above the Demarcation Zone (DMZ) 500 are under the control of the business entity using the present invention. And all users, administrators and components below the Demarcation Zone (DMZ) 500 are under the control of the operators of the present invention. All data that is private to the business using the present invention, exists only above the Demarcation Zone (DMZ) 500 and is never passed across the Demarcation Zone (DMZ) 500.
[0050] Now referring to FIG. 1, in one embodiment of the present invention the Data Analysts 100 are responsible for passing the training data to the suite of DNN components of the present invention. Training of the DNN components is done both initially before any user questions are submitted to the system and on a continual basis after the system is in operation. The training data provided by the Data Analysts 100 includes but is not limited to the Schema Description data 112 and the Gold Data Set 122. The Schema Description data 112 includes but is not limited to schema descriptions and Where value description data for each column of the Business Database 424. The Gold Data Set 122 includes but is not limited to frequently asked questions, their corresponding SQL queries and paraphrases of the frequently asked questions. Data Analysts 100 pass the Schema Description data 112 and the Gold Data Set 122 via the Schema Portal component 102 across the DMZ 500 to the components of the present invention in order to train the various DNNs.
[0051] Still referring to FIG. 1, in one embodiment of the present invention after the Schema Description data 112 is passed by the Schema Portal component 102 across the DMZ 500 it is first persisted in the Schema Repository 110. The Schema Description data 112 is then passed to the Schema Based Query Generator component 114 which creates a set of queries based on the Schema Description data 112.
[0052] Still referring to FIG. 1, the Schema Based Query Generator component 114, then takes the Schema Description data 112 and automatically generates a large number of hypothetical sample SQL queries. In one embodiment of the invention the number of sample SQL queries generated is in the range of thousands to ten of thousand sample queries. The distribution of SQL queries that are generated span the range of every possible query type including but not limited to queries with Joins, queries without Joins, queries with Order By, queries without Order By, queries with Aggregators, queries without Aggregators, queries with sub-queries, queries without sub-queries, etc.
[0053] Still referring to FIG. 1, in one embodiment of the present invention, the Template Based Labeling component 116 generates natural language utterances corresponding to a subset of the queries generated by Schema Based Query Generator component 114. A query along with its corresponding natural language utterance generated by the Schema Based Query Generator component 114 forms a tuple. The collection of these tuples is stored in the Bronze Data Set 128. These tuples are also referred to as the Auto Labeled Tuples.
[0054] Still referring to FIG. 1, in one embodiment of the invention, the remaining subset of queries generated by the Schema Based Query Generator component 114 are then passed to the Label Expert 118. The Label Expert 118 looks at each sample query and turns it into one or more natural language utterances using the Labeling Portal component 120. Each query and its corresponding natural language utterance form a tuple and the collection of these tuples are stored in the Bronze Data Set 128.
[0055] Still referring to FIG. 1, in one embodiment of the present invention, the Question Question Pairing (QQP) Model Training component 130 uses the Auto Labeled Tuples from the Bronze Data Set 128 to train a Question Question Pairing (QQP) Model 330 in FIG. 2. The QQP Model 330 in FIG. 2 is able to fuzzy match a given question with questions from the Bronze Data Set 128.
[0056] For example, if the Bronze Data Set 128 contains a question “Total sales for year 2019”, and the user asks the question, “Total sale for year 2019”. The QQP Model 330 in FIG. 2 matches the user question with Bronze Data Set 128 questions using fuzzy text matching. For instance, in this case, the words “sale” and “sales” produce the same result “sale” when stemmed. Thus, the QQP Model 330 in FIG. 2 would predict that the user question is close enough to Bronze Data Set 128 question “Total sales for year 2019”. When such a match is found, the corresponding Query 422 in FIG. 2 in the Bronze Data Set 128 is sent to the Results Generator Model component 426 in FIG. 2.
[0057] Now referring to FIG. 2, in one embodiment of the invention, the QQP Model 330 also attempts to match questions using exact match where each character in the questions matches.
[0058] Still referring to FIG. 2, in one embodiment of the invention, the QQP Model 330 also attempts to match questions using known synonyms.
[0059] Still referring to FIG. 2, in one embodiment of the invention, the QQP component 331 performs the necessary preprocessing steps on the input, uses the QQP Model 330 to generate a predicted output and then post-processes the output into the appropriate output format.
[0060] Still referring to FIG. 2, in one embodiment of the present invention the NL2Query Model component 310 is trained by the NL2Query Model Training component 136 of FIG. 1 using Schema Description data 112 of FIG. 1 and tuples of Natural Language Utterances and Query Templates 312. A Query Template 312 is a query with all values replaced by a placeholder. During inference, the model is presented with the Schema Description data 112 of FIG. 1 and a User's 200 utterance and it generates a Query Template 312. The Query Values Generator component 400 then finds out the values for each placeholder in the Query Template 312. A Query 412 is produced by taking the Query Template 312 and replacing placeholders with values generated by the Query Values Generator component 400.
[0061] Still referring to FIG. 2, in one embodiment of the invention, the NL2Query component 311 performs the necessary preprocessing steps on the input and uses the NL2Query Model 310 to generate a predicted output and then post-processes the output into the appropriate output format.
[0062] Now referring to FIG. 1, in one embodiment of the present invention, the Intent Discovery component 124 takes the Gold Data Set 122 and the Schema Description 112 and discovers user intents corresponding to each query in the Gold Data Set 122.
[0063] The Intent Discovery component 124 analyzes each query in the Gold Data Set 122 and splits each query into components such as WHERE conditions, tables and table joins used, SELECT columns and aggregations, etc. The components are then grouped as a set of intents that are stored in the Intent Data Set 126.
[0064] For example, consider the following example Gold Data Set utterance-query tuples: [0065] “Which product is abcd sku?”—“SELECT name FROM products WHERE sku=‘abcd’” [0066] “Give me product name for sku xyz.”—“SELECT name FROM products WHERE sku=‘xyz’” [0067] “Total sales”—“SELECT sum(sales) FROM transactions”
Here, the first two queries represent the same user intent, “get name of product for a given sku”. The third query represents a different intent “get total sales”. In this case, the Intent Discovery component 124 would produce an Intent Data Set 126 with two intents with following templates [0068] Intent 0—“SELECT name from products sku=‘[query value]’” [0069] Intent 1—“SELECT sum(sales) FROM transactions”
[0070] Still referring to FIG. 1, in one embodiment of the present invention, the IDSF Model Training component 132 takes the Intent Data Set 126, the Gold Data Set 122 and the Schema Description data 112 and trains models including but not limited to IDSF Model 320 in FIG. 2 to predict user intent and query values from an utterance from a User 200 in FIG. 2.
[0071] In the above examples, the IDSF Model 320 in FIG. 2 would be trained to predict Intent 0 for the first two utterances (“Which product is abcd sku?” and “Give me product name for sku xyz.”), and to predict Intent 1 for the last utterance “Total sales.” The IDSF Model 320 in FIG. 2 would also be trained to predict query values “abcd” and “xyz” for the first two utterances (“Which product is abcd sku?” and “Give me product name for sku xyz.”).
[0072] Similarly, the QVEnum Model Training component 134 takes the Intent Data Set 126, the Gold Data Set 122 and the Schema Description 112 and trains models including but not limited to the QVEnum Model 322 in FIG. 2. The QVEnum Model 322 in FIG. 2 is trained by using utterance values as input and corresponding enum column values as the target.
[0073] Now referring to FIG. 2, in one embodiment of the invention, the IDSF+QVEnum component 321 performs the necessary preprocessing steps on the input utterance. It then uses the IDSF Model 320 to predict Intent and Query Values for the utterance. The Query Values produced by the IDSF Model 320 are passed through the QVEnum Model 322 to produce corrected Query Values for any enumerated columns. The IDSF+QVEnum component 321 then looks up the Query Template corresponding to the Intent predicted by the IDSF Model 320 from the Intent Data Set 126. It then produces the Query Template 324 and the corrected Query Values 324 as output.
[0074] Each Deep Neural Network (DNN) component, including but not limited to the NL2Query DNN 311, the IDSF+QVEnum DNN 321 and the QQP DNN 331 are essentially pattern matching machines. (Note in FIG. 1 331, 321 and 311 should be labelled as DNNs)
[0075] After each of the Deep Neural Network (DNN) components have been trained the system is ready for user submitted natural language questions. When a user submits a natural language question to be converted into a SQL query and its associated where values, the Deep Neural Network (DNN) components look for similarities between the submitted question and the data it has been trained with in order to generate a corresponding query.
[0076] Still referring to FIG. 2, in one embodiment of the present invention, after the system has been trained, Users 200 can submit natural language questions to be processed by the system. Users 200 interact with various components and other users and administrators of the system including but not limited to the Data Visualization Tool component 440, the Voice Processing Tool components 202, the Conversational Agent component 204 and Business Analysts 430.
[0077] For example, a user might ask “How many buys greater than $100 did Leo Swanson make after Jan. 12, 2019.” The Conversational Agent component 204 removes any data that would be sensitive or private to the business and substitutes generic parameters to be passed over the DMZ 500.
[0078] The rules for which data is considered sensitive or private is configurable and is set by the Business. In this example the actual dollar value and the specific customer name would be masked. Resulting in a masked question of the form: “How many buys greater than $VALUE did $NAME make after Jan. 12, 2019.” At the same time the Conversational Agent component 204 persists the original sensitive data in a data store to be re-inserted later on in the process.
[0079] Still referring to FIG. 2, in one embodiment of the present invention, the Conversational Agent component 204 passes the natural language question with the masked parameters across the DMZ 500 to the Conversational Coordinator component 300.
[0080] Still referring to FIG. 2, in one embodiment of the present invention, the QQP component 331 is the first of the suite of the cascading Deep Neural Network Suite components. The Conversational Coordinator component 300 first queries the QQP component 331 to retrieve the Query 422 if a similar question already exists in the Bronze Data Set 128 in FIG. 1. In that case, the Query 422 is executed against the Business Database 424 by the Results Generator component 426 and the results are displayed to the User 200 using the Data Visualization Tool component 440.
[0081] Still referring to FIG. 2, in one embodiment of the present invention, if the QQP component 331 does not match the user question to a Bronze Data Set 128 question in FIG. 1, the Conversational Coordinator component 300 cascades to the second Deep Neural Network component which is the IDSF+QVEnum component 321, with the question, i.e. utterance.
[0082] Within the the IDSF+QVEnum component 321, the IDSF Model 320 predicts a known intent from the utterance and predicts query values as utterance phrases. The QVEnum Model 322 predicts any enumerated column values from the utterance phrases. The Query Template 324 corresponding to the predicted intent and the set of Query Values 324 are then sent to the Query Values Validator component 410 across the Demarcation Zone (DMZ) 500. The Query Values Validator component 410 checks Query Values 324 against corresponding column values in the Business Database 424 and generates a list of suggested values to be shown to the User 200 if a value in not found.
[0083] For instance, if Query Values 324 are {$BUYER NAME=“Leo Swanson”}, the Query Values Validator component 410 first looks if a “Leo Swanson” is in the Business Database 424. If instead of “Leo Swanson” the database returns a “Leonard Swanson”, the Query Values Validator substitutes “Leonard Swanson” for “Leo Swanson”. The Query Builder component 420 is then invoked to produce the Query 422 by combining the Query Template and validated, fixed query values. The Query 422 is executed against the Business Database 424 by the Results Generator 426 and the results are displayed to the User 200 using the Data Visualization Tool component 440.
[0084] Still referring to FIG. 2, in one embodiment of the present invention, if the QQP component 331 does not match the user question to a Bronze Data Set 128 question in FIG. 1, and the IDSF+QVEnum component 321 fails to predict a known intent, the Conversational Coordinator component 300 cascades next to the third Deep Neural Network component which is the NL2Query component 311 with the question, i.e. utterance.
[0085] The NL2Query component 311 uses the NL2Query Model 310 to generate a Query Template 312 that it expects to answer the user's question. The Query Values Generator component 400 predicts phrases from the utterance that can be used as Query Values. The Query Builder component 420 is then invoked to produce the Query 422 by combining the Query Template 312 and Query Values produced by the Query Values Generator component 420. The Query 422 is executed against the Business Database 424 by the Results Generator 426. However, the NL2Query Model 310 and the Query Values Generator component 420 are not reliable enough to show the results directly to the User 200. So, the question, the Query 422 and the results are sent to the Business Analyst Portal 432 instead. The Business Analyst 430 approves or rejects the question, inspects the query and results, and makes any necessary changes to the Query 422. The approved Query 422 is then sent to the Results Generator 426, where the Query 422 is executed against the Business Database 424 by the Results Generator 426 and the results are displayed to the User 200 using the Data Visualization Tool component 440.
[0086] Still referring to FIG. 2, in one embodiment of the present invention, Business Analysts 430 may interact with Users 200 to approve or deny a natural language question if the IDSF Model 320 indicates that the question does not map to a known intent. In such cases, Business Analysts 430 judge the semantic appropriateness of a question. For example, a Business Analyst 430 would deny a question like “Is there a god?” on the grounds that it knows that the Business Database 424 does not contain information to answer that question. In addition to interacting with Users 200, the Business Analysts 430 interacts with various components of the system including but not limited to the Business Analyst portal 432, the Business Database 424, and the Result Generator component 426.
[0087] Now referring to FIG. 1, in one embodiment of the present invention, user questions and corresponding Business Analyst 430 in FIG. 2 validated queries are logged and periodically sent to the Data Analyst 100. The Data Analyst 100 uses the Schema Portal 102 to add the question and query pairs to the Gold Data Set 122 and triggers the process of Intent Discovery 124 and model training 130, 132, 134 and 136. Once new models are trained and enabled for use, the system can answer questions similar to the newly added set of questions.
[0088] Still referring to FIG. 1, in one embodiment of the present invention, in case the schema for the Business Database 424 changes in FIG. 2, the Data Analyst 100 updates Schema Description 112 using the Schema Portal 102. Then Bronze Data Set 128 and Gold Data Set 122 are updated and DNN model training 126 invoked. Once new models are trained and enabled for use, the system can generate Queries 422 of FIG. 2 conforming to the updated schema for the Business Database 424 of FIG. 2.
[0089] The foregoing descriptions, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the invention. However, it will be apparent to one skilled in the art that specific details are not required in order to practice the invention. Thus, the foregoing descriptions of specific embodiments of the invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in view of the above teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications, they thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.
