An embodiment of the present invention is a prime representation data structure in a computer architecture. The prime representation data structure has a plurality of records where each record contains a prime representation and where the prime representation is a product of two or more selected prime factors. Each of the selected prime factor associated with an n-gram of a domain representation of a domain string. The domain representation of the domain string is a domain string of ordered, contiguous domain characters. The n-gram being a subset of n number of the ordered, contiguous domain characters in the domain string. The computer architecture performs string searching and includes one or more central processing units (CPUs) with one or more operating systems, one or more input/output device interfaces, one or more memories, and one or more input/output devices. The architecture further includes the prime representation data structure, one or more prime target query data structures and a search process performed by one or more of the CPUs. The CPUs can be organized in a hierarchical structure. The prime target query data structure has one or more target prime queries. Each target prime query is the product of one or more target selected prime factors. Each target selected factor is associated with a target n-gram of a target domain representation of a target domain string. The search process, performed by one or more of the CPUs, determines whether one or more of the target selected prime factors is common with one of the selected prime factors. By performing this efficient testing, the computer system can determine if one or more small strings are included in one or more large strings.

An embodiment of the present invention is a prime representation data structure in a computer architecture. The prime representation data structure has a plurality of records where each record contains a prime representation and where the prime representation is a product of two or more selected prime factors. Each of the selected prime factor associated with an n-gram of a domain representation of a domain string. The domain representation of the domain string is a domain string of ordered, contiguous domain characters. The n-gram being a subset of n number of the ordered, contiguous domain characters in the domain string. The computer architecture performs string searching and includes one or more central processing units (CPUs) with one or more operating systems, one or more input/output device interfaces, one or more memories, and one or more input/output devices. The architecture further includes the prime representation data structure, one or more prime target query data structures and a search process performed by one or more of the CPUs. The CPUs can be organized in a hierarchical structure. The prime target query data structure has one or more target prime queries. Each target prime query is the product of one or more target selected prime factors. Each target selected factor is associated with a target n-gram of a target domain representation of a target domain string. The search process, performed by one or more of the CPUs, determines whether one or more of the target selected prime factors is common with one of the selected prime factors. By performing this efficient testing, the computer system can determine if one or more small strings are included in one or more large strings.

A question answering method that is performed by a question answering apparatus includes: receiving a data set including video content and question-answer pairs; generating input time-series sequences from the video content of the input data set and also generating a question-answer time-series sequence from the question-answer pair of the input data set; calculating weights by associating the input time-series sequence with the question-answer time-series sequence and also calculating first result values by performing operations on the calculated weights and the input time-series sequences; calculating second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers; and calculating third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags and selecting a final answer based on the third result values.

A question answering method that is performed by a question answering apparatus includes: receiving a data set including video content and question-answer pairs; generating input time-series sequences from the video content of the input data set and also generating a question-answer time-series sequence from the question-answer pair of the input data set; calculating weights by associating the input time-series sequence with the question-answer time-series sequence and also calculating first result values by performing operations on the calculated weights and the input time-series sequences; calculating second result values by paying attention to portions of the input time-series sequences that are directly related to characters appearing in questions and answers; and calculating third result values by concatenating the time-series sequences, the first result values, the second result values, and Boolean flags and selecting a final answer based on the third result values.

A mechanism is provided for implementing an optical character recognition (OCR) error correction mechanism for correcting OCR errors. Responsive to receiving a document in which OCR has been performed, the mechanism assesses the document to identify a set of OCR errors generated by an OCR engine that performed the OCR using a set of visual embeddings. Responsive to identifying the set of OCR errors, the mechanism analyzes each character of a plurality of sentences within the document to generate a high-dimensional embedding for the characters of the plurality of sentences within the document. The mechanism then linguistically corrects each OCR error in the set of OCR error. The mechanism utilizes ground truth information and the set of visual embeddings to verify that character stream is linguistically correct. Responsive to verifying that the character stream is linguistically correct, the mechanism outputs an OCR error corrected document to a user.

A mechanism is provided for implementing an optical character recognition (OCR) error correction mechanism for correcting OCR errors. Responsive to receiving a document in which OCR has been performed, the mechanism assesses the document to identify a set of OCR errors generated by an OCR engine that performed the OCR using a set of visual embeddings. Responsive to identifying the set of OCR errors, the mechanism analyzes each character of a plurality of sentences within the document to generate a high-dimensional embedding for the characters of the plurality of sentences within the document. The mechanism then linguistically corrects each OCR error in the set of OCR error. The mechanism utilizes ground truth information and the set of visual embeddings to verify that character stream is linguistically correct. Responsive to verifying that the character stream is linguistically correct, the mechanism outputs an OCR error corrected document to a user.

[Problem(s)] To provide a text analysis system that is low cost and able to detect text with a normal expressive or structural features.

[Solution] A text analysis system 100 according to the present invention includes a text acquisition portion 110 for acquiring text data; a feature extraction portion 120 for converting the text data acquired by the text acquisition portion 110 into a time series signal to extract a feature from the converted time series signal; a feature storage portion 130 for storing the feature extracted by feature extraction portion 120; and an anomalous text detection portion 140 for detecting anomalous text based on the feature in the feature storage portion 130.

Described herein are systems and methods that provide a natural language processing system (NLPS) that employs compressed word embeddings. An auto-encoder that includes encoder circuitry and decoder circuitry can be used to produce the compressed word embeddings. The decoder circuitry is trained to decompress the word embeddings with reduced or minimal differences between the original uncompressed word embeddings and the corresponding decompressed word embeddings. One or more parameters of the trained decoder circuitry are transferred to the NLPS, where the NLPS is then trained using the compressed word embeddings to improve the correctness of the responses or actions determined by the NLPS.

Described herein are systems and methods that provide a natural language processing system (NLPS) that employs compressed word embeddings. An auto-encoder that includes encoder circuitry and decoder circuitry can be used to produce the compressed word embeddings. The decoder circuitry is trained to decompress the word embeddings with reduced or minimal differences between the original uncompressed word embeddings and the corresponding decompressed word embeddings. One or more parameters of the trained decoder circuitry are transferred to the NLPS, where the NLPS is then trained using the compressed word embeddings to improve the correctness of the responses or actions determined by the NLPS.

A session message generation method, apparatus and device and storage medium. The method includes: acquiring historical session messages between a session robot and a target user in the current man-machine session process and a session reference information set related to a session topic when it is detected that there is a trigger event in the current man-machine session process; determining a target session role of the session robot at the time of performing a message input operation according to the historical session messages and the session reference information set; and generating a target session message corresponding to the target session role based on the historical session messages and the session reference information set, and outputting the target session message.