Computer-implemented system and method for generating document groupings for display

Abstract

A computer-implemented system and method for generating document groupings is provided. A lexicon of terms extracted from a set of documents is generated. The lexicon includes a frequency of each extracted term within each document in the set. Concepts each having two or more of the extracted terms are generated. A subset of the documents in the set is selected based on the term frequencies. The subset of documents is grouped into clusters based on the concepts. A similarity of each document cluster is calculated with one or more documents based on a distance by summing the frequency of each term in that document and a weight of the cluster for each of the terms. The weights are updated until a rate of change for each cluster becomes constant.

Claims

1. A computer-implemented system for generating document groupings, comprising: a database to store a set of document, a lexicon of terms extracted from the set of documents and comprising a frequency of each extracted term within each document, and concepts each comprising two or more of the extracted terms; and a server comprising a central processing unit, memory, an input port to receive the documents, lexicon and concepts from the database, and an output port, wherein the central processing unit is configured to: select a subset of the documents in the set based on the term frequencies; group the subset of documents into clusters based on the concepts; calculate a similarity of each document cluster with at least one document based on a distance by summing the frequency of each term in that document and a weight of the cluster for each of the terms; and update the weights until a rate of change for each cluster becomes constant.

2. A system according to claim 1, wherein the central processing unit further identifies one or more of the clusters as meaningful by defining a variance, selects those clusters having one or more documents with inner products that fall within the variance as meaningful, and presents those documents identified as meaningful.

3. A system according to claim 1, wherein the central processing unit further represents semantic content of each document by mapping the terms in that document in order of decreasing frequency.

4. A system according to claim 1, wherein the central processing unit further represents latent semantics of the document set by mapping each term in the document set against a total frequency occurrence, wherein the total frequency occurrence is calculated as a sum of the term frequencies within each document in the set.

5. A system according to claim 4, wherein the central processing unit further selects a median value and edge conditions for the total frequency occurrences of the terms and generates the subset of documents from those documents in the set that satisfy the edge conditions.

6. A system according to claim 5, wherein the central processing unit further re-centers the median value and to generate a different subset of documents for grouping.

7. A system according to claim 5, wherein the central processing unit further sets the edge conditions based on a size of the documents.

8. A system according to claim 7, wherein larger documents have tighter edge conditions than shorter documents.

9. A system according to claim 1, wherein the central processing unit further calculates the distance using the following equation: $d_{\mathrm{cluster}}=\underset{i.fwdarw.n}{.Math.}{\mathrm{doc}}_{{\mathrm{term}}_i}.Math.{\mathrm{cluster}}_{{\mathrm{term}}_i}$ where doc.sub.term represents the frequency of a given term i in one such document and cluster.sub.term represents the weight of a given cluster for a given term i.

10. A system according to claim 1, wherein the central processing unit further calculates the rate of change by determining a first derivative of the inner products over successive iterations.

11. A computer-implemented method for generating document groupings, comprising: generating a lexicon of terms extracted from a set of documents and comprising a frequency of each extracted term within each document; generating concepts each comprising two or more of the extracted terms; selecting a subset of the documents in the set based on the term frequencies; grouping the subset of documents into clusters based on the concepts; calculating a similarity of each document cluster with at least one document based on a distance by summing the frequency of each term in that document and a weight of the cluster for each of the terms; and updating the weights until a rate of change for each cluster becomes constant.

12. A method according to claim 11, further comprising: identifying one or more of the clusters as meaningful, comprising: defining a variance; and selecting those clusters having one or more documents with inner products that fall within the variance as meaningful; and displaying those documents identified as meaningful.

13. A method according to claim 11, further comprising: representing semantic content of each document by mapping the terms in that document in order of decreasing frequency.

14. A method according to claim 11, further comprising: representing latent semantics of the document set by mapping each term in the document set against a total frequency occurrence, wherein the total frequency occurrence is calculated as a sum of the term frequencies within each document in the set.

15. A method according to claim 14, further comprising: selecting a median value and edge conditions for the total frequency occurrences of the terms; and generating the subset of documents from those documents in the set that satisfy the edge conditions.

16. A method according to claim 15, further comprising: re-centering the median value; and generating a different subset of documents for grouping.

17. A method according to claim 15, further comprising: setting the edge conditions based on a size of the documents.

18. A method according to claim 17, wherein larger documents have tighter edge conditions than shorter documents.

19. A method according to claim 11, further comprising: calculating the distance using the following equation: $d_{\mathrm{cluster}}=\underset{i.fwdarw.n}{.Math.}{\mathrm{doc}}_{{\mathrm{term}}_i}.Math.{\mathrm{cluster}}_{{\mathrm{term}}_i}$ where doc.sub.term represents the frequency of a given term i in one such document and cluster.sub.term represents the weight of a given cluster for a given term i.

20. A method according to claim 11, further comprising: calculating the rate of change by determining a first derivative of the inner products over successive iterations.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram showing a system for dynamically evaluating latent concepts in unstructured documents, in accordance with the present invention.

(2) FIG. 2 is a block diagram showing the software modules implementing the document analyzer of FIG. 1.

(3) FIG. 3 is a process flow diagram showing the stages of text analysis performed by the document analyzer of FIG. 1.

(4) FIG. 4 is a flow diagram showing a method for dynamically evaluating latent concepts in unstructured documents, in accordance with the present invention.

(5) FIG. 5 is a flow diagram showing the routine for performing text analysis for use in the method of FIG. 4.

(6) FIG. 6 is a flow diagram showing the routine for creating a histogram for use in the routine of FIG. 5.

(7) FIG. 7 is a data structure diagram showing a database record for a concept stored in the database 30 of FIG. 1.

(8) FIG. 8 is a data structure diagram showing, by way of example, a database table containing a lexicon of extracted concepts stored in the database 30 of FIG. 1.

(9) FIG. 9 is a graph showing, by way of example, a histogram of the frequencies of concept occurrences generated by the routine of FIG. 6.

(10) FIG. 10 is a table showing, by way of example, concept occurrence frequencies generated by the routine of FIG. 6.

(11) FIG. 11 is a graph showing, by way of example, a corpus graph of the frequency of concept occurrences generated by the routine of FIG. 5.

(12) FIG. 12 is a flow diagram showing a routine for creating a matrix for use in the routine of FIG. 5.

(13) FIG. 13 is a table showing, by way of example, the matrix of themes generated by the routine of FIG. 12.

(14) FIG. 14 is a flow diagram showing a routine for determining results for use in the routine of FIG. 5.

DETAILED DESCRIPTION

Glossary

(15) Keyword: A literal search term which is either present or absent from a document. Keywords are not used in the evaluation of documents as described herein. Term: A root stem of a single word appearing in the body of at least one document. Phrase: Two or more words co-occurring in the body of a document. A phrase can include stop words. Concept: A collection of terms or phrases with common semantic meanings Theme: Two or more concepts with a common semantic meaning Cluster: All documents for a given concept or theme.
The foregoing terms are used throughout this document and, unless indicated otherwise, are assigned the meanings presented above.

(16) FIG. 1 is a block diagram showing a system 11 for dynamically evaluating latent concepts in unstructured documents, in accordance with the present invention. By way of illustration, the system 11 operates in a distributed computing environment 10 which includes a plurality of heterogeneous systems and document sources. The system 11 implements a document analyzer 12, as further described below beginning with reference to FIG. 2, for evaluating latent concepts in unstructured documents. The system 11 is coupled to a storage device 13 which stores a document warehouse 14 for maintaining a repository of documents and a database 30 for maintaining document information.

(17) The document analyzer 12 analyzes documents retrieved from a plurality of local sources. The local sources include documents 17 maintained in a storage device 16 coupled to a local server 15 and documents 20 maintained in a storage device 19 coupled to a local client 18. The local server 15 and local client 18 are interconnected to the system 11 over an intranetwork 21. In addition, the document analyzer 12 can identify and retrieve documents from remote sources over an internetwork 22, including the Internet, through a gateway 23 interfaced to the intranetwork 21. The remote sources include documents 26 maintained in a storage device 25 coupled to a remote server 24 and documents 29 maintained in a storage device 28 coupled to a remote client 27.

(18) The individual documents 17, 20, 26, 29 include all forms and types of unstructured data, including electronic message stores, such as electronic mail (email) folders, word processing documents or Hypertext documents, and could also include graphical or multimedia data. Notwithstanding, the documents could be in the form of structured data, such as stored in a spreadsheet or database. Content mined from these types of documents does not require preprocessing, as described below.

(19) In the described embodiment, the individual documents 17, 20, 26, 29 include electronic message folders, such as maintained by the Outlook and Outlook Express products, licensed by Microsoft Corporation, Redmond, Wash. The database is an SQL-based relational database, such as the Oracle database management system, release 8, licensed by Oracle Corporation, Redwood Shores, Calif.

(20) The individual computer systems, including system 11, server 15, client 18, remote server 24 and remote client 27, are general purpose, programmed digital computing devices consisting of a central processing unit (CPU), random access memory (RAM), non-volatile secondary storage, such as a hard drive or CD ROM drive, network interfaces, and peripheral devices, including user interfacing means, such as a keyboard and display. Program code, including software programs, and data are loaded into the RAM for execution and processing by the CPU and results are generated for display, output, transmittal, or storage.

(21) FIG. 2 is a block diagram showing the software modules 40 implementing the document analyzer 12 of FIG. 1. The document analyzer 12 includes three modules: storage and retrieval manager 41, text analyzer 42, and display and visualization 43. The storage and retrieval manager 41 identifies and retrieves documents 44 into the document warehouse 14 (shown in FIG. 1). The documents 44 are retrieved from various sources, including both local and remote clients and server stores. The text analyzer 42 performs the bulk of the text mining processing. The display and visualization 43 complements the operations performed by the text analyzer 42 by presenting visual representations of the information extracted from the documents 44. The display and visualization 43 can also generate a graphical representation which preserves independent variable relationships, such as described in common-assigned U.S. Pat. No. 6,888,548, issued May 3, 2005, the disclosure of which is incorporated by reference.

(22) During text analysis, the text analyzer 42 identifies terms and phrases and extracts concepts in the form of noun phrases that are stored in a lexicon 18 maintained in the database 30. After normalizing the extracted concepts, the text analyzer 42 generates a frequency table 46 of concept occurrences, as further described below with reference to FIG. 6, and a matrix 47 of summations of the products of pair-wise terms, as further described below with reference to FIG. 10. Similarly, the display and visualization 43 generates a histogram 47 of concept occurrences per document, as further described below with reference to FIG. 6, and a corpus graph 48 of concept occurrences over all documents, as further described below with reference to FIG. 8.

(23) Each module is a computer program, procedure or module written as source code in a conventional programming language, such as the C++ programming language, and is presented for execution by the CPU as object or byte code, as is known in the art. The various implementations of the source code and object and byte codes can be held on a computer-readable storage medium or embodied on a transmission medium in a carrier wave. The document analyzer 12 operates in accordance with a sequence of process steps, as further described below with reference to FIG. 5.

(24) FIG. 3 is a process flow diagram showing the stages 60 of text analysis performed by the document analyzer 12 of FIG. 1. The individual documents 44 are preprocessed and noun phrases are extracted as concepts (transition 61) into a lexicon 45. The noun phrases are normalized and queried (transition 62) to generate a frequency table 46. The frequency table 46 identifies individual concepts and their respective frequency of occurrence within each document 44. The frequencies of concept occurrences are visualized (transition 63) into a frequency of concepts histogram 48. The histogram 48 graphically displays the frequencies of occurrence of each concept on a per-document basis. Next, the frequencies of concept occurrences for all the documents 44 are assimilated (transition 64) into a corpus graph 49 that displays the overall counts of documents containing each of the extracted concepts. Finally, the most relevant concepts are summarized (transition 65) into a matrix 46 that presents the results as summations of the products of pair-wise terms.

(25) FIG. 4 is a flow diagram showing a method 70 for dynamically evaluating latent concepts in unstructured documents 44 (shown in FIG. 2), in accordance with the present invention. As a preliminary step, the set of documents 44 to be analyzed is identified (block 71) and retrieved into the document warehouse 14 (shown in FIG. 1) (block 72). The documents 44 are unstructured data and lack a common format or shared type. The documents 44 include electronic messages stored in messaging folders, word processing documents, hypertext documents, and the like.

(26) Once identified and retrieved, the set of documents 44 is analyzed (block 73), as further described below with reference to FIG. 5. During text analysis, a matrix 47 (shown in FIG. 2) of term-document association data is constructed to summarize the semantic content inherent in the structure of the documents 44. As well, the frequency of individual terms or phrases extracted from the documents 44 are displayed and the results are optionally visualized (block 74). The routine then terminates.

(27) FIG. 5 is a flow diagram showing the routine 80 for performing text analysis for use in the method 70 of FIG. 4. The purpose of this routine is to extract and index terms or phrases for the set of documents 44 (shown in FIG. 2). Preliminarily, each document in the documents set 44 is preprocessed (block 81) to remove stop words. These include commonly occurring words, such as indefinite articles (a and an), definite articles (the), pronouns (I, he and she), connectors (and and or), and similar non-substantive words.

(28) Following preprocessing, a histogram 48 of the frequency of terms (shown in FIG. 2) is logically created for each document 44 (block 82), as further described below with reference to FIG. 6. Each histogram 48, as further described below with reference to FIG. 9, maps the relative frequency of occurrence of each extracted term on a per-document basis.

(29) Next, a document reference frequency (corpus) graph 49, as further described below with reference to FIG. 10, is created for all documents 44 (block 83). The corpus graph 49 graphically maps the semantically-related concepts for the entire documents set 44 based on terms and phrases. A subset of the corpus is selected by removing those terms and phrases falling outside either edge of predefined thresholds (block 84). For shorter documents, such as email, having less semantically-rich content, the thresholds are set from about 1% to about 15%, inclusive. Larger documents may require tighter threshold values.

(30) The selected set of terms and phrases falling within the thresholds are used to generate themes (and concepts) (block 85) based on correlations between normalized terms and phrases in the documents set. In the described embodiment, themes are primarily used, rather than individual concepts, as a single co-occurrence of terms or phrases carries less semantic meaning than multiple co-occurrences. As used herein, any reference to a theme or concept will be understood to include the other term, except as specifically indicated otherwise.

(31) Next, clusters are created (block 86) from groups of highly-correlated concepts and themes. Individual concepts and themes are categorized based on, for example, Euclidean distances calculated between each pair of concepts and themes and defined within a pre-specified range of variance, such as described in commonly-assigned U.S. Pat. No. 6,778,995, issued Aug. 17, 2004, the disclosure of which is incorporated by reference.

(32) A matrix 47 of the documents 44 is created (block 87), as further described below with reference to FIG. 13. The matrix 47 contains the inner products of document concept frequency occurrences and cluster concept weightings mapped into a multi-dimensional concept space for each theme. Finally, the results of the text analysis operations are determined (block 88), as further described below with reference to FIG. 14, after which the routine returns.

(33) FIG. 6 is a flow diagram showing the routine 90 for creating a histogram 48 (shown in FIG. 2) for use in the routine of FIG. 5. The purpose of this routine is to extract noun phrases representing individual concepts and to create a normalized representation of the occurrences of the concepts on a per-document basis. The histogram represents the logical union of the terms and phrases extracted from each document. In the described embodiment, the histogram 48 need not be expressly visualized, but is generated internally as part of the text analysis process.

(34) Initially, noun phrases are extracted (block 91) from each document 44. In the described embodiment, concepts are defined on the basis of the extracted noun phrases, although individual nouns or tri-grams (word triples) could be used in lieu of noun phrases. In the described embodiment, the noun phrases are extracted using the LinguistX product licensed by Inxight Software, Inc., Santa Clara, Calif.

(35) Once extracted, the individual terms or phrases are loaded into records stored in the database 30 (shown in FIG. 1) (block 92). The terms stored in the database 30 are normalized (block 93) such that each concept appears as a record only once. In the described embodiment, the records are normalized into third normal form, although other normalization schemas could be used.

(36) FIG. 7 is a data structure diagram showing a database record 100 for a concept stored in the database 30 of FIG. 1. Each database record 100 includes fields for storing an identifier 101, string 102 and frequency 103. The identifier 101 is a monotonically increasing integer value that uniquely identifies each term or phrase stored as the string 102 in each record 100. The frequency of occurrence of each term or phrase is tallied in the frequency 103.

(37) FIG. 8 is a data structure diagram showing, by way of example, a database table 110 containing a lexicon 111 of extracted concepts stored in the database 30 of FIG. 1. The lexicon 111 maps out the individual occurrences of identified terms 113 extracted for any given document 112. By way of example, the document 112 includes three terms numbered 1, 3 and 5. Concept occurs once in document 112, concept 3 occurs twice, and concept 5 occurs once. The lexicon tallies and represents the occurrences of frequency of the concepts 1, 3 and 5 across all documents 44.

(38) Referring back to FIG. 6, a frequency table is created from the lexicon 111 for each given document 44 (block 94). The frequency table is sorted in order of decreasing frequencies of occurrence for each concept 113 found in a given document 44. In the described embodiment, all terms and phrases occurring just once in a given document are removed as not relevant to semantic content. The frequency table is then used to generate a histogram 48 (shown in FIG. 2) (block 95) which visualizes the frequencies of occurrence of extracted concepts in each document. The routine then returns.

(39) FIG. 9 is a graph showing, by way of example, a histogram 48 of the frequencies of concept occurrences generated by the routine of FIG. 6. The x-axis defines the individual concepts 121 for each document and the y-axis defines the frequencies of occurrence of each concept 122. The concepts are mapped in order of decreasing frequency 123 to generate a curve 124 representing the semantic content of the document 44. Accordingly, terms or phrases appearing on the increasing end of the curve 124 have a high frequency of occurrence while concepts appearing on the descending end of the curve 124 have a low frequency of occurrence.

(40) FIG. 10 is a table 130 showing, by way of example, concept occurrence frequencies generated by the routine of FIG. 6. Each concept 131 is mapped against the total frequency occurrence 132 for the entire set of documents 44. Thus, for each of the concepts 133, a cumulative frequency 134 is tallied. The corpus table 130 is used to generate the document concept frequency reference (corpus) graph 49.

(41) FIG. 11 is a graph 140 showing, by way of example, a corpus graph of the frequency of concept occurrences generated by the routine of FIG. 5. The graph 140 visualizes the extracted concepts as tallied in the corpus table 130 (shown in FIG. 10). The x-axis defines the individual concepts 141 for all documents and the y-axis defines the number of documents 44 referencing each concept 142. The individual concepts are mapped in order of descending frequency of occurrence 143 to generate a curve 144 representing the latent semantics of the set of documents 44.

(42) A median value 145 is selected and edge conditions 146a-b are established to discriminate between concepts which occur too frequently versus concepts which occur too infrequently. Those documents falling within the edge conditions 146a-b form a subset of documents containing latent concepts. In the described embodiment, the median value 145 is document-type dependent. For efficiency, the upper edge condition 146b is set to 70% and the 64 concepts immediately preceding the upper edge condition 146b are selected, although other forms of threshold discrimination could also be used.

(43) FIG. 12 is a flow diagram showing the routine 150 for creating a matrix 47 (shown in FIG. 2) for use in the routine of FIG. 5. Initially, those documents 44 having zero values for frequency counts are removed through filtering (block 151). The inner products of document concept frequency occurrences and cluster concept weightings mapped into a multi-dimensional concept space for each theme are calculated and used to populate the matrix (block 152). The individual cluster weightings are iteratively updated (block 153) to determine best fit. Those documents having the smallest inner products are deemed most relevant to a given theme and are identified (block 154). The routine then returns.

(44) FIG. 13 is a table 170 showing the matrix 47 generated by the routine of FIG. 12. The matrix 47 maps a cluster 171 to documents 172 based on a calculated inner product. Each inner product quantifies similarities between documents, as represented by a distance. The distance is mapped into a multi-dimensional concept space for a given document, as measured by the magnitude of a vector for a given term drawn relative to an angle , held constant for the given cluster.

(45) For a set of n documents, the distance d.sub.cluster is calculated by taking the sum of products (inner product) by terms between document concept frequency occurrences and cluster concept weightings, using the following equation:

(46) $d_{\mathrm{cluster}}=\underset{i.fwdarw.n}{.Math.}{\mathrm{doc}}_{{\mathrm{term}}_i}.Math.{\mathrm{cluster}}_{{\mathrm{term}}_i}$
where doc.sub.term represents the frequency of occurrence for a given term i in the selected document and cluster.sub.term represents the weight of a given cluster for a given term i. The weights of the individual inner products are iteratively updated until the clusters settle. The goal is to calculate the minimum distances between as few clusters as possible until the rate of change goes constant. The rate of change can be calculated, for example, by taking the first derivative of the inner products over successive iterations.

(47) FIG. 14 is a flow diagram showing the routine 180 for determining results for use in the routine of FIG. 5. Duplicate documents 44 are removed from the results (block 181). The results are re-run (block 182), as necessary by repeating the text analysis operations (block 183), beginning with creating the corpus graph 49 (block 84 in FIG. 5). After satisfactory results have been obtained (block 182), the routine returns.

(48) Satisfactory results are shown when a meaningful cluster of documents is found. Objectively, each document within a given theme will have an inner product falling within a pre-defined variance of other related documents, thereby reflecting a set amount of similarity. The cluster itself represents a larger grouping of document sets based on related, but not identical, themes.

(49) If necessary, the results are re-run (block 182). One reason to re-run the results set would be to re-center the median value 145 of the corpus graph 140 (shown in FIG. 11) following the filtering of further documents 44. The filtering of edge condition concept frequency occurrences will cause the curve 144 to be redefined, thereby requiring further processing.

(50) While the invention has been particularly shown and described as referenced to the embodiments thereof, those skilled in the art will understand that the foregoing and other changes in form and detail may be made therein without departing from the spirit and scope of the invention.