COMPUTER READABLE ELECTRONIC RECORDS AUTOMATED CLASSIFICATION SYSTEM

Abstract

Classifying an electronic document in a computer-based system is disclosed. For each classification instance in a plurality of classification instances, a confidence data indicating a degree of confidence that the electronic document is associated with that classification instance is determined. A classification, based on a first classification instance in the plurality of classification instances, is assigned without human intervention to the electronic document if the confidence data associated with the first classification instance exceeds a first threshold.

Claims

1. A method for classifying an electronic document in a computer-based system, comprising: generating a set of confidence data comprising for each of one or more classification instances a corresponding determined confidence data, wherein a confidence data indicates a degree of confidence that the electronic document is associated with a corresponding classification instance; in the event at least one of the confidence data in the set exceeds a first predetermined threshold, assigning a classification to the electronic document without user input; and in the event none of the confidence data in the set exceeds the first predetermined threshold and that at least one of the confidence data in the set exceeds a second predetermined threshold that is lower than the first predetermined threshold, receiving via a user interface a user input indicating a selection of a classification comprising the set and determining, based at least in part on the user input, that the user-selected classification should be assigned to the electronic document, wherein determining the confidence data includes using information obtained by analyzing at least one training document associated with a pre-assigned classification.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0020] FIG. 1 is a functional block diagram of a tree structure depicting a records classification structure including excerpts from an official filing plan for an organization.

[0021] FIG. 2 shows a representative symbol diagram of enterprise records management, file management, and database systems incorporating features of the invention.

[0022] FIG. 3 is a table providing an example of the classification agent evaluation mode output result list.

[0023] FIG. 4 is a decision control table, which is user configurable, to control automated electronic document classification and processing.

[0024] FIG. 5 shows the decision control table of FIG. 4 provided with example confidence factor ranges and illustrating a user configuration where an optional user selected document action is disabled.

[0025] FIG. 6 is a decision flow diagram depicting the process of categorizing a document and performing the actions or operations to the document based on the configuration shown in FIG. 4.

[0026] FIG. 7 is open dialog box representing a user input interface to obtain user input to select a candidate document to be processed by the classification process of FIG. 6.

[0027] FIG. 8 is dialog box representing a user input interface to obtain user input to select the document processing options of decision box 46 of FIG. 6.

DETAILED DESCRIPTION

[0028] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term processor refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.

[0029] A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

[0030] FIG. 1 shows a functional block diagram of a tree structure depicting a records subject classification structure for an organization. The classification structure preferably includes an official file plan. The filing plan forms part of a document classification structure or list. The classification structure provides specific instances of records classifications into which particular documents are classified and into which future documents should be classified The filing plan tree structure includes a root node 10 labelled Official Records. Root node 10 connects to a plurality of classification folders 12. Each classification folder 12 defines document groups or organizational groups, which contain official records of the organization. The classification folders 12 can point to records 18 or to other classification folders, or sub-classification folders, such as the rsums classification folder 14 of the human resources classification folder 12. Classification sub-folders, such as rsums classification folder 14, can in turn point to other classification sub-folders 16 or records 18. For example, rsums classification folder 14 points to classification sub-folders 16. Classification sub-folders 16, contains two classification folders, one to contain solicited rsums and the other classification folder to contain unsolicited rsums. In the classification tree structure, each classification folder may contain other classification folders or records. The records or documents within a classification folder are collected within or under the classification folder as the records relate to the folder contents. The objective of the classification folder contents, that is to define the retention rules, security and access limitations and like attributes, and definitions of the documents to be placed into a classification folder, are defined by the file management rules of the organisation.

[0031] For example, contracts classification folder 12 contains records 18 corresponding to agreements and contracts that the organization has entered into. The documents contained within the contracts classification folder can be in any suitable computer readable representation of the document such as a text file or even an image file that has an associated text file, for example an optical character recognition (OCR) text file produced from the image contained in the image file. Preferably, the documents maintained within a classification folder are subject to file management rules for the classification folder. The file management rules are specified in the file plan that includes the classification folder. Commonly, the file management rules provide for document retention periods. In the example of FIG. 1, the retention period for contract documents is set to maintain those documents for a period of five years after the contract or agreement expiry period. File management rules of a file plan also generally include other items such as an authority designation to delineate the person or organizational unit responsible for the documents filed within the classification, what constitutes a document that is to be included in the classification, who has the authority to add, change or remove the documents in the classification and other parameters relating to documents kept within the classification. The file plan file management rules generally provide security rules that designate who have the right to access the document. The classification folders also may conveniently include text description to define the subject contents of the documents maintained within the classification folder. For example, the contract subject is exemplified as contracts and agreements, which the organization has entered into.

[0032] Shown in FIG. 1 is a Review Classification folder 21, the function of which will become apparent in the description of the invention; In accordance with the invention, a candidate document is processed by a classification agent and, in certain circumstances which are configured by the user, the candidate document presented to the classification agent will be automatically placed into the Review Classification folder 21. Documents in the Review Classification folder 21 are available for subsequent inspection and review by a designated user. In FIG. 1, the available documents to be reviewed 23 are graphically depicted the box symbols labelled as RA and RB.

[0033] FIG. 2 shows a representative symbol diagram of enterprise records management, file management, and database systems incorporating features of the invention. The enterprise systems for providing sources of and electronic data repositories for electronic records that an enterprise may have available to it are numerous. For example, one source of electronic records includes e-mail that is accessed by users connected to, or in communication with, a mail server 20. Other electronic records data repositories include various enterprise resource planning systems (ERP) database systems such as a SAP database 22 maintained using the proprietary systems of SAP AG of Germany, for example. Other systems for providing storage of and access to data records of an enterprise are shown representatively by the generic reference to an electronic data management system or EDMS 24.

[0034] In the preferred embodiment, an enterprise document server 26 accesses the data contained in these electronic document repositories. The enterprise document server 26 controls the appearance of the electronic document data in an enterprise records database 28. The appearance of a record in the enterprise document server 26 can be accomplished by copying the document from one of the source repositories 20, 22 or 24 into the enterprise records database 28. When a copy of the source document is taken, the source document in repository 20, 22 or 24 can remain, or the source document can be deleted from the source repository 20, 22 or 24 with the only copy remaining in the enterprise records database 28. Alternately, a pointer reference record can be inserted into the enterprise records database 28 that points or refers to the source document in its native repository 20, 22 or 24. When a pointer reference record is inserted into the enterprise document database 28, the enterprise document server 26 will use the pointer reference record on subsequent access requests for that document to obtain the document from the source repository 20, 22 or 24 as located by the pointer reference record. It will be understood by those skilled in the art that the presence of an electronic document in the enterprise records database will enable an ERS file plan to be implemented by applying the file plan to the enterprise records database 28 for automated management by the enterprise document server 26.

[0035] The processing performed by enterprise document server 26 is directed by using various forms of user input, depicted in the figure as control input 30 and which will be described in more detail subsequently. Enterprise document server 26 controls electronic document process flow to classification agent 32 and the process flow is based on user control input 30. When the classification agent, 32 is invoked by the enterprise document server 26, the text of a document is provided to classification agent 32. Classification agent 32 operates in two basic modes, namely, learning mode and evaluation or classification mode. When classification agent 32 is provide with the text of a document, which is passed to it by the enterprise document server 26, the classification agent will be instructed to process the document text in either the learning mode or the evaluation mode. Classification agent 32 will be instructed to process the contents of the electronic document in the manner directed by the enterprise document server 26.

User Control

[0036] FIG. 2 shows user control input 30 as a separate control input to enterprise document server 26 independent of the various repositories 20, 22, 24 and 28, where an electronic document is stored. However, it will be understood by those skilled in the art that control input 30 can originate from user interaction or user initiated processes when operating any one of these repositories 20, 22, 24 and 28. Using a distributed processing environment, such as the Component Object Model (COM) paradigm available from Microsoft Corporation, and the Object Linking and Embedding (OLE) functionality and facilities of the Microsoft Windows (trademark) operating system, an application programming interface (API) for the enterprise document server 26 can enable user control input 30 to be provided to the enterprise document server 26 from numerous other client applications. For example, the user interface to mail system or server 20 can be adapted to include facilities for generating control input to enterprise document server 26 by a user operating the email system when reviewing email documents.

Training Mode

[0037] The classification agent 32 is operable in two modes, one of which is a training mode or a learning mode. In the training mode, the text contents of a document are passed to the classification agent 32 together with a pre-assigned classification instance, which corresponds to or has been associated with the document. One way to effect training of the classification agent is to traverse a classification structure, for example the tree structure depicted in FIG. 1 and provide the classification agent 32 with each document together with the classification instance from which the document was retrieved. For classification agent training, a suitable classification structure selected to train the classification agent has several documents populating each node or folder of the classification structure. There must be at least 3 or 5 documents per classification instance and preferably numerous documents, such as at least 20 or more, depending on the algorithm employed by the classification agent.

[0038] There are several computer based algorithms that are suitable to perform the function of the classification agent, including: neural networks, document key word indexing algorithms providing word tuples or statistical analysis of document key words and word tuples. For each training document, the classification agent algorithm processes the text contents of the training document along with being provided with the classification instance assigned to the document. Using these inputs, the classification agent builds an association or preference between the document contents and the pre-assigned classification which is stored in a file plan taxonomy database 34. Preferably, the classification agent will also build a disassociation, or preference to exclude, the other classification instances to which the training document does not belong. Thus, in training mode, the classification agent training will develop state information establishing a probabilistic association or linkages between classification instances and document contents the result of which is kept in a data file which is referred to as the file plan taxonomy database 34. This stored probabilistic association will form the basis for assigning a classification instance and a confidence factor to subsequently presented documents that do not have assigned classifications. This subsequent candidate document classification instance assignment is the other mode of operation of the classification agent, and is called the evaluation or classification mode.

Evaluation Mode

[0039] In the other mode of operation, the evaluation mode or classification mode, classification agent 32 is provided with the text contents of a candidate document or record. When classification agent 32 is in the evaluation mode, the classification agent will evaluate the text contents of the candidate document or record with reference to past training data contained in the file plan taxonomy database 34 to produce a result list of classification instances for the candidate document. For each classification instance in the result list, the classification agent provides a numeric result establishing a probability, or confidence level, to associate the text content of the document with the classification instance.

[0040] FIG. 3 shows a table providing an example of the classification agent evaluation mode output result list. In this example, the classification agent has processed a candidate document and produced a result list containing three classification instances for the document, namely the contracts, solicited and unsolicited classification instances or folders. This result list is shown in the three rows of the table of FIG. 3. For each row of the classification agent return result list, there is also provided a confidence level that has been produced by the classification agent. The confidence level is a numeric representation falling within a pre-determined range. In the preferred embodiment, the larger the number in the numeric representation the better the association or confidence level the classification agent places in the association between the content of the candidate document and the indicated classification instance or folder name. For example, the classification agent has provided a confidence level of 93% for the contracts classification instance and a confidence level of 35% for the solicited classification instance and 5% for the unsolicited classification instance. Thus, in the evaluation mode, the operation of the classification agent is to produce a list of classification instances together with a confidence level for each classification instance based on the contents of a candidate document. For a given candidate document, the output of the classification agent can include a null list or a table that has no row entries. In such a return, the classification agent has no classification, which it associates with the text contents of the candidate document. A functional equivalent to a null table result would be a return table that contains one row, which assigns a confidence level of zero to the classification instance of the row.

[0041] FIG. 4 depicts a decision control table, which provides the user with the ability to configure automated electronic document classification and processing based on the confidence factor output from the classification agent. The table provides for up to 4 actions or cases that can be made in response to the confidence factor produced for a candidate document by the classification agent. Row 1, indicated by the entry 1 in the Case column, provides a user configurable lower limit aa.a % in the From: column which defines the lowest value of a confidence factor produced by the classification agent that will result in the action specified in the Action column being taken. In the preferred embodiment, a confidence factor of 100% is the largest value of a confidence factor, which can be produced, and 0% is the smallest value. Of course, other ranges can be used, which can be numeric or even alphabetic symbols or other forms to express a range of values. The highest value of a confidence factor that can be produced by the classification agent is 100% which is the highest value of a confidence factor that is specified in the To column for case 1. Therefore, when the classification agent returns a value within the range specified for case one, namely aa.a % to 100%, then the action specified in the Action column will be taken for the document. The case 1 action is the automatic classification option. Therefore, candidate documents which meet the user configurable confidence factor range specified for case 1 will be automatically placed into the enterprise records database 28 by the enterprise data server 26.

[0042] For case 2, the confidence factor returned by the classification agent falls within the range specified as bb.b % to aa.a-0.1%. The upper bound of the case 2 range is less than the lower bound of case 1. There is no overlap or gap between the upper bound of the range of case 2 and the lower bound of the range of case 1. These two ranges, and all of the ranges, are contiguous. In the preferred embodiment, the confidence factor for each classification produced by the classification agent is selected from a universe of the 1,001 values represented by a single decimal point number having a value between 0.0 and 100.0 inclusive. Other confidence factor value universes could be provided with suitable changes to the case selection confidence factor ranges. In the preferred embodiment, case 2 actions are optional. That is, the user can configure the confidence factor range associated with this case to prevent this case action from being taken.

[0043] For case 3, the confidence factor returned by the classification agent falls within the range specified as cc.c % to bb.b-0.1%. The upper bound of the case 3 range is less than the lower bound of case 2. There is no overlap or gap between the upper bound of the range of case 3 and the lower bound of the range of case 2. As previously stated, these two ranges, and all of the ranges, are contiguous. For case 3, the action taken in relation to the document presented is to place the document in the Review Classification folder 21. Documents in the Review Classification folder 21 are documents which may be records that should be placed into the enterprise records database 28 but which require review by a user to determine whether the document is such a record, and, if so, what classification the document should be assigned to. In the preferred embodiment, case 3 actions are optional. That is the user can configure the confidence factor range associated with this case to prevent this case action from being taken.

[0044] For case 4, no action will be taken for the document in relation to the enterprise document database 28. If no action is selected then the inventive system preferably produces a message to confirm that the document has been reviewed by the system and the system review result is that the document does not require placement into the enterprise records database 28. No action may be confirmed, for example, by producing a confirmatory message such as: This Document Does Not Meet the Criteria to become an Official Record. A no action confirmatory message is preferable to confirm that the system received and processed the candidate electronic document. The no action confirmatory message provides an indication that the processing result for the candidate document is a confidence factor of zero percent or a confidence factor that is within the zero to cc.c-0.1% range of case 4. Additionally, the user can manually assign the document to a records subject category or select the Review Classification folder 21 and have the document filed as a record into either of these choices. This option is made available to process documents, which may be non-textual in content but nevertheless should become an official record or for documents that may be of a new currently untrained category that was recently added to the records file plan.

[0045] FIG. 5 shows the decision control table of FIG. 4 provided with example confidence factor ranges and illustrating a user configuration where optional case 3 action is disabled. The user has configured the decision control table action cases by providing numeric ranges that specify the confidence factor results produced by classification agent 32 which will result in the specified actions being taken for candidate documents presented to the classification agent operating in classification or evaluation mode. For confidence factor return results in the range 85.0 to 100, the system will process the candidate document to file the document to the enterprise records database server under the classification instance that resulted in the confidence factor within the range specified. This classification instance would be the row 1 entry in the classification agent return result list exemplified in the table of FIG. 3. The case 2 optional action, which in this figure is described as Present top 3-5 Categories for user selection has been disabled as indicated by the blank entries for the confidence factor range limits. The case 3 optional action is enabled and active for classification agent return result confidence factors falling in the 20.0 to 84.9 range. The case 4 candidate document action is selected when the candidate document returns a classification agent return result confidence factor, which is in the range 0 to 19.9.

[0046] FIG. 6 shows a decision flow diagram depicting document process flows and database interactions in relation to the classification agent when operating in the training or retraining mode as shown in the portion of the diagram within the dotted line box 35 and in the evaluation or classification mode in the balance of the diagram. With reference to the user control input 30 depicted in FIG. 2, a user selects a candidate document to be processed at step 36. The submission of candidate documents, or selection of candidate documents as depicted by step 36 of FIG. 6 can be accomplished using several different options known to those skilled in the art. For example, the application can be provided with a file-open dialogue box, such as that shown in FIG. 7, to prompt the user for input to select a file. Another way to select a file for submission is to supply the file name as a command-line parameter when the classification agent evaluation or classification process task of FIG. 6 is initiated. This method of submission allows advance users to incorporate elaborate techniques for document submission such as using batch files. Another method of submission is to provide the classification agent evaluation or classification process task of FIG. 6 as a COM object which can be activated when the user is working with another application. For example by incorporating the Microsoft Corporation Explorer Shell Extension Interface in the COM object implementation of the classification agent evaluation or classification process task to enable the user to initiate the classification agent evaluation or classification process when operating the Explorer feature of Microsoft Windows (trademark).

[0047] The selected candidate document is presented to the classification agent (32 of FIG. 2) for classification processing as depicted in process box 38. The classification agent examines the content of the document with reference to historical data maintained in the file plan taxonomy database 34 to produce a result list which forms the basis on which the decisions depicted in decision box 40 are taken. The decision box 40 depicts the document processing actions one of which is selected based on the result returned by the classification agent the user configuration of the decision control table shown in FIG. 4. If the return result of the classification agent is within the Case 1 range, then the Case 1 exit path is taken and the document is inserted in the Enterprise database 28. If the classification agent result compared to the decision table parameter configuration is within a range to select the Case 2 outcome, then a list of classification instances will be presented to the user to select from as depicted in the process box 42 of the Case 2 outcome path from decision box 42. With a Case 2 outcome, the user can provide an input to indicate what disposition is to be made of the document when the user input is evaluated at decision box 46. As indicated by the outcome paths of decision box 46, the user input can choose to Cancel further processing of the document which will have the effect of leaving the candidate document selected at box 36 unchanged but will not place a copy of that document into the enterprise database 28. Alternately, the user input can choose to delete the document which will have the effect of deleting the candidate document selected at box 36 and, naturally, will not place a copy of the deleted candidate document into the enterprise database 28. As well the user input can choose to Submit the candidate document selected at 36 to the enterprise database 28.

[0048] If the classification agent result falls within the configured decision table range to route the candidate document selected at 36 to the classification review folder 21, then the Case 3 exit will be taken from decision box 40 and a copy of the candidate document will be placed into the classification review folder 21.

[0049] If the classification agent result falls within the configured decision table range to rejecte the document, then the Case 4 exit will be taken from decision box 40 and a message will be produced confirming that the document has been reviewed but will not be processed as shown by the report box 42 inscribed with Not an Official Record.

[0050] FIG. 6 also depicts the relationship between the enterprise records database 28 in FIG. 2, the file plan taxonomy engine and the classification agent 32 during the training mode of operation as shown in the diagram area surrounded by dotted box 35.

[0051] FIG. 7 is open dialog box representing a user input interface to obtain user input to select a candidate document to be processed by the classification process of FIG. 6. A file list 50 is presented in a list box area 52. The user can select a particular file from file list 50 using a computer mouse or keyboard. Once the desired file has been selected by the user, the Open control button 54 is activated causing the selected file to be processed. The file selection process can be cancelled, if desired, by selecting the Cancel control button 56.

[0052] FIG. 8 is dialog box representing a user input interface to obtain user input to select the document processing options of decision box 46 of FIG. 6. In this dialog box, the path name of the candidate document or file being processed is identified in the banner area 58 of the dialog box, namely: c:.backslash.fmtemp.backslash.legal-Incorp-1000-76.txt. The processing of the file by the classification agent 38 has resulted in two classifications being proffered for the candidate document, the higher confidence level of 76.8 being assigned to the FileID or classification instance of Softdev-prov-1100 and the next higher confidence level of 64.0 being assigned to the classification instance of Softdev-prov-1000. For further guidance of the user that is to select the proper classification from the proffered list, each classification instance includes an associated Subject which is a text describing the contents of the particular classification instance. The user can select the appropriate classification instance from the list using the mouse or keyboard and initiate further processing of the candidate document by selecting a disposition button. For example, once the desired classification instance has been selected, the user can activate the Submit button to cause the document to be submitted to the enterprise database 28 which is the Submit disposition of the decision box 46 of FIG. 6. Alternately, the user may select the Delete control button 60 to delete the candidate document or the Cancel button to abort further processing of the document.

Retraining

[0053] As will be understood, entries in the enterprise records database 28 can be used as a document collection that can be used to effect the training mode operation of the classification agent 32. To begin using the system, a sample records database can be used as was described previously with reference to the description of the training mode operation of the classification agent 32. However, as the system operates and the enterprise records database 28 becomes populated with more and more records, the enterprise records database 28 itself can be provided to the classification agent operating in training mode to retrain the classification agent based on a larger and larger database to refine the ability of the classification agent to classify candidate documents in the evaluation mode. The benefit of retraining the classification agent will be improved automated classification of candidate documents as well as to enable the classification agent to accommodate new classifications or reclassifications of records.

[0054] For example, the classification agent may produce significant numbers of case 3 file to Classification Review folder results that will cause such documents to be placed into the Classification Review folder 21 and require review by assigned users. Once the documents placed into the Classification Review folder have been reviewed and filed to existing or to newly established classifications, the classification agent can then be activated in training mode to enable the classification agent to incorporate the filing classification that was made to the documents it had previously filed to the Classification Review folder. As a result of this retraining, classification agent 32 can develop a probabilistic association to form the basis for assigning a classification instance and a confidence factor based on the document classifications that were effected by user review of the records in the Classification Review folder. After retraining, operation of the classification agent in evaluation mode will tend to decrease the number of documents that are placed into the Classification Review folder.

[0055] As will be understood from the above, the particular language of the documents presented to the system for training and classification is not a limitation of the system, which relies on the text contents of the documents. Thus the text of the documents may be in any language and, consequently, the operation of the invention is language independent and not restricted or limited to any particular language such as English, French, or German.

[0056] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.