Evaluating XML full text search
11481439 · 2022-10-25
Assignee
Inventors
Cpc classification
G06F16/81
PHYSICS
International classification
Abstract
Techniques are described to improve query evaluation in computer systems. In an embodiment, a system receives a full text query for evaluation against a collection of hierarchically marked data object sets. The query specifies token(s) and context(s) which indicate hierarchical location(s) to match within a queried hierarchical data structure. To evaluate the query, the system determines a) data object set(s) that contain the query specified token(s) using token list(s), and/or b) data object set(s) that contain the query specified context(s) using label list(s).
Claims
1. A computer-implemented method comprising: receiving a query for evaluation against a collection of hierarchically marked data object sets, wherein the query specifies one or more query tokens and one or more query contexts, each query context, of said one or more query contexts, identifying one or more hierarchical locations to match within a queried hierarchical data structure; identifying first one or more hierarchically marked data object sets in the collection of hierarchically marked data object sets that contain at least one query token of the one or more query tokens by using one or more token lists that specify tokens contained in the collection of hierarchically marked data object sets; identifying second one or more hierarchically marked data object sets in the collection of hierarchically marked data object sets that contain at least one query context of the one or more query contexts by using one or more label lists, the one or more label lists indexing labels, of contexts, contained in the collection of hierarchically marked data object sets; identifying third one or more hierarchically marked data object sets that contain both the at least one query token and the at least one query context by determining each of the third one or more hierarchically marked data object sets are both in the first one or more hierarchically marked data object sets of the collection and in the second one or more hierarchically marked data object sets in the collection; and evaluating the query based at least in part on the third one or more hierarchically marked data object sets of the collection.
2. The method of claim 1, wherein the one or more label lists include mapping information that maps a particular label to one or more locations or one or more location ranges that the particular label appears in the collection of hierarchically marked data object sets.
3. The method of claim 1, wherein the one or more token lists include mapping information that maps a particular token to one or more nodes, one or more locations, or one or more location ranges that the particular token appears in the collection of hierarchically marked data object sets.
4. The method of claim 1, further comprising: identifying, based on the one or more label lists, one or more label locations or one or more label location ranges mapped to the one or more query tokens; identifying, based on the one or more token lists, one or more token locations, or one or more token location ranges mapped to the one or more query tokens; wherein evaluating the query comprises determining whether the one or more token locations or the one or more token location ranges are within the one or more label locations or the one or more label location ranges.
5. The method of claim 1, further comprising: determining, within the third one or more hierarchically marked data object sets, one or more locations of one or more candidate nodes that contain the at least one query token by using the one or more token lists that map the one or more locations of the one or more candidate nodes to the at least one query token; and evaluating the query at least by traversing a tree of nodes, within the third one or more hierarchically marked data object sets, upwardly from the one or more candidate nodes to determine whether the one or more candidate nodes match the at least one query context.
6. The method of claim 5, wherein traversing the tree of nodes upwardly from the one or more candidate nodes, comprises: traversing a node tree index, of the tree of nodes, that stores a plurality of entries comprising, for each candidate node of the one or more candidate nodes, a corresponding entry that specifies a location of the candidate node within the third one or more hierarchically marked data object sets, and a link to an entry for a node that is a parent of said each candidate node.
7. The method of claim 1, further comprising: using a path-based index of the third one or more hierarchically marked data object sets to determine one or more label location ranges for one or more label hierarchies, of the third one or more hierarchically marked data object sets, that match the at least one query context; determining, within the third one or more hierarchically marked data object sets, one or more token locations that contain the at least one query token by using the one or more token lists that map the one or more token locations to the at least one query token; and evaluating the query at least by determining whether the one or more label location ranges contain the one or more token locations.
8. The method of claim 1, further comprising: analyzing a model of the third one or more hierarchically marked data object sets to determine one or more label location ranges for one or more label hierarchies, of the third one or more hierarchically marked data object sets, that match the at least one query context; and determining, within the third one or more hierarchically marked data object sets, one or more token locations that contain the at least one query token by using the one or more token lists that map the one or more token locations to the at least one query token; and evaluating the query at least by determining whether the one or more label location ranges contain the at least one query token.
9. The method of claim 1, further comprising: determining, within the third one or more hierarchically marked data object sets, one or more label location ranges using the one or more label lists that map the at least one query context to the one or more label location ranges; and evaluating the query at least by determining whether the one or more label location ranges contain the at least one query token.
10. The method of claim 1, wherein the one or more token lists further specify one or more storage locations of the one or more query tokens in the collection, and wherein the one or more label lists further specify one or more storage ranges of storage locations of one or more labels in the collection, wherein evaluating the query further comprises: determining fourth one or more documents that contain the at least one query token in the at least one query context from the third one or more hierarchically marked data object sets based at least in part on the one or more storage locations and the one or more storage ranges of storage locations.
11. One or more non-transitory computer-readable media storing a set of instructions, wherein the set of instructions includes instructions, which when executed by one or more hardware processors, cause: receiving a query for evaluation against a collection of hierarchically marked data object sets, wherein the query specifies one or more query tokens and one or more query contexts, each query context, of said one or more query contexts, identifying one or more hierarchical locations to match within a queried hierarchical data structure; identifying first one or more hierarchically marked data object sets in the collection of hierarchically marked data object sets that contain at least one query token of the one or more query tokens by using one or more token lists that specify tokens contained in the collection of hierarchically marked data object sets; identifying second one or more hierarchically marked data object sets in the collection of hierarchically marked data object sets that contain at least one query context of the one or more query contexts by using one or more label lists, the one or more label lists indexing labels, of contexts, contained in the collection of hierarchically marked data object sets; identifying third one or more hierarchically marked data object sets that contain both the at least one query token and the at least one query context by determining each of the third one or more hierarchically marked data object sets are both in the first one or more hierarchically marked data object sets of the collection and in the second one or more hierarchically marked data object sets in the collection; and evaluating the query based at least in part on the third one or more hierarchically marked data object sets of the collection.
12. The one or more non-transitory computer-readable media of claim 11, wherein the one or more label lists include mapping information that maps a particular label to one or more locations or one or more location ranges that the particular label appears in the collection of hierarchically marked data object sets.
13. The one or more non-transitory computer-readable media of claim 11, wherein the one or more token lists include mapping information that maps a particular token to one or more nodes, one or more locations, or one or more location ranges that the particular token appears in the collection of hierarchically marked data object sets.
14. The one or more non-transitory computer-readable media of claim 11, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: identifying, based on the one or more label lists, one or more label locations or one or more label location ranges mapped to the one or more query tokens; identifying, based on the one or more token lists, one or more token locations, or one or more token location ranges mapped to the one or more query tokens; wherein evaluating the query comprises determining whether the one or more token locations or the one or more token location ranges are within the one or more label locations or the one or more label location ranges.
15. The one or more non-transitory computer-readable media of claim 11, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: determining, within the third one or more hierarchically marked data object sets, one or more locations of one or more candidate nodes that contain the at least one query token by using the one or more token lists that map the one or more locations of the one or more candidate nodes to the at least one query token; and evaluating the query at least by traversing a tree of nodes, within the third one or more hierarchically marked data object sets, upwardly from the one or more candidate nodes to determine whether the one or more candidate nodes match the at least one query context.
16. The one or more non-transitory computer-readable media of claim 15, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: traversing a node tree index, of the tree of nodes, that stores a plurality of entries comprising, for each candidate node of the one or more candidate nodes, a corresponding entry that specifies a location of the candidate node within the third one or more hierarchically marked data object sets, and a link to an entry for a node that is a parent of said each candidate node.
17. The one or more non-transitory computer-readable media of claim 11, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: using a path-based index of the third one or more hierarchically marked data object sets to determine one or more label location ranges for one or more label hierarchies, of the third one or more hierarchically marked data object sets, that match the at least one query context; determining, within the third one or more hierarchically marked data object sets, one or more token locations that contain the at least one query token by using the one or more token lists that map the one or more token locations to the at least one query token; and evaluating the query at least by determining whether the one or more label location ranges contain the one or more token locations.
18. The one or more non-transitory computer-readable media of claim 11, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: analyzing a model of the third one or more hierarchically marked data object sets to determine one or more label location ranges for one or more label hierarchies, of the third one or more hierarchically marked data object sets, that match the at least one query context; and determining, within the third one or more hierarchically marked data object sets, one or more token locations that contain the at least one query token by using the one or more token lists that map the one or more token locations to the at least one query token; and evaluating the query at least by determining whether the one or more label location ranges contain the at least one query token.
19. The one or more non-transitory computer-readable media of claim 11, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: determining, within the third one or more hierarchically marked data object sets, one or more label location ranges using the one or more label lists that map the at least one query context to the one or more label location ranges; and evaluating the query at least by determining whether the one or more label location ranges contain the at least one query token.
20. The one or more non-transitory computer-readable media of claim 11, wherein the one or more token lists further specify one or more storage locations of the one or more query tokens in the collection, and wherein the one or more label lists further specify one or more storage ranges of storage locations of one or more labels in the collection, wherein the set of instructions further includes instructions, which when executed by said one or more hardware processors, cause: determining fourth one or more documents that contain the at least one query token in the at least one query context from the third one or more hierarchically marked data object sets based at least in part on the one or more storage locations and the one or more storage ranges of storage locations.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) In the drawings:
(2)
(3)
(4)
(5)
(6)
(7)
DETAILED DESCRIPTION
(8) In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.
(9) General Overview
(10) Processes, machines, and stored instructions are provided for storing posting lists for tokens in XML document(s) and using the posting lists to process queries. A document processor parses XML document(s) and, for each occurrence of a token in the XML document(s), adds an entry to a list for the token. The entry for the token maps the token to document(s), location(s), range(s), or node(s) within the document(s) where the token(s) can be found. The document processor may also detect tag(s) in the XML document(s) and, for each occurrence of a tag, add an entry to a list for the tag. The entry for the tag may also specify document(s), location(s), range(s), or node(s) within the document(s) that are covered by the tag.
(11) A query processor may then receive a full text query for evaluation against XML document(s). The full text query specifies token(s) and context(s). The query processor may determine a first set of one or more documents, location(s), range(s), or node(s) within document(s) that contain the token(s) by using token list(s) that specify such information for the specified token(s). The query processor may also determine a second set of one or more documents, location(s), range(s), or node(s) within document(s) that contain the context(s) by using tag list(s) that specify such information for the specified token(s). Then, the query processor may determine a third set of document(s) or node(s) within document(s) that contain both the token(s) and the context(s) based at least in part on the first set of document(s) or node(s) and the second set of document(s) or node(s). The query may then be evaluated based at least in part on the third set of document(s) or node(s).
(12) In one embodiment, an XML document is parsed, and, for each occurrence of a token in the XML document, a document processor adds an entry to a list for the token The entry maps the token to a location of a parent node that contains the token in the XML document. A query processor may then receive a full text query for evaluation against the XML document. The query processor then uses the token lists to determine location(s) of parent node(s) that contain token(s) specified by the query. The query processor may then traverse a node tree index away from the parent node(s) to determine whether the parent node(s) satisfy context(s) specified by the query. In the same or a different embodiment, the query processor may first determine node(s) or range(s) of locations of the node(s) that match the context(s) specified in the query and then use the token list(s) to determine whether the token(s) are under the determined node(s) or within the range(s) of locations of the node(s).
(13)
(14) First Approach of Scanning Documents Row-by-Row
(15) XQuery FullText has become the recommended standard way of querying XML with full text search support by W3C XQuery working group. The SQL/XML standard, which defines the standard operators in SQL to process XML type in RDBMS, has defined to use XMLQuery( ), XMLExists( ), XMLTABLE construct with XQuery full text to semantically search text within XML documents that are stored in XMLType column of a relational table.
(16) Various techniques described herein allow users to use the standard XMLQuery( ), XMLExists( ), and XMLTABLE constructs in SQL/XML to do xquery full text search over XML.
(17) Therefore, a set of techniques are provided that capture both indexing technique and SQL/XML query compile time and run time optimizations to optimize XQuery Full Text operations used in SQL/XML query that searches XML documents in XMLType storage. These optimization techniques are diagnosable by end users via explain plans and event tracing.
(18) In one example, a table PO is created that has an XMLType column X storing purchaseOrder XML documents:
(19) Create table po (x xmltype);
(20) The user can issue the following SQL SELECT statements with XQuery Full Text search in XMLExists( ) and XMLQuery( ) to search purchaseOrder XML documents.
(21) TABLE-US-00001 Q1: SELECT XMLQUERY(‘for $li in $doc/purchaseOrder/lineItems where $li/description contains text “iPhone 5” return $li’ PASSING PO.X as “doc” RETURNING CONTENT) FROM PO WHERE XMLEXISTS(‘ $doc/purchaseOrder/lineItems[description contains text “iPhone 5” ] ’ PASSING PO.X as “doc”)
(22) According to a first approach, this SELECT statement could be processed separately on a row-by-row basis for each row of XML content from the PO table. A query processor would perform a functional evaluation of the query by compiling the query into an execution plan that uses a document object model (DOM) for each document. For each row, a query processor would run the XQuery full text search in XMLEXISTS( ) to see if the document for that row satisfies the XQuery expression. If a row does satisfy the XQuery expression, then the query processor would return the row by running the XQuery in the XMLQuery( ) operator of the select list on the XML document. In other words, the query or at least a portion thereof is evaluated row-by-row in this example.
(23) Running the XQuery full text search for a given row may involve traversing a DOM tree to first find a purchaseOrder node. The query processor may further search for a lineItem node under each purchaseOrder node that is found in the document, and a description node under each of these lineItem nodes. For each description node that is found, the descendant text nodes are combined into a text object, and the text object is broken up into tokens such as separate words. These tokens are analyzed to determine whether, for example, adjacent tokens contain “iPhone” and “5.” If so, the document may be added to the result list. Other documents may be similarly analyzed for other rows as the query is processed.
(24) Functional evaluation of XQFT according to this first approach may have the following limitations:
(25) Each row may be separately fetched from disk, and XQFT functional evaluation is applied separately to each row. This first approach might not scale if there are many XML documents stored in the base table.
(26) Depending on the XMLtype physical storage model, the XML DOM representation may be constructed for each XML document stored in the table because XQFT functional evaluation uses a DOM mode of XMLType. A DOM construction can be computationally expensive (in terms of processor time, memory, and/or input/output) for XML text storage which XML Parser has to be invoked. The DOM construction can be computationally expensive for XML structured (object relational) storage because XML contents are fetched from various storage tables to construct a DOM.
(27) Other approaches are provided as alternatives to the first approach of doing functional evaluation of XQFT by scanning documents row-by-row.
Example Token and/or Tag Indices Approach
(28) In one embodiment, an XMLFullText index or indices are created that may index not only words, but also XML structures (in the form of element and attributes) and their hierarchical relationships. In this way, the keywords and XML path element/attribute tags can be searched together with hierarchical relationship among them.
(29) In one example, the indices are created by creating and storing separate lists for each keyword or token, and separate lists for each unique tag. The lists may cover content from an entire collection of documents, such as all of the documents in an XMLType column, or a subset of those documents. Each token list may include information that identifies the token, such as “iPhone,” and recorded location(s) that instance(s) of the token appear in document(s), such as location(s) of beginning(s) of the instance(s) of the token. The token lists may also include information such as an identity of a parent node that contains the instance of the token, or a size of the token. The token lists may exclude all other information, such as hierarchical information, tag information, or node navigation information, or the token lists may include additional information such as this. Keeping the sizes of the token lists and tag lists small reduces I/O between the query processor or database server and the storage device(s) that persistently store the indices or memory device(s) that temporarily store the indices, and/or between the storage device(s) and memory device(s). Token list(s) and/or tag list(s) may be included as part of the same database object or may be maintained as separate objects. For example, different tokens and tags may represent different rows in a table that stores token lists and tag lists. In other examples, one table may store token lists, and another table may store tag lists, or each list may have its own table. Each token or tag list may include hits from multiple documents, and/or multiple hits per document. Some of the lists may be document-specific, for example, by providing information about tag(s) or token(s) in a particular document.
(30) A unique tag may be a tag identifier or a tag name. A tag identifier, such as a QName, identifies a unique namespace and a local name, and different local prefixes may be mapped to the same namespace. The namespace may be unique globally among a collection of documents, and the local name may be re-used in different ways among different documents. A table may store a mapping of prefixes to namespaces for different documents, and this mapping may differ for different documents. In the examples below, documents 1 and 2 have different prefixes mapped to the namespace “a.com,” and document 3 has no prefix but maps a tag to the prefix “a.com.”
(31) Document 1
(32) <pfx:po xmlns:pfx=“a.com”>mytext</pfx:po>
(33) Document 2
(34) <xyz:po xmlns:xyz=“a.com”>mytext</xyz:po>
(35) Document 3
(36) <po xmlns=“a.com”>mytext</po>
(37) In an example XQuery, “declare element namesapce uvw=“a.com”;/uvw:po[. contains text “mytext”],” the result should match all documents even though the query references “uvw” instead of “pfx” or “xyz.” The prefix information may or may not be retained in a list for the tag identifier, such as a list for the tag identified by namespace “a.com” and local name “po.”
(38) In one embodiment, if no namespace is referenced for a given node, a query processor may map the node to an empty namespace or use a local name rather than the QnodeID for the node. In one embodiment, the QName incorporates a character that cannot be included in a local tag name (i.e., a character that is outside of the local tag name character set) to ensure that there is no collision between local tag names and QNames. For example, the QName may incorporate a “#” character or other syntax, which may not occur in or is not valid for local tag names. In this manner, identifiers that refer to a (namespace, local name) combination may be distinguished from local names that are not in namespaces, without any other special distinction between the two.
(39) For tag names that do not reference a namespace, the local tag name may be used to identify the tag. For example, the tag “<CONTACT>John Smith</CONTACT>” may be identified as “CONTACT” in a tag list for the CONTACT tag.
(40) Each tag list may include information that identifies the tag, such as the tag name or tag identifier, and recorded location(s) or range(s) of location(s) that the tag appears in document(s). A range for a tag may cover the region from where the beginning of the tag occurs in a document to where the end of the tag occurs in the document, and may or may not include the tag itself. The tag lists may exclude all other information, such as token information, hierarchical information, or node navigation information, or the tag lists may include additional information such as this. In a search for given token(s) that match given context(s), a larger set of documents may be narrowed down, using the lists, to those documents that contain items matching each of the separate token(s) and/or context(s). Similarly, a set of possible locations, ranges, or nodes in a document may be narrowed down to only those locations, ranges, or nodes that contain the token(s) and/or match the context(s). Each document that contains items matching each of the separate token(s) and/or context(s) may be further processed using the lists to determine whether the token(s) are within the context(s).
(41) For example, a search for keywords k.sub.1 and k.sub.2 under context c.sub.1 may cause a query processor to determine, optionally in parallel: (1) using a stored list for k.sub.1, set s.sub.1 of all documents and/or locations, ranges, or nodes within the documents that contain k.sub.1, regardless of whether k.sub.1 appears in text with k.sub.2 or under context c.sub.1; (2) using a stored list for k.sub.2, set s.sub.2 of all documents and/or locations, ranges, or nodes within the documents that contain k.sub.2, regardless of whether k.sub.2 appears in text with k.sub.1 or under context c.sub.1; (3) using a stored list for c.sub.1, set s.sub.3 of all documents and/or locations, ranges, or nodes within the documents that contain c.sub.1, regardless of whether c.sub.1 contains k.sub.1 or k.sub.2. Sets s.sub.1, s.sub.2, and s.sub.3 can be logically AND-ed to find set s.sub.4 of all documents and/or locations, ranges, or nodes within the documents that contain k.sub.1, k.sub.2, and c.sub.1, regardless of whether k.sub.1 and k.sub.2 appears in text under context c.sub.1 in those documents, locations, ranges, or nodes.
(42) The lists may also be used to determine whether k.sub.1 and k.sub.2 appears in text under context c.sub.1 in those documents, locations, ranges, or nodes. For example, for each recorded range of locations for c.sub.1 in documents that contain k.sub.1 and k.sub.2, the query processor may compare the range (from the list for c.sub.1) to the location(s) of k.sub.1 and k.sub.2 in those documents (from the lists for k.sub.1 and k.sub.2) to determine whether the locations fall within the range. If so, then k.sub.1 and k.sub.2 appear within c.sub.1 in that document. The determination of whether k.sub.1 and k.sub.2 appear within c.sub.1 in a document may be performed for any documents known to contain k.sub.1, k.sub.2, and c.sub.1, regardless of whether the entire set s.sub.4 has been or is ever determined.
(43) The lists for k.sub.1 and k.sub.2 may also be used to determine whether k.sub.1 and k.sub.2 appear adjacent to each other. The sizes of k.sub.1 and k.sub.2 in the text may be determined from the token itself, such as “iPhone,” or from stored information about the sizes of k.sub.1 and k.sub.2, optionally in the lists. For example, because “iPhone” is 6 characters, the query processor can determine whether k.sub.2 appears immediately after k.sub.1 by determining whether k.sub.2 begins immediately after or shortly after 6 characters from the stored location of the beginning of k.sub.1. The proximity of k.sub.1 and k.sub.2 in matching documents may be determined serially or in parallel with the other determinations, for those documents known to contain both k.sub.1 and k.sub.2.
(44)
XQFT Search Using the Token and/or Tag Indices
(45) The XMLFullText index exposes index search primitives to the SQL/XML query processing layer so that it can leverage XML Full Text Index to satisfy XMLExists( ) query in the WHERE clause. XML Full Text index is used as pre-filter to get a set of candidate documents for post index evaluation, which may be used if the query requests information that cannot be or otherwise is not gleaned from the indices. XML FullText search may return one or multiple documents with one or multiple nodes in the documents. The nodes may include multiple occurrences of the same node or occurrences of different nodes that satisfy the context based on their hierarchical position within the document.
(46) In one embodiment, the lists for each token or tag are posting lists that include a set of docids with token or tag offset or range information. These posting lists are built by a database server that manages storage of the underlying XML documents, and used to support XML full text search on those documents.
(47) In one embodiment, XML element and attribute tags are indexed in a manner similar to tokens, except that the database server maintains offset range information defined by each open and close tag so that hierarchical relationships among tags and tokens can be maintained. A query processor may be configured to evaluate two index primitives HASPATH(xpath) and ‘keyword INPATH(xpath)’ constructs to support SQL/XML XQFT processing. The HASPATH(xpath) parameter finds only those documents that contain the xpath parameter. The keyword INPATH(xpath) construct causes the query processor to search for only those documents where the searched keyword is within the range of the xpath parameter, according to posting list(s) for tag(s) in the path.
(48) In one embodiment, the XML Full Text Index does not index element or attribute tag names AS IS because the XML document may have namespace declaration(s). In other words, the pure lexical form of element or attribute name may have namespace prefix associated with it, and the XQFT path element may not match the pure lexical form of those names. The database server may maintain a mapping, as part of the XML Full Text Index, between lexical forms of element and attribute names and an identifier of an internal name that is stored in the XML Full Text Index. The SQL/XML XQFT processing layer consults the mapping table maintained by XML Full Text Index and does QName remapping so that the QName passed to the index layer is the mapped internal name.
(49) In one example, a full-text search for a specified token having a specified context in a set of documents may find that documents 10 and 13 have the specified context and documents 4, 5, 7, and 10 have the specified token. In the example, document 10 is the only document that contains both the token and the context. Document 10 may be further analyzed to determine whether the token is within the context in document 10.
(50) For each overlapping document that contains both the specified context and the specified token, a query processor may analyze a word position of the token within the document and a range covered by the context in the document. If the position of the token is within the range covered by the context, then the query processor may determine that the token is under the context.
(51) For multiple tokens in a phrase, the query processor may analyze a position of the multiple tokens in the document to determine whether the tokens are in a specified order, within a specified distance of each other, and/or in a same paragraph. For example, the query processor may filter out pairs of tokens that are not within a threshold distance of each other. The threshold distance may be based on the size of the tokens. In a first example, in a search for “Sacramento Kings,” the query processor may search for “Kings” that starts within 12 characters of the beginning of Sacramento, which is 10 characters long. In a second example, in a search for “Palo Alto,” the query processor may search for “Alto” that starts within 6 characters of the beginning of Palo, which is 4 characters long. In the examples, a buffer distance of 2 is used to allow for spacing between tokens. In other examples, larger or smaller buffer distances may be used, and the buffer distances may be user-configurable.
(52)
Example Document-Level Indexing
(53) In one embodiment, the posting lists for the collection of documents do not provide a result for the entire query. For example, the query may require evaluation on a row-by-row basis in a manner that cannot utilize the progressive filtering effect of the lists. However, posting lists may be utilized even if query evaluation is limited to the evaluation of a single document at a time.
(54) In one embodiment, posting lists are created and retained by the database server on a document-by-document basis, optionally in addition to posting lists for an entire collection of documents. In one example, token lists are created on the document-by-document basis. These document-level posting lists might only contain the token location, for example, identified using the address of a tree node that immediately contains the token (i.e., the immediate parent node as opposed to ancestor tree nodes that contain the immediate parent node).—In other words, in a tree hierarchy where a root node is at a highest level and leaf nodes are at lowest levels, the immediate parent node is the lowest level node that contains the token or an occurrence of the token. Alternatively or additionally, the document-level posting lists might contain information about where the token is located within the immediate parent node, where the token is located within the document as a whole, and/or identities of other ancestor nodes that contain the token and the immediate parent node.
(55) In one embodiment, a document parser parses an XML document and, for each occurrence of a token in the document, adds an entry to a list for the token. The entry may map the token to a location of a parent node that contains the token in the XML document. The parent node may be an immediate parent node of the occurrence of the token. In other words, the parent node may be a lowest level node in the XML document that contains the occurrence of the token. The entry in the list may map the token to the location of the parent node at least in part by identifying an entry in a node tree index for the parent node. The entry for the node tree index stores a location of the parent node in the XML document.
(56) For posting document-level posting lists based on a single document, the document identifier would be the same for each item in the list. Therefore, the document identifier may be stored once for each document-level list rather than for each occurrence of a token or tag that is represented by the list. The XML tag list might not be maintained at the document-level because a document has an XML tree index which identifies each XML node by a tree node address. The tree node address may be stored in a document-level posting list to verify the node containment relationship (i.e., whether the node is under a specified context).
(57) To verify the node containment relationship, the XML tree index may be traversed away from a node that is found to contain the specified context, downwardly towards a node that contains the specified token; or, the XML tree index may be traversed away from a node that is found to contain the specified token, upwardly towards a node that contains the specified context. If the node containing the token is linked to the node containing the context, directly or indirectly through a series of node-to-node links, then the node containing the token may be determined to satisfy a query for the context.
(58) In another embodiment, rather than using document-level lists, posting lists for the collection of documents may be utilized and filtered to find only the information that pertains to a particular document that is being evaluated.
(59) In one embodiment, the token lists that are maintained at the document-level contain information that identifies a location of the node/tag that contains the text and/or the location of the text within the document or node/tag. For example, a token list might specify that text “xyz” is located 20 characters into a node at a particular address. When evaluating a query for “xyz” under a particular context, even if there are no tag lists at the document-level, the token list may be used to identify a node address of the immediate parent of the node that contains “xyz.” Then, an XML tree index or DOM model, which may be maintained as an index, may be used to navigate upwardly from that node to ancestor nodes to determine whether “xyz” is within a given context. By navigating upwardly through the hierarchy of the XML document, the query processor may determine whether or not “xyz” is contained within a given context.
(60) In one embodiment, the database server creates and maintains XML tree indices for stored XML documents. An XML Tree index for a document may be used when evaluating documents on a document-by-document basis, or if the text of the document needs to be loaded or analyzed separately from the posting lists in order to finish evaluation of the query. Even if there are no tag lists for a given document, a contextual portion of a query may be evaluated for that document by using the XML tree index to navigate from a given node that contains a searched text upwardly through the hierarchy of nodes within the document.
(61) For example, a query that searches for text under context “A/B/C,” the query may use a posting list to determine a location or identity of a node in the document that contains the text. That location or identity of the node may be used to find an entry in the XML tree index or DOM model. The XML Tree index or DOM model may be used to navigate upwardly from the found node to parent node(s), toward the root node. At some point in the navigation, the query processor may be able to determine whether or not the searched text is within the context in the document by using token list maintained at the document level. One way of determining whether the searched text is within the context is to use a bottom-up strategy. For example, if the query processor finds text in the C node, navigates upwardly to the B node, and navigates upwardly to the D node and then to the root node, then the query processor may determine that the text was not contained within the context “A/B/C.” However, if the query processor finds text in the C node, navigates upwardly to the B node, and navigates upwardly to the A node, the query processor may determine that the text was within the context “A/B/C.”
(62) In another example, the query processor may do a top-down evaluation by finding node addresses of C nodes via XPath “/A/B/C” evaluation first. Then, the query processor may use a token list maintained at the document level to find node addresses P containing the text. Then, the query processor may determine whether walking from node address P upward can reach node address C.
(63) In one example, an XQuery, “/a/b/d[c contains text “tx1” and e contains text “tx2”],” a document with the snippet “<a><b><d><c>txt1</c></d><d><e>txt2</e></d></b></a>” does not satisfy the query, but a document with the snippet “<a><b><d><c>txt1</c><e>txt2</e></d></b></a>” does satisfy the query. In the example, the XML tree index may be traversed from node <e> in each document to discover whether or not the found text is within the specified context “/a/b/d/c.”
(64) In another embodiment, a query processor receives a full text query for evaluation against an XML, document. The full text query specifies token(s) and context(s). The query processor determines location(s) of parent node(s) that contain the token(s) by using token list(s) that specify the location(s) of the parent node(s) that contain the token(s). The query processor may then traverse a node tree index away from the parent node(s) to determine whether or not the parent node(s) satisfy the context(s). The node tree index may store a plurality of entries comprising, for each parent node, an entry that specifies a location of the parent node within the XML document, and a link to an entry for a node that is a parent of the parent node.
(65) In one embodiment, a query processor receives a full text query for evaluation against an XML document. The full text query specifies token(s) and context(s). The query processor may first determine range(s) of node(s) that match the context(s) in a path-based evaluation. The path-based evaluation may use a path-based index or may analyze a model of the document to determine which nodes could satisfy the context(s) specified in the query. The query processor may then determine whether the range(s) contain the token(s) by using token list(s) that specify location(s) of parent node(s) that contain the token(s). If the location of a parent node is within a range of the range(s), the parent node may be determined to be within the range(s). Some of the ranges may include nodes other than the parent node(s). For example, a particular range may include the parent node and other node(s) that nest the parent node within the context(s). In a particular example, a search for nodes within the path “A/B” may be matched by a D node that is nested under a C node, which, in turn, is nested under the B node. In the example, the path to the parent D node would be “A/B/C/D,” and this node may be determined to match the context(s) and contain the token(s) specified in the query.
(66) The query processor may generate a result of the query based at least in part on whether or not the parent node(s) satisfied the context(s). For example, information from or about node(s) that immediately contained the token(s) and satisfied the context(s) may be included in the query result.
(67)
(68)
Example Post Index Evaluation
(69) In various embodiments, there can be different XMLType storage models (text clob, object relational and binary XML), the XMLFullText index needs to handle different storage models and provides an efficient XML DOM model to handle post index evaluation.
(70) In one embodiment, an XMLFullText index layer of the database server is able to handle post index evaluation because such evaluation may be more efficient to handle in the index layer rather than in a SQL layer.
(71) In one embodiment, XML Full Text index does post index evaluation instead of passing it onto the SQL layer. XML Full Text index posting list stores docids for efficient pre-sorted merge join. Then the XML Full Text index maps docids into ROWIDs when returning qualified docs to SQL layer which always consumes ROWIDs. The XML full text layer may be enhanced to do post evaluation of XQFT constructs so that the XML full text layer may reduce a number of ROWIDs of qualified XML documents before passing the ROWIDs to the SQL layer. In one embodiment, XML has different storage models. Post index XQFT evaluation may utilize DOM tree of XML documents as XQDM is based on DOM mode. XML Full Text Index creates a persistent DOM tree as index for different XML storage models. This persistent DOM tree can be efficiently converted into in-memory DOM tree that is used by XML Full text index as post evaluation.
Example Conversion of XQFT Expression
(72) In one embodiment, compile time analysis of XQFT usage is performed in SQL/XML so that the XQFT usage can be decomposed using XMLFullText index search primitives and post index evaluation.
(73) In one embodiment, the query processor or optimizer combines multiple XMLExists( ) operators in WHERE clause and converts XQFT usage in XMLTABLE row expression into XMLExists( ) operator using XQFT index. If an XQFT expression appears in an expression that does not null-pad non-matching rows (such as an expression other than an outer join), then that expression may be copied into a filter predicate in the WHERE clause (which also does not preserve null-padding). For a query that operates on a set of rows, null-padding non-matching rows (such as in outer join operations) involves storing, in a result set of rows, null entries for the rows in that do not satisfy join predicates in the query. If the query does not null-pad non-matching rows, then the rows that do not satisfy the join predicates are excluded from the result set. In one example, the full-text expression may appear in the EXISTS operator in the XMLTABLE function, which does not preserve null padding, and the full-text expression may be pushed down into the WHERE clause. The filter predicate may be applied using the progressively narrowing posting list indices rather than requiring evaluation on a row-by-row basis.
(74) In various embodiments, the SQL/XML processor is enhanced with following features to handle XQFT constructs.
(75) In one embodiment, each XMLExists( ) operator having XQFT constructs is analyzed during compile time, and the query processor rewrites these operators such that they use XML Full Text Index primitives. In addition, the query processor may determine whether post index evaluation should be done, or if the query can be completely evaluated using the posting lists. If post-processing should be done, then a post index evaluation query plan is built and sent to XML, full text index layer to do post filtering during index lookup time.
(76) In one embodiment, XMLExists( ) operators under the same WHERE clause are analyzed and combined if they query the same XMLType column using XQFT constructs to form a set of XML Full Text Index primitives.
(77) In one embodiment, XQFT usage in row expression of XMLTABLE on the XMLType column is pushed down to XMLEXISTS( ) in a filter predicate to leverage XML Full Text Index.
(78) For example,
(79) TABLE-US-00002 SELECT v.liname, v.liprice FROM PO, XMLTABLE(‘$doc/purchaseOrder/lineItems[description contains text “iPhone 5” ] ’ PASSING PO.X as “doc” COLUMNS liname varchar2(20) PATH ‘linename’, liprice NUMBER PATH ‘liprice’) v is transformed into SELECT v.liname, v.liprice FROM PO, XMLTABLE(‘$doc/purchaseOrder/lineItems[description contains text “iPhone 5” ] ’ PASSING PO.X as “doc” COLUMNS liname varchar2(20) PATH ‘linename’, liprice NUMBER PATH ‘liprice’) v WHERE XMLEXISTS(‘$doc/purchaseOrder/lineItems[description contains text “iPhone 5” ] ’ PASSING PO.X as “doc”)
Example Range Value Search
(80) Range value searches for certain path expressions may be performed against XML data. In one embodiment, the XMLFullText index is enhanced to organize range value(s) for certain path expressions as posting list(s) so that range value search with XQFT keyword search is feasible.
(81) In one embodiment, XML Full Text Index supports range value index with certain xpath expressions using posting list form so that XQFT search and range value search can be combined.
(82) For example, a database server that stores range-compatible data such as dates, times, or numbers may organize the range data sequentially and determine how to divide the data into separate buckets that may be stored as separate posting lists. The range-compatible data may be user-defined as data of a certain type. Also, the posting lists may be created automatically when the data of the certain type is created, or when the user requests creation of an index for a specific dataset. The number of buckets or size of the buckets to hold the ranges of values may be user-configurable or selected by default. For example, a user may specify in a user interface that buckets should have at least or at most 1000 values, or that there should be at least or at most 10 buckets. The data may be split at edges of data values that support the number of buckets or number of values to be stored in each bucket. When range buckets are used, the database server does not need to create and store separate posting lists for each of the values represented in the range buckets.
(83) In one embodiment, a database server detects that one or more of the buckets have expanded or contracted to a size that is outside of a default or user-configurable threshold size. For example, birth date information may initially be stored in separate lists for each month. After time, if few people are born in February, the database server may change the posting list ranges such that the February list also includes some entries from the beginning of the March list. For example, the range February 1-February 29 may be changed to February 1-March 5. The database server may split or combine existing buckets or shift value assignments among the different buckets.
(84) When a query is received that checks for a value that falls within one of the ranges, a posting list that contains that range may be analyzed to determine which documents satisfy the query. Documents are excluded from the result set if they do not fit within the searched range.
Hardware Overview
(85) According to one embodiment, the techniques described herein are implemented by one or more special-purpose computing devices. The special-purpose computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or may include one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. The special-purpose computing devices may be desktop computer systems, portable computer systems, handheld devices, networking devices or any other device that incorporates hard-wired and/or program logic to implement the techniques.
(86) For example,
(87) Computer system 600 also includes a main memory 606, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 602 for storing information and instructions to be executed by processor 604. Main memory 606 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 604. Such instructions, when stored in non-transitory storage media accessible to processor 604, render computer system 600 into a special-purpose machine that is customized to perform the operations specified in the instructions.
(88) Computer system 600 further includes a read only memory (ROM) 608 or other static storage device coupled to bus 602 for storing static information and instructions for processor 604. A storage device 610, such as a magnetic disk, optical disk, or solid-state drive is provided and coupled to bus 602 for storing information and instructions.
(89) Computer system 600 may be coupled via bus 602 to a display 612, such as a cathode ray tube (CRT), for displaying information to a computer user. An input device 614, including alphanumeric and other keys, is coupled to bus 602 for communicating information and command selections to processor 604. Another type of user input device is cursor control 616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 604 and for controlling cursor movement on display 612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane.
(90) Computer system 600 may implement the techniques described herein using customized hard-wired logic, one or more ASICs or FPGAs, firmware and/or program logic which in combination with the computer system causes or programs computer system 600 to be a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system 600 in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. Such instructions may be read into main memory 606 from another storage medium, such as storage device 610. Execution of the sequences of instructions contained in main memory 606 causes processor 604 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions.
(91) The term “storage media” as used herein refers to any non-transitory media that store data and/or instructions that cause a machine to operate in a specific fashion. Such storage media may comprise non-volatile media and/or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, or solid-state drives, such as storage device 610. Volatile media includes dynamic memory, such as main memory 606. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, any other memory chip or cartridge.
(92) Storage media is distinct from but may be used in conjunction with transmission media. Transmission media participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise bus 602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infra-red data communications.
(93) Various forms of media may be involved in carrying one or more sequences of one or more instructions to processor 604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid-state drive of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to computer system 600 can receive the data on the telephone line and use an infra-red transmitter to convert the data to an infra-red signal. An infra-red detector can receive the data carried in the infra-red signal and appropriate circuitry can place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 either before or after execution by processor 604.
(94) Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides a two-way data communication coupling to a network link 620 that is connected to a local network 622. For example, communication interface 618 may be an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 618 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.
(95) Network link 620 typically provides data communication through one or more networks to other data devices. For example, network link 620 may provide a connection through local network 622 to a host computer 624 or to data equipment operated by an Internet Service Provider (ISP) 626. ISP 626 in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet” 628. Local network 622 and Internet 628 both use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 620 and through communication interface 618, which carry the digital data to and from computer system 600, are example forms of transmission media.
(96) Computer system 600 can send messages and receive data, including program code, through the network(s), network link 620 and communication interface 618. In the Internet example, a server 630 might transmit a requested code for an application program through Internet 628, ISP 626, local network 622 and communication interface 618.
(97) The received code may be executed by processor 604 as it is received, and/or stored in storage device 610, or other non-volatile storage for later execution.
(98) In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The sole and exclusive indicator of the scope of the invention, and what is intended by the applicants to be the scope of the invention, is the literal and equivalent scope of the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction.