CONTENT SENSITIVE DOCUMENT RANKING METHOD BY ANALYZING THE CITATION CONTEXTS

Abstract

This invention relates to a method which provides, showing as well the relevant documents to the user even if the said documents that are closely related to the subject do not contain the keywords that are entered for search in their content, by analyzing citation contexts of every document in a data pool containing documents that are citing or linking to at least one document. In an alternative embodiment of this method, in the case when the documents cited by using the entered keywords are cited by other documents using other keywords, these said other keywords are considered as similar terms and the search is conducted as well by including these said similar terms.

Claims

1. A computer implemented method for accessing to related documents by chosen keywords, the method comprising: extracting citation contexts of each document, that cites at least one document, in a database containing documents, identifying meaningful keywords/word groups from a citation context of each document, forming a citation network, an edge-labeled directed graph, where nodes are the documents; there is an edge from a citing article to a cited article and the edge is labeled by inferred keywords, constructing a table T which contains a citing document, a cited document, and the inferred keywords in a citation, receiving the keywords/word groups to initiate a search identifying citing and cited documents by searching entered keywords in the table T, forming a related document pool by adding the cited documents identified in the table T and the related document pool, ranking the documents taken to the related document pool by using any ranking algorithm.

2. The computer implemented method of claim 1, wherein in the step of “identifying meaningful keywords/word groups from a citation context of each document”, every term in the citation context is inferred.

3. The computer implemented method of claim 1, wherein in the step of “identifying meaningful keywords/word groups from a citation context of each document”, every term in the citation context is used as a definitive term for articles in the citation context.

4. The computer implemented method of claim 1, wherein the step of “identfying meaningful keywords/word groups from a citation context of each document”, terms are single words, bigrams or n-grams.

5. The computer implemented at method of claim 1, wherein in the step of “forming a citation network, an edge-labeled directed graph, where nodes are the documents; there is an edge from a citing article to a cited article and the edge is labeled by inferred keywords”, the edge is labeled by terms used in the citation context of the citing document while citing.

6. The computer implemented method of claim 1, wherein in the step of “identifying citing and cited documents by searching entered keywords in a table T”, the entered keywords are searched in the table constructed by using term labeling the citation network.

7. The computer implemented method of claim 1, wherein in the step of “identifying citing and cited documents by searching entered keywords in a table T”, for a keyword α entered for the search, a term-α specific citation network is formed and after that, the citing documents, the cited documents and entered α in the citation network are written in the table and the citing documents corresponding to the keyword α are determined.

8. The computer implemented method of claim 1, wherein in the step of “ranking the documents taken to the related document pool by using any ranking algorithm”, a simple ranking module which takes a bigram and gives a ranked list of scientific articles in return is used.

9. A computer implemented method for finding documents which are closely related to a subject, by using other keywords along with chosen keywords, in the case where a same document is referred to by multiple documents and by multiple keywords, the method comprising, extracting citation contexts of each document, that cites at least one document, in a database containing the documents, identifying meaningful keywords/word groups from a citation context of each documents, forming a citation network, an edge-labeled directed graph, where nodes are the documents; there is an edge from a citing article to a cited article and the edge is labeled by inferred keywords, constructing a table T which contains the citing document, the cited document, and the inferred keywords in a citation, receiving the keywords/word groups from a user computer to initiate a search by a server, inferring terms which are similar to entered keywords by the server, identifying citing and cited documents by the server for searching the entered keywords in a table T; identifying the citing and cited documents in the table T by searching inferred similar terms, forming a relating document pool fir all results in the table T corresponding to both the entered keywords and similar terms, ranking the documents taken from the document pool using any ranking algorithm.

10. The computer implemented method of claim 9, wherein in the step of “inferring terms which are similar to entered keywords by the server”, the terms with which other documents cite every document that cites another document, and the documents that are cited in the citation context using the entered keywords, are investigated and these terms are taken as the similar terms,

11. The computer implemented method of claim 9, wherein in the step of “inferring terms which are similar to entered keywords by the server”, a term matrix F=[f.sub.αj] is constructed where an entry f.sub.αj shows a count regarding how many articles use a term α in a related citation context, in order to cite an article j, and F is fundamentally taken from a non-directed weighted bipartite graph between article nodes and term nodes.

12. The computer implemented method of claim 9, wherein in the step of “inferring terms which are similar to entered keywords by the server”, for finding distinguishing terms which are used for defining smaller article sets that are present as similar terms, in order to lower a term weight of a term by a factor increasing with a frequency of appearance of the term in the citation contexts, a term frequency-inverse document frequency technique is used

13. The computer implemented method of claim 9, wherein in the step of “inferring terms which are similar to entered keywords by the server”, as a measure towards a power linear correlation between two sample terms, Pearson Correlation Coefficient is used.

14. The computer implemented method, of claims 9, wherein the keywords that are entered in the step “identifying citing and cited documents by the server for searching the entered keywords in a table T” are searched in the table T in which, words inside the citation context of every document in an information pool which contains documents which are citing at least one document, every citing document and every cited document are present.

15. The computer implemented method of claim 9, wherein after forming the citation network of the similar terms for a given term α in the step “identifying citing and cited documents by the server for searching the entered keywords a table T”, the citing documents using the term α in the citation context and the cited documents are written in the table T, and the citing documents corresponding to the term α are determined.

16. The computer implemented method of claim 9, wherein the keywords entered in the step “identifying the citing and cited documents in the table T by searching inferred similar terms” are searched in the table which, words inside the citation context of every document in an information pool which contains documents which are citing at least one document, every citing document and every cited document are present.

17. The computer implemented method of claim 9, wherein after forming the citation network of the similar terms for a given term α in the step “identifying the citing and cited documents in the table T searching inferred similar terms”, writing the terms similar to the term α, the citing documents using the similar terms in the citation context and the cited documents in the table T and determining the documents corresponding to an entered term α and the terms similar to the term α.

18. The computer implemented method of claim 10, wherein in the step of “inferring terms which are similar to entered keywords by the server”, a term matrix F=[f.sub.αj] is constructed where an entry f.sub.αj shows a count regarding how many articles use a term α in a related citation context, in order to cite an article j, and F is fundamentally taken from a non-directed weighted bipartite graph between article nodes and term nodes.

19. The computer implemented method of claim 10, wherein in the step of “inferring terms which are similar to entered keywords by the server”, for finding distinguishing terms which are used for defining smaller article sets that are present as similar terms, in order to lower a term weight of a term by a factor increasing with a frequency of appearance of the term in the citation contexts, a term frequency-inverse document frequency technique is used.

20. The computer implemented method of claim 11, wherein in the step of “inferring terms which are similar to entered keywords by the server”, for finding distinguishing terms which are used for defining smaller article sets that are present as similar terms, in order to lower a term weight of a term by a factor increasing with a frequency of appearance of the term in the citation contexts, a term frequency-inverse document frequency technique is used.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Methods in order to fulfill the objects of the present invention are illustrated in the attached figures, where:

[0021] FIG. 1. Citation network of the exemplary pseudo-articles

[0022] FIG. 2. Citation Networks

[0023] FIG. 2a. Term labeled citation network G(A,C).

[0024] FIG. 2b. Term-α1 specific citation network G.sub.α1(A, C.sub.α1)

[0025] FIG. 2c. Term-α4 specific citation network G.sub.α4(A, C.sub.α4)

[0026] FIG. 2d. Like term citation network. G.sub.Sα1(A, C.sub.Sα1) for set of like terms S.sub.60 1={α.sub.1,α.sub.4}

[0027] FIG. 2e. Bipartite graph of documents versus terms

[0028] FIG. 2f. Matrix F of terms versus documents

[0029] FIG. 3. Intersecting terms

[0030] FIG. 4. Steps of the context sensitive search method.

[0031] FIG. 5. Steps of the method of searching with like terms.

ELEMENTS SHOWN IN THE FIGURES ARE NUMBERED AS FOLLOWS

[0032] 100. Context sensitive search method

[0033] 110. Separating the citation contexts

[0034] 120. Identifying the meaningful keywords/word groups from the citation contexts

[0035] 130. Forming the citation network, as a directed graph, from the citing article to the cited articles

[0036] 140. Writing the keywords, citing and cited documents in a table T

[0037] 150. Entering the keywords/word groups to start a search

[0038] 160. Searching the keywords word groups in the table T

[0039] 170. Taking the cited documents corresponding to the keywords in a pool of related documents

[0040] 180. Ranking using a ranking algorithm

[0041] Method of searching with like terms

[0042] 210. Entering the keywords/word groups

[0043] 220. Inferring the terms which are similar to the entered keywords

[0044] 230. Conducting a search in the table T for the first keywords/word groups entered.

[0045] 240. Conducting a search in the table T for the inferred similar terms

[0046] 250. Forming a relating document pool for all results in the table T corresponding to the entered keywords and similar terms

[0047] 260. Ranking the documents taken from the document pool using a ranking algorithm

DETAILED DESCRIPTION OF THE INVENTION

[0048] Citation context sensitive search method (100), which provides access to the related documents by the chosen keywords, essentially comprises the steps of,

[0049] extracting the citation contexts of each document, that cites at least one document, in a database containing the documents (110),

[0050] identifying meaningful keywords/word groups from the citation context of each document (120),

[0051] forming the citation network, an edge-labeled directed graph, where nodes are the documents. There is an edge from the citing article to the cited article and the edge is labeled by the inferred keywords (130),

[0052] constructing a table T (see Table T) which contains the citing document, the cited document, and the inferred keywords in the citation (140),

[0053] entering the keywords/word groups to initiate a search (150),

[0054] identifying citing and cited documents by searching the entered keywords in the table (160),

[0055] forming a relating document pool by adding the cited documents identified in (160), (170),

[0056] ranking the documents taken to the related document pool by using any ranking algorithm (180).

[0057] In the subject matter of citation context sensitive searching method (100), in order to provide access to the related documents via selected keywords, firstly, a table T consisting of keywords used for citing, along with citing and cited documents based on the keywords that are already present in the citation context, needs to be constructed. Once the table T is formed, all searches are conducted through this table T.

[0058] In order to form the table T, first of all, the citation context of every document, that cite another document, is extracted (110), and then meaningful keywords/word groups (terms) are inferred from the citation context (120).

[0059] The best articles are cited by numerous articles with relevant terms in the citation context. This, in turn, shows that the cited document is relevant to the subject. For this, in the method subject to the invention, a simple method is followed for determining/defining the citation terms in the citation context. The terms used for explaining the cited article stand dose to the citation point. As also shown in FIG. 1, citation context is the text around the citation marker that is present inside the window, the size of which is predefined. The size of this text can be defined by a specific sentence, word or character number around the citation point. As a result of this, as present in the previous studies, the method subject to the invention also takes the citation context as a window with a fixed size, for example 400 characters in length, around the citation point.

[0060] Once citation context is obtained, the next step is the identification of meaningful keywords/word groups (terms) from the citation context of each document (120). A word or a word group, which states something or some concept in a specific field is referred to as “term”. Every bigram, that is present in citation context, is used as a definitive term for the cited article. For example, if three articles are cited in the same citation context, every bigram it this content is taken as definitive terms for the three articles. The number of cords, taken as a term, is one or two but it can be expanded for using n-grams of any n. However, bigrams are preferred since they are the most used n-grams for explaining specific terms such as “scale free”, “map reduce”, and “preferential attachment”. While this situation helps inferring meaningful terms from the citation context, it eliminates problems such as synonymous words in the case of single words.

[0061] After inferring meaningful terms from the citation context of each document (120), a citation network is formed (130). A citation network is an edge-labeled directed graph, where nodes are the documents. There is a directed edge from the citing article to the cited article. The edge is labeled by the inferred keywords. An edge in a citation network carries a lot more information than merely a binary relationship. The terms that the citing author used for explaining the cited document could be taken from the citation context.

[0062] In FIG. 1, an example of a small citation network, formed by six pseudo-articles is given. The citation contexts are underlined, cited articles indicated in brackets, such as “[2]”, together with the terms taken from the citation context are emphasized by bold font, such as “term-1”.

[0063] For example, the corresponding term labeled citation network of FIG. 1 is given in FIG. 2a, where the set of vertices is A={a.sub.1, a.sub.2, . . . , a.sub.|A|}, and T={α.sub.1, α.sub.2, . . . , α.sub.|T|} is the set of terms. The directed edge from a.sub.1 to a.sub.3 is labeled with terms in set T.sub.1,3={α.sub.1, α.sub.4, α.sub.5}. The edge a.sub.1, a.sub.2 is not there since there is no such citation, that is T.sub.1,2 is the empty set. In other words, edges are labeled with the terms used in the citation context of the citing document while citing. The conventions for A and T are given below.

[0064] A={a.sub.1, a.sub.2, . . . , a.sub.|A|} is the set of all documents. Lower case letters of the Latin alphabet such as i, j∈A are used to denote the members of A.

[0065] T={α.sub.1, α.sub.2, . . . , α.sub.|T|} is the set of all terms used in all documents in A. Letters of the Greek alphabet such as α, β∈T are used to denote the members of T.

[0066] The edge from i to j is labeled by the terms in T.sub.ij. Set T.sub.ih ∈T is the set of all terms, which are in at least one citation context of article i citing article j. If no citation is made from article i to article j, then T.sub.ij is the empty set. In the situations where article i cites article j, however there are no terms inside the citation context, then T.sub.ij again the empty set. It is also possible that the article i cites article j more than once and a term might be presented in each of these citations. Having used at least once is enough for the term to be in T.sub.ij.

[0067] More formally, the term labeled citation network G(A,C), shown in FIG. 2a, is a directed graph by C⊂A×A where the edge is (i, j)∈C, if and only if article i contains at least on citation referring to article j. The edge (i, j)∈C is labeled with all the terms inside T.sub.ij.

[0068] In the exemplary term labeled citation network G(A,C) given in FIG. 2a, the directed edges used for forming citation network between documents a.sub.1, a.sub.2, a.sub.3, a.sub.4, a.sub.5 and a.sub.6 are labeled with the terms α.sub.1, α.sub.2, α.sub.3, α.sub.4 and α.sub.5. Document a.sub.1 cites document a.sub.3. In the citation context it terms uses α.sub.1, α.sub.4 and α5 In the citation context for a.sub.4, there are terms α.sub.1 and α.sub.2. Similarly, document a.sub.2 uses terms α.sub.1 for a.sub.4 and α.sub.1 and α.sub.3 for a.sub.6.

[0069] After constructing the citation network (130), table T constructed T (140). For every citing and cited document pair, there is a row. The terms used in the citation context are inserted into the corresponding row. These terms, though they change according to the content of the cited document, in one example of the invention are “scale free”, “preferential attachment”, “map reduce”. The directed edges used between the citing documents and cited documents are labeled with the terms used in the citation context. An exemplary table T formed for FIG. 2a is shown below in the exemplary table T shown below, α.sub.1, α.sub.2, α.sub.3, α.sub.4 and α.sub.5 each represent a term.

TABLE-US-00001 TABLE 1 The table T which is formed for the G(A, C) term labeled citation network given in FIG. 2a. Contains the citing and cited documents and the terms used in the citation context. Cited Words used Citing documents in citation context documents Document a.sub.3 α.sub.1, α.sub.4, α.sub.5 Document a.sub.1 Document a.sub.4 α.sub.1, α.sub.2 Document a.sub.1 Document a.sub.6 α.sub.2 Document a.sub.1 Document a.sub.4 α.sub.1 Document a.sub.2 Document a.sub.6 α.sub.1, α.sub.3 Document a.sub.2 Document a.sub.5 α.sub.4, α.sub.5 Document a.sub.3 Document a.sub.6 α.sub.3 Document a.sub.4

[0070] In the inventive citation context sensitive searching method (100), after forming the table T containing citing and cited documents and relating terms, a content sensitive search can be initiated.

[0071] In the method, in order to initiate a search, first of all keywords/word groups/terms of the subject to be searched are entered (150). In the preferred embodiment of the invention, the entered terms are searched in a table T (160).

[0072] In another embodiment of the invention, for the terms entered for search (for example for α) a term-α specific citation network is formed and after that, citing documents, cited documents and entered α in this citation network are written in a table and citing documents corresponding to this value α are determined (160). In an exemplary embodiment of the invention, an example of the term-α specific citation network for α.sub.1 and α.sub.4 as entered terms, are shown in FIG. 2b and FIG. 2c respectively.

[0073] Suppose α.sub.1 is the term/keyword for the search. Term α.sub.1 is searched inside the table T. All the documents corresponding to α.sub.1 in table T are selected and considered as the related document pool (170). Thereby, not only the documents that contain α.sub.1, documents related to the subject but do not contain α.sub.1 are also selected. Hence access to all documents closely relating to the subject α.sub.1 are provided.

[0074] After gathering all documents relation to α.sub.1 in a document pool, the documents taken from the document pool can be ranked by using any ranking algorithm (180).

[0075] In one embodiment of the invention, for ranking the documents related to the subject (180), a simple ranking module, which takes a bigram and gives a ranked list of scientific articles in return is used.

[0076] So far search related to term is considered (100). Another method (200), which provides access to the relevant documents by using similar terms, is explained below. Here not only documents gathered in (100), but also documents related to the terms that are similar to the entered terms are considered. So that documents, which are closely related to the subject but which do not contain the keywords can also be reached.

[0077] A searching method with similar terms (200), which enables finding the documents, which are closely related to the subject, by using other keywords along with the chosen keywords, reuses steps (110) trough (140) and essentially comprises the steps of;

[0078] extracting the citation contexts of each document, that cites at least one document, in a database containing the documents (110),

[0079] identifying meaningful keywords/word groups from the citation context of each document (120),

[0080] forming the citation network, an edge-labeled directed graph, where nodes are the documents. There is an edge from the citing article to the cited article and the edge is labeled by the inferred keywords (130),

[0081] constructing a table T (see Table T) which contains the citing document, the cited document, and the inferred keywords in the citation (140),

[0082] entering the keywords/word groups to initiate a search (210),

[0083] inferring the terms which are similar to the entered keywords (220),

[0084] as in the case of (160) identifying citing and cited documents in the table T by searching the entered keywords (230),

[0085] as in the case (160), identifying citing and cited documents in the table T by searching the inferred similar terms (240),

[0086] forming a relating document pool for all results in the table T corresponding to both the entered keywords and the similar terms (250),

[0087] as in the case of (180) ranking the documents taken from the document pool using any ranking algorithm (260).

[0088] One of the main approaches of the inventive searching method with similar terms (200) is using both the entered and the similar terms in the process of network forming. This helps to expand the selected document set to include documents related to the similar terms.

[0089] In scientific publications, one term generally is not sufficient to explain a subject by itself and usage of only a single term is prone to noises because of the natural usage of the language such as synonymous words. For every term, there is a set of articles that contain it. In FIG. 3, as can be seen related to the terms “law of force”, “scale free” and “preferential attachment”, these article sets substantially coincide for some terms.

[0090] In the searching method with similar terms (200), table T is constructed by means of sequence of (110) through (140). After that, in order to initialize the searching process keywords/word groups/terms are entered (210) and the terms that are similar to the entered keywords are inferred (220). In principle two terms are similar if they appear together in a considerable number of citation. Given a term, inferring similar terms requires some tools.

[0091] In order to infer the terms which are similar to the entered keywords (220), first a term-article matrix is formed. Similar terms for the given term α, is the set of terms which have the article scope which substantially coincides with the article scope of the term α.

[0092] Term frequencies are related to the articles by a document matrix F=[f.sub.αj] which has a size of |T|×|A|. In this matrix, the entry f.sub.αj is the count regarding how many articles use the term α in the citation context that cites article j. That is, f.sub.αj is the in-degree of article j in term-α graph G.sub.α. Therefore F is actually the adjacency matrix of the weighted bipartite graph between the article nodes and term nodes.

[0093] An example the bipartite graph given in FIG. 2e is obtained from term labeled citation network as in FIG. 2a. The corresponding related term document matrix F is shown in FIG. 2f.

[0094] In the inventive searching method with similar terms (200), there are distinguishing terms which are used for especially defining smaller article sets are present as similar terms. Simple term frequency has a problem of assuming every one of the terms to have the same importance, however some terms have very little or no distinguishing power. For example, it is possible for almost the entire citation context of an article collection on the topic “cancer” to contain the term “cancer”. For this, the weights of the terms, which are present in numerous citation contexts, are lowered. In principle, the idea is reducing term frequency weight of a term by a factor that grows with its citation context frequency it appears. Term frequency-inverse document frequency (tf-idf) is a technique which is based on this idea. This method is widely used in information retrieval and text mining and it reflects how important a word is to a document in a collection. For this reason, in the inventive method (200), this technique is used for weighting the term frequencies.

[0095] The inverse document frequency for term α is defined by g(α),

[00001]$g\left(\alpha \right)=\log .Math..Math.\frac{.Math.A.Math.}{{.Math.}_{j=1}^{.Math.A.Math.}.Math.\mathrm{sgn}\left(f_{\alpha .Math..Math.j}\right)}$

[0096] where sgn(x) is a signum function designed as

[00002]$\mathrm{sgn}\left(x\right)=\{\begin{array}{cc}1,& x>0,\\ 0,& x=0,\\ -1,& x<0.\end{array}$

[0097] Afterwards, let us assume that D=[d.sub.αβ] is a |T|×|T| diagonal matrix defined with below:

[00003]$d_{\alpha .Math..Math.\beta }=\{\begin{array}{cc}g\left(\alpha \right),& \alpha =\beta ,\\ 0,& \alpha \ne \beta .\end{array}$

[0098] We define the weighted term document Matrix N=[n.sub.αβ] of size |T|×|A| with N=D×F.

[0099] Afterwards, a relationship between the terms is established. α and β are assumed, to be the α.sup.th and β.sup.th row vectors of N respectively, and the members α and β show the related weighted term frequencies of α and β for the articles inside the data set. In order to learn how much of the scopes of the articles of these terms coincide, comparison of the corresponding row vectors of α and β is realized. For this, in a preferred embodiment of the invention, as a measure towards the power linear correlation between two sample terms. Pearson Correlation Coefficient, which is widely used in the field of science, is used,

[0100] Afterwards, a Sample Pearson Correlation Matrix P=[p.sub.αβ] of size |T|×|T| is defined and P.sub.αβ

[00004]$p_{\alpha .Math..Math.\beta }=\frac{{.Math.}_{i=1}^{.Math.A.Math.}.Math.\left({\alpha }_i-\stackrel{\_}{\alpha }\right).Math.\left({\beta }_i-\stackrel{\_}{\beta }\right)}{\sqrt{{.Math.}_{i=1}^{.Math.A.Math.}.Math.{\left({\alpha }_i-\stackrel{\_}{\alpha }\right)}^2}.Math.\sqrt{{.Math.}_{i=1}^{.Math.A.Math.}.Math.{\left({\beta }_i-\stackrel{\_}{\beta }\right)}^2}}$

[0101] is the Sample Pearson Correlation between term α and β where α and β are the α.sup.th β.sup.th row vectors of N. The vector α is the average of the entrance of the vector α.

[0102] The sample Pearson Correlation Coefficient is the measure of the linear correlation between two samples X and Y, and it can give a value between −1 and 1 (including −1 and 1). A value of 1 means that a linear equation defines the relationship between X and Y, and all data points are located on a line where Y increases with increasing X. A value of −1 means that all data points are located on a line where Y decreases with increasing X. This case is irrelevant for our data set, because, in order to take the value −1 for two terms α and β, they need to be complementary to each other. This is not possible for large article collections. The value 0 means that there is no linear correlation present between the samples.

[0103] For a given term α, the similar term set S.sub.α is defined as S.sub.α={β∈T|p.sub.αβ>δ} for some value 0<δ<1. δ is the cross validation parameter and the value of δ changes between topics. Additionally, similarity point p.sub.αα for term α equals to 1. For this reason, since α∈S.sub.α, S.sub.α is not empty,

[0104] Weighted citation network which takes term α as basis and which is directed from the term set S.sub.α is defined as follows:

[0105] The sub graphic G.sub.Sα (A, C.sub.Sα) of G(A, C) is named as the citation network of the set of similar terms, where [0106] (i) C.sub.Sα=∪.sub.βαSαC.sub.β [0107] (ii) the weight of the edge (i, j)∈C.sub.Sα equals to the sum of weights of the edges combined w.sub.if=Σ.sub.(ij)∈T.sub.ij.sub.∩S.sub.αP.sub.αβ.

[0108] For example from FIG. 2a, suppose S.sub.α1={α.sub.1, α.sub.4} is the similarity set for a given term α.sub.1 and δ. Then, the network of similar terms set G.sub.Sα1 (A, C.sub.Sα1) for term α.sub.1 is shown in FIG. 2d.

[0109] The keywords entered, in the inventive method (200), are searched in table T and related documents are identified (230).

[0110] In another embodiment of the invention, the citation network of terms similar to given term α is formed in (220). Therefore table T contains the documents related to similar terms to the entered keyword.

[0111] In another embodiment of the invention, after forming the citation network of similar terms set is formed for a given term α, terms (for example α.sub.1 and α.sub.4) similar to the entered keyword (α), the documents citing using these similar terms in the citation context and cited documents are written in the table T. Thus, cited documents corresponding to the entered keyword (α) and the terms that are similar to the entered keyword/keywords (α.sub.1 and α.sub.4), in other words documents closely related to the subject are determined.

[0112] Thus, documents closely related to the subject can be determined by means of another search in the table T for the terms similar to the entered (240). Thus, the documents related to the entered keywords, and the documents related to similar terms, that is, closely related to the subject, are determined (250).

[0113] After the cited documents corresponding to the entered keyword and the similar terms are collected in a data pool (250).

[0114] The related documents, in the sense of both entered keywords and the similar terms, taken into the data pool. Then so selected documents are ranked by using any ranking algorithm (260).