METHOD FOR AUTOMATICALLY IDENTIFYING WORD REPETITION ERRORS

Abstract

A method for automatically identifying word repetition errors includes the following steps; after performing word segmentation on a large-scale training corpus, performing statistics to obtain two-tuple and three-tuple structures including repeated words in the training corpus, and repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy in the training corpus; performing statistics and recording words containing repeated characters in a Chinese dictionary and establishing a repeated word library of the Chinese dictionary; judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary; and judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics.

Claims

1. A method for automatically identifying word repetition errors, comprising the following steps: after performing word segmentation on a training corpus, performing statistics to obtain a two-tuple structure and a three-tuple structure comprising repeated words in the training corpus, and to obtain repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy in the repeated words; performing statistics to obtain words containing repeated characters in a Chinese dictionary, recording the words, and establishing a repeated word library of the Chinese dictionary; judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary; and judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, the left contextual adjacent word information entropy and the right contextual adjacent word information entropy obtained by performing statistics.

2. The method for automatically identifying the word repetition errors of claim 1, wherein step of, after performing word segmentation on the training corpus, performing statistics to obtain the two-tuple structure and the three-tuple structure including the repeated words in the training corpus, and the repeated combination degrees, the left contextual adjacent word information entropy and the right contextual adjacent word information entropy in the repeated words, comprises the following steps: 11) scanning all sentences in the training corpus to obtain all 2-tuples and 3-tuples containing the repeated words, and performing statistics on frequencies that each tuple of the 2-tuples and 3-tuples appears in the training corpus, respectively; wherein performing word segmentation on a sentence S in the training corpus to obtain S=W.sub.1 . . . W.sub.n, where W.sub.i is a word in the Chinese dictionary, 1<=i<=n; for the sentence S, if there is i enabling W.sub.i=W.sub.i+1 to be satisfied, performing statistics on frequencies freq(W.sub.i,W.sub.i+1) that a word string gram1 corresponding to a 2-tuple (W.sub.i,W.sub.i+1) appears in the training corpus, frequencies freq(W.sub.i−1,W.sub.i,W.sub.i+1) that a word string gram2 corresponding to a 3-tuple (W.sub.i−1,W.sub.i,W.sub.i+1) appears in the training corpus, and frequencies freq(W.sub.i,W.sub.i+1,W.sub.i+2) that a word string gram3 corresponding to a 3-tuple (W.sub.i,W.sub.i+1,W.sub.i+2) appears, respectively; 12) calculating the repeated combination degree of the 2-tuple (W.sub.i,W.sub.i+1) as follows: $\mathrm{Degree}\left(W_i,W_{i+1}\right)=\frac{\log P\left(W_i,W_{i+1}\right)}{\log P\left(W_i\right)*\log P\left(W_{i+1}\right)},\mathrm{wherein},P\left(W_i,W_{i+1}\right)=\frac{\mathrm{freq}\left(W_i,W_{i+1}\right)}{N1},P\left(W_i\right)=\frac{\mathrm{freq}\left(W_i\right)}{N},P\left(W_{i+1}\right)=\frac{\mathrm{freq}\left(W_{i+1}\right)}{N},$ wherein, freq(W.sub.i) is the frequency that a word W.sub.i appears in the training corpus; freq(W.sub.i+1) is the frequency that a word W.sub.i+1 appears in the training corpus: N1 is a sum of frequencies that all the 2-tuples (W.sub.i,W.sub.i+1) containing the repeated words in the training corpus appear in the training corpus; N is a total frequency that all words in the training corpus appear in the training corpus; 13) for the 3-tuple (W.sub.i−1,W.sub.i,W.sub.i+1) and the 3-tuple (W.sub.i,W.sub.i+1,W.sub.i+2), for each pair of W.sub.i=W.sub.i+1, marking W.sub.i as W without loss of generality, marking all left contextual words W.sub.i−1 as {C1, . . . , Cn}, and marking all right contextual words W.sub.i+2 as {D1 . . . Dn}, and calculating the left contextual adjacent word information entropy LE(WW) and the right contextual adjacent word information entropy RE(WW), respectively, as follows: $\mathrm{LE}\left(\mathrm{WW}\right)=-\underset{i=0}{\overset{n}{.Math.}}P\left(C_i.Math.\mathrm{WW}\right)*\log P\left(C_i.Math.\mathrm{WW}\right),\mathrm{RE}\left(\mathrm{WW}\right)=-\underset{i=0}{\overset{n}{.Math.}}P\left(D_i.Math.\mathrm{WW}\right)*\log P\left(D_i.Math.\mathrm{WW}\right),\mathrm{wherein}P\left(C_i.Math.\mathrm{WW}\right)=\frac{P\left(C_i,\mathrm{WW}\right)}{P\left(\mathrm{WW}\right)}=\frac{\mathrm{freq}\left(C_i,\mathrm{WW}\right)}{\mathrm{freq}\left(\mathrm{WW}\right)},P\left(D_i.Math.\mathrm{WW}\right)=\frac{P\left(D_i,\mathrm{WW}\right)}{P\left(\mathrm{WW}\right)}=\frac{\mathrm{freq}\left(\mathrm{WW},D_i\right)}{\mathrm{freq}\left(\mathrm{WW}\right)},$ wherein, freq(C.sub.i,WW) is the frequency that a word string corresponding to a 3-tuple (C.sub.i,W,W) appears in the training corpus; freq(WW,D.sub.i,) is the frequency that a word string corresponding to a 3-tuple (W,W,D.sub.i) appears in the training corpus.

3. The method for automatically identifying the word repetition errors of claim 1, wherein step of performing statistics to obtain the words containing the repeated characters in the Chinese dictionary, recording the words and establishing the repeated word library of the Chinese dictionary comprises: 21) performing statistics on the words containing the repeated characters in the Chinese dictionary, 22) establishing the repeated word library of the Chinese dictionary and an index structure of the repeated word library for recording and storing.

4. The method for automatically identifying the word repetition errors of claim 3, wherein step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary comprises performing word segmentation on the sentence corresponding to the text to be subjected to error checking, and judging the repeated words appearing in the text to be subjected to error checking based on the repeated word library of the Chinese dictionary; and specifically comprises: 31) performing word segmentation on a sentence S′ corresponding to the text to be subjected to error checking to obtain S′=W.sub.1′ . . . W.sub.n′; 32) if W.sub.i′=W.sub.i+1′ is satisfied, judging whether W.sub.i′W.sub.i+1′ is a word in the repeated word library of the Chinese dictionary; if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 33); 33) if W.sub.i′W.sub.i+1′ is not a word in the repeated word library of the Chinese dictionary, and if a word on a left of W.sub.i′W.sub.i+1′ is not empty, judging whether W.sub.i−1W.sub.i′ W.sub.i+1′ is a word in the repeated word library of the Chinese dictionary; if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, if a word on a right of W.sub.i′W.sub.i+1′ is not empty, judging whether W.sub.i′W.sub.i+1′W.sub.i+2′ is a word in the repeated word library of the Chinese dictionary; if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to the step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, the left contextual adjacent word information entropy and the right contextual adjacent word information entropy obtained by performing statistics.

5. The method for automatically identifying the word repetition errors of claim 4, wherein step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics comprises the following steps: 41) for the sentence S′=W.sub.1′ . . . W.sub.n′ corresponding to a segmented text subjected to error checking, and W.sub.i′=W.sub.i+1′ existing in the sentence S′=W.sub.1′ . . . W.sub.n′, judging whether W.sub.i′ W.sub.i+1′ exists in the training corpus; if W.sub.i′W.sub.i+1′ does not exist in the training corpus, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors; if W.sub.i′W.sub.i+1′ exists in the training corpus, judging whether the repeated combination degree Degree(W.sub.i′,W.sub.i+1′) is equal to 0, and if yes, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors; otherwise, turn to step 42); 42) judging whether the repeated combination degree Degree(W.sub.i′,W.sub.i+1′) is greater than α, wherein α is a first preset threshold, and if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 43); 43) judging the left contextual adjacent word information entropy and the right contextual adjacent word information entropy, if the left contextual adjacent word information entropy LE(W.sub.i′W.sub.i+1′)>β or the right contextual adjacent word information entropy RE(W.sub.i′W.sub.i+1′)>β, wherein β is a second preset threshold, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 44); 44) judging frequencies that the 3-tuple W.sub.i−1′W.sub.i′W.sub.i+1′ and the 3-tuple W.sub.i′W.sub.i+1′W.sub.i+2′ appear in the training corpus, if freq(W.sub.i−1′,W.sub.i′,W.sub.i+1′)>c or freq(W.sub.i′,W.sub.i+1′,W.sub.i+2′)>c, wherein c is a third preset threshold, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors.

6. The method for automatically identifying the word repetition errors of claim 5, wherein the first preset threshold α is 3.0; the second preset threshold β is 3.0; and the third preset threshold c is 3.0.

7. The method for automatically identifying the word repetition errors of claim 5, wherein in step 44), if W.sub.i−1′W.sub.i′W.sub.i+1′ does not exist in the training corpus, freq(W.sub.i−1′,W.sub.i′,W.sub.i+1′)=0, and if W.sub.i′W.sub.i+1′W.sub.i+2′ does not exist in the training corpus, freq(W.sub.i′, W.sub.i+1′, W.sub.i+2′)=0.

8. The method for automatically identifying the word repetition errors of claim 2, wherein in step 11), if i=1, W.sub.i−1 is a first character string representing a beginning of the sentence; if i+1=n, W.sub.i+2 is a second character string representing an ending of the sentence.

9. The method for automatically identifying the word repetition errors of claim 2, wherein step of performing statistics to obtain the words containing the repeated characters in the Chinese dictionary, recording the words and establishing the repeated word library of the Chinese dictionary comprises: 21) performing statistics on the words containing the repeated characters in the Chinese dictionary; 22) establishing the repeated word library of the Chinese dictionary and an index structure of the repeated word library for recording and storing.

Description

DESCRIPTION OF THE DRAWINGS

[0035] FIG. 1 is a flow diagram of a method for automatically identifying word repetition errors provided by an embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] The present invention will be further described in detail in conjunction with the embodiments and the drawings, and the following embodiments do not constitute a limitation on the present invention.

[0037] The present invention provides a method for automatically identifying word repetition errors, and the method includes the following steps:

[0038] after performing word segmentation on a large-scale training corpus, performing statistics to obtain two-tuple and three-tuple structures including repeated words in the training corpus, and repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy in the repeated words;

[0039] performing statistics to obtain words containing repeated characters in a Chinese dictionary, recording the words, and establishing a repeated word library of the Chinese dictionary;

[0040] judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary; and

[0041] judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics.

[0042] In the method for automatically identifying word repetition errors provided by the present invention, the step of, after performing word segmentation on the large-scale training corpus, performing statistics to obtain a two-tuple structure and a three-tuple structure including repeated words in the training corpus, and repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy in the repeated words, includes the following steps:

[0043] 11) scanning all sentences in the training corpus to obtain all 2-tuples and 3-tuples containing the repeated words in the training corpus, and performing statistics on frequencies that each tuple appears in the training corpus, respectively; wherein performing word segmentation on a sentence S in the training corpus to obtain S=W.sub.1 . . . W.sub.n, where W.sub.i is the word in the Chinese dictionary, I<=i<=n;

[0044] for the sentence S, if there is i enabling W.sub.i=W.sub.i+1 to be satisfied, performing statistics on frequencies freq(W.sub.i,W.sub.i+1) that a word string gram1 corresponding to a 2-tuple (W.sub.i,W.sub.i+1) appears in the training corpus, frequencies freq(W.sub.i−1,W.sub.i,W.sub.i+1) that a word string gram2 corresponding to a 3-tuple (W.sub.j−1,W.sub.j,W.sub.j+1) appears in the training corpus, and frequencies freq(W.sub.i,W.sub.i+1,W.sub.i+2) that a word string gram3 corresponding to a 3-tuple(W.sub.i,W.sub.i+1,W.sub.i+2) appears respectively;

[0045] in the text, freq(gram) can also be used to represent the frequency that the words or tuples corresponding to gram (words or word string) appear in the training corpus without loss of generality; gram represents the word string corresponding to a certain word or a certain tuple;

[0046] in the present embodiment, if i=1, W.sub.i−1 is a first character string representing a beginning of the sentence; if i+1=n, W.sub.i+2 is a second character string representing an ending of the sentence. In the present embodiment, the first character string is “#Begin #”, and the second character string is “#End #”. That is to say, in the present embodiment, if i=1, W.sub.i−1 is “#Begin #” representing the beginning of the sentence; if i+1=n, W.sub.i+2 is “#End #” representing the ending of the sentence;

[0047] 12) calculating the repeated combination degree of the 2-tuple (W.sub.j,W.sub.i+1), which is:

[00003]$\mathrm{Degree}\left(W_i,W_{i+1}\right)=\frac{\log P\left(W_i,W_{i+1}\right)}{\log P\left(W_i\right)*\log P\left(W_{i+1}\right)},\mathrm{wherein},P\left(W_i,W_{i+1}\right)=\frac{\mathrm{freq}\left(W_i,W_{i+1}\right)}{N1},P\left(W_i\right)=\frac{\mathrm{freq}\left(W_i\right)}{N},P\left(W_{i+1}\right)=\frac{\mathrm{freq}\left(W_{i+1}\right)}{N},$

wherein, freq(W.sub.1) is the frequency that a word W.sub.i appears in the training corpus; freq(W.sub.i+1) is the frequency that a word W.sub.i+1 appears in the training corpus; N1 is a sum of frequencies that all the 2-tuples (W.sub.i,W.sub.i+1) containing the repeated words in the training corpus appear in the training corpus; N is a total frequency that all words in the training corpus appear in the training corpus;

[0048] as result, N1 is a sum of frequencies that the word string gram1 corresponding to all the 2-tuples (W.sub.j,W.sub.i+1) containing the repeated words in the training corpus appear in the training corpus;

[0049] in other word, N1 is a sum of frequencies that all word strings gram1 in the training corpus appear in the training corpus, wherein the word strings gram1 are word strings gram1 corresponding to several 2-tuples (W.sub.i,W.sub.i+1) satisfying W.sub.i=W.sub.i+1 in the training corpus. Further, in the present embodiment, N2 is a sum of frequencies that all word strings gram2 in the training corpus appear in the training corpus, wherein the word strings gram2 are word strings gram2 corresponding to several 3-tuples (W.sub.i−1,W.sub.i,W.sub.i+1) satisfying W.sub.i=W.sub.i+1 in the training corpus; and N3 is a sum of frequencies that all word strings gram3 in the training corpus appear in the training corpus, wherein the word strings gram3 are word strings gram3 corresponding to several 3-tuples (W.sub.i,W.sub.i+1,W.sub.i+2) satisfying W.sub.i=W.sub.i+1 in the training corpus.

[0050] 13) for the 3-tuples (W.sub.i−1,W.sub.i,W.sub.i+1) and the 3-tuples (W.sub.i,W.sub.i+1,W.sub.i+2), for each pair of W.sub.i=W.sub.i+1, marking W.sub.j as W without loss of generality, marking all left contextual words W.sub.i−1, as {C1, . . . , Cn} and marking all right contextual words W.sub.i+2 as {D1 . . . Dn}, and calculating the left contextual adjacent word information entropy LE(WW) and the right contextual adjacent word information entropy RE(WW), respectively:

[00004]$\mathrm{LE}\left(\mathrm{WW}\right)=-\underset{i=0}{\overset{n}{.Math.}}P\left(C_i.Math.\mathrm{WW}\right)*\log P\left(C_i.Math.\mathrm{WW}\right)\mathrm{RE}\left(\mathrm{WW}\right)=-\underset{i=0}{\overset{n}{.Math.}}P\left(D_i.Math.\mathrm{WW}\right)*\log P\left(D_i.Math.\mathrm{WW}\right)\mathrm{wherein}P\left(C_i.Math.\mathrm{WW}\right)=\frac{P\left(C_i,\mathrm{WW}\right)}{P\left(\mathrm{WW}\right)}=\frac{\mathrm{freq}\left(C_i,\mathrm{WW}\right)}{\mathrm{freq}\left(\mathrm{WW}\right)},P\left(D_i.Math.\mathrm{WW}\right)=\frac{P\left(D_i,\mathrm{WW}\right)}{P\left(\mathrm{WW}\right)}=\frac{\mathrm{freq}\left(\mathrm{WW},D_i\right)}{\mathrm{freq}\left(\mathrm{WW}\right)};$

wherein freq(C.sub.i,WW) is the frequency that a word string corresponding to the 3-tuple (C.sub.i,W,W) appears in the training corpus; freq(WW,D.sub.i,) is the frequency that a word string corresponding to the 3-tuple (W,W,D.sub.i) appears in the training corpus.

[0051] At this time, because W.sub.i is marked as W without loss of generality, W.sub.i=W.sub.i+1=W, and W.sub.iW.sub.i+1 in the text can be expressed as WW without loss of generality.

[0052] In the method for automatically identifying word repetition errors provided by the present invention, the step of performing statistics to obtain the words containing repeated characters in the Chinese dictionary, recording the words, and establishing a repeated word library of the Chinese dictionary specifically includes:

[0053] 21) performing statistics to obtain the words containing repeated characters in the Chinese dictionary;

[0054] in the present embodiment, that is, performing statistics and looking for words containing continuously identical repeated characters in the Chinese dictionary, such as “man tun tun”, “gao gao xing xing”, “duo duo yi shan”, “xin xin xiang rong”, “ha-ha”, and/or “Bye Bye”, etc.;

[0055] 22) establishing a repeated word library of the Chinese dictionary and the index structure thereof for recording and storing,

[0056] wherein the index structure can improve the efficiency of matching, and in the present embodiment, the index structure is Set<String>wordSet.

[0057] In the method for automatically identifying word repetition errors provided by the present invention, the step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary in the present embodiment is specifically as follows: performing word segmentation on a sentence corresponding to the text to be subjected to error checking, and judging the repeated words appearing in the text to be subjected to error checking based on the repeated word library of the Chinese dictionary; and as shown in FIG. 1 of the present embodiment, it specifically includes:

[0058] 31) performing word segmentation on a sentence S′ corresponding to the text to be subjected to error checking to obtain S′=W.sub.1′ . . . W.sub.n′, wherein W.sub.i′ is a word in the Chinese dictionary, 1<=i<=n;

[0059] 32) if W.sub.i=W.sub.i+1′ is satisfied, judging whether W.sub.i′ is a word in the repeated word library of the Chinese dictionary (in the present embodiment, it can also be to say, judging whether W.sub.i′ W.sub.i+1′ is a word in wordSet); if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 33);

[0060] 33) if W.sub.i′W.sub.i+1′ is not words in the repeated word library of the Chinese dictionary, and if a word on a left thereof is not empty, judging whether W.sub.i−1′W.sub.i′W.sub.i+1′ is a word in the repeated word library of the Chinese dictionary; if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, if a word on a right thereof is not empty, judging whether W.sub.i′W.sub.i+1′W.sub.i+2′ is a word in the repeated word library of the Chinese dictionary; if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to the step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics, so as to continuously perform judgments by performing statistics on amount of information.

[0061] In the method for automatically identifying word repetition errors provided by the present invention, the step of judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics, as shown in FIG. 1 of the present embodiment, specifically includes the following steps:

[0062] 41) for the sentence S′=W.sub.1′ . . . W.sub.n′ corresponding to the segmented text subjected to error checking, and W.sub.i′=W.sub.i+1′ existing therein, judging whether W.sub.i′W.sub.i+1′ exists in the training corpus; if W.sub.i′W.sub.i+1′ does not exist in the training corpus, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors; if W.sub.i′W.sub.i+1′ exists in the training corpus, judging whether the repeated combination degree Degree(W.sub.i′,W.sub.i+1′) is equal to 0, and if Degree(W.sub.i′,W.sub.i+1′)=0, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors; otherwise, turn to step 42);

[0063] 42) judging whether the repeated combination degree Degree(W.sub.i′,W.sub.i+1′) is greater than α, wherein α is a first preset threshold, and if yes, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 43);

[0064] 43) judging the left contextual adjacent word information entropy and the right contextual adjacent word information entropy, if the left contextual adjacent word information entropy LE(W.sub.i′W.sub.i+1′)>β or the right contextual adjacent word information entropy RE(W.sub.i′W.sub.i+1′)>β, wherein β is a second preset threshold, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, turn to step 44);

[0065] 44) judging frequencies that a 3-tuple W.sub.i−1W.sub.i′W.sub.i+1′ and a 3-tuple W.sub.i′W.sub.i+1′W.sub.i+2′ appear in the training corpus, if freq(W.sub.i−1′,W.sub.i,′,W.sub.i+1′)>c or freq(W.sub.1′,W.sub.i+1′,W.sub.1+2′)>c, wherein c is a third preset threshold, judging W.sub.i′W.sub.i+1′ to be a correct repeated word; otherwise, judging W.sub.i′W.sub.i+1′ to be an incorrect repeated word, and marking W.sub.i′ and W.sub.i+1′ as errors; wherein if W.sub.i−1′W.sub.i′W.sub.i+1′ does not exist in the training corpus (i.e., training corpus library), freq(W.sub.i−1′,W.sub.i′,W.sub.i+1′)=0, and if W.sub.i′W.sub.i+1′W.sub.i+2′ does not exist in the training corpus (i.e., training corpus library), freq(W.sub.i′,W.sub.i+1′,W.sub.i+2′)=0.

[0066] The 3-tuple W.sub.i−1′W.sub.i′W.sub.i+1′ in the text can also be expressed as a 3-tuple (W.sub.i−1′W.sub.i′W.sub.i+1′) or a 3-tuple (W.sub.i−1W.sub.i′,W.sub.i′,W.sub.i+1′); the 3-tuple W.sub.i′W.sub.i+1′W.sub.i+2′ can also be expressed as a 3-tuple (W.sub.i′W.sub.i+1′W.sub.i+2′) or a 3-tuple (W.sub.i′,W.sub.i+1′,W.sub.i+2′); freq(W.sub.i−1′,W.sub.i′,W.sub.i+1′) can also be expressed as freq(W.sub.i−1′W.sub.i′W.sub.i+1′), and freq(W.sub.i′,W.sub.i+1′,W.sub.i−2′) can also be expressed as freq(W.sub.i′W.sub.i+1′W.sub.i+2′).

[0067] The above α is a first preset threshold, and the first preset threshold a in the present embodiment is 3.0; the above β is a second preset threshold, and the second preset threshold β in the present embodiment is 3.0; and the above c is a third preset threshold, and the third preset threshold c in the present embodiment is 3.0.

[0068] In the text, the training corpus can also be called a corpus or a training corpus.

[0069] Test: firstly, the present invention is used to perform statistical training on a large-scale training corpus (8G) (the 8G large-scale corpus is the training corpus), and all sentences in the test set include the preset 1000 incorrect repeated words. The method for automatically identifying word repetition errors provided by the present invention is used to perform error checking/identifying of repeated words on the test set, and the test result shows that the recall rate thereof reaches 84%, the precision rate reaches 77%. Hence, the present invention can effectively discover the word repetition errors.

[0070] The above embodiments are only the preferred embodiments of the present invention, it should be noted that the above implementation does not constitute a limitation on the present invention, and the various changes and modifications made by the person skilled in the art within the scope of the technical idea of the present invention fall within the protection scope of the present invention.