System and method for three-way call detection

Abstract

A system for detecting three-way calls in a monitored telephone conversation includes a speech recognition processor that transcribes the monitored telephone conversation and associates characteristics of the monitored telephone conversation with a transcript thereof, a database to store the transcript and the characteristics associated therewith, and a three-way Call detection processor to analyze the characteristics of the conversation and to detect therefrom the addition of one or more parties to the conversation. The system preferably includes at least one domain-specific language model that the speech recognition processor utilizes to transcribe the conversation. The system may operate in real-time or on previously recorded conversations. A query and retrieval system may be used to retrieve and review call records from the database.

Claims

1. A system for detecting a three-way call in a monitored telephone conversation, the system comprising: one or more circuits configured to: convert an audio of the monitored telephone conversation to a transcript using a domain specific language model; detect a pattern within the monitored telephone conversation based on the transcript, the pattern being an indicator of the three-way call; generate output information based on the pattern, the output information including timing information and weighting information corresponding to the pattern; and determine that the monitored telephone conversation includes the three-way call based on the output information.

2. The system of claim 1, wherein the one or more circuits are further configured to tag the transcript with a timestamp at a starting point of the three-way call in the monitored telephone conversation.

3. The system of claim 1, wherein the one or more circuits are further configured to store at least one of the audio and the transcript to a database.

4. The system of claim 1, wherein to convert the audio of the monitored telephone conversation to the transcript using the domain specific language model, the one or more circuits are configured to: translate the transcript from a first language into a second language.

5. The system of claim 1, wherein to detect the three-way call based on the output information, the one or more circuits are configured to: determine a value corresponding to the pattern; and determine that the value is greater than a predetermined threshold.

6. The system of claim 1, wherein the output information is a matrix comprising a plurality of characteristics or patterns identified within the transcript.

7. The system of claim 1, wherein the pattern includes one or more of a frequency of an utterance of a phrase during the monitored telephone conversation, a timing of the utterance of the phrase during the monitored telephone conversation, or an entity extracted from the utterance of the phrase.

8. The system of claim 1, wherein the monitored telephone conversation is one of a monitored voice over IP (VOIP) telephone call or a monitored plain old telephone system (POTS) call.

9. A method for detecting a three-way call in a monitored telephone conversation, the method comprising: converting an audio of the monitored telephone conversation to a transcript using a domain specific language model; detecting a pattern within the monitored telephone conversation based on the transcript, the pattern being an indicator of the three-way call; generating output information based on the pattern, the output information including timing information and weighting information corresponding to the pattern; and determining that the monitored telephone conversation includes the three-way call based on the output information.

10. The method according to claim 9, further comprising: storing at least one of the audio and the transcript to a database.

11. The method according to claim 9, further comprising: tagging the transcript with a timestamp at a starting point of the three-way call in the monitored telephone conversation.

12. The method according to claim 11, further comprising: providing a synchronized presentation of the audio and the transcript.

13. The method according to claim 9, wherein converting the audio of the monitored telephone conversation to the transcript using the domain specific language model comprises: translating the transcript from a first language into a second language.

14. The method according to claim 9, wherein detecting the three-way call based on the output information comprises: determining a value corresponding to the pattern; and determining that the value is greater than a predetermined threshold.

15. The method according to claim 9, wherein the output information is a matrix comprising a plurality of characteristics or patterns identified within the transcript.

16. The method according to claim 9, wherein the pattern includes one or more of a frequency of an utterance of a phrase during the monitored telephone conversation, a timing of the utterance of the phrase during the monitored telephone conversation, or an entity extracted from the utterance of the phrase.

17. A non-transitory computer-readable device having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising: converting an audio of a monitored telephone conversation to a transcript using a domain specific language model; detecting a pattern within the monitored telephone conversation based on the transcript, the pattern being an indicator of the three-way call; generating output information based on the pattern, the output information including timing information and weighting information corresponding to the pattern; and determining that the monitored telephone conversation includes the three-way call based on the output information.

18. The non-transitory computer-readable device of claim 17, the operations further comprising: tagging the transcript with a timestamp at a starting point of the three-way call in the monitored telephone conversation.

19. The non-transitory computer-readable device of claim 18, wherein the output information is a matrix comprising a plurality of characteristics or patterns identified within the transcript.

20. The non-transitory computer-readable device of claim 17, wherein the pattern includes one or more of a frequency of an utterance of a phrase during the monitored telephone conversation, a timing of the utterance of the phrase during the monitored telephone conversation, or an entity extracted from the utterance of the phrase.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram illustrating a speech recognition system according to an embodiment of the present invention.

(2) FIG. 2 is a flow diagram illustrating a three way call detection method according to an embodiment of the present invention.

(3) FIG. 3 is a drawing illustrating a scoring algorithm according to an embodiment of the present invention.

(4) FIG. 4 is a block diagram illustrating a query and retrieval system according to an embodiment of the present invention.

(5) FIG. 5 is a sample output screen according to an embodiment of the query and retrieval system of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

(6) The following description illustrates the invention with reference to a real world domain, and in particular with respect to correctional facilities. However, it should be understood that the practice of the invention is not limited to any particular domain or application. Rather, it is intended to be more generally applicable to any domain where there is a need for three way call detection by examining the contents and call characteristics of recorded or live telephone conversations.

(7) Unlike traditional three-way call detection systems, the primary purpose of which is to terminate a telephone conversation upon establishment of a three-way call, the present invention identifies the three-way call and allows the call to proceed. A user of the system, such as a corrections officer or investigator, may then search for monitored calls during which at least one three-way call took place, retrieve such a call, and proceed directly to the point where a third party was added, for example to gather intelligence or evidence of inappropriate conduct.

(8) The following is a portrayal of one example of a typical sequence of events that might occur when an inmate or similar person makes a three way call:

(9) 1. The inmate first places a call to a telephone number that is not blocked or otherwise restricted by the correctional facility. The recipient of the call answers the phone.

(10) 2. At some point during the telephone conversation, the inmate asks the recipient to place a three way call. When making this request, the inmate might dictate a telephone number or mention the name or organization of the third party to call.

(11) 3. The recipient then might put the inmate on hold while the call is being made. Alternatively, the recipient might use a separate phone (e.g., a separate cell phone) to place the three way call. Certain call characteristics suggestive of a three-way call, such as a gap in speech or a series of dial tones or pulses, may be observed, and may possibly be followed by a ring tone.

(12) 4. The third party answers the phone and commences conversing with the inmate. The third party will typically answer with a phrase commonly used when answering the telephone such as “hello,” “hi,” or another salutation or greeting, or stating the name of a person or organization. At this point a speaker recognition (or speaker turn) component of the speech recognition system may detect that a third and previously undetected speaker has been added to the call. It is important to note that the addition of a third and previously undetected speaker in the conversation by itself is not dispositive of a three way call. It could simply be another member of the household of the original recipient of the call.

(13) 5. After a while the inmate or three way call recipient decides to terminate the three way call. At this point the inmate may do one of three things: terminate the original call, continue to converse with the original recipient, or make another three way call.

(14) 6. If another three way call is made, then the same or similar sequence of events from (1) through (5) applies.

(15) The present invention focuses on and analyzes the three way call characteristics described above (e.g., verbal and non-verbal cues, such as telephone numbers, names of people or organizations to call, salutations, dial tones, dial pulses, ring tones, gaps in conversation, speaker turns/recognition of new speakers, and other three way call characteristics). A speech recognition processor, in combination with a plurality of other algorithms and processors, searches the monitored call for clues that would indicate the addition of one or more third parties to a telephone conversation, while a three-way call detection processor evaluates the clues to determine a probability that the call included at least one three-way call.

(16) FIG. 1 illustrates a block diagram of a speech recognition system 100 according to an embodiment of the present invention. The speech recognition system 100 may be software-implemented (e.g., a software program executed by one or more computer systems or speech recognition processors), hardware-implemented (e.g., as a series of instructions stored in one or more solid-state devices), or a combination of both. It should also be understood that multiple instances of the speech recognition system 100 may be simultaneously executed on a single computer or on multiple computers.

(17) In FIG. 1, the speech recognition process 104 processes previously recorded telephone conversations 106 or ongoing telephone conversations 102, which are referred to interchangeably herein as “monitored telephone conversations,” “telephone conversations,” “conversations,” or simply “calls.” The speech recognition process 104 performs a number of functions, of which one is converting the spoken audio to text (transcription). In doing so, the speech recognition process utilizes at least one language model 108. When transcribing speech to text, it is desirable to ensure that the language model used is domain-specific, which enhances the accuracy of the transcription process. A “domain-specific” language model is a language model that accurately reflects the linguistic nuances of the participants of a telephone conversation, for example a language model that is domain-specific to inmate telephony. Preferably, therefore, at least one domain specific language model is used by the system in transcribing the audio of the monitored telephone conversation to text.

(18) In some embodiments of the invention, it is contemplated that multiple domain-specific language models may be used, which may be trained for a plurality of ethnic groups, a plurality of regional dialects, or other language differences. Using multiple domain-specific language models has been shown to significantly improve speech recognition and transcription accuracy. It is also contemplated that, in instances where foreign languages are spoken, multiple domain-specific language models trained for a plurality of foreign languages may be used. Further, a translation processor may be utilized to translate the transcript of the monitored telephone conversation from a first language to a second language (e.g., to translate a conversation in Spanish into English).

(19) In addition to converting spoken audio to text, the speech recognition process extracts a number of verbal and non-verbal characteristics from the telephone conversation. These include, but are not limited to, speaker turns (e.g., as determined by voice-recognition of the speakers in the telephone conversation); gaps in audio; dial, pulse, and ring tones; verbal cues (e.g., mentions of telephone numbers, mentions of people or organizations to call, or telephone salutations such as “hello”); speech and phraseology patterns; and timing information that includes the beginning and end times (e.g., as measured in either seconds or milliseconds from the beginning of the telephone conversation) of utterances, audio gaps, dial, pulse and ring tones, and speaker turns. The characteristics are preferably associated with the transcript of the monitored telephone conversation.

(20) Once the speech recognition process has completed processing the telephone conversation, it outputs the results to a file. One suitable format for the output file is an XML file 112. The output file is then processed by an output processor 110 that extracts each component from the XML file and inserts it as a call record into a multimedia database 114, for example as a binary large object (BLOB). That is, as illustrated in FIG. 1, the multimedia database stores the transcript of the monitored telephone conversation, the associated characteristics of the monitored telephone conversation, and, in some embodiments of the invention, a sound recording of the monitored telephone conversation. The sound recording may, of course, be analog or digital. It is also within the spirit and scope of the present invention to store the recordings of the monitored telephone conversation in a file system external to the multimedia database, in which case the multimedia database preferably includes appropriate references to the external file system. The telephone conversation is now ready for three way call detection.

(21) FIG. 2 illustrates a flow chart diagram of a three way call detection method 200 according to an embodiment of the present invention. As with the speech recognition method illustrated in FIG. 1 and described above, it is contemplated that the three-way call detection method illustrated in FIG. 2 and described below may be hardware-implemented, software-implemented, or both hardware- and software-implemented. It is also contemplated that multiple instances of the three-way call detection method may run simultaneously on one or more computers or by one or more three-way call detection processors.

(22) In the embodiment illustrated in FIG. 2, the three way call detection method begins by utilizing a call retrieval process 202 to retrieve a previously unprocessed call from the multimedia database 114. The transcribed text, as well as information regarding the associated call characteristics, may be forwarded to the call processing algorithms 210. The call processing algorithms employ a variety of techniques including, but not limited to: entity extraction (e.g., algorithms that extract people, places, organizations, telephone numbers and other entities from the transcribed text); algorithms that measure the frequency of phrases uttered during the call; algorithms that measure the timing of phrases uttered during the call; pattern detection techniques that compare the timing of phrases and entities uttered with the timing of extracted telephone conversation characteristics such as gaps in speech, dial tones, pulses and speaker turns; and any combinations thereof For example, one call processing algorithm may compare the timing of the utterance of a telephone number with the timing of a gap in the conversation; if the two are close in time (e.g., a telephone number is uttered and, a few seconds later, a gap in the conversation occurs), it may indicate establishment of a three-way call.

(23) The call processing algorithms seek characteristics or patterns indicative of a three-way call. If any characteristics or patterns indicative of a three-way call are identified at a particular point in the telephone conversation, the call processing algorithms may output a matrix of the identified characteristics and/or patterns, along with timing information and a weighting structure that weights each pattern and characteristic with respect to its relative importance as a characteristic or pattern indicative of a three-way call. For example, a gap in the conversation may have a lower weight than an utterance of a telephone number followed by a gap in the conversation, which may have a lower weight than an utterance of a telephone number followed by a gap in the conversation in which dial tones are detected, which may have a lower weight than an utterance of a telephone number followed by a gap in the conversation in which dial tones are detected and after which a new speaker is identified by the speech recognition processor. The output matrix is then forwarded to the scoring algorithm 208. If no three way call characteristics are identified, control is returned to the call retrieval process, which then retrieves the next unprocessed telephone conversation from the multimedia database.

(24) The scoring algorithm 208 then computes a score based on the output matrix of the extracted call characteristics and patterns. The score is indicative of a likelihood or probability that at least one third party was added to the monitored telephone conversation. In some embodiments of the invention, the score is computed as follows:

(25) $S=\underset{i=l}{\overset{n}{.Math.}}{w}_i{x}_i,$
where S is the interim score, n is the number of characteristics and patterns in the output matrix of the call processing algorithms 210, w.sub.i is the weight of the i.sup.th characteristic or pattern in the matrix, and x.sub.i is the i.sup.th characteristic or pattern in the matrix. Note that x.sub.i can be represented by binary, integer, or continuous values. In other embodiments of the invention, the scoring algorithm may calculate the value of S using an associative artificial neural network, such as an associative network, for example Kohonen networks.

(26) The interim score S may optionally be further refined by the use of a logistic function to produce values of between zero and one as follows:

(27) $SF=\frac{1}{1+e^{-s}},$
where SF is the final score. A graphical representation of the two scoring equations outlined above is illustrated in FIG. 3, where F(S) is represented by the preceding equation. Also in FIG. 3, a bias is added to the calculation of S. If the interim score S is not further refined, then the interim score S becomes the final score SF.

(28) Referring once again to FIG. 2, after the final score has been calculated, it is determined whether it exceeds a minimum requirement to indicate that a three way call took place. For example, the final score may be compared to a threshold in order to minimize the potential for false positives (e.g., identifications of three-way calls where no three-way calls in fact occurred). If the threshold is exceeded, control is passed to the tagging algorithm 206, which will be further described below. Otherwise, if the end of the current call has been reached, control is passed back to the call retrieval process 202. If the end of the call has not been reached, control is passed back to the call processing algorithms 210, which will then continue processing the call.

(29) Upon successful detection of a three way call, information regarding the addition of a third party to the telephone conversation may be generated. For example, the tagging process 206 may tag the telephone conversation being processed as including at least one three-way call, and may also tag each phrase, pattern, or point in the transcript that is pertinent to a three way call. The tags may then be added to the multimedia database 114 as part of the call record along with the final score for the telephone conversation output by the scoring algorithm.

(30) The tagging process 206 then passes control to the synchronization process 204. The synchronization process identifies the likely beginning of the detected three way call and updates the call record produced by the tagging process 206 with information regarding the likely beginning of the detected three-way call. For example, the synchronization process 204 may add a time stamp or other timing information that identifies the number of seconds from the beginning of the call at which the three-way call took place. The synchronization process 204 may also add information about how many words or characters into the transcript the three way call occurred. The multimedia database 114 may also be updated to include at least one index of detected three-way calls and the characteristics associated with those calls.

(31) If the end of the call has been reached, control is returned to the call retrieval process. Otherwise, control is returned to the call processing algorithms.

(32) FIG. 4 is a block diagram of a generalized query and retrieval system 400 according to an embodiment of the present invention that may be used to retrieve records from the multimedia database 114, and in particular may be used to retrieve records of three-way calls from the multimedia database 114. The call browser and display 402 of FIG. 4 may be used to query and retrieve detected three way calls from the multimedia database 114. The call browser and display 402 preferably accepts queries based upon scores. For example, a user may request calls having scores that fall within a particular range or that exceed a particular threshold. It is contemplated, however, that the call browser and display 402 may accept any type of criteria on which to search the multimedia database 114 (e.g., call dates, call times, or the identity of a party to the call). A suitably-programmed processor may retrieve one or more call records meeting the specified criteria from the multimedia database 114.

(33) FIG. 5 shows a sample output screen 500 of a query and retrieval system according to the present invention. Panel 502 displays the call records retrieved in response to a particular query and permits a user to select one of the retrieved call records. Panel 504 shows the transcript of the selected call record. The “Play Call” button 506 will initiate playback of the recording of the selected telephone conversation. Preferably, the query and retrieval system includes an output device capable of providing synchronized playback and visualization, respectively, of the recording and transcript of the selected telephone conversation. For example, as the audio recording of the call is played back, a moving highlight may track through the transcript in panel 504.

(34) The output device further preferably provides a visual identification of the one or more characteristics of the monitored telephone conversation that are indicative of the establishment of a three-way call. For example, the transcript in panel 504 may be highlighted or otherwise flagged at the point where the three-way call was established or throughout the three-way call (e.g., the entire portion of the transcript covering the three-way call may be bold faced). Similarly, the panel 508 may show a time stamp 510 of when in the call the three-way call was established, and, optionally, when it was terminated. In some embodiments of the invention, the time stamp 510 may be used to navigate directly (or “jump”) to the point in the conversation where the three-way call took place, thereby advantageously permitting law enforcement officials to review only the portion of the conversation that is of particular interest.

(35) Although only a few illustrative embodiments of this invention have been described above with a certain degree of particularity, those skilled in the art could make numerous alterations to the disclosed embodiments without departing from the spirit or scope of this invention. It is intended that all matter contained in the above description or shown in the accompanying drawings shall be interpreted as illustrative only and not limiting. Changes in detail or structure may be made without departing from the spirit of the invention as defined in the appended claims.