Method for encoding multiple microphone signals into a source-separable audio signal for network transmission and an apparatus for directed source separation

Abstract

A method is provided for encoding multiple microphone signals into a composite source-separable audio (SSA) signal, conducive for transmission over a voice network. The embodiments enable the processing of source separation of the target voice signal from its ambient sound to be performed at any point in the voice communication network, including the internet cloud. A multiplicity of processing is possible over the SSA signal, based on the intended voice application. The level of processing is adapted with the availability of the processing power at the chosen processing node in the network in one embodiment. An apparatus for separating out the target source voice from its ambient sound is also provided. The apparatus includes a directed source separation (DSS) unit, which processes the two virtual microphone signals in the SSA representation, to generate a new SSA signal including the enhanced target voice and the enhanced ambient noise.

Claims

1. A method for network transmission of voice .Iadd.captured through a plurality of microphones spatially disposed in a first group and a second group.Iaddend., comprising: combining two digital audio signals into a composite source separable audio (SSA) signal, each digital audio signal of the two digital audio signals representing an independent mixture of a target source voice and an ambient noise, wherein outputs of the plurality of microphones within the first group are summed together as a first .[.output.]. .Iadd.digital audio signal of the two digital audio signals .Iaddend.and .[.the.]. outputs of the plurality of microphones within the second group are summed together as a second .[.output.]. .Iadd.digital audio signal of the two digital audio signals.Iaddend., thereby defining a first virtual microphone and a second virtual microphone, respectively, and wherein the combining .[.process.]. comprises interleaving the two .Iadd.digital .Iaddend.audio signals to generate the composite SSA signal.Iadd.; and separating the two digital audio signals within the composite SSA signal into two mono audio signals by performing a first instance of directed source separation (DSS) on the composite SSA signal, the DSS comprising: generating one or more control signals indicating an instantaneous signal-to-noise ratio, in the composite SSA signal, between the target source voice and the ambient noise; under direction of the one or more control signals, separating the target source voice of the composite SSA signal into a first mono audio signal; and under direction of the one or more control signals, separating the ambient noise of the composite SSA signal into a second mono audio signal.Iaddend..

.[.2. A method of claim 1, further comprising: separating the two audio signals within the composite SSA signal into two mono audio signals by performing directed source separation (DSS)..].

3. A method of claim 1, wherein .[.the two audio signals are analog signals and the combining process comprises delaying the second audio signal and summing the.]. .Iadd.a first output of a first microphone of the first group is .Iaddend.delayed .Iadd.before being summed with a second output of a .Iaddend.second .[.audio signal with the first audio signal to generate the composite SSA signal.]. .Iadd.microphone of the first group to generate the first digital audio signal.Iaddend..

4. A method of claim 1, wherein the composite SSA signal is intelligible for human listening without requiring any further processing.

.[.5. A method of claim 2, comprising: performing a first ambient sound separation process for human listening intelligibility; and performing a second ambient sound separation process for a machine voice application..].

6. A method of claim 1, wherein a quality of ambient sound separation is traded off gracefully, depending on .[.the.]. .Iadd.an .Iaddend.availability of processing power.

.[.7. A method of claim 1, wherein the separating is performed in an intermediate server in a network cloud..].

.Iadd.8. The method of claim 1, comprising: separating the two digital audio signals within the composite SSA signal into another two mono audio signals by performing a second instance of DSS on the composite SSA signal; wherein the two mono audio signals include a first level of the ambient noise of the composite SSA signal; wherein the other two mono audio signals include a second level of the ambient noise of the composite SSA signal; wherein the first level is different than the second level..Iaddend.

.Iadd.9. The method of claim 1, wherein the separating is performed in an intermediate server in a network cloud..Iaddend.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The embodiments of the invention are illustrated by way of examples and not be interpreted by way of limitation in the accompanying drawings.

(2) FIG. 1 describes the use case scenarios, where a single microphone is not able to deal well with ambient noise and acoustic echo.

(3) FIG. 2 illustrates the use of a second microphone and associated beam forming to mitigate the ambient noise and acoustic echo.

(4) FIG. 3 reviews the concept of blind source separation (BSS).

(5) FIGS. 4A and 4B illustrate the concept of a virtual microphone for an exemplary tablet computer in accordance with one embodiment.

(6) FIG. 5 and FIG. 6 illustrate the concept of virtual microphone for an exemplary binaural headset in accordance with one embodiment.

(7) FIG. 7 depicts the block schematic representation of the directed source separation (DSS) processing in accordance with one embodiment.

(8) FIG. 8. illustrates the concept of loudspeaker signal pre-processing to further facilitate DSS for acoustic echo suppression in accordance with one embodiment.

(9) FIG. 9 illustrates the simplification of connectivity introduced by this invention in harnessing the benefits of a multiplicity of microphones in accordance with one embodiment.

(10) FIG. 10 shows the different representations of the SSA signal in accordance with one embodiment.

(11) FIG. 11 shows how a mono SSA signal can be converted back to composite (stereo) SSA in accordance with one embodiment.

(12) FIG. 12 depicts the flow of the SSA signal through the network in accordance with one embodiment.

(13) FIG. 13 shows that multiple SSA signals may be mixed for voice conferencing in accordance with one embodiment.

(14) FIG. 14 shows an application where two independent calls can benefit from SSA in accordance with one embodiment.

(15) FIG. 15 depicts the notion the DSS processing may be specialized for different applications in accordance with one embodiment.

(16) FIG. 16 shows how a slowly varying sensor signal may be multiplexed into a SSA signal in accordance with one embodiment.

(17) FIG. 17 depicts the process by which a composite audio signal is generated in accordance with one embodiment.

(18) FIG. 18 depicts the use of a statistical signal processing technique for generating a noise estimate from the composite audio signal for performing the required voice and noise separation in accordance with one embodiment.

DETAILED DESCRIPTION

(19) While several details are set forth, it is understood that some embodiments of the invention may be practiced without these details. In some instances, well-known circuits and techniques have not been shown in detail so as not to obscure the understanding of this description.

(20) As mentioned above two microphones in the beam forming array may provide some mitigation, however, it is possible to do much better with more than two microphones. Increasing the number of microphones brings several scaling hurdles with it, such as: 1. Hardware Hurdle: The standard stereo audio jacks do not support more than two channels. There is also the cost of wiring and the need for multiple channel codec. 2. Bandwidth Hurdle: Wireless connectivity such as Bluetooth and digital enhanced cordless telecommunication (DECT) do not support more than two channels. Also, this is expensive to route more than two audio channels over the internet. 3. Processing Hurdle: The availability of processing power on small form-factor devices is limited due to the battery life constraint.

(21) With advances in server technology, the processing hurdle may be overcome by moving processing to the cloud, making the consumer clients thinner and lighter. With the advent of personal WiFi routers connected to the internet via 3G/4G cellular network, it is becoming more and more feasible to defer voice processing to the cloud.

(22) To overcome the hardware and bandwidth hurdle, it is desirable to reduce the outputs of multiple microphones into a signal, whose required bandwidth does not increase with the increase in the number of microphones. This reduction or encoding should be achievable using hardware circuitry, such as a summer. The encoding needs to preserve the useful information from multiple microphones with respect to the applications mentioned herein which benefit from the use of multiple microphones.

(23) In the embodiments described above, a plurality of microphones is bifurcated into two groups. FIGS. 4A and 4B, depicts two such groupings for the use case of a tablet computer or a net TV. In FIG. 4A microphones 041 are positioned to assume the need to discriminate target voice from ambient noise along the horizontal direction. In FIG. 4B microphones 049 are positioned to assume that the target voice needs to be discriminated from ambient noise along both horizontal and vertical directions. In both these cases, the preferred direction of the target voice is perpendicular to the device. However, the voice source could itself be moving in the vicinity of the preferred direction. The algorithm adapts dynamically to the changing angles of incidence of target voice. As can be seen, the microphone groupings are organized to be roughly symmetrical with respect to the preferred angle of incidence of the target voice. The summed outputs of the microphones in each of the groups are called virtual microphone 1 (042 and 047, respectively) and virtual microphone 2 (043 and 048, respectively). For a second embodiment of the invention, consider four microphones placed on a wired headset 051, as illustrated in FIG. 5 and FIG. 6. The microphones are bifurcated into two groups, namely virtual microphone group 1, 065 (microphone 052) and virtual microphone group 2, .[.064.]. .Iadd.066 .Iaddend.(microphones 053, 054 and 055).

(24) In all the above cases, the impact of target voice from the desired look direction is similar on both the virtual microphones. The impact of ambient noise is relatively dissimilar on the two virtual microphones. A shown in FIG. 7, the outputs of the two virtual microphones, 072 and 073, are bundled together into one entity, i.e., the composite Source Separable Audio (SSA). The dissimilarity between the two virtual microphones is exploited by block 075, to generate control signals indicating the presence, or likelihood, of target voice and ambient noise. The control signals indicate the instantaneous signal-to-noise ratio between target voice and ambient noise. The cross coupled Directed Source Separator (DSS), 071, directed by the control signals is used to separate out the target voice signal into the output Channel A and the ambient noise into Channel B, collectively the output SSA, 078. There are several algorithmic approaches to source separation (often referred in literature as Blind Source Separation (BSS)).

(25) In another embodiment, the acoustic feedback from loud speakers is treated as another source of ambient noise. The plurality of microphones are placed and grouped in such a fashion that the acoustic feedback has maximally disparate impact on the two virtual microphones. In one embodiment, as shown in pre-processing module 82 in FIG. 8, the maximum disparity is achieved by pre-processing the loudspeaker channels to maximize the disparity between the acoustic outputs, while minimizing the artifacts audible to the listener. There are several pre-processing techniques to achieve the disparity. Inversion of a portion of the signal between the two channels, introducing phase difference between the two channels, and injection of a small amount of dissimilar white noise in the two channels, are exemplary pre-processing techniques to achieve the disparity.

(26) One aspect of the embodiments is the ability of simplify the hardware requirement for grouping multiple microphones into a virtual microphone. One embodiment is to passively gang or wire-sum the outputs of analog microphones, 091, as shown in FIG. 9. For example, the two terminal and three terminal electret microphones are connected in parallel to generate the virtual microphone output. Similarly, a three terminal silicon or micro electrical mechanical (MEMS) microphone is also connected in parallel. In another embodiment, for the case of a digital microphone interface, where a digital pulse digital modulation (PDM) signal is required, a plurality of analog MEMS microphone can be ganged together, 092; the output of which is fed to an analog summing input of a digital MEMS microphone, 093. Then the digital PDM output 095 will represent the output of the virtual microphone. In an alternate embodiment, it is also possible to connect multiple digital MEMS microphone by providing a circuitry to interleave the PDM outputs of the plurality of digital microphones. This multiplexer circuitry may be distributed in a modular fashion in all the component digital microphones, so they can be daisy chained together.

(27) Logically, SSA is a composite or a bundle of two audio streams, Channel A and Channel B. As shown in FIG. 10, SSA may be represented as stereo, 103, in a system which supports streaming of stereo audio. Alternatively, in a system which only supports mono, the two channels may be interleaved, 104, to create a mono stream of twice the original sampling rate. In another embodiment, the SSA signal may also be converted to a mono analog SSA signal 105, by converting the mono digital SSA 104, to analog. As shown in FIG. 11, a method is provided by which an analog audio signal of the type SSA can be detected. This is done by detecting if a target voice is panned almost similarly in the two channels. In the case of mono digital, or analog, an oversampling operation 111 is executed, clock recovery synchronization is performed, 113, and resampling 112 is executed to extract the two constituent channels.Iadd., 114 and 115.Iaddend..

(28) In another embodiment, the SSA signal may be transmitted end to end, i.e., from the plurality of microphones on the transmit end to the receiving end, through the voice communication network. Along the way, the SSA signal may be transmitted using the two channel stereo format or the mono audio format. The SSA format is such that the intermediate processing is optional. In others words, the SSA signal degenerates gracefully to a voice signal (with ambient noise) in the absence of any DSS processing. The SSA composite is agnostic to the existing voice communication network, requiring no change at the system level. The SSA composite works with any existing voice communication standard, including bluetooth and voice over Internet Protocol (VoIP). When the DSS signal processing needs to be performed, it can be done so at any point in the network shown in FIG. 12, including the audio accessory 122, transmit host 121, the intermediate server 124, in the internet cloud or the receiving host 123. The DSS processing may be performed at a quality level consistent with the availability of the processing power in the chosen processing node in the network.

(29) In another embodiment, where the inputs from the two virtual microphones are analog, an analog SSA signal is generated as shown in FIG. 17. The first audio signal (175) captured by the virtual microphone 1 (171) is an independent mixture of voice and noise, relative to the second audio signal (176) captured by the virtual microphone 2 (172). For example, there may be a built-in delay 173 of d between the voice signals arriving at the two virtual microphones. In the present embodiment, the second audio signal (176) is delayed by D and then summed with the signal 175, to generate the composite analog SSA (177). The delay D is chosen to be large enough, so the autocorrelation of the voice (speech) signal is sufficiently small. The directed separation process (DSS) to revert the SSA signal (181) into its constituents is shown in FIG. 18. With the delay D known a priori, a correlation process results in the voice estimate (182) and an anti-correlation process into a noise estimate (183). The estimates are then run through a directed source separation process to generate enhanced voice (184) and enhanced ambient noise (185).

(30) In another embodiment, it is possible for the receiving end to recover the ambient noise, while suppressing the primary source voice. For example, it may be socially interesting for the receiving listener to experience the party ambience around the transmitting talker. The ambient noise may be used by an application to determine the proximity of two talkers in one embodiment. In another example, an internal map of a shopping mall may be annotated with the ambient noise in several critical spots such as shops, to guide a phone user in reaching their target destination.

(31) In another embodiment, the SSA representation enables effective processing required for audio conferencing, as illustrated in FIG. 13. The DSS signal processing 136 is performed on two of the transmit host SSA signals 137 and then mixed together, 138, component by component to realize an .Iadd.SSA composite audio .Iaddend.output .[.SSA.]. signal .Iadd.139 .Iaddend.for the host .[.139.]. .Iadd.133.Iaddend.. A similar processing path is provided for generating the outputs required for the hosts 131 and 134.

(32) In another embodiment, the signal processing on a primary call is enhanced by taking advantage of the reference ambient sound present in another secondary call, when the two transmit parties are located in proximity. For example, if two parties are transmitting voice from the same social gathering, they are sharing the ambient noise environment. In fact, a target voice may be another's ambient noise. If the call server is aware of the situation, the server can take advantage of one call's SSA to perform better enhancement in the other call. In today's consumer gadget deployment, one can use global positioning satellite (GPS) to locate whether the two transmit hosts are in physical proximity. In the example of FIG. 14, the transmit host 141 is collocated in the proximity of the second transmit host 143. A special application running in the cloud, 145, is aware of this collocation, which takes advantage of the ambient noise estimates from both to present a better output signal to the receive host 149 and the receive host 148.

(33) The DSS signal processing requirement is different for different applications. While speech recognition is better off with silence insertion between speech segments, the discontinuity caused by the silence insertion is extremely annoying to human listener. Also, the quality of left over ambient noise is extremely important for human listening. Unlike speech recognition or voice search, voice command recognition is typically much more robust in the presence of ambient noise, hence it does not require as much processing. In another embodiment, as shown in FIG. 15, the SSA signal representation allows different applications to perform the necessary level and type of DSS signal processing. On one instance of an SSA .Iadd.composite audio .Iaddend.signal 153, the DSS 154 is optimized for human intelligibility, DSS 155 is optimized for command recognition and the DSS 156 is optimized for voice search.

(34) In another embodiment, a slowly varying (voice-band compatible) non-voice signal 161 is mixed into the Channel A 162 of the SSA composite, and it's inversion 164 is mixed into the Channel B 163, to generate a new SSA (166,167) be carried end-to-end. It is best to modulate these signals into the higher bands of the wide-band voice, so it has the least interference with voice. The said slowly varying signal is not audible to the listener, since it is suppressed by the DSS process for voice enhancement. The slow non-voice sensor signal may be GPS, Gyro, temperature, barometer, accelerometer, illumination, gaming controller, etc.

(35) With the above embodiments in mind, it should be understood that the embodiments might employ various computer-implemented operations involving data stored in computer systems. These operations are those requiring physical manipulation of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Further, the manipulations performed are often referred to in terms, such as producing, identifying, determining, or comparing. Any of the operations described herein that form part of the invention are useful machine operations. The embodiments also relates to a device or an apparatus for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations

(36) The invention can also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data, which can be thereafter read by a computer system. Examples of the computer readable medium include hard drives, network attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion. Embodiments of the present invention may be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers and the like. The invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

(37) Although the method operations were described in a specific order, it should be understood that other operations may be performed in between described operations, described operations may be adjusted so that they occur at slightly different times or the described operations may be distributed in a system which allows the occurrence of the processing operations at various intervals associated with the processing.

(38) Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.