A sound processing system, method and program product for estimating parameters from binaural audio data. A system is provided having: a system for inputting binaural audio; and a binaural signal analyzer (BICAM) that: performs autocorrelation on both the first channel and second channel to generate a pair of autocorrelation functions; performs a first layer cross-correlation between the first channel and second channel to generate a first layer cross-correlation function; removes the center peak from the first layer cross-correlation function and a selected autocorrelation function to create a modified pair; performs a second layer cross-correlation between the modified pair to determine a temporal mismatch; generates a resulting function by replacing the first layer cross correlation function with the selected autocorrelation function using the temporal mismatch; and utilizes the resulting function to determine ITD parameters and interaural level difference ILD parameters of the direct sound components and reflected sound components.

A sound processing system, method and program product for estimating parameters from binaural audio data. A system is provided having: a system for inputting binaural audio; and a binaural signal analyzer (BICAM) that: performs autocorrelation on both the first channel and second channel to generate a pair of autocorrelation functions; performs a first layer cross-correlation between the first channel and second channel to generate a first layer cross-correlation function; removes the center peak from the first layer cross-correlation function and a selected autocorrelation function to create a modified pair; performs a second layer cross-correlation between the modified pair to determine a temporal mismatch; generates a resulting function by replacing the first layer cross correlation function with the selected autocorrelation function using the temporal mismatch; and utilizes the resulting function to determine ITD parameters and interaural level difference ILD parameters of the direct sound components and reflected sound components.

A method for detecting a voice activity in an input audio signal composed of frames includes that a noise characteristic of the input signal is determined based on a received frame of the input audio signal. A voice activity detection (VAD) parameter is derived based on the noise characteristic of the input audio signal using an adaptive function. The derived VAD parameter is compared with a threshold value to provide a voice activity detection decision. The input audio signal is processed according to the voice activity detection decision.

A method for detecting a voice activity in an input audio signal composed of frames includes that a noise characteristic of the input signal is determined based on a received frame of the input audio signal. A voice activity detection (VAD) parameter is derived based on the noise characteristic of the input audio signal using an adaptive function. The derived VAD parameter is compared with a threshold value to provide a voice activity detection decision. The input audio signal is processed according to the voice activity detection decision.

A method of speech enhancement for target speakers is presented. A blind source separation (BSS) module is used to separate a plurality of microphone recorded audio mixtures into statistically independent audio components. At least one of a plurality of speaker profiles are used to score and weight each audio components, and a speech mixer is used to first mix the weighted audio components, then align the mixed signals, and finally add the aligned signals to generate an extracted speech signal. Similarly, a noise mixer is used to first weight the audio components, then mix the weighted signals, and finally add the mixed signals to generate an extracted noise signal. Post processing is used to further enhance the extracted speech signal with a Wiener filtering or spectral subtraction procedure by subtracting the shaped power spectrum of extracted noise signal from that of the extracted speech signal.

A method of speech enhancement for target speakers is presented. A blind source separation (BSS) module is used to separate a plurality of microphone recorded audio mixtures into statistically independent audio components. At least one of a plurality of speaker profiles are used to score and weight each audio components, and a speech mixer is used to first mix the weighted audio components, then align the mixed signals, and finally add the aligned signals to generate an extracted speech signal. Similarly, a noise mixer is used to first weight the audio components, then mix the weighted signals, and finally add the mixed signals to generate an extracted noise signal. Post processing is used to further enhance the extracted speech signal with a Wiener filtering or spectral subtraction procedure by subtracting the shaped power spectrum of extracted noise signal from that of the extracted speech signal.

A computing device receives a first audio waveform representing a first utterance and a second utterance. The computing device receives identity data indicating that the first utterance corresponds to a first speaker and the second utterance corresponds to a second speaker. The computing device determines, based on the first utterance, the second utterance, and the identity data, a diarization model configured to distinguish between utterances by the first speaker and utterances by the second speaker. The computing device receives, exclusively of receiving further identity data indicating a source speaker of a third utterance, a second audio waveform representing the third utterance. The computing device determines, by way of the diarization model and independently of the further identity data of the first type, the source speaker of the third utterance. The computing device updates the diarization model based on the third utterance and the determined source speaker.

A computing device receives a first audio waveform representing a first utterance and a second utterance. The computing device receives identity data indicating that the first utterance corresponds to a first speaker and the second utterance corresponds to a second speaker. The computing device determines, based on the first utterance, the second utterance, and the identity data, a diarization model configured to distinguish between utterances by the first speaker and utterances by the second speaker. The computing device receives, exclusively of receiving further identity data indicating a source speaker of a third utterance, a second audio waveform representing the third utterance. The computing device determines, by way of the diarization model and independently of the further identity data of the first type, the source speaker of the third utterance. The computing device updates the diarization model based on the third utterance and the determined source speaker.

Apparatus and methods related to separation of audio sources are provided. The method includes receiving an audio waveform associated with a plurality of video frames. The method includes estimating, by a neural network, one or more audio sources associated with the plurality of video frames. The method includes generating, by the neural network, one or more audio embeddings corresponding to the one or more estimated audio sources. The method includes determining, based on the audio embeddings and a video embedding, whether one or more audio sources of the one or more estimated audio sources correspond to objects in the plurality of video frames. The method includes predicting, by the neural network and based on the one or more audio embeddings and the video embedding, a version of the audio waveform comprising audio sources that correspond to objects in the plurality of video frames.

An estimation device includes a memory, and processing circuitry coupled to the memory and configured to receive an input of an input audio signal that is an audio signal in which sounds from a plurality of sound sources are mixed, and an input of supplemental information, and output an estimation result of mask information that identifies a mask for extracting a sound of any one of the sound sources included in an entire or a part of a signal included in the input audio signal, the signal being identified by the supplemental information, cause a neural network to iterate a process of outputting the estimation result of the mask information, and cause the neural network to output an estimation result of the mask information for a different sound source, by inputting a different piece of the supplemental information to the neural network at each iteration.