Mask estimation apparatus, model learning apparatus, sound source separation apparatus, mask estimation method, model learning method, sound source separation method, and program

Abstract

A mask estimation apparatus for estimating mask information for specifying a mask used to extract a signal of a specific sound source from an input audio signal includes a converter which converts the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model and a mask calculator which calculates the mask information by fitting the embedded vectors to a mixed Gaussian model.

Claims

1. A mask estimation apparatus for estimating mask information for specifying a mask used to extract a signal of a specific sound source from an input audio signal, the mask estimation apparatus comprising: a processor; and a memory that includes instructions, which when executed, cause the processor to execute the following steps: converting the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model; and calculating the mask information by fitting the embedded vectors to a mixed Gaussian model.

2. The mask estimation apparatus according to claim 1, wherein the steps executed by the processor further include: performing calculation of the mask information such that a likelihood of parameters of the mixture Gaussian distribution with respect to the embedded vectors is maximized.

3. The mask estimation apparatus according to claim 1, wherein the trained neural network model has been trained with respect to input audio signals for learning prepared in advance such that mask information obtained by performing processing of the processor approaches mask information of a correct solution prepared in advance.

4. A model learning apparatus for learning a neural network model used to estimate mask information for specifying a mask used to extract a signal of a specific sound source from an input audio signal, the model learning apparatus comprising: a processor; and a memory that includes instructions, which when executed, cause the processor to execute the following steps: converting input audio signals for learning and for which mask information is known in advance into embedded vectors of a predetermined dimension using the neural network model; calculating a mask information estimation result by fitting the embedded vectors to a mixed Gaussian model; and updating parameters of the neural network model until a result of comparison between the mask information estimation result and the mask information known in advance satisfies a predetermined criterion.

5. A sound source separation apparatus for extracting a signal of a specific sound source from an input audio signal, the sound source separation apparatus comprising: a processor; and a memory that includes instructions, which when executed, cause the processor to execute the following steps: converting the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model; calculating mask information for specifying a mask used to extract a signal of a specific sound source from the input audio signal by fitting the embedded vectors to a mixture Gaussian model; and extracting the signal of the specific sound source from the input audio signal using the mask information.

6. A mask estimation method executed by a mask estimation apparatus for estimating mask information for specifying a mask used to extract a signal of a specific sound source from an input audio signal, the mask estimation method comprising: converting the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model; and calculating the mask information by fitting the embedded vectors to a mixed Gaussian model.

7. A model learning method executed by a model learning apparatus for learning a neural network model used to estimate mask information for specifying a mask used to extract a signal of a specific sound source from an input audio signal, the model learning method comprising: converting input audio signals for learning and for which mask information is known in advance into embedded vectors of a predetermined dimension using the neural network model; calculating a mask information estimation result by fitting the embedded vectors to a mixed Gaussian model; and updating parameters of the neural network model until a result of comparison between the mask information estimation result and the mask information known in advance satisfies a predetermined criterion.

8. A sound source separation method executed by a sound source separation apparatus for extracting a signal of a specific sound source from an input audio signal, the sound source separation method comprising: converting the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model; calculating mask information for specifying a mask used to extract a signal of a specific sound source from the input audio signal by fitting the embedded vectors to a mixture Gaussian model; and extracting the specific sound source from the input audio signal using the mask information.

9. A non-transitory storage medium for storing a program for causing a computer to execute a mask estimation method, wherein the mask estimation method comprising: converting the input audio signal into embedded vectors of a predetermined dimension using a trained neural network model; and calculating the mask information by fitting the embedded vectors to a mixed Gaussian model.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a diagram illustrating a system configuration example in an embodiment of the present invention.

(2) FIG. 2 is a diagram illustrating a functional configuration of a mask estimation apparatus in Embodiment 1 of the present invention.

(3) FIG. 3 is a diagram illustrating a functional configuration of a mask calculator of the mask estimation apparatus in Embodiment 1 of the present invention.

(4) FIG. 4 is a diagram illustrating a functional configuration of a model learning apparatus in Embodiment 2 of the present invention.

(5) FIG. 5 is a diagram illustrating a functional configuration of a sound source separation apparatus in Embodiment 3 of the present invention.

(6) FIG. 6 is a diagram illustrating a hardware configuration example of each apparatus.

DESCRIPTION OF EMBODIMENTS

(7) Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(8) FIG. 1 is a diagram illustrating a system configuration example in an embodiment of the present invention. In FIG. 1, a microphone M can collect sounds from a plurality of sound sources that can be clustered into C classes S1 to SC. The microphone M outputs the collected sounds to a sound source separation apparatus 30 as an audio signal. However, the sound source separation apparatus 30 of the present embodiment is not limited to cases in which sounds collected through a microphone are directly input to the sound source separation apparatus 30, and the operation may be performed by storing sounds collected through a microphone or the like in media, a hard disk or the like and reading the stored audio signals with the sound source separation apparatus 30, for example.

(9) The sound source separation apparatus 30 is an apparatus that receives audio signals as an input and outputs a signal of a specific sound source. The sound source separation apparatus 30 can output a signal of a specific sound source alone or output a signal of each sound source. The sound source separation apparatus 30 uses a mask for extracting a signal of a specific sound source from input audio signals. The sound source separation apparatus 30 uses a neural network model in order to estimate mask information for specifying a mask. Meanwhile, the mask information includes parameters for determining a mask as well as information on the mask. The sound source separation apparatus 30 learns the neural network model using input audio signals for learning prepared in advance during learning (mask information is assumed to be known in advance). During operation, the sound source separation apparatus 30 calculates mask information using the neural network model (a neural network for which parameters have been set) trained during learning.

(10) Meanwhile, learning of the neural network model and calculation of the mask information in the sound source separation apparatus 30 may be performed by separate apparatuses or the same apparatus. In the description of the following embodiment, they are performed by separate devices: a model learning apparatus; and a mask estimation apparatus.

Embodiment 1: Mask Estimation Apparatus

(11) In Embodiment 1, a mask estimation apparatus which receives input audio signals that are mixed audio signals in which sounds from a plurality of sound sources are mixed and outputs mask information for specifying a mask used to extract a specific sound source from the input audio signals or mask information for specifying a mask used to separate a signal for each sound source from the input audio signal will be described.

(12) FIG. 1 illustrates a functional configuration of the mask estimation apparatus 10 in Embodiment 1 of the present invention. The mask estimation apparatus 10 includes a converter 110 and a mask calculator 120.

Converter

(13) The converter 110 is a neural network that receives input audio signals as an input and outputs vectors of a predetermined dimension (embedded vectors). Although the type of the neural network is not particularly limited, for example, the bi-directional long short-term memory (BLSTM) described in NPL 1 may be used. In the following description, the BLSTM will be exemplified.

(14) The neural network model (each parameter of the BLSTM in this example) is trained in advance by a model learning apparatus which will be described later. The converter 110 converts input audio signals into embedded vectors through a trained neural network as described below.

(15) First, the converter 110 performs time frequency analysis of input audio signals, inputs a feature quantity of each time frequency point to the neural network and obtains an embedded vector corresponding to an index n of a time frequency point, which is represented by the following equation.
v.sub.n=(υ.sub.n,1, . . . , ν.sub.n,D).sup.T  [Formula 1]
Here, n∈{1, . . . , N} is an index of a time frequency point and D is a degree of embedded vectors output from the neural network.

Mask Calculator

(16) The mask calculator 120 is a processor which receives embedded vectors output from the converter 110 as an input and outputs mask information. The mask calculator 120 fits embedded vectors to a mixture Gaussian model in order to calculate mask information.

(17) Specifically, an average μ.sub.c and a covariance matrix Σ.sub.c which are parameters of the mixture Gaussian distribution and can more appropriately represent embedded vectors are estimated on the assumption that embedded vectors conform to a mixture Gaussian distribution composed of C Gaussian distributions. Here, c∈{1, . . . , C} is an index of a cluster corresponding to each sound source and C is a total number of clusters of sound sources. In addition, μ.sub.c represents the average of Gaussian distributions corresponding to a cluster c and Σ.sub.c represents a covariance matrix of the Gaussian distributions corresponding to the cluster c. A mask estimation value is obtained from the estimated parameters μ.sub.c and Σ.sub.c of the mixture Gaussian distribution according to the following equation.

(18) $\begin{array}{cc}\left[\mathrm{Formula}2\right]& \\ {\hat{M}}_{n,c}=\frac{𝒩\left(v_n;{\mu }_c,{.Math.}_c\right)}{{.Math.}_{c^{\prime }}𝒩\left(v_n;{\mu }_{c^{\prime }},{.Math.}_{c^{\prime }}\right)}& \left(1\right)\end{array}$

(19) Here,
(v; μ, Σ)  [Formula 3]
represents a probability density function of a Gaussian distribution in the case of an average μ and a covariance matrix Σ.

(20) In this manner, a mask
{circumflex over (M)}.sub.n,c  [Formula 4]
can be specified by the estimated parameters μ.sub.c and Σ.sub.c of the mixture Gaussian distribution, and thus mask information may be the mask itself or parameters of the mixture Gaussian distribution. Hereinafter, estimates of the parameters μ.sub.c and Σ.sub.c of the mixture Gaussian distribution calculated by the mask calculator 120 are represented as {circumflex over ( )}μ.sub.c and {circumflex over ( )}Σ.sub.c. Meanwhile, the symbol {circumflex over ( )} is meant to be indicated above the immediately following symbols (μ and Σ).

(21) The mask calculator 120 obtains parameters of each Gaussian distribution such that a likelihood of the parameters of the mixture Gaussian distribution with respect to embedded vectors is maximized for fitting of the mixture Gaussian model. Specifically, the parameters {circumflex over ( )}μ.sub.c and {circumflex over ( )}Σ.sub.c of each Gaussian distribution are repeatedly updated on the basis of embedded vectors with respect to all time frequency points obtained by the converter 110 such that the following log-likelihood function is maximized.
[Formula 5]
J.sub.test(μ.sub.1:C,Σ.sub.1:C)=log P(v.sub.1:N|μ.sub.1:C, Σ.sub.1:C)  (2)

(22) Parameter update may be performed, for example, using the EM (expectation-maximization) algorithm. FIG. 3 illustrates a functional configuration of the mask calculator 120 when the EM algorithm is used. The mask calculator 120 includes a first update unit 121, a second update unit 122, a determination unit 123, and a storage unit 124.

(23) The storage unit 124 is a storage device that stores initial values or updated values of parameters used in processing of the first update unit 121, the second update unit 122 and the determination unit 123. Initial values of the parameters {circumflex over ( )}μ.sub.c and {circumflex over ( )}Σ.sub.c of the mixture Gaussian distribution are initially stored in the storage unit 124. Initial values can be arbitrarily determined.

First Update Unit

(24) The first update unit 121 is a processor which receives embedded vectors output from the converter 110 as an input and outputs an estimate of a mask. The first update unit 121 performs processing corresponding to E-step of the EM algorithm. Specifically, the first update unit 121 calculates an estimate of the mask which is a posterior probability using the following equation.

(25) $\begin{array}{cc}\left[\mathrm{Formula}6\right]& \\ {\hat{M}}_{n,c}=\frac{𝒩\left(v_n;{\hat{\mu }}_c,{\widehat{.Math.}}_c\right)}{{.Math.}_{c^{\prime }}𝒩\left(v_n;{\hat{\mu }}_{c^{\prime }},{.Math.}_{c^{\prime }}\right)}& \left(3\right)\end{array}$

(26) Here,
{circumflex over (μ)}.sub.c,{circumflex over (Σ)}.sub.c  [Formula 7]
represents parameters stored in the storage unit 124.

Second Update Unit

(27) The second update unit 122 is a processor which receives the estimate of the mask obtained by the first update unit 121 and embedded vectors output from the converter 110 as inputs and outputs estimates {circumflex over ( )}μ.sub.c and {circumflex over ( )}Σ.sub.c of the parameters. The second update unit 122 performs processing corresponding to M-step of the EM algorithm. Specifically, the second update unit 122 updates {circumflex over ( )}.sub.c and {circumflex over ( )}Σ.sub.c using the following equation.

(28) $\begin{array}{cc}\left[\mathrm{Formula}8\right]& \\ {\hat{\mu }}_c=\frac{{.Math.}_n{\hat{M}}_{n,c}{v}_n}{{.Math.}_n{\hat{M}}_{n,c}}& \left(4\right)\\ {\widehat{.Math.}}_c=\frac{1}{{.Math.}_n{\hat{M}}_{n,c}}\underset{n}{.Math.}{\hat{M}}_{n,c}\left(v_n-{\hat{\mu }}_c\right){\left(v_n-{\hat{\mu }}_c\right)}^T& \left(5\right)\end{array}$
(Determination Unit)

(29) The determination unit 123 performs processing of repeatedly updating mask information by causing the first update unit and the second update unit to repeatedly operate until a predetermined criterion is satisfied. Then, when the predetermined criterion is satisfied, the determination unit 123 outputs the values of parameters of Gaussian distributions stored in the storage unit 124 at that time and ends processing.

(30) For example, the predetermined criterion is maximization of a log-likelihood calculated on the basis of estimates of parameters of Gaussian distributions obtained by the second update unit 122 and embedded vectors output from the converter 110. The log-likelihood can be conceived as an evaluated value of likelihood when embedded vectors are clustered on the basis of current estimates of Gaussian distribution parameters and represents repeated update of estimates of Gaussian distribution parameters for more likely clustering.

(31) Specifically, the log-likelihood of Equation (2) is obtained on the basis of estimates of Gaussian distribution parameters obtained by the second update unit 122 and the embedded vectors. Here, it is assumed that the parameters μ.sub.c and Σ.sub.c in Equation (2) are replaced with the estimates {circumflex over ( )}μ.sub.c and {circumflex over ( )}Σ.sub.c thereof. Then, if a log-likelihood calculated through previous processing of the determination unit 123 is smaller, it is determined that the criterion has been satisfied, Gaussian distribution parameters stored in the storage unit 124 or a mask estimate obtained from the Gaussian distribution parameters is output as mask information, and processing is ended. If not (if the criterion is not satisfied), it is desirable to return to the first update unit 121 to repeat processing. Alternatively, if a value obtained by subtracting the log-likelihood calculated through previous processing of the determination unit 123 from the currently calculated log-likelihood is less than a predetermined threshold value, it is determined that the criterion has been satisfied. If not, it may be determined that the criterion has not been satisfied. Alternatively, the number of repetition processes may be counted and it may be determined that a predetermined criterion has been satisfied when the number of repetition processes reaches a predetermined number.

Embodiment 2: Model Learning Apparatus

(32) In Embodiment 2, a model learning apparatus which learns the neural network model of Embodiment 1, specifically, the parameters of the neural network constituting the converter 110 when estimation of Embodiment 1 is performed, will be described.

(33) FIG. 4 illustrates a functional configuration of the model learning apparatus 20 in Embodiment 2 of the present invention. The model learning apparatus 20 includes a converter 210, a mask calculator 220, and a parameter update unit 230.

Converter

(34) The converter 210 is a neural network that receives input audio signals for learning which are mixed audio signals prepared for learning in advance as an input and outputs vectors (embedded vectors) of a predetermined degree. Here, it is assumed that the input audio signals for learning are prepared in advance for learning and mask information (teacher mask information) that is a correct solution is known in advance. The converter 210 converts the input audio signals for learning into embedded vectors using a neural network (e.g., BLSTM) as in Embodiment 1. Here, although a predetermined initial value is initially used as each parameter of the neural network, updated values of parameters are used when the parameters have been updated in the parameter update unit 230 which will be described later.

Mask Calculator

(35) The mask calculator 220 is a processor which receives embedded vectors output from the converter 210 and outputs mask information estimation results (estimated mask information). The mask calculator 120 fits embedded vectors to a mixture Gaussian model through the same method as that in Embodiment 1 in order to calculate estimated mask information. In this manner, learning of each parameter of a neural network optimal for input audio signals during operation can be realized by estimating a mask through the same method during learning and during operation.

(36) Specifically, although the mask calculator 220 obtains estimated mask information through fitting to the mixture Gaussian model, fitting of the mixture Gaussian model uses an average μ.sub.c obtained from a covariance matrix Σ.sub.c and a teacher mask M.sub.n,c provided in advance instead of the EM algorithm.

(37) The covariance matrix Σ.sub.c is common for all of c∈{1, . . . , C} and Σc=σI. Here, I is a unit matrix and σ is a scalar variable that can be set to an arbitrary value. The average μ.sub.c is calculated from the teacher mask M.sub.n,c according to the following equation.

(38) $\begin{array}{cc}\left[\mathrm{Formula}9\right]& \\ {\mu }_c=\frac{{.Math.}_n{M}_{n,c}{v}_n}{{.Math.}_n{M}_{n,c}}& \left(6\right)\end{array}$

(39) Accordingly, the estimated mask information can be obtained using the following equation.

(40) $\begin{array}{cc}\left[\mathrm{Formula}10\right]& \\ {\hat{M}}_{n,c}=\frac{𝒩\left(v_n;{\mu }_c,{.Math.}_c\right)}{{.Math.}_{c^{\prime }}𝒩\left(v_n;{\mu }_{c^{\prime }},{.Math.}_{c^{\prime }}\right)}& \left(7\right)\end{array}$
(Parameter Update Unit)

(41) The parameter update unit 230 is a processor that receives teacher mask data known in advance and estimated mask information output from the mask calculator 220 as inputs and updates each parameter of a neural network.

(42) Specifically, the parameter update unit 230 repeatedly updates each parameter of the neural network such that mask information obtained by performing processing of the converter and the mask calculator approaches mask information of a correct solution prepared in advance.

(43) For example, the parameter update unit 230 repeatedly updates each parameter of the neural network until a result of comparison between estimated mask information and teacher mask information satisfies a predetermined criterion. For example, each parameter of the neural network may be updated by defining a cost function that represents a distance between an estimated mask and a teacher mask or by defining a cost function that represents a distance between a signal calculated using the estimated mask and an input audio signal for learning. This is the same as learning processing generally performed in neural networks and it is desirable that the processing be performed through the error propagation learning method or the like. For example, it is desirable that each parameter Θ of the neural network be updated on the basis of cross entropy of the estimated mask represented by the following formula and the teacher mask M.sub.n,c using the following equation.

(44) $\begin{array}{cc}\left[\mathrm{Formula}11\right]& \\ {\hat{M}}_{n,c}& \\ \left[\mathrm{Formula}12\right]& \\ J_{\mathrm{train}}\left(\Theta \right)=-\underset{n,c}{.Math.}{M}_{n,c}\log {\hat{M}}_{n,c}& \left(8\right)\end{array}$

Embodiment 3: Sound Source Separation Apparatus

(45) In Embodiment 3, the sound source separation apparatus 30 that extracts a signal of a specific sound source from input audio signals using the mask estimation apparatus 10 of Embodiment 1 will be described. The sound source separation apparatus 30 can extract a signal of a specific sound source alone from input audio signals, separate a signal for each sound source from the input audio signals and output the signal.

(46) FIG. 5 illustrates a functional configuration of the sound source separation apparatus 30 in Embodiment 3 of the present invention. The sound source separation apparatus 30 includes a sound source separator 310 in addition to the mask estimation apparatus 10.

Sound Source Separator

(47) The sound source separator 310 is a processor that receives input audio signals and mask information output from the mask estimation apparatus 10 as inputs and outputs a signal of a specific sound source or a signal of each sound source.

(48) Specifically, the sound source separator 310 obtains and outputs a result of estimation of a signal of each sound source by applying a mask configured on the basis of mask information estimated by the mask estimation apparatus 10 to input audio signals. When only a signal from a specific sound source is desired to be output, it is desirable to obtain an estimate of a signal of the sound source by further receiving information c identifying the sound source as an input, configuring a mask from mask information corresponding to the input information identifying the sound source c from among mask information and applying the mask to input audio signals.

Hardware Configuration Example

(49) FIG. 6 illustrates a hardware configuration example of each apparatus (the mask estimation apparatus 10, the model learning apparatus 20, and the sound source separation apparatus 30) in the embodiment of the present invention. Each apparatus may be a computer including a processor such as a CPU (Central Processing Unit) 151, a memory device 152 such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a storage device 153 such as a hard disk, and the like. For example, functions and processing of each apparatus may be realized by the CPU 151 executing data and programs stored in the storage device 153 or the memory device 152. In addition, information necessary for each apparatus may be input from an input/output interface device 154 and results obtained in each apparatus maybe output from the input/output interface device 154.

Supplement

(50) Although the mask estimation apparatus, the model learning apparatus and the sound source separation apparatus according to the embodiments of the present invention have been described using functional block diagrams for convenience of description, the mask estimation apparatus, the model learning apparatus and the sound source separation apparatus according to the embodiments of the present invention may be realized by hardware, software or a combination thereof. For example, the embodiments of the present invention may be realized by a program causing a computer to execute the functions of the mask estimation apparatus, the model learning apparatus and the sound source separation apparatus according to the embodiments of the present invention, a program causing a computer to execute each procedure of the methods according to the embodiments of the present invention, and the like. In addition, functional units may be combined and used as necessary. Further, the methods according to the embodiments of the present invention may be implemented in different orders from the orders described in the embodiments.

(51) Although the method for improving the accuracy of sound source separation during operation by estimating a mask using the same method during learning and during operation has been described, the present invention is not limited to the above-described embodiments and various modifications and applications can be made without departing from the scope of the claims.

REFERENCE SIGNS LIST

(52) 10 Mask estimation apparatus

(53) 110 Converter

(54) 120 Mask calculator

(55) 121 First update unit

(56) 122 Second update unit

(57) 123 Determination unit

(58) 124 Storage unit

(59) 20 Model learning apparatus

(60) 210 Converter

(61) 220 Mask calculator

(62) 230 Parameter update unit

(63) 30 Sound source separation apparatus

(64) 310 Sound source separator

(65) 151 CPU

(66) 152 Memory

(67) 153 Storage device

(68) 154 Input/output interface device