Anomalous sound detection training apparatus, acoustic feature extraction apparatus, anomalous sound sampling apparatus, and methods and programs for the same

Abstract

An anomalous sound detection training apparatus includes: a first acoustic feature extraction unit that extracts an acoustic feature of normal sound based on training data for normal sound by using an acoustic feature extractor; a normal sound model updating unit that updates a normal sound model by using the acoustic feature extracted; a second acoustic feature extraction unit that extracts an acoustic feature of anomalous sound based on simulated anomalous sound and extracts the acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor; and an acoustic feature extractor updating unit that updates the acoustic feature extractor by using the acoustic feature of anomalous sound and the acoustic feature of normal sound that have been extracted, in which processing by the units is repeatedly performed.

Claims

1. An anomalous sound detection training apparatus comprising: processing circuitry configured to extract an acoustic feature of normal sound based on training data for normal sound by using an acoustic feature extractor; update a normal sound model by using the extracted acoustic feature; extract an acoustic feature of anomalous sound based on simulated anomalous sound and extract an acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor; and update the acoustic feature extractor by using the extracted acoustic feature of anomalous sound and the extracted acoustic feature of normal sound, wherein the extraction of the acoustic feature of the normal sound based on the training data, the updating the normal sound model using the extracted acoustic feature, the extraction of the acoustic feature of the anomalous sound and the extraction of the acoustic feature of the normal sound, and the updating the acoustic feature extractor by using the extracted acoustic feature of the anomalous sound and the extracted acoustic feature of the normal sound are repeatedly performed to simultaneously optimize the acoustic feature extractor and the normal sound model, based on a probability density function followed by a feature amount extracted from normal sound and a probability density function followed by a feature amount extracted from anomalous sound, so that a degree-of-anomaly calculated from the feature amount extracted from the normal sound is less than a degree-of-anomaly calculated from the feature amount extracted from the anomalous sound, the training data for normal sound is sound signals collected by a microphone installed near or inside a device for which anomalous sound is to be detected, and the simultaneous optimization of the acoustic feature extractor and the normal sound model is performed so as to minimize a predetermined objective function L.

2. An anomalous sound detection training apparatus comprising: processing circuitry configured to extract an acoustic feature of normal sound based on training data for normal sound by using an acoustic feature extractor; update a normal sound model by using the extracted acoustic feature; extract an acoustic feature of anomalous sound based on training data for anomalous sound and extract an acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor; and update the acoustic feature extractor by using the extracted acoustic feature of anomalous sound and the extracted acoustic feature of normal sound, wherein the extraction of the acoustic feature of the normal sound based on the training data, the updating the normal sound model using the extracted acoustic feature, the extraction of the acoustic feature of the anomalous sound and the extraction of the acoustic feature of the normal sound, and the updating the acoustic feature extractor by using the extracted acoustic feature of the anomalous sound and the extracted acoustic feature of the normal sound are repeatedly performed to simultaneously optimize the acoustic feature extractor and the normal sound model, based on a probability density function followed by a feature amount extracted from normal sound and a probability density function followed by a feature amount extracted from anomalous sound, so that a degree-of-anomaly calculated from the feature amount extracted from the normal sound is less than a degree-of-anomaly calculated from the feature amount extracted from the anomalous sound, the training data for normal sound is sound signals collected by a microphone installed near or inside a device for which anomalous sound is to be detected, and the simultaneous optimization of the acoustic feature extractor and the normal sound model is performed so as to minimize a predetermined objective function L.

3. The anomalous sound detection training apparatus according to claim 1, wherein the processing circuitry is further configured to: sample the simulated anomalous sound, wherein the simulated anomalous sound is anomalous sound simulated by making assumption on one of occurrence frequency, sparsity, and continuity in temporal frequency direction of amplitude values.

4. The anomalous sound detection training apparatus according to claim 3, wherein the simulated anomalous sound is X.sub.ω,n as sampled based on a distribution: $\begin{array}{cc}{X}_{\omega ,n}~\frac{1}{\beta }\exp \left\{-\frac{1}{\beta }{X}_{\omega ,n}\right\}& \left(26\right)\end{array}$ where ω is frequency, n is frame number, and β is an integer.

5. An anomalous sound detection training method comprising: extracting an acoustic feature of normal sound based on training data for normal sound by using an acoustic feature extractor; updating a normal sound model by using the extracted acoustic feature; extracting an acoustic feature of anomalous sound based on simulated anomalous sound and extracting an acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor; and updating the acoustic feature extractor by using the extracted acoustic feature of anomalous sound and the extracted acoustic feature of normal sound, wherein the extracting the acoustic feature of the normal sound based on the training data, the updating the normal sound model using the extracted acoustic feature, the extracting the acoustic feature of the anomalous sound and the extracting of the acoustic feature of the normal sound, and the updating the acoustic feature extractor by using the extracted acoustic feature of the anomalous sound and the extracted acoustic feature of the normal sound are repeatedly performed to simultaneously optimize the acoustic feature extractor and the normal sound model, based on a probability density function followed by a feature amount extracted from normal sound and a probability density function followed by a feature amount extracted from anomalous sound, so that a degree-of-anomaly calculated from the feature amount extracted from the normal sound is less than a degree-of-anomaly calculated from the feature amount extracted from the anomalous sound, the training data for normal sound is sound signals collected by a microphone installed near or inside a device for which anomalous sound is to be detected, and the simultaneous optimization of the acoustic feature extractor and the normal sound model is performed so as to minimize a predetermined objective function L.

6. An anomalous sound detection training method comprising: extracting an acoustic feature of normal sound based on training data for normal sound by using an acoustic feature extractor; updating a normal sound model by using the extracted acoustic feature; extracting an acoustic feature of anomalous sound based on training data for anomalous sound and extracting an acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor; and updating the acoustic feature extractor by using the extracted acoustic feature of anomalous sound and the extracted acoustic feature of normal sound, wherein the extracting the acoustic feature of the normal sound based on the training data, the updating the normal sound model using the extracted acoustic feature, the extracting the acoustic feature of the anomalous sound and the extracting of the acoustic feature of the normal sound, and the updating the acoustic feature extractor by using the extracted acoustic feature of the anomalous sound and the extracted acoustic feature of the normal sound are repeatedly performed to simultaneously optimize the acoustic feature extractor and the normal sound model, based on a probability density function followed by a feature amount extracted from normal sound and a probability density function followed by a feature amount extracted from anomalous sound, so that a degree-of-anomaly calculated from the feature amount extracted from the normal sound is less than a degree-of-anomaly calculated from the feature amount extracted from the anomalous sound, the training data for normal sound is sound signals collected by a microphone installed near or inside a device for which anomalous sound is to be detected, and the simultaneous optimization of the acoustic feature extractor and the normal sound model is performed so as to minimize a predetermined objective function L.

7. A non-transitory computer-readable medium storing computer-readable instructions thereon, which, when executed by a computer, cause the computer to function as the apparatus according to claim 1.

8. An anomalous sound detection apparatus which detects whether or not input sound is anomalous sound, the anomalous sound detection apparatus comprising: processing circuitry configured to extract an acoustic feature of the input sound based on an acoustic feature extractor, wherein the input sound is sound of a machine acquired through a microphone; calculate a degree of anomaly from the acoustic feature of the input sound; and determine whether or not the input sound is anomalous sound based on the degree of anomaly and a threshold, wherein the acoustic feature extractor is optimized, by relating a probability density function followed by a normal sound to a probability density function followed by an anomalous sound, so that the acoustic feature extractor outputs a degree of anomaly to be determined as anomalous sound based on the threshold in case of normal sound and the acoustic feature extractor outputs a degree of anomaly to be determined as anomalous sound based on the threshold in case of the anomalous sound, wherein the probability density function followed by the normal sound is obtained from a first number of normal sound data, a probability distribution of the degree of anomaly followed by the anomalous sound is obtained from a number of anomalous sound data sampled, the number of anomalous sound data sampled being substantially same as the first number of normal sound data, and the determination that the input sound is anomalous sound indicates a potential failure in the machine.

9. The anomalous sound detection apparatus according to claim 8, wherein the anomalous sound data is generated by an anomalous sound sampling apparatus configured to generate an anomalous sound data where an amplitude spectrum of the anomalous sound data is based on normal sound, differs for each of anomalous sound data, is derived to be sparse and is smoothed in a temporal frequency direction, and an average spectrum of stationary noise is added to the amplitude spectrum of the anomalous sound data.

10. The anomalous sound detection training apparatus according to claim 1, wherein an optimized acoustic feature extractor is used in the extraction of the acoustic feature of the normal sound based on the training data and used in the extraction of the acoustic feature of the anomalous sound.

11. The anomalous sound detection training apparatus according to claim 2, wherein an optimized acoustic feature extractor is used in the extraction of the acoustic feature of the normal sound based on the training data and used in the extraction of the acoustic feature of the anomalous sound.

12. The anomalous sound detection training apparatus according claim 1, wherein the predetermined objective function L is given by
L=−∫σ{g(A,x)−ϕ}(p(F(A,x)|z=1)−p(F(A,x)|z=0))dx where $\sigma \left\{\alpha \right\}=\frac{1}{1+\exp \left(-\alpha \right)},$ g(A,x) is a degree of anomaly, ϕ is a threshold value, p(F(A,x)|z=1) is an anomalous sound model, and p(F(A,x)|z=0) is a normal sound model, and simultaneous optimization of the normal sound model and the acoustic feature extractor is performed so as to simultaneously satisfy both: $\begin{array}{c}p\left(F\left(A,x\right).Math.z=0\right)=\underset{p\left(F\left(A,x\right).Math.z=0\right)}{\arg \min }L\\ A=\underset{A}{\arg \min }L\end{array},$ where matrix A is a matrix of values that is used as a multiplier in the acoustic feature extractor.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram for describing an example of an anomalous sound detection training apparatus.

(2) FIG. 2 is a flowchart for describing an example of an anomalous sound detection training method.

(3) FIG. 3 is a block diagram for describing an example of an anomalous sound detection apparatus.

(4) FIG. 4 is a flowchart for describing an example of an anomalous sound detection method.

(5) FIG. 5 is a diagram for describing an example of the structure of the present invention.

(6) FIG. 6 is a diagram for describing an example of an acoustic feature function.

(7) FIG. 7 shows an example of occurrence frequency distribution of an amplitude spectrum spectrum for a single sound signal.

(8) FIG. 8 shows an example of sparsity of an amplitude spectrum.

(9) FIG. 9 is a diagram for describing a conventional technique.

DETAILED DESCRIPTION OF THE EMBODIMENTS

(10) [Overview of the Invention]

(11) [[Structure of the Invention]]

(12) A hierarchical structure of the present invention is shown in FIG. 5.

(13) (I) Irrespective of supervised/unsupervised, a “normal sound model” and an “acoustic feature extractor” are optimized simultaneously from training data.

(14) (II) In the case of unsupervised, training data for anomalous sound is sampled.

(15) (III) Irrespective of supervised/unsupervised, the “acoustic feature extractor” is represented as a matrix product and a logarithm operation, or as a multilayer perceptron, for example.

(16) (IV) A training/detection algorithm that simultaneously implements (I) to (III) is provided.

(17) The following will describe (I) to (III). Note that for the sake of description, they will be described in the order of (III), (I), (H). It will be followed by description on [Anomalous sound detection training apparatus and method] for performing them.

(18) [[(III) Definition of Acoustic Feature Extractor]]

(19) A series derived by Fourier-transformation of an observation signal and extraction of a power spectrum or amplitude spectrum is expressed as X.sub.ω,t∈R.sub.+.sup.Ω×T. Here, ω={1, 2, . . . , Ω} and t={1, 2, . . . , T} represent indices of frequency and time, respectively, Ω represents the number of frequency bins, and T represents the number of observed frames. R.sub.+ represents a set of non-negative real numbers.

(20) Most cases of acoustic feature extraction can be represented as a matrix operation on this series with a logarithmic function. For example, log-MFBO can be described as spectrum X.sub.t=(X.sub.1,t, . . . , X.sub.Ω,t).sup.T of all frequencies in a certain frame multiplied by M Mel filter banks mm, of which the log is then taken.

(21) $\begin{array}{cc}F\left(M,X_t\right)=\left[\begin{array}{c}\ln \left(\max \left(m_1^T{X}_t,\mathrm{.Math.}\right)\right)\\ \mathrm{.Math.}\\ \ln \left(\max \left(m_M^T{X}_t,\mathrm{.Math.}\right)\right)\end{array}\right]=\ln \left(\max \left({\mathrm{MX}}_t,\mathrm{.Math.}\right)\right)& \left(1\right)\end{array}$

(22) Here, T represents transposition. Also, when ΔMFBO is defined as the temporal difference of Mel filter bank's output, it can be described as follows using a difference matrix D.

(23) $\begin{array}{cc}F\left(M,D,X_t\right)=\ln \left(\max \left(D\left[\begin{array}{c}{\mathrm{MX}}_{t-1}\\ {\mathrm{MX}}_t\end{array}\right],\mathrm{.Math.}\right)\right),D={\left[\begin{array}{c}-E_M\\ E_M\end{array}\right]}^T& \left(2\right)\end{array}$

(24) Here, E.sub.M is a unit matrix of M dimensions.

(25) Herein, an acoustic feature extractor is defined with extension of this concept. First, past P.sub.1 frame through future P.sub.2 frame are taken from determination target frame t, and a vertical connection of them is defined as x.sub.t, where x.sub.t is represented as shown in FIG. 6 and the formula below.

(26) $\begin{array}{cc}\begin{array}{c}{x}_t={\left(\underset{\underset{X_{t-P_1}}{︸}}{X_{1,t-P_1},\mathrm{.Math.},X_{\Omega ,t-P_1}},\underset{\underset{X_{t-P_1+1}}{︸}}{X_{1,t-P_1+1},\mathrm{.Math.},X_{\Omega ,t-P_1+1}},\mathrm{.Math.},\underset{\underset{X_{t+P_2}}{︸}}{X_{1,t+P_2},\mathrm{.Math.},X_{\Omega ,t+P_2}}\right)}^T\\ ={\left(x_{t,1},\mathrm{.Math.},x_{t,Q}\right)}^T\in R_{+}^Q\end{array}& \left(1^{\prime }\right)\end{array}$

(27) Here, Q=Ω×(P.sub.1+P.sub.2+1) holds. The modification to the first to second lines of formula (1′) is rewriting for the purpose of simplifying the notation; the contents and order of the elements are unchanged. An operation that multiplies x.sub.t by a matrix A∈R.sup.D×Q and takes the log thereof is defined as the acoustic feature extractor. That is, it will be as follows.

(28) $\begin{array}{cc}F\left(A,x_i\right)=\left[\begin{array}{c}f\left(a_1,x_i\right)\\ \mathrm{.Math.}\\ f\left(a_D,x_i\right)\end{array}\right]=\left[\begin{array}{c}\ln \left(\max \left(a_1^T{x}_i,\mathrm{.Math.}\right)\right)\\ \mathrm{.Math.}\\ \ln \left(\max \left(a_D^T{x}_i,\mathrm{.Math.}\right)\right)\end{array}\right]=\ln \left(\max \left({\mathrm{Ax}}_i,\mathrm{.Math.}\right)\right)& \left(3\right)\\ A=\left[\begin{array}{ccc}{a}_{1,1}& \mathrm{.Math.}& a_{1,Q}\\ \mathrm{.Math.}& \ddots & \mathrm{.Math.}\\ a_{D,1}& \mathrm{.Math.}& a_{D,Q}\end{array}\right]=\left[\begin{array}{c}{a}_1\\ \mathrm{.Math.}\\ a_D\end{array}\right]& \left(4\right)\end{array}$

(29) Thus, optimization of the acoustic feature extractor can be said to be optimization of matrix A.

(30) [[(I) Simultaneous Optimization of Normal Sound Model and Acoustic Feature Extractor]]

(31) First, a normal sound model (a probability density function followed by a feature amount extracted from normal sound) is defined as p(F(A,x)|z=0). Then, a negative logarithmic likelihood (the degree of anomaly) can be computed as below.
g(A,x)=−ln p(F(A,x)|z=0)  (5)

(32) Here, although the normal sound model may be any of various types, when it is a Gaussian mixture distribution, for example, it can be described as:

(33) $\begin{array}{cc}p\left(F\left(A,x\right).Math.z=0\right)=\underset{k=1}{\overset{K}{.Math.}}{W}_k\exp \left\{-\frac{1}{2}{\left(F\left(A,x\right)-{\mu }_k\right)}^T\underset{k}{\overset{-1}{.Math.}}\left(F\left(A,x\right)-{\mu }_k\right)\right\}& \left(6\right)\\ W_k=\frac{w_k}{{\left(2\pi \right)}^{D/2}\sqrt{\det \left(\underset{k}{.Math.}\right)}}& \left(7\right)\end{array}$

(34) Here, K is the number of mixtures, and k, Σ.sub.k, and w.sub.k are a mean vector, a covariance matrix, and a weight parameter for the kth distribution, respectively.

(35) In anomalous sound detection, when the value of g(A,x) is greater than threshold ϕ, it is determined to be “anomalous”; when the value is smaller than threshold ϕ, it is determined to be “normal”. Although this determination can be described as a step function, for the simplicity of the subsequent calculations, it is described with a sigmoid function:

(36) $\begin{array}{cc}\sigma \left\{\alpha \right\}=\frac{1}{1+\exp \left(-\alpha \right)}& \left(8\right)\end{array}$

(37) The sigmoid function 0≤σ{α}≤1 outputs a value closer to 0 as α is smaller and outputs a value closer to 1 as α is greater. That is, when σ{g(A,x−ϕ)} is equal to or greater than 0.5, it is determined to be anomalous; when it is equal to or smaller than 0.5, it is determined to be normal.

(38) An aim of an anomalous sound detection algorithm is to output a great a {g(A,x−ϕ)} when the observation represents anomalous sound and to output a small σ{g(A,x−ϕ)} when it represents normal sound. That is, minimizing the following formula is the aim of optimization of anomalous sound detection.
L=−∫σ{g(A,x)−ϕ}(p(F(A,x)|z=1)−p(F(A,x)|z=0))dx  (9)

(39) Here, p(F(A,x)|z=1) is an anomalous sound model (a probability density function followed by a feature amount extracted from anomalous sound).

(40) From the foregoing, simultaneous optimization of the normal sound model and the acoustic feature extractor should be performed so as to simultaneously satisfy both:

(41) $\begin{array}{cc}p\left(F\left(A,x\right).Math.z=0\right)=\underset{p\left(F\left(A,x\right).Math.z=0\right)}{\arg \min }L& \left(10\right)\\ A=\underset{A}{\arg \min }L& \left(11\right)\end{array}$

(42) Optimization of the acoustic feature extraction can be said to be design of the matrix A such that formula (11) is optimized.

(43) Optimization of formula (10) can be carried out with a maximum likelihood method, for example. Thus, consider the optimization of formula (11). The foremost challenge in calculating formula (9) is the integration operation. Thus, this integration is replaced with arithmetic mean of training data. First, an objective function is defined as:
L=−∫σ{g(A,x)−ϕ}p(F(A,x)|z=1)dx+∫σ{g(A,x)−ϕ}p(F(A,x)|z=0)dx  (12)

(44) Each of the terms can then be regarded as an expectation operation for σ{G(A,x−ϕ)}, thus they can be described as:
L=−E[σ{g(A,x)−ϕ}].sub.x|z=1+E[σ{g(A,x)−ϕ}].sub.x|z=0  (13)

(45) Here, if E[.Math.].sub.x|z=0 is an expectation operation relating to normal sound, while E[.Math.].sub.x|z=1 is an expectation operation relating to anomalous sound.

(46) Here, if replacing the expectation operations with arithmetic means, formula (11) can be approximated as follows:

(47) $\begin{array}{cc}A=\underset{A}{\arg \min }-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}\sigma \left\{g\left(A,x_n\right)-\varphi \right\}+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}\sigma \left\{g\left(A,x_t\right)-\varphi \right\}& \left(14\right)\\ =\underset{A}{\arg \min }\underset{\underset{J}{︸}}{-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}{\Psi }_n+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}{\Psi }_t}& \left(15\right)\end{array}$

(48) Here, N is a time index of training data for anomalous sound. That is, the accuracy of anomalous sound detection will be maximized by calculating the arithmetic means of Ψ.sub.1, . . . , T using training data for normal sound, calculating the arithmetic means of Ψ.sub.1, . . . , T using training data for anomalous sound, and designing A so that the sum of the two arithmetic means is minimized. In the case of supervised anomalous sound detection where training data for anomalous sound has been obtained, this can be performed without modification. In the case of unsupervised anomalous sound detection, N pieces of anomalous sound data are sampled (simulated) in accordance with the sampling algorithm described below. In that case, N should be set about N=T.

(49) Now that the objective function has been set with formula (15), it is then minimized. For the minimization, a gradient method may be used. When a steepest descent method is used as the gradient method, the matrix may be optimized for each dimension d as:

(50) $\begin{array}{cc}{a}_d\leftarrow a_d-\lambda \frac{\partial J}{\partial a_d}& \left(16\right)\end{array}$

(51) Here, λ is step size. For this optimization, AdaGrad, that is, an improved algorithm of the steepest descent method, or the like may be used, of course.

(52) In a case where the normal sound model is represented as a Gaussian mixture distribution, a gradient vector ∂J/∂a.sub.d can be calculated follows using the chain rule of partial differentiation.

(53) 0$\begin{array}{cc}\frac{\partial J}{\partial a_d}=-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}\frac{\partial {\Psi }_n}{\partial g\left(A,x_n\right)}\frac{\partial g\left(A,x_n\right)}{\partial F\left(A,x_n\right)}\frac{\partial F\left(A,x_n\right)}{\partial a_d}+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}\frac{\partial {\Psi }_t}{\partial g\left(A,x_t\right)}\frac{\partial g\left(A,x_t\right)}{\partial F\left(A,x_t\right)}\frac{\partial F\left(A,x_t\right)}{\partial a_d}& \left(17\right)\end{array}$

(54) Here, each of the components can be calculated by:

(55) $\begin{array}{cc}\frac{\partial {\Psi }_i}{\partial g\left(A,x_i\right)}=\sigma \left\{g\left(A,x_i\right)-\varphi \right\}\left(1-\sigma \left\{g\left(A,x_i\right)-\varphi \right\}\right)& \left(18\right)\\ \frac{\partial g\left(A,x_i\right)}{\partial F\left(A,x_i\right)}=\underset{k=1}{\overset{K}{.Math.}}{\Gamma }_{k,i}\left\{{\left(F\left(A,x_i\right)-{\mu }_k\right)}^T{\Sigma }_k^{-1}\right\}& \left(19\right)\\ {\Gamma }_{k,i}=\frac{W_k\exp \left\{-\frac{1}{2}{\left(F\left(A,x_i\right)-{\mu }_k\right)}^T{\Sigma }_k^{-1}\left(F\left(A,x_i\right)-{\mu }_k\right)\right\}}{\underset{j=1}{\overset{K}{.Math.}}{W}_j\exp \left\{-\frac{1}{2}{\left(F\left(A,x_i\right)-{\mu }_j\right)}^T{\Sigma }_j^{-1}\left(F\left(A,x_i\right)-{\mu }_j\right)\right\}}& \left(20\right)\\ \frac{\partial F\left(A,x_i\right)}{\partial a_d}=\left[\begin{array}{ccc}\frac{\partial f\left(a_1,x_i\right)}{\partial a_{d,1}}& \mathrm{.Math.}& \frac{\partial f\left(a_1,x_i\right)}{\partial a_{1,Q}}\\ \mathrm{.Math.}& \ddots & \mathrm{.Math.}\\ \frac{\partial f\left(a_D,x_i\right)}{\partial a_{d,1}}& \mathrm{.Math.}& \frac{\partial f\left(a_D,x_i\right)}{\partial a_{1,Q}}\end{array}\right]& \left(21\right)\\ \frac{\partial f\left(a_j,x_i\right)}{\partial a_{d,q}}=\frac{C_{i,j,d,q}}{\mathrm{ma}\stackrel{.}{x}\left(a_j^T{x}_i,\mathrm{.Math.}\right)}& \left(22\right)\\ C_{i,j,d,q}=\{\begin{array}{cc}{x}_{i,q}\delta \left(d,j\right)& \left(a_j^T{x}_i>\mathrm{.Math.}\right)\\ 0& \left(\mathrm{otherwise}\right)\end{array}& \left(23\right)\\ \delta \left(d,j\right)=\{\begin{array}{cc}1& d=j\\ 0& \left(\mathrm{otherwise}\right)\end{array}& \left(24\right)\end{array}$

(56) [[(II) Sampling of Training Data for Anomalous Sound]]

(57) For calculation of formula (15) in unsupervised anomalous sound detection, how training data for anomalous sound is sampled will be discussed.

(58) The distribution used with anomalous sound model p(F(A,x)|z=1) is arbitrary. In unsupervised training, no previous knowledge on anomalous sound has been obtained. In order to detect every kind of anomalous sound, it is desirable to construct an anomalous sound model with as least assumptions as possible. That is, it is desirable to sample anomalous sound only from generally known findings about the physical characteristics or frequency of appearance of an amplitude spectrum and/or power spectrum.

(59) In the following, an anomalous sound model is constructed with assumptions made only for the occurrence frequency, sparsity, and continuity in temporal frequency direction of amplitude values.

(60) First, consider the magnitude of X.sub.ω,n. FIG. 7 shows the occurrence frequency distribution of the amplitude spectrum for a single sound signal. The horizontal axis is strength and the vertical axis is the frequency of appearance. From FIG. 7, it can be seen that strengths around 0 are most likely to occur in the amplitude spectrum of a sound signal and the frequency of appearance exponentially decreases as the strength increases. X.sub.ω,n should be sampled from a probability distribution that satisfies these characteristics.

(61) A distribution that satisfies the characteristics is an exponential distribution, for example. An exponential distribution is a distribution such that:

(62) $\begin{array}{cc}p\left(o;\beta \right)=\frac{1}{\beta }\exp \left\{-\frac{o}{\beta }\right\}& \left(25\right)\end{array}$

(63) In the distribution, the frequency of appearance of o decreases as the value of o increases. β is an average value. In the case of sampling X.sub.ω,n from an exponential distribution, it will be as follows.

(64) $\begin{array}{cc}{X}_{\omega ,n}~\frac{1}{\beta }\exp \left\{-\frac{1}{\beta }{X}_{\omega ,n}\right\}& \left(26\right)\end{array}$

(65) Since β is a parameter relating to the volume of observed sound, it can be set from the training data for normal sound. For example, it can be set as:

(66) $\begin{array}{cc}\beta =\frac{1}{\Omega T}\underset{\omega =1}{\overset{\Omega }{.Math.}}\underset{t=1}{\overset{T}{.Math.}}{X}_{\omega ,t}& \left(27\right)\end{array}$

(67) If previous knowledge on stationary noise contained in observation signals or the like is available, this sampling may be modified as desired. For example, if an average spectrum Y.sub.ω of stationary noise is known, sampling may be performed such that:

(68) $\begin{array}{cc}{X}_{\omega ,n}=Y_{\omega }+E_{\omega ,n}& \left(28\right)\\ E_{\omega ,n}~\frac{1}{{\beta }_{\omega }}\exp \left\{-\frac{1}{{\beta }_{\omega }}{E}_{\omega ,n}\right\}& \left(29\right)\end{array}$

(69) β here can be calculated by:

(70) $\begin{array}{cc}{\beta }_{\omega }=\sqrt{\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}{\left(\max \left(0,X_{\omega ,t}-Y_{\omega }\right)\right)}^2}& \left(30\right)\end{array}$

(71) When sampled with formula (29), a volume average β.sub.ω of anomalous sound will be constant independently of time. Depending on the distance to a microphone or the type of anomalous sound, the amplitude value, or the volume, might increase or decrease. Hence, it is desirable to vary the value of β.sub.ω,n on a per-n basis. For representing this, formula (29) may be modified as follows, for example:

(72) $\begin{array}{cc}{E}_{\omega ,n}~\frac{1}{{\beta }_{\omega ,n}}\exp \left\{-\frac{1}{{\beta }_{\omega ,n}}{E}_{\omega ,n}\right\}& \left(31\right)\\ {\beta }_{\omega ,n}=g_{\omega }\times b_{\omega }{10}^{\theta /20}& \left(32\right)\\ g_{\omega }=\sqrt{\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}{\left(X_{\omega ,t}-N_{\omega }\right)}^2}& \left(33\right)\\ b_{\omega }~\mathrm{Uniform}\left(0,1\right)& \left(34\right)\end{array}$

(73) Here, θ is a parameter in unit of dB for correcting the maximum volume of anomalous sound, and Uniform(a,b) is a continuous uniform distribution of maximum a and minimum b. For example, θ may be set to about 0 to 12. As another way, similar effects could be provided by sampling θ from a normal distribution or the like and setting it always to, for example, 1, without performing sampling of b.sub.ω.

(74) It is also possible to focus on the sparsity of an amplitude spectrum (almost all of the elements being 0). FIG. 8 shows an example of plotting X.sub.1, . . . , Ω,t for a certain frame. FIG. 8 indicates that a sound signal often has only one or several frequency components in a single frame. Focusing on this, the sampling of formula (31) may be modified as follows:

(75) $\begin{array}{cc}{E}_{\omega ,n}~\frac{1}{{\beta }_{\omega ,n}}\exp \left\{-\frac{1}{{\beta }_{\omega ,n}}{E}_{\omega ,n}\right\}& \left(35\right)\\ {\beta }_{\omega ,n}=\{\begin{array}{cc}{g}_{\omega }& w=z_n\\ 0& \left(\mathrm{otherwise}\right)\end{array}& \left(36\right)\\ z_n~\mathrm{Cat}\left(1_{\Omega }/\Omega \right)& \left(37\right)\end{array}$

(76) Here, Cat(π) is category distribution and 1.sub.Ω is a vector with a length of Ω whose elements are all 1. This makes only one spectrum bin become active at each time via z.sub.n, thereby driving the anomalous sound spectrum E to be sparse.

(77) Also, sampling with formula (35) results in generation of a value of E.sub.ω,n that is completely independent from the temporal frequency direction. A sound signal typically changes smoothly in terms of time, so that the spectrum thereof smoothly changes in the temporal frequency direction. For achieving this, F.sub.ω,n may be smoothed in the temporal frequency direction.

(78) While the smoothing can be done in various ways, it can be performed using a Gaussian filter as follows, for example:
X.sub.ω,n=Y.sub.ω+conv2(E.sub.ω,n,G(F,T))  (38)

(79) Here, conv2 is a convolution operation of two dimensions, and G(F,T) is a two-dimensional Gaussian filter with a size of F×T. Although the parameters should be controlled experimentally, they may be set at about F=11 and T=31, for example.

(80) [Anomalous Sound Detection Training Apparatus and Method]

(81) As exemplarily shown in FIG. 1, an anomalous sound detection training apparatus includes a spectrum calculation unit 1, an initialization unit 2, a first acoustic feature extraction unit 3, a normal sound model updating unit 4, an anomalous sound sampling unit 5, a second acoustic feature extraction unit 6, an acoustic feature extractor updating unit 7, and a control unit 8. An anomalous sound detection training method is implemented by the units of the anomalous sound detection training apparatus executing the processing at step S1 to S82 described in FIG. 2 and below.

(82) <Spectrum Calculation Unit 1>

(83) Training data for normal sound is input to the spectrum calculation unit 1.

(84) The training data for normal sound is sound signals collected by a microphone installed near (or alternatively inside) a device for which anomalous sound is to be detected. While the sampling frequency for collecting sound signals should be set so that the frequencies of normal sound can be recorded, it may be set at 16 kHz, for example.

(85) The spectrum calculation unit 1 converts the training data into a frequency domain via an approach such as short-time Fourier transformation (STFT).

(86) It then takes an absolute value of a complex spectrum to determine an amplitude spectrum X.sub.ω,t (step S1). Alternatively, X.sub.ω,t may be a power spectrum derived by determining the square root of the amplitude spectrum. Here, Fourier transformation length may be set at 32 ms and shift width may be set at 16 ms, for example.

(87) In the following, X.sub.ω,t as either an amplitude spectrum or a power spectrum will be abbreviated as spectrum X.sub.ω,t. The spectrum X.sub.ω,t is output to the first acoustic feature extraction unit 3 and the second acoustic feature extraction unit 6.

(88) If training data for anomalous sound is available, namely in the case of supervised training, the spectrum calculation unit 1 calculates the spectrum of anomalous sound based on the training data for anomalous sound. The spectrum of anomalous sound is the amplitude spectrum or the power spectrum of the anomalous sound, for example. The calculated spectrum of anomalous sound is output to the first acoustic feature extraction unit 3 and the second acoustic feature extraction unit 6.

(89) <Initialization Unit 2>

(90) Preset parameters for a training algorithm and preset parameters for an acoustic feature extractor are input to the initialization unit 2.

(91) The various parameters may be set such that P.sub.1=3, P.sub.2=3, ε=min(X.sub.ω,t)/2, ϕ=100, and λ=5.0×10.sup.−3, for example.

(92) The initialization unit 2 initializes the matrix A for the acoustic feature extractor and the normal sound model (step S2). After being initialized, the matrix A and the normal sound model are output to the first acoustic feature extraction unit 3.

(93) The matrix A may be initialized with uniform random numbers, for example. While initialization of the normal sound model varies depending on the type of the probability density function, when a Gaussian mixture distribution is used, for example, the normal sound model can be initialized via a k-means algorithm or the like.

(94) <First Acoustic Feature Extraction Unit 3>

(95) To the first acoustic feature extraction unit 3, the acoustic feature function initialized by the initialization unit 2 and the training data for normal sound are input.

(96) The first acoustic feature extraction unit 3 extracts an acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor (step S3). In this example, as the training data for normal sound, the first acoustic feature extraction unit 3 uses the spectrum X.sub.ω,t calculated by the spectrum calculation unit 1 based on the training data for normal sound to extract the acoustic feature of normal sound. The extracted acoustic feature is output to the normal sound model updating unit 4.

(97) A first round of processing by the first acoustic feature extraction unit 3 is performed by using the acoustic feature extractor initialized by the initialization unit 2. Second and subsequent rounds of processing by the first acoustic feature extraction unit 3 are performed by using the acoustic feature extractor as updated by the acoustic feature extractor updating unit 7.

(98) If the spectrum of anomalous sound has been calculated by the spectrum calculation unit 1 based on training data for anomalous sound, the first acoustic feature extraction unit 3 extracts the acoustic feature of anomalous sound based on this calculated spectrum of anomalous sound by using the acoustic feature extractor.

(99) <Normal Sound Model Updating Unit 4>

(100) The normal sound model updating unit 4 updates the normal sound model by using the acoustic feature extracted by the first acoustic feature extraction unit 3 (step S4).

(101) While the way of update varies depending on the type of the probability density function for the normal sound model, it may be updated by maximum likelihood estimation, for example. In a case where a Gaussian mixture distribution is used as the normal sound model, one to several steps of update may be performed in an EM algorithm (see Reference Literature 1, for instance). [Reference Literature 1] Sadanori Konishi, “Introduction to Multivariate Analysis, Appendix C: EM algorithm”, pp. 294-298, Iwanami Shoten, 2010.

(102) <Anomalous Sound Sampling Unit 5>

(103) The anomalous sound sampling unit 5 samples simulated anomalous sound (step S5). The sampled anomalous sound is output to the second acoustic feature extraction unit 6.

(104) The anomalous sound sampling unit 5 performs sampling of simulated anomalous sound based on formula (26), for example. The parameter β in formula (26) can be determined based on formula (27), for example. In this case, for X.sub.ω,t in formula (27), the spectrum X.sub.ω,t of normal sound calculated by the spectrum calculation unit 1 may be used.

(105) If training data for anomalous sound is available, namely in the case of supervised training, sampling is not performed. That is, the subsequent processing may be performed using the training data for anomalous sound as the sampling result.

(106) <Second Acoustic Feature Extraction Unit 6>

(107) The second acoustic feature extraction unit 6 extracts the acoustic feature of anomalous sound based on simulated anomalous sound and extracts the acoustic feature of normal sound based on the training data for normal sound by using the acoustic feature extractor (step S6). The acoustic feature of anomalous sound and the acoustic feature of normal sound that have been extracted are output to the acoustic feature extractor updating unit 7.

(108) In this example, as the training data for normal sound, the second acoustic feature extraction unit 6 uses the spectrum X.sub.ω,t calculated by the spectrum calculation unit 1 based on the training data for normal sound to extract the acoustic feature of normal sound.

(109) The first round of processing by the second acoustic feature extraction unit 6 is performed by using the acoustic feature extractor initialized by the initialization unit 2. The second and subsequent rounds of processing by the second acoustic feature extraction unit 6 are performed by using the acoustic feature extractor as updated by the acoustic feature extractor updating unit 7.

(110) <Acoustic Feature Extractor Updating Unit 7>

(111) The acoustic feature extractor updating unit 7 updates the acoustic feature extractor by using the acoustic feature of anomalous sound and the acoustic feature of normal sound that have been extracted by the second acoustic feature extraction unit 6 (step S7). The updated acoustic feature extractor is output to the first acoustic feature extraction unit 3 and the second acoustic feature extraction unit 6. Also, the acoustic feature extractor as finally updated after repeated control by the control unit 8 is output as a final result of training performed by the anomalous sound detection training apparatus and method.

(112) The acoustic feature extractor updating unit 7 updates the acoustic feature extractor based on formula (16), for example. Prior to execution of this update, threshold ϕ may be updated according to the formula below using training data for normal sound x.sub.1, . . . , x.sub.T and sampled anomalous sound data x.sub.1, . . . x.sub.N. This stabilizes convergence.

(113) $\begin{array}{cc}\varphi =\frac{\mathrm{median}\left(g\left(A,x_{1,\mathrm{.Math.},T}\right)\right)+\mathrm{median}\left(g\left(A,x_{1,\mathrm{.Math.},N}\right)\right)}{2}& \left(39\right)\end{array}$

(114) <Control Unit 8>

(115) The control unit 8 performs control so as to repeatedly perform processing by the first acoustic feature extraction unit 3, the normal sound model updating unit 4, the second acoustic feature extraction unit 6, and the acoustic feature extractor updating unit 7 (steps S81 and S82). This repeated processing is performed until the acoustic feature extractor and the normal sound model converge.

(116) For example, the control unit 8 performs control so as to repeatedly perform the processing by the second acoustic feature extraction unit 6 and the acoustic feature extractor updating unit 7 (hereinafter referred to as a first repeated processing). Setting a first convergence determination condition as that the number of executions of processing has reached a certain number of times (for example, 50) or that an update norm for the matrix A has fallen below a certain value, the control unit 8 performs control so as to repeatedly perform the first repeated processing until the first convergence condition is satisfied, or in other words, until it is determined that convergence has been reached. The first convergence determination condition may also be a different condition.

(117) The control unit 8 subsequently performs control so as to repeatedly perform the processing by the first acoustic feature extraction unit 3 and the normal sound model updating unit 4 and the first repeated processing (hereinafter referred to as a second repeated processing). Setting a second convergence condition as that the number of executions of processing has reached a certain number of times (for example, 200), the control unit 8 performs control so as to repeatedly perform the second repeated processing until the second convergence condition is satisfied, or in other words, until it is determined that convergence has been reached. The second convergence determination condition may also be a different condition.

(118) [Anomalous Sound Detection Apparatus and Method]

(119) As exemplarily shown in FIG. 3, an anomalous sound detection apparatus includes a spectrum calculation unit 11, an acoustic feature extraction unit 12, a degree-of-anomaly calculation unit 13, and a determination unit 14. An anomalous sound detection method is implemented by the units of the anomalous sound detection apparatus executing the processing at step S11 to S14 described in FIG. 4 and below.

(120) <Spectrum Calculation Unit 11>

(121) Operational sound of a machine for which anomalous sound is to be detected is collected through a microphone. The sampling rate used for this collection is similar to the one used for training. The collected sound signal is input to the spectrum calculation unit 11.

(122) The spectrum calculation unit 11 obtains spectrum. X.sub.ω,t based on the collected sound signal in a similar manner to the spectrum calculation unit 1 (step S11). The obtained spectrum X.sub.ω,t is output to the acoustic feature extraction unit 12.

(123) <Acoustic Feature Extraction Unit 12>

(124) The acoustic feature extraction unit 12 uses the Obtained spectrum X.sub.ω,t to extract the acoustic feature of the collected sound signal based on the acoustic feature extractor that has been output by the anomalous sound detection training apparatus and method as the final result of training (step S12). The extracted acoustic feature is output to the degree-of-anomaly calculation unit 13.

(125) <Degree-of-Anomaly Calculation Unit 3>

(126) The degree-of-anomaly calculation unit 13 calculates a negative logarithmic likelihood g(A,x) by using the extracted acoustic feature and a normal sound model that has been output by the anomalous sound detection training apparatus and method as the final result of training (step S13). Hereinafter, a negative logarithmic likelihood will be referred to as the degree of anomaly. The calculated degree of anomaly is output to the determination unit 14.

(127) <Determination Unit 14>

(128) The determination unit 14 outputs “anomalous” if the degree of anomaly of a current frame is equal to or greater than threshold ϕ (step S14). Although the threshold should be adjusted in accordance with the machine and/or environment in question, it may be set at about 1500, for example.

(129) As with voice activity detection, “hangover”, which suppresses detection errors with heuristic rules, may be used. While the hangover processing to be applied may be any of various types, the hangover processing should be set in accordance with the types of false detection of anomalous sound.

(130) As an example, musical noise that occurs during noise suppression could be determined as a sporadic anomalous sound. Since sound like a sporadic hitting sound often exhibits change in spectrum shape for 100 ms or more, the degree of anomaly would remain equal to or greater than the threshold for .sub.┌(100/the frame shift width for STFT).sub.┐ frames continuously. However, because in musical noise an anomalous amplitude spectrum value only occurs in the relevant frames, there would be several frames at most where the degree of anomaly remains equal to or greater than the threshold continuously. Thus, a rule for anomaly determination may be set as “outputting “anomalous” if the degree of anomaly remains equal to or greater than the threshold for F.sub.1 frames or more continuously”, for example.

(131) As another example, it is also conceivable that anomalous sound continues for a long period of time with the degree of anomaly slightly below the threshold due to a low volume of the anomalous sound. In such a case, a rule like “detecting a sound as anomalous if a total sum of its degree of anomaly over the last F.sub.2 frames is equal to or greater than ϕ.sub.1” may be added as a determination rule for continuous anomalous sound. Although ϕ.sub.1 should be determined by tuning, it may be set at about ϕ.sub.1=F.sub.2×(ϕ−250), for example.

(132) By detecting anomalous sound from a large-sized manufacturing or molding machine installed in a factory or the like using the anomalous sound detection apparatus and method as described above, faster handling of a failure and/or failure prediction become possible. This can contribute to increased efficiency of an industry, for example, manufacturing industry in particular.

(133) [Program and Recording Medium]

(134) In the case of implementing the processing in the anomalous sound detection training apparatus or the anomalous sound detection apparatus with a computer, the processing details of the functions to be provided by the anomalous sound detection training apparatus or the anomalous sound detection apparatus are described by a program. By the computer then executing the program, the processing is implemented on the computer.

(135) The program describing the processing details may be recorded in a computer-readable recording medium. The computer-readable recording medium may be any kind of medium, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory.

(136) In addition, individual processing means may be embodied by execution of a predetermined program on a computer or at least some of their processing details may be implemented in hardware.

(137) [Modifications]

(138) Using the optimization metrics for acoustic feature described above, a multilayer perceptron can also be trained unsupervised. Specifically, although the acoustic feature extractor was described using formula (3), it may instead be implemented with a multilayer perceptron. For example, when a simplest fully-connected multilayer perceptron is used, the acoustic feature extractor would be as shown below. In this manner, the acoustic feature extractor may also be represented by a multilayer perceptron.
F(x.sub.τ)=W.sup.(L)g.sub.τ.sup.(L-1)+b.sup.(L)  (40)
g.sub.τ.sup.(k)=σ.sub.θ{W.sup.(k)g.sub.τ.sup.(k-1)+b.sup.(k)}  (41)

(139) Here, L, (k), and b(k) represent the number of layers of a neural network, a weight matrix, and a bias vector, respectively. L is a predetermined positive integer equal to or greater than 2. σ.sub.θ is an activation function and may be implemented as a sigmoid function, for example. Input to the input layer may be such that g.sub.τ.sup.(1)=x.sub.τ or g.sub.τ.sup.(1)=ln x.sub.τ, for example. g.sub.τ.sup.(k) is input to the kth layer of the multilayer perceptron. In this case, optimization of the acoustic feature extractor is optimization of Θ={(W(k),b(k)}, (k∈{2, . . . , L}).

(140) For this optimization, a gradient method may be used as in formula (16). If a steepest descent method is used as the gradient method, it may be done such that:

(141) 0$\begin{array}{cc}{W}^{\left(k\right)}\leftarrow W^{\left(k\right)}-\lambda \frac{\partial J}{\partial W^{\left(k\right)}}& \left(42\right)\\ b^{\left(k\right)}\leftarrow b^{\left(k\right)}-\lambda \frac{\partial J}{\partial b^{\left(k\right)}}& \left(43\right)\end{array}$

(142) For this optimization, AdaGrad, that is, an improved algorithm of the steepest descent method, or the like may be used, of course. The objective function J is given by

(143) $\begin{array}{cc}J=-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}{\Psi }_n+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}{\Psi }_t& \left(44\right)\end{array}$

(144) Thus, formulae (42) and (43) can be described as:

(145) $\begin{array}{cc}{W}^{\left(k\right)}\leftarrow W^{\left(k\right)}-\lambda \left[-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}\frac{\partial {\Psi }_n}{\partial W^{\left(k\right)}}+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}\frac{\partial {\Psi }_t}{\partial W^{\left(k\right)}}\right]& \left(45\right)\\ b^{\left(k\right)}\leftarrow b^{\left(k\right)}-\lambda \left[-\frac{1}{N}\underset{n=1}{\overset{N}{.Math.}}\frac{\partial {\Psi }_n}{\partial b^{\left(k\right)}}+\frac{1}{T}\underset{t=1}{\overset{T}{.Math.}}\frac{\partial {\Psi }_t}{\partial b^{\left(k\right)}}\right]& \left(46\right)\end{array}$

(146) The gradients:

(147) $\frac{\partial {\Psi }_i}{\partial W^{\left(k\right)}},\frac{\partial {\Psi }_i}{\partial b^{\left(k\right)}}$

(148) can be calculated as follows using the chain rule of partial differentiation from formula (17).

(149) $\begin{array}{cc}\frac{\partial {\Psi }_i}{\partial W^{\left(k\right)}}=\frac{\partial {\Psi }_i}{\partial G\left(x_i\right)}\frac{\partial G\left(x_i\right)}{\partial F\left(x_i\right)}\frac{\partial F\left(x_i\right)}{\partial W^{\left(k\right)}}& \left(47\right)\\ \frac{\partial {\Psi }_i}{\partial b^{\left(k\right)}}=\frac{\partial {\Psi }_i}{\partial G\left(x_i\right)}\frac{\partial G\left(x_i\right)}{\partial F\left(x_i\right)}\frac{\partial F\left(x_i\right)}{\partial b^{\left(k\right)}}& \left(48\right)\end{array}$

(150) In a multilayer perceptron, gradients in the form of formulae (47) and (48) can be efficiently calculated by using error back propagation (see Reference Literature 2, for instance). [Reference Literature 2] Hideki Asoh and seven others, “Deep Learing”, the Japanese Society for Artificial Intelligence, Kindai kagaku sha, pp. 5, 2015.

(151) In addition to being executed chronologically in the order described, the processing described for the anomalous sound detection training apparatus or the anomalous sound detection apparatus may also be executed in parallel or individually depending on the processing ability of the apparatus executing the processing or on any necessity.

(152) It goes without saying that other modifications may be made as desired without departing from the scope of this invention.