SOUND PROCESSING METHOD AND DEVICE USING DJ TRANSFORM

Abstract

According to research findings, it is known that human hearing ability is not restricted by the Fourier uncertainty principle. The present disclosure intends to propose the sound processing method and device using the DJ transform method, a new frequency extraction method from understanding of the human hearing ability that improves the temporal resolution as well as the frequency resolution simultaneously based on the operating principle of hair cells constituting the cochlea.

Claims

1. A sound processing device comprising: a spring modeling unit that calculates displacement and velocity of each of the plurality of springs by modeling a plurality of springs, each of which has a different natural frequency and vibrates according to an input sound, and calculates displacement, velocity, energy, and amplitude of each of the plurality of springs by modeling a plurality of springs, each of which has a different natural frequency and vibrates according to an input pure tone; a frequency extraction unit that extracts the natural frequency of the spring corresponding to the local maximum among the filtered pure tone amplitudes calculated by the spring modeling unit; a sound recognition and synthesis unit that recognizes and synthesizes sound by using the amplitude or natural frequency of the input pure tone; and an error inspection unit that checks the excess error of the conversion result of the frequency when the frequency of the plurality of input springs changes and inspects the error between the pure tone frequencies.

2. The device according to claim 1, the spring modeling unit comprises: a spring frequency modeling module that models natural frequencies of a plurality of springs having different natural frequencies and vibrating according to input sound; a filtered pure tone amplitude determination module that determines filtered pure tone amplitudes of the plurality of springs; an amplitude calculation module that calculates transient pure tone amplitudes of the modeled plurality of springs, calculates expected steady-state amplitudes of the modeled plurality of springs, calculates predicted pure tone amplitudes based on the expected steady-state amplitudes, and calculates filtered pure tone amplitudes by multiplying the transient pure tone amplitude by the predicted pure tone amplitude; an expected steady-state amplitude estimation module that estimates the expected steady-state amplitude of a spring having the largest amplitude among the modeled plurality of springs; a spring energy calculation module that calculates the energy of at least one spring having the largest amplitude among the plurality of springs based on the expected steady-state amplitude; and an input pure tone amplitude calculation module that calculates the amplitude of the input pure tone based on the energy.

3. The device according to claim 1, the sound recognition and synthesis unit is characterized by performing speech recognition; speaker verification; speaker identification; source separation; sound direction detection; sound-based nomenclature diagnostics; sound-based machine fault diagnostics; or Sonar for navigation undersea terrain or ranging objects.

4. The device according to claim 1, the error inspection unit is characterized in that, when the frequency of the plurality of input springs is maintained at a first value until a certain point of time and turns to a second value at the certain point, the frequency conversion result up to the certain point is indicated as the first value, and immediately after the turning point, the transient error from the first value to the second value is checked to be within 10%, thereby inspecting the error between pure tone frequencies.

5. A sound processing method comprising the steps of: modeling, by a spring modeling unit, natural frequencies of a plurality of springs, the plurality of springs having natural frequencies different from each other and oscillating according to an input sound; determining, by the spring modeling unit, filtered pure-tone amplitudes of the plurality of springs: calculating, by the spring modeling unit, transient-state-pure-tone amplitudes of the plurality of modeled springs; calculating, by the spring modeling unit, expected steady-state amplitudes of the plurality of modeled springs; calculating, by the spring modeling unit, predicted pure-tone amplitudes based on the expected steady-state amplitudes; calculating, by the spring modeling unit, filtered pure-tone amplitudes by multiplying the transient-state-pure-tone amplitudes with the predicted pure-tone amplitudes; extracting, by a frequency extraction unit a natural frequency of at least one spring of the plurality of springs which corresponds to a local maximum value among the filtered pure-tone amplitudes; and using, by a sound recognition and synthesis unit, the natural frequency for sound recognition or sound synthesis.

6. The method according to claim 5, wherein said expected steady-state amplitude is calculated based on the amplitudes at least two time points within a duration of the input sound.

7. The method according to claim 5, wherein said expected steady-state amplitude is calculated by the equation below: $A_{i,s}=\frac{A_i\left(t_2\right)-A_i\left(t_1\right)e^{-\left(t_2-t_1\right)}}{1-e^{-\left(t_2-t_1\right)}}$ where Ai,s is the expected steady-state amplitude of i-th spring Si among the plurality of springs, wherein I is a positive integer, where t1 and t2 are two different time points within a duration of the input sound, t2>t1, Ai(t1) is an amplitude of said spring Si at t.sub.1, Ai(t.sub.2) is an amplitude of said spring Si at t.sub.2, is a damping ratio of said spring Si, and satisfies the equation =.sub.i{square root over (12.sup.2)}, where i is the natural frequency of said spring Si.

8. The method according to claim 6, wherein a difference between the two different time points is a period of the natural frequency of the corresponding spring.

9. The method according to claim 6, wherein if one of the two time points is t1, a sampling rate of the input sound is SR, and a period of the natural frequency of the corresponding spring is T, then the other t2 of the two time points is calculated by the equation below: $t_2=\left[t_1+\mathrm{SR}T+0.5\right].$

10. The method according to claim 6, wherein the expected steady-state amplitude is calculated by substituting amplitudes at least two points in the duration of the input sound into the following equation and using a linear regression analysis: $A\left(t\right)=A_s+\left(A_c-A_s\right)e^{-\left(t-t_c\right)}$ where A(t) is an amplitude of any spring among said plurality of springs at t, As is the expected steady-state amplitude of said spring, A.sub.c is an amplitude of said spring at tc, tc is a time point before the at least two points in the duration of the input sound, is a damping ratio of said spring, and satisfies the equation =.sub.i{square root over (12.sup.2)}, where i is the natural frequency of said spring.

11. The method according to claim 5, wherein the spring modeling unit is characterized by performing the steps of: measuring displacements and velocities at time points for each of the plurality of springs; calculating an energy at each time point for each of the plurality of springs based on the displacements and the velocities; and calculating an amplitude at each time point for each of the plurality of springs based on the energy.

12. The method according to claim 5, wherein the number of the plurality of springs is determined based on a range and a resolution of the frequency to be extracted.

13. A sound processing method comprising the steps of: sampling, by a spring modeling unit, natural frequencies of a plurality of springs, the plurality of springs having natural frequencies different from each other and oscillating according to an input sound; estimating, by the spring modeling unit, an expected steady-state amplitude of the spring of which the amplitude is the highest among the plurality of modeled springs; calculating, by the spring modeling unit, an energy of at least one spring of the plurality of springs of which the amplitude is the highest based on the expected steady-state amplitudes; calculating, by the spring modeling unit, an amplitude of the input pure tone based on the energy; and using, by a sound recognition and synthesis unit, the amplitude of the input pure tone for sound recognition or sound synthesis.

14. The method according to claim 13, wherein said expected steady-state amplitude is calculated by the equation below: $A_{i,s}=\frac{A_i\left(t_2\right)-A_i\left(t_1\right)e^{-\left(t_2-t_1\right)}}{1-e^{-\left(t_2-t_1\right)}}$ in which Ai,s is the expected steady-state amplitude of a spring Si among the plurality of springs, said spring Si of which amplitude being the highest among amplitudes of the plurality of springs at each time point, wherein I is a positive integer, t1 and t2 are two time points within a duration of input sound satisfying t2>t1, Ai(t1) is an amplitude of said spring Si at t1, Ai(t2) is an amplitude of said spring Si at t2, is a damping ratio of said spring, and satisfies the equation =.sub.i{square root over (12.sup.2)}, where is the natural frequency of said spring of which the amplitude is the highest.

15. The method according to claim 13, wherein the spring modeling unit is characterized by performing the steps of: measuring a displacement and a velocity at each time point for each of the plurality of springs; calculating an energy at each time point for each of the plurality of springs based on the displacement and the velocity; and calculating an amplitude at each time point for each of the plurality of springs based on the energy.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0025] FIG. 1 is a configuration diagram of a sound processing device of the present invention. FIG. 2 is a configuration diagram of a spring modeling unit of the present

[0026] invention.

[0027] FIG. 3 is a configuration diagram of a sound recognition and synthesis unit of the present invention.

[0028] FIG. 4 is an example of a graph showing the displacement of a spring when an external force is zero;

[0029] FIG. 5 is an example of a graph showing the changes in amplitude of a spring when an external force is applied and disappears;

[0030] FIG. 6 is a flowchart showing sound processing method according to an embodiment of the present disclosure;

[0031] FIG. 7A is a graph showing the transient-state-pure-tone amplitude, and FIG. 7B showing the amplitude of the input pure tone;

[0032] FIGS. 8A, 8B, 8C, 8D, 8E, 8F, 8G, 8H and 8I show graphs for the transient-state-pure-tone amplitude, the predicted pure-tone amplitude, and the filtered pure-tone amplitude according to an embodiment of the present disclosure when 1 kHz sound with constant amplitude is input;

[0033] FIG. 9 shows a graph of the filtered pure-tone amplitude when a complex tone is input;

[0034] FIG. 10 shows a graph of the filtered pure-tone amplitude when a complex tone which is different from FIG. 9 is input;

[0035] FIG. 11 is a flowchart showing a sound processing method according to an embodiment of the present disclosure;

[0036] FIGS. 12A, 12B, 12C, 12D, 12E and 12F, are drawings which show the result of the STFT, the frequencies of the input sounds, and the result of the DJ transform according to the present disclosure when pure tones are input;

[0037] FIGS. 13A, 13B, 13C and 13D are drawings which show the results of the DJ transform according to the present disclosure when the frequencies of the input pure tones are changed;

[0038] FIGS. 14A, 14B, 14C and 14D are drawings which show the results of the STFT when the frequencies of the input pure tones are changed;

[0039] FIGS. 15A, 15 B and 15C are drawings which show the frequency components of the input signals, the results of DJ transform, and the results of the STFT when a flickering signal and a lasting signal are input;

[0040] FIGS. 16A, 16B and 16C are drawings which show the frequency components of an input sound, the results of the DJ transform and the STFT when 1 kHz and 2 kHz sounds are alternately input;

[0041] FIGS. 17A, 17B and 17C are drawings which show the results of the DJ transform and the STFT when a pure tone and a complex tone are input;

DETAILED DESCRIPTION

[0042] The following description with reference to the accompanying drawings is provided to assist in a comprehensive understanding of various embodiments of the present disclosure.

[0043] Referring to FIG. 1, the sound processing device (100) of the present invention includes a spring modeling unit (110), a frequency ex raction unit (120), a sound recognition and synthesis unit (130), and an error inspection unit (140).

[0044] The spring modeling unit (110) models the movement of hair cells using a plurality of springs that have different natural frequencies and vibrate according to input sounds.

[0045] Hair cells change mechanical signals generated from the basilar membrane into electrical signals and transmit signals to the primary auditory cortex. Hair cells are composed of approximately 3,500 inner hair cells and 12,000 outer hair cells, and each hair cell is sensitive to sounds of its own characteristic frequency. This characteristic of hair cells is similar to the phenomenon in which a spring resonates and its amplitude increases when it receives an external force of a frequency that matches its own natural frequency. Utilizing this similarity, the spring modeling unit (100) models the movement of hair cells using a plurality of springs.

[0046] The spring modeling unit (110) can calculate the displacement and velocity of each of the plurality of springs by modeling a plurality of springs, each of which has a different natural frequency and vibrates according to the input sound. In addition, the spring modeling unit (110) can calculate the displacement, velocity, energy, and amplitude of each of the plurality of springs by modeling a plurality of springs, each of which has a different natural frequency and vibrates according to the input pure tone.

[0047] The spring modeling unit (110) can calculate the transient pure tone amplitude of the modeled plurality of springs, calculate the expected steady-state amplitude of the modeled plurality of springs, calculate the predicted pure tone amplitude based on the expected steady-state amplitude, multiply the transient pure tone amplitude by the predicted pure tone amplitude to calculate the filtered pure tone amplitude, and estimate the expected steady-state amplitude of the spring with the largest amplitude.

[0048] To this end, as illustrated in FIG. 2, the spring frequency modeling unit (110) includes a spring frequency modeling module (111), a filtered pure tone amplitude determination module (112), an amplitude calculation module (113), an expected steady-state amplitude estimation module (114), a spring energy calculation module (115), and an input pure tone amplitude calculation module (116).

[0049] The spring frequency modeling module (111) performs a function of modeling the natural frequencies of a plurality of springs that have different natural frequencies and vibrate according to the input sound.

[0050] The filtered pure tone amplitude determination module (112) performs a function of determining the filtered pure tone amplitude of a plurality of springs. The amplitude calculation module (113) performs a function of

[0051] calculating transient pure tone amplitudes of a plurality of modeled springs, a function of calculating expected steady state amplitudes of a plurality of modeled springs, a function of calculating expected pure tone amplitudes based on the expected steady state amplitudes, and a function of calculating filtered pure tone amplitudes by multiplying the expected pure tone amplitudes by the transient pure tone amplitudes.

[0052] The expected steady state amplitude estimation module (114) performs a function of estimating the expected steady state amplitude of a spring having the largest amplitude among the plurality of modeled springs.

[0053] The spring energy calculation module (115) performs a function of calculating the energy of at least one spring having the largest amplitude among the plurality of springs based on the expected steady state amplitudes.

[0054] Here, the spring energy calculation module (115) can measure displacement and velocity for each of the plurality of springs at each point in time, and calculate energy for each of the plurality of springs at each point in time based on the displacement and velocity.

[0055] The input pure tone amplitude calculation module (116) performs a function of calculating the amplitude of the input pure tone based on the energy.

[0056] The frequency extraction unit (120) extracts the natural frequency of the spring corresponding to the local maximum among the filtered pure tone amplitudes calculated by the water spring modeling unit (110).

[0057] The sound recognition and synthesis unit (130) determines the filtered pure tone amplitudes of several springs and performs sound recognition or sound synthesis using the natural frequencies.

[0058] To this end, as shown in FIG. 3, the sound recognition and synthesis unit (130) includes a sound recognition module (131) and a sound synthesis module (132).

[0059] The sound recognition module (131) performs a function of recognizing sound using the amplitude or natural frequency of the input pure tone.

[0060] Here, sound recognition includes voice recognition in a narrow sense of converting human speech into text, speaker recognition that determines whose voice the input sound corresponds to, sound source separation such as distinguishing a specific person's voice when multiple speakers' voices are mixed, separating voice from noise when noise is mixed in the voice, or separating vocals excluding instruments in a song, sound direction detection, sound-based disease diagnosis such as coughing or breathing sounds, sound-based machine failure diagnosis using machine sounds, sonar for underwater terrain exploration or object distance measurement, etc.

[0061] The sound synthesis module (132) performs a function of synthesizing sound using the amplitude or natural frequency of the input pure tone.

[0062] The error checking unit (140) determines the frequency of multiple springs applied as input sounds. When input, maintains the first value until a certain point in time and changes to a second value at said certain point in time, the frequency conversion result up to said certain point in time is represented as the first value, and immediately after said changing point in time, checks whether the transient error from said first value to said second value is within 10% to examine the error between pure tone frequencies. In one embodiment, the sound processing device (100) of the present

[0063] invention may be configured as a SoC (System-on-a-chip) that receives sound in the form of wav data and extracts frequency at a constant cycle (e.g., 1 msec). Therefore, each of the components, a spring modeling unit (110), a frequency extraction unit (120), a sound recognition and synthesis unit (130), and an error inspection unit (140), may be components that operate through the hard-wired logic of the SoC.

[0064] In another embodiment, the sound processing device (100) of the present invention may be a DSP (Digital Signal Processor) that receives sound in the form of wav data and extracts frequency at a constant cycle (e.g., 1 msec). Therefore, each of the components, a spring modeling unit (110), a frequency extraction unit (120), a sound recognition and synthesis unit (130), and an error inspection unit (140), may be components of a programming code that operates in the DSP.

[0065] Through the above examples, the sound processing device (100) of the present invention can be used as a main component of a voice device such as a volume amplifier, a voice recognizer, a noise canceller, etc.

[0066] The process in which the sound processing device (100) of the present invention performs sound recognition and sound synthesis by utilizing the similarity between hair cells and water springs is as follows.

[0067] Hair cells convert mechanical signals generated in the basement membrane into electrical signals and transfer the signals to the primary auditory cortex. Hair cells consist of about 3,500 inner hair cells and 12,000 external hair cells, and each hair cell reacts sensitively to the sound of its own natural frequency. This characteristic of hair cells is similar to a phenomenon occurred in a spring of which amplitude increases because of resonance when the spring receives an external force with a frequency that matches the natural frequency of the spring. Using this similarity, the sound processing device (100) of present invention models the behavior of hair cells using a plurality of springs.

[0068] The human audible frequency is known to be in the range of 2020,000 Hz and the human voice frequency is known to be in the range of 808,000 Hz. The frequency range covered in the field such as speech recognition is within 8 kHz. Considering the same, when used for a voice processing, the natural frequencies of the springs from 50 Hz to 8 kHz are classified by 1 Hz intervals, and 7,951 different types of springs can be used based on those natural frequencies. This means that the frequency resolution is 1 Hz unit. However, this is only an example, and widening the frequency range or increasing the resolution by using more springs is possible.

[0069] The behavior of a hair cell modeled by a spring can be represented as a differential equation of motion for driven harmonic oscillations. A sound corresponds to an external force made up of a combination of various sine waves which are applied to a spring. Each spring has its own natural frequency and draws its own motion trajectory by a series of sound samples. The motion trajectory of each spring can be obtained by calculating the solution of the differential equation of motion for driven harmonic oscillations using numerical analysis techniques such as the Runge-Kuta method.

[0070] Assume that i is the natural frequency of a spring Si (1iN). The spring Si is used to model the response of a hair cell that are most sensitive to the sound of i frequency among the hair cells constituting the human hearing system.

[0071] When the sound F0cos(t) is input, the reaction xi(t) of the spring Si to the sound can be represented by the equation of motion of the following equation (1):

[00004]$\begin{array}{cc}\frac{d^2{x}_i}{{\mathrm{dt}}^2}+2{}_i\frac{{\mathrm{dx}}_i}{\mathrm{dt}}+{}_i^2{x}_i=\frac{F_0\cos \left(t\right)}{m}& \left(1\right)\end{array}$

where x.sub.i is the length of the spring which deviates from the balance point (displacement), and m is the mass of the object suspended in the spring. is a damping ratio and when a friction coefficient is b.sub.i,

[00005]$=\frac{b_i}{2\sqrt{{\mathrm{mk}}_i}}.$

k.sub.i is a spring constant. i is the natural frequency of the spring when both and F.sub.i are zero, and .sub.i={square root over (k.sub.i/m)}.

[0072] Equation (1) is a differential equation with a general solution. When <1, the solution is the same as the equation (2) below.

[00006]$\begin{array}{cc}{x}_i\left(t\right)=A_i{e}^{-{}_{i}t}\cos \left(\sqrt{1-{}^2}{}_{i}t+{}_i\right)+\frac{F_0}{{\mathrm{mZ}}_i}\cos \left(t+{}_i\right)& \left(2\right)\end{array}$

where A.sub.i and .sub.i are determined by the initial conditions of the spring, and Z.sub.i and .sub.i are as below:

[00007]$\begin{array}{cc}{Z}_i=\sqrt{{\left(2{}_i\right)}^2+{\left({}_i^2-{}^2\right)}^2}& \left(3\right)\end{array}$$\begin{array}{cc}{}_i=\arctan \left(\frac{2{}_i}{{}^2-{}_i^2}\right)+n& \left(4\right)\end{array}$

[0073] The integer n is specified so that qi is between 180 and 0. If F0=0, the spring is subjected to periodically damping oscillation as shown in FIG. 1. If F0>0 and the spring reaches a steady state after a certain period of time, the first term in the equation (2) disappears and the second term remains only so that the trajectory Xi,s(t) of the spring in a steady state follows the equation (5).

[00008]$\begin{array}{cc}{x}_{i,s}\left(t\right)=\frac{F_0}{{\mathrm{mZ}}_i}\cos \left(t+{}_i\right)& \left(5\right)\end{array}$

[0074] Consider a situation in which a sound having a frequency identical with the natural frequency i of a spring Si in a stop state is applied to the spring as an external force. The behavior of the spring in the process of reaching a steady state is described by the equation (6) below.

[00009]$\begin{array}{cc}{x}_i\left(t\right)=\left(1-e^{-{}_{i}t}\right)x_{i,s}\left(t\right)& \left(6\right)\end{array}$

[0075] Therefore, the amplitude A.sub.i(t) of the spring gradually increases along the trajectory of

[00010]$A_i\left(t\right)=\frac{F_0}{{\mathrm{mZ}}_i}\left(1-e^{-{}_{i}t}\right)$

and finally becomes

[00011]$\frac{F_0}{{\mathrm{mZ}}_i}.$

[0076] As the external force disappears at the point t.sub.o, the amplitude of the spring gradually decreases to zero. This corresponds F.sub.0=0 in the equation (2), and the amplitude change in this process follows the equation below.

[00012]$\begin{array}{cc}{A}_i\left(t\right)=A_i\left(t_0\right)e^{-{}_i\left(t-t_0\right)}& \left(7\right)\end{array}$

[0077] FIG. 5 is an example of a graph showing the changes in amplitude of a spring when an external force is applied and disappears.

[0078] According to the embodiments of the sound processing device (100) of the present invention, two methods for extracting the frequency and amplitude of the input sound are proposed based on the behavior of the spring modeled as hair cells.

Method I for Extracting the Frequency and Amplitude of the Input Sound

[0079] 1. In a steady state [0080] (1) Extraction of frequency

[0081] Based on the characteristic that a resonating spring oscillates with a greater amplitude than other springs, a frequency of an input sound can be extracted.

[0082] Given a pure sound F.sub.ocos(t), an amplitude of a spring Si in a steady state becomes

[00013]$\frac{F_0}{{\mathrm{mZ}}_i}$

by the equation (5). If the mass m of the object suspended in each spring is equal to each other, the spring with the greatest amplitude is the spring having the minimum Z.sub.i. The relationship between the natural frequency .sub.i of the spring and the frequency of pure tone can be obtained by differentiating Equation (3) with respect to .sub.i, and the result is as follows:

[00014]$\begin{array}{cc}={}_i\sqrt{1-2{}^2}& \left(8\right)\end{array}$

where <1/{square root over (2)}. If (is a small value near zero, then .sub.i. For example, could be 0.001.

[0083] In order to find out the spring having the greatest amplitude, a numerical analysis method such as Runge-Kuta, which solves differential equations, is used. Given a pure sound F.sub.ocos(t), the displacement x.sub.i(t) and the velocity v.sub.i(t) of each spring S.sub.i which corresponds to the solution of equation (1) are calculated using the numerical analysis method. Since an energy of each spring is the sum of a kinetic energy and a potential energy, the energy of spring S.sub.i can be obtained by equation (9).

[00015]$\begin{array}{cc}{E}_i=\frac{1}{2}{k}_i{x_i}^2+\frac{1}{2}{{\mathrm{mv}}_i}^2& \left(9\right)\end{array}$

[0084] The energy of the spring that has reached a steady state maintains a constant value. Thus, the displacement x.sub.i at the time when the velocity v.sub.i is 0 becomes the amplitude of the spring S.sub.i. Therefore, the amplitude A.sub.i of spring S.sub.i in a steady state can be calculated by the equation below:

[00016]$\begin{array}{cc}{A}_i=\sqrt{\frac{2E_i}{k_i}}& \left(10\right)\end{array}$

[0085] The spring having the largest amplitude among the extracted amplitudes of the springs is the resonating spring. Therefore, it is possible to obtain the frequency of an input pure tone by using both the natural frequency i of the spring having the largest amplitude and the equation (8).

(2) Extraction of Amplitude

[0086] In a steady state, the trajectory of the spring is given by the equation (5). Therefore, the relationship between an energy of a spring in a steady state, E.sub.i,s, and an amplitude F.sub.o of a given pure tone can be represented by the equation (11).

[00017]$\begin{array}{cc}{E}_{i,s}=\frac{1}{2}{k_i\left(\frac{F_o}{{\mathrm{mZ}}_i}\right)}^2& \left(11\right)\end{array}$

[0087] In addition, the energy in a steady state, E.sub.i,s, can be obtained by putting the displacement x.sub.i and the velocity v.sub.i in the steady state, which are obtained by solving the equation (1) with the numerical analysis method, into the equation (9). Therefore, the amplitude F.sub.o of a given pure tone becomes as below:

[00018]$\begin{array}{cc}{F}_o={\mathrm{mZ}}_i\sqrt{\frac{2E_{i,s}}{k_i}}& \left(12\right)\end{array}$

[0088] The natural frequency .sub.i of the spring that resonates with an external force is almost the same with the frequency of the external force. Therefore, if putting w.sub.i into the equation (3), then Zi=2.sub.i.sup.2. If putting both of this result and i={square root over (k.sub.i/m )} into the equation (12), the amplitude F.sub.0 of the input pure tone can be calculated by the equation (13).

[00019]$\begin{array}{cc}{F}_o=2{}_i=\sqrt{2{\mathrm{mE}}_{i,s}}& \left(13\right)\end{array}$ [0089] 2. In a transient state [0090] (1) Extraction of frequency

[0091] Assume that a pure tone F.sub.ocos(t) is given over a time interval [t.sub.a, t.sub.b]. All springs start to move in an initial state where both displacements and velocities are zero. Using the numerical analysis technique, the energies of the springs are calculated at each time point, and the calculated results are put into the equation (10) to obtain the amplitudes of the springs at each time point. After that, the natural frequency of the spring having the largest amplitude is substituted into the equation (8) to calculate the frequency of the given pure tone.

[0092] (2) Extraction of amplitude

[0093] Assume that an energy of a resonating spring S.sub.i found by the numerical analysis is E.sub.i(t). The amplitude A.sub.i(t) of a spring S.sub.i at time t can be calculated from E.sub.i(t) using the equation (10).

[0094] According to the general solution of the equation (1), the amplitude A.sub.i(t) of the spring S.sub.i resonating with a given sound wave follows the trajectory of the equation (6), so that the spring S.sub.i follows the trajectory of A.sub.i(t)=(1e.sup.(tt.sup.a.sup.))A.sub.i,s in a time interval [t.sub.a, t.sub.b] starting from the initial state until it reaches the steady state. Here, A.sub.i,s is the amplitude of the spring when it reaches the steady state. We call it an expected steady-state amplitude.

[0095] The energies E.sub.i(t.sub.1) and E.sub.i(t.sub.2) at two time points t.sub.1, t.sub.2 within the time interval [t.sub.a, t.sub.b] can be obtained with the numerical analysis method. Therefore, the amplitudes Ai(t.sub.1) and Ai(t.sub.2) can be obtained by substituting these results into the equation (10). The expected steady-state amplitude, A.sub.i,s, can be obtained by putting the result into A.sub.i(t)=(1e.sup.(tt.sup.a.sup.))A.sub.i,s, and the result is as the equation below:

[00020]$\begin{array}{cc}{A}_{i,s}=\frac{A_i\left(t_2\right)-A_i\left(t_1\right)e^{-\left(t_2-t_1\right)}}{1-e^{-\left(t_2-t_1\right)}}& \left(14\right)\end{array}$

[0096] Next, regarding the case where the frequency is the same but the volume of the sound changes, assume that the amplitude of the sound given at the point t.sub.c has changed from F.sub.1to F.sub.2. Let A.sub.c be the amplitude of a spring at the time point t.sub.c and let A.sub.s be the amplitude of a spring at the time the spring will have approached a steady state after the external force changes to F.sub.2. The behavior of the amplitude over time can be described by the following equation.

[00021]$\begin{array}{cc}A\left(t\right)=A_s+\left(A_c-A_s\right)e^{-\left(t-t_c\right)}& \left(15\right)\end{array}$

[0097] Given the amplitudes A(t.sub.1) and A(t.sub.2) at two time points t.sub.1 and t.sub.2 within the time interval that the amplitude changes from A.sub.c to A.sub.s, it can be seen that the obtained A.sub.s is the same as Equation (14).

[0098] For example, consider the case where the external force F.sub.2=0 at the time point t.sub.c. When the external force disappears, the energy of the spring decreases exponentially according to the equation (7). Namely, the measured amplitude of the spring after T seconds from the time when the external force disappears will be A(t.sub.c+T)=A(t.sub.c)e.sup.t. Putting this measurement result into the equation (14) makes A.sub.s=0, and it means the external force has disappeared.

[0099] Therefore, the expected steady-state amplitude, As, can be obtained by measuring the energy of the spring more than once. Using equation (10) which represents the correlation between amplitude and energy, the energy in the steady state, Es, can be calculated and consequently the amplitude Fo of a given pure tone can be calculated using the equation (13).

[0100] Since the force applied to the spring is in the form of a periodic function, the energy does not increase uniformly within a period of a transient state. Considering this characteristic, when selecting the two time points ti and t.sub.2 described above, the time interval is made to be the same with the period.

[0101] In this regard, it may not be able to select two time points of which a time difference between them is one period due to the relationship between the sampling rate of the sound data and the natural frequency of the spring. In this case, an error may occur, and two methods can be used to correct this error.

[0102] The first method is to select an adjacent sample which shows a less difference with a period. When the position S.sub.1 of a sample and the period T of an audio data are given, the position S.sub.2 of the second sample is calculated as [S.sub.1+sampling rateT+0.5]. The expected steady-state amplitude, A.sub.s, is calculated by putting the time information of the two points and the amplitudes at the two points into the equation (14).

[0103] The second method uses a linear regression analysis. After extracting the amplitude at several points and putting the extracted data into the equation (15), the expected steady-state amplitude, A.sub.s, is calculated by the linear regression analysis.

[0104] Based on the above theoretical background, a method for extracting a frequency of an input sound can be proposed as below.

[0105] Referring FIG. 6, a method, of which each step is performed by the sound processing device (100), for extracting a frequency of an input sound according to an embodiment of the present disclosure may comprise the steps of: [0106] (a) modeling, by a spring modeling unit (110), a plurality of springs which have natural frequencies different from each other and oscillate in accordance with the input sound; [0107] (b) estimating, by a spring modeling unit (110), an expected steady-state amplitude, A.sub.i,s, of the spring of which amplitude A.sub.i(t) is the highest among the plurality of modeled springs; [0108] (c) calculating, by a spring modeling unit (110), an energy E.sub.i,s of said spring of which amplitude is the highest based on the expected steady-state amplitude, A.sub.i,s; and [0109] (d) calculating, by a spring modeling unit (110), the amplitude F.sub.0 of the input sound based on said energy E.sub.i,s.

[0110] The step (a) may comprise the steps of: measuring displacements x.sub.i(t) and velocities v.sub.i(t) at time points for each of the plurality of springs (see the equation 1); calculating energy E.sub.i(t) at each time point for each of the plurality of springs based on the displacements and the velocities (see the equation 9); and calculating an amplitude A.sub.i(t) of each of the plurality of springs based on the energies E.sub.i(t) (see the equation 10).

[0111] The step (b) can be calculated with the equation (14).

[0112] In the step (b), said expected steady-state amplitude, A.sub.i,s(t), can be calculated based on the amplitudes at two different time points within a duration of the input sound.

[0113] A difference between the two different time points can be a period of the natural frequency of the corresponding spring.

[0114] When one of the two time points is t.sub.1, a sampling rate of the input sound is SR, and the period of the natural frequency of the corresponding spring is T, the other t.sub.2 of the two time points can be calculated by means of the equation below.

[00022]$t_2=\left[t_1+\mathrm{SR}T+0.5\right]$

[0115] The number of the plurality of springs N may be determined based on a range and a resolution of the frequency to be extracted.

[0116] FIGS. 7A to 7C are graphs representing the experimental results according to embodiments of the present disclosure.

[0117] FIG. 7A shows the result obtained by putting the energy E2000 (t) of a spring, of which natural frequency is 2 kHz over time when a pure tone having a frequency of 2 kHz with a constant amplitude is input between 0.2 and 0.8 seconds, into the equation (13). This result is called a transient-state-pure-tone amplitude. The transient-state-pure-tone amplitude is an amplitude of the input pure tone which is calculated under the assumption that there is no change in the energy of the spring. As time goes by, the energy of the spring reaches a steady state. Therefore, as shown in FIG. 7A, the transient-state-pure-tone amplitude gradually reaches a steady state, and the amplitude at this time corresponds to the amplitude F.sub.m(t) of the input pure tone. Here, m indicates a natural frequency of a spring.

[0118] FIG. 7B shows the amplitude F.sub.m(t) of the input pure tone that is obtained by putting the measured amplitude of the spring into the equation (14) to obtain the expected steady-state amplitude of the spring, A.sub.m,s(t), and applying the results to the steps (c) and (d) of the frequency extraction method above. As shown in FIG. 7B, the amplitude of the input pure tone is extracted from the starting point of the pure tone.

Method II for Extracting the Frequency and Amplitude of the Input Sound

[0119] According to the method I for extracting the frequency and amplitude of the input sound described above, if the input sound is a pure tone, the frequency and amplitude of the input sound can be effectively extracted.

[0120] Now, assume that there are n types of pure tones constituting a complex tone F(t)=.sub.jF.sub.jcos(.sub.jt+.sub.j). If n =1, the pure tone of a given sound can be found by selecting the spring having the largest amplitude among the springs. However, if n>1, it is difficult to find out pure tones constituting the complex tone by selecting top n springs in the order of amplitude.

[0121] The first reason is that the amplitude of a spring of which the frequency is adjacent to the spring having the largest amplitude could be greater than the amplitude of the spring which resonates with other pure tones constituting the complex tone. The second reason is that, as shown in the trajectory after 0.8 seconds in FIG. 5, even though the external force disappears, it takes time until the amplitude of the spring reaches 0, so the amplitude of the sound that does not exist anymore could be greater than the amplitude of other pure tones.

[0122] Accordingly, in this embodiment, instead of finding the local maximum value among the spring amplitudes at each time point, a method of finding the local maximum value from the results of multiplying an expected steady-state amplitude and a transient-state-pure-tone amplitude is proposed.

1. Expected Steady-State Amplitude and Filtered Pure-Tone Amplitude

[0123] First, in order to extract the pure tones constituting a complex tone, the amplitude A.sub.i(t) of each spring S.sub.i is calculated by applying the step (a) of the method I to each spring for extracting the frequency of an input sound. FIG. 8A shows the amplitudes of springs of which natural frequencies are around 1 kHz as a result measured at 215 milliseconds when a sound having a frequency of 1 kHz with a constant amplitude starts at 200 milliseconds. FIG. 8A shows that the amplitude of the spring that does not resonate is lower than that of the spring that resonates.

[0124] Next, an expected steady-state amplitude, A.sub.i,s(t), is calculated by applying the step (b) of the method I for extracting the frequency of an input sound to the amplitude A.sub.i(t) of each spring Si. However, the equation (14) which calculates the expected steady-state amplitude is an equation derived from the equation (7) which describes the behavior of a resonating spring. Therefore, high amplitudes could be resulted even at the frequencies away from the resonant frequency as in FIG. 8B.

[0125] Accordingly, the following steps are performed. The third step is to calculate a transient-state-pure-tone amplitude, F.sub.i,t(t), by putting the amplitude A.sub.i(t) of the spring S.sub.i into the equation (13). In addition, a predicted pure-tone amplitude, F.sub.i,s(t), is calculated by applying steps (c) and (d) of the method I for extracting the frequency of the input sound to the expected steady-state amplitude, A.sub.i,s(t).

[0126] As the final step, a filtered pure-tone amplitude, F.sub.i,p(t), is calculated by multiplying the transient-state-pure-tone amplitude, F.sub.i,t(t), with the predicted pure-tone amplitude, F.sub.i,s(t), as in F.sub.i,p(t)=F.sub.i,st(t)F.sub.i,s(t). Additionally, the result of multiplication of the amplitudes may be divided by the maximum amplitude of the sound in order not to exceed 1 but to be normalized. For example, if the sound is expressed as a 16-bit integer, the result is divided by 32,767.

[0127] A filtered pure-tone amplitude has the characteristic that 1) the amplitude becomes 0 when the sound disappears, and 2) the amplitudes of frequencies away from a resonant frequency in the frequency domain are low.

[0128] FIG. 8C shows the filtered pure-tone amplitude, which is the result of multiplication of the amplitudes in FIGS. 8A and 8B with respect to the same frequency.

[0129] FIGS. 8D to 8F show the transient-state-pure-tone amplitude, the predicted pure-tone amplitude, and the filtered pure-tone amplitude obtained by the spring with a natural frequency of 1 kHz, respectively. Especially, it is shown that, after the input sound disappears at 0.8 seconds, the amplitude in FIG. 8D remains not to be zero, but the amplitudes in FIG. 8E and 8F become zero. FIG. 8G to 8I show the results for the spring with the natural frequency of 1,020 Hz. Apparently, the filtered pure-tone amplitude, F.sub.1020,p(t), is very small compared to the filtered pure-tone amplitude, F.sub.1000,p(t), of the resonating spring of FIG. 8F.

2. Finding a Pure Tone from Local Maximum Values

[0130] FIG. 9 is a graph showing frequency vs. filtered pure-tone amplitude of a complex tone composed of five pure tones of 100 Hz, 250 Hz, 500Hz, 1 kHz, and 4 kHz. As shown in FIG. 9, if frequency intervals of the sounds constituting the complex tone are broad, each pure tone frequency generates a local maximum value among local maximum values in a frequency. Using these characteristics, several local maxima are obtained from a frequency vs. amplitude graph obtained by using the filtered pure-tone amplitude. Then the local maxima of those several local maxima are obtained again. Finally, frequencies corresponding to the local maxima are regarded as frequencies of the pure tones constituting the complex tone.

[0131] However, if the frequency interval is narrow, no local maximum might exist between two adjacent local maxima. FIG. 10 is a part of the graph for frequency vs. filtered pure-tone amplitude of a complex tone composed of five pure tones of 112 Hz, 181 Hz, 1,034 Hz, 5,017 Hz, and 5,034 Hz. It shows that no local maximum exists between the two local maxima that are generated by the two adjacent frequencies, 5,017 Hz and 5,034 Hz. The characteristic of this case is that the frequency interval is narrow and the two filtered pure-tone amplitudes are similar. Therefore, if the frequency difference between of two adjacent local maxima in filtered pure-tone amplitudes is within a certain width (e.g., the bandwidth of a high-amplitude frequency) and the ratio of those filtered pure-tone amplitudes is equal to or greater than a certain level (e.g. 0.5), both frequencies are treated as the frequencies of pure tones constituting the complex tone.

[0132] Based on the theoretical background described above, the following method for extracting the frequency of the input sound is proposed.

[0133] Referring FIG. 11, a method, of which each step is performed by the sound processing device (100), for extracting a frequency of an input sound according to an embodiment of the present disclosure comprises the steps of: [0134] (1) modeling, by a spring modeling unit (110), a plurality of springs, each spring S.sub.i(1iN) of which has natural frequencies .sub.i, being different from each other, and oscillates according to the input sound; [0135] (2) calculating, by a spring modeling unit (110), transient-state-pure-tone amplitudes of the plurality of modeled springs at each time t, {F.sub.i,t(t)|1iN}, based on displacements and velocities of the modeled springs; [0136] (3) calculating, by a spring modeling unit (110), expected steady-state amplitudes of the plurality of modeled springs at each time t, {A.sub.i,s(t)|1iN}; [0137] (4) calculating, by a spring modeling unit (110), predicted pure-tone amplitudes, {F.sub.i,s(t)|1iN}, based on the expected steady-state amplitudes at each time t, {A.sub.i,s(t)|1iN}; [0138] (5) calculating, by a spring modeling unit (110), filtered pure-tone amplitudes at each time t, {F.sub.i,s(t)|1iN}, by multiplying the transient-state-pure-tone amplitude, F.sub.i,t(t), with the predicted pure-tone amplitude, F.sub.i,s(t), for each spring S.sub.i; [0139] (6) extracting, by a frequency extraction unit (120), natural frequencies of the springs, each filtered pure-tone amplitude of which is a local maximum in a frequency range; and [0140] (7) using, by a sound recognition and synthesis unit (130), the natural frequency for sound recognition or sound synthesis.

[0141] The step (1) may comprise the steps of: measuring displacements x.sub.i(t) and velocities v.sub.i(t) at different time points for each of the plurality of springs (see the equation 1); calculating an energy E.sub.i(t) at each time point for each of the plurality of springs based on the displacements x.sub.i(t) and the velocities v.sub.i(t) (see the equation 9); and calculating an amplitude A.sub.i(t) at each time point for each of the plurality of springs based on the energy E.sub.i(t) (see the equation 10).

[0142] The equation 13 can be used in the step (2), the equation 14 can be used in the step (3), and the equation 13 can be used in the step (4).

[0143] The number of the plurality of springs, N, may be determined based on a range and a resolution of the frequencies to be extracted.

[0144] In the step (3), the expected steady-state amplitudes, A.sub.i,s(t), can be calculated based on the amplitudes at two time points within a duration of the input sound.

[0145] In the step (3), the expected steady-state amplitudes, A.sub.i,s(t), can be calculated by means of the equation below:

[00023]$A_{i,s}=\frac{A_i\left(t_2\right)-A_i\left(t_1\right)e^{-\left(t_2-t_1\right)}}{1-e^{-\left(t_2-t_1\right)}}$

where t.sub.1 and t.sub.2 are the two different time points within the duration of input sound, t.sub.2>t.sub.1, Ai(t.sub.1) is an amplitude of any spring among the plurality of springs at t.sub.1, Ai(t.sub.2) is an amplitude of said spring at t.sub.2, is a damping ratio of said spring, and satisfies the equation =.sub.i{square root over (12.sup.2)}, where .sub.i is the natural frequency of said spring.

[0146] A difference between the two different time points can be a period of the natural frequency of the corresponding spring.

[0147] When one of the two time points is t1, a sampling rate of the input sound is SR, and a period of the natural frequency of the corresponding spring is T, the other t2 of the two time points is calculated by the equation below.

[00024]$t_2=\left[t_1+\mathrm{SR}T+0.5\right]$

[0148] In step (7), the natural frequency may be used for sound recognition or sound synthesis.

[0149] The sound processing method and sound processing apparatus according to the sound processing device (100) of the present invention can be applied not only to human voice but also to all types of sounds such as objects such as musical instruments and animals. In the present disclosure, sound recognition includes: speech recognition in a sense of converting human speech into text; speaker verification/speaker identification for determining whose voice an input sound corresponds to; source separation such as discrimination of a specific person's voice in a state in which the voices of a plurality of speakers are mixed, separation of voice from noise when noise is mixed, and separation of vocals from songs excluding instruments; sound direction detection; sound-based nomenclature diagnostics such as coughing or breathing; sound-based machine fault diagnostics based on mechanical sounds; and Sonar for navigating undersea terrain, ranging objects and more.

[0150] Sound recognition or sound synthesis are example to which the natural frequency obtained by the present invention can be applied, and the scope of the present invention is not limited thereto. The present invention can be applied to any field in which periodic properties or Fourier transforms are used such as price prediction for cryptocurrencies and stocks and image processing such as denoising.

[0151] Hereinafter, the experimental results according to the sound processing device (100) of the present invention will be described. To show the performance of the DJ transform according to the present disclosure, the results of the DJ transform and that of the STFT were compared. In the DJ transform, 7,951 springs of which natural frequencies are from 50 Hz to 8,000 Hz were used, respectively. The frequency interval of springs was 1 Hz. A 25 milliseconds window was used for the STFT.

[0152] The DJ transform was performed in an NVIDIA M40 GPU environment with 3,072 cores and 12 GB of memory and was implemented using the C language API of Cuda Toolkit 8.0. It took about 0.6 seconds to do the DJ transform for a 1 second audio data.

[0153] FIGS. 12A to 12F are diagrams showing the results of the STFT and the DJ transform in terms of the frequency resolution. In FIGS. 12A to 12F, the first rows show the results of the STFT, the second rows show the frequencies of the input sounds, and the third rows show the results of the DJ transform according to an embodiment of the present disclosure.

[0154] As shown in FIGS. 12A to 12F, the frequency resolution of the STFT result was 40 Hz. In addition, when the frequencies of pure tones were 400 Hz, 408 Hz, and 416 Hz, the peak was output at 400 Hz, and when the frequencies of pure tones were 424 Hz, 432 Hz, and 440 Hz, the peak was output at 440 Hz. However, the DJ transform results were matched with all the frequencies of pure tones. That means the frequency resolution of the DJ transform result was 1 Hz.

[0155] Three experiments were conducted to compare the results of the DJ transform with the STFT in terms of temporal resolution.

[0156] The first experiment was to check the frequency extracted at the time point where an input frequency changes. FIG. 13A shows a result of extracted frequencies by the DJ transform when a 1 kHz pure tone had been input for 500 milliseconds and a 2 kHz pure tone was input just after 500 milliseconds, FIG. 13B shows a result of extracted frequencies by the DJ transform when a 2 kHz pure tone had been input for 500 milliseconds and a 1 kHz pure tone was input just after 500 milliseconds, FIG. 13C shows a result of extracted frequencies by the DJ transform when a 4 kHz pure tone had been input for 500 milliseconds and a 2 kHz pure tone was input just after 500 milliseconds, and FIG. 13D shows a result of extracted frequencies by the DJ transform when a 2 kHz pure tone had been input for 500 milliseconds and a 4 kHz pure tone was input just after 500 milliseconds. Obviously, FIGS. 13A to 13D show that the boundaries between the two frequencies were at 500-milliseconds. Specifically, until 500 milliseconds, the frequencies of 1 kHz, 2 kHz, 4 kHz and 2 kHz of the input pure tones were clearly displayed, and immediately after 500 milliseconds, the frequencies of 2 kHz, 1 kHz, 2 kHz and 4 kHz of the changed pure tone were displayed with about 10% error only. However, in the STFT results shown in FIGS. 14A to 14D, two frequencies are simultaneously extracted on the 500-millisecond boundary.

[0157] The second experiment is to extract frequencies from the sounds that appear and disappear rapidly. The first rows of FIGS. 15A to 15C show the frequency extraction results when a 1 kHz pure tone is generated for 5 milliseconds, and silent for the next 5 milliseconds from 200 milliseconds to 800 milliseconds (when a flicker signal is repeatedly input). The second rows show the results when a 1 kHz pure tone continuously is input from 200 milliseconds to 800 milliseconds (when a continuous signal is input). FIG. 15A is for the frequency components of the input sound over time, FIG. 15B is for the DJ transform results, and FIG. 15C is for the STFT result.

[0158] In FIG. 15B showing the results of DJ transform, the repeated flicker signal results in a broken line while the continuous signal results in a solid line thereby two signals are distinguished apparently. On the other hand, the results of the STFT shown in FIG. 15C show a solid line at 1 KHz, therefore, the distinction between the flicker signal and the continuous signal is not clear.

[0159] The upper drawing in FIG. 15B shows relatively weak broken lines at 1.1 kHz and 0.9 kHz. These lines are interpreted as the result of 100 Hz signal due to the repeated input of every 10 milliseconds cycle. On the other hand, in the STFT result, solid lines appear at 0.88 kHz, 0.92 kHz, 0.96 kHz, 1.04 kHz, 1.08 kHz and 1.12 kHz when looking at the upper drawing in FIG. 15C. It is conjectured that the reason the STFT result occurs is because 0.9 kHz and 1.1 kHz frequency components are generated by the 100 Hz signal and those components are represented by 40 Hz intervals due to the 40 Hz frequency resolution of the STFT.

[0160] The third experiment is an extension of the second experiment, which shows the results in frequency extraction when a 1 kHz and a 2 kHz pure tones are alternately generated for 5 milliseconds from 200 milliseconds to 800 milliseconds (FIGS. 16A to 16C). FIG. 16B shows that the DJ transform produces the 1 kHz pure tone and the 2 kHz pure tone that are clearly separated in 5 milliseconds units. On the other hand, when the STFT is used, boundaries between the pure tones are not distinguishable as shown in FIG. 16C.

[0161] The first rows of FIGS. 17A to 17C show the input waveform, the result of the DJ transform, and the result of the STFT when a 420 Hz pure tone is input, and the second rows show the input waveform, the DJ transform result, and the STFT result when a complex tone composed of 400 Hz and 440 Hz is input. FIG. 17A shows input waveforms, and FIGS. 17B and 17C show the DJ transform results and the STFT results, respectively.

[0162] As can be seen in FIG. 17B and 17C, the DJ transform extracts 420 Hz frequencies from a pure tone, and 400 Hz and 440 Hz frequencies from a complex tone. On the other hand, there is little difference between the results extracted from both pure tones and the complex tone with the STFT.

[0163] Since the complex tone is composed of 400 Hz and 440 Hz, the amplitude fluctuates in a 40 Hz cycle as shown in the bottom of FIG. 17A. On the other hand, as in the bottom of FIG. 17B, the DJ transform well reflects the characteristic of the amplitude fluctuation. Although the present disclosure has been described in detail through

[0164] preferred embodiments, the present disclosure is not limited thereto, and various changes and applications can be made without departing from the technical spirit of the present disclosure, which is obvious to a person skilled in the art. Therefore, the scope of protection for the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present disclosure.