Sound signal modelling based on recorded object sound
11140495 · 2021-10-05
Assignee
Inventors
Cpc classification
G10L21/02
PHYSICS
H04R25/50
ELECTRICITY
H04R2225/43
ELECTRICITY
International classification
Abstract
A hearing device configured to be worn by a user, includes: a first input transducer for providing an input signal; a first processing unit configured for processing the input signal according to a first sound signal model; and an acoustic output transducer coupled to an output of the first processing unit for conversion of an output signal from the first processing unit into an audio output signal; wherein the hearing device is configured to obtain an input signal comprising a first signal part and a second signal part, the first signal part corresponding at least partly to a first object signal recorded by a recording unit; and wherein the hearing device is also configured to apply a first set of parameter values of a second sound signal model to the first sound signal model, and process the input signal according to the first sound signal model.
Claims
1. A method for signal modelling in a hearing device configured to be worn by a user, the hearing device comprising a first input transducer, a first processing unit coupled to the first input transducer and configured to perform signal processing according to a first sound signal model, and an acoustic output transducer for conversion of an output signal from the first processing unit into an audio output signal, the method comprising: recording a first object signal by a recording unit; determining, by a second processing unit, a first set of parameter values of a second sound signal model for the first object signal; receiving, in the first processing unit of the hearing device, an input signal comprising a first signal part and a second signal part, the first signal part corresponding at least partly to the first object signal; applying the determined first set of parameter values of the second sound signal model to the first sound signal model; and processing the input signal according to the first sound signal model; wherein the input signal is modelled in the hearing device using a generative probabilistic modelling approach.
2. The method according to claim 1, further comprising: recording a second object signal by the recording unit; determining, by the second processing unit, a second set of parameter values of the second sound signal model for the second object signal; receiving, in the first processing unit of the hearing device, an additional input signal comprising a first signal part and a second signal part, the first signal part of the additional input signal corresponding at least partly to the second object signal; applying the determined second set of parameter values of the second sound signal model to the first sound signal model; and processing the additional input signal according to the first sound signal model.
3. The method according to claim 2, further comprising generating a library of sets of parameter values for the second sound signal model for respective object signals, the object signals comprising at least the first object signal and the second object signal, wherein the library of sets of parameter values comprises at least the first set of parameter values and the second set of parameter values.
4. The method according to claim 1, further comprising determining whether the input signal corresponds at least partly to the first object signal, wherein the act of applying the determined first set of parameter values to the first sound signal model is performed if the input signal corresponds at least partly to the first object signal.
5. The method according to claim 1, wherein the first set of parameter values of the second sound signal model is stored in a storage, and wherein the first set of parameter values of the second sound signal model is configured to be retrieved from the storage by the second processing unit.
6. The method according to claim 1, wherein the second processing unit is in an electronic device, and wherein the first set of parameter values of the second sound signal model is sent from the electronic device to the hearing device to be applied to the first sound signal model.
7. The method according to claim 1, wherein the recording unit comprises a second input transducer in an electronic device.
8. The method according to claim 1, further comprising modifying the first set of parameter values of the second sound signal model based on an interface output from a user interface.
9. A method for signal modelling in a hearing device configured to be worn by a user, the hearing device comprising a first input transducer, a first processing unit coupled to the first input transducer and configured to perform signal processing according to a first sound signal model, and an acoustic output transducer for conversion of an output signal from the first processing unit into an audio output signal, the method comprising: recording a first object signal by a recording unit; determining, by a second processing unit, a first set of parameter values of a second sound signal model for the first object signal; receiving, in the first processing unit of the hearing device, an input signal comprising a first signal part and a second signal part, the first signal part corresponding at least partly to the first object signal; applying the determined first set of parameter values of the second sound signal model to the first sound signal model; and processing the input signal according to the first sound signal model; wherein the act of processing the input signal according to the first sound signal model comprises estimating a set of average spectral power coefficients in each frequency band of a filter bank of the first sound signal model.
10. The method according to claim 9, wherein the act of processing the input signal according to the first sound signal model comprises applying the estimated average spectral power coefficients in a spectral subtraction calculation, wherein a fixed object spectrum is subtracted from a time-varying frequency spectrum of the input signal.
11. The method according to claim 10, wherein the spectral subtraction calculation is performed to estimate a time-varying impact factor based on feature(s) in the input signal.
12. The method according to claim 1, wherein the first object signal is a noise signal to be suppressed in the input signal.
13. The method according to claim 1, wherein the first object signal is a desired signal to be enhanced in the input signal.
14. The method according to claim 1, wherein the act of recording is initiated by the user of the hearing device.
15. A hearing device configured to be worn by a user, the hearing device comprising: a first input transducer for providing an input signal; a first processing unit configured for processing the input signal according to a first sound signal model; and an acoustic output transducer coupled to an output of the first processing unit for conversion of an output signal from the first processing unit into an audio output signal; wherein the hearing device is configured to obtain an input signal comprising a first signal part and a second signal part, the first signal part corresponding at least partly to a first object signal recorded by a recording unit; and wherein the hearing device is also configured to apply a first set of parameter values of a second sound signal model to the first sound signal model; wherein the hearing device is configured to model the input signal using a generative probabilistic modelling approach, and/or wherein the first processing unit is configured to process the input signal according to the first sound signal model by estimating a set of average spectral power coefficients in each frequency band of a filter bank of the first sound signal model.
16. The hearing device of claim 15, wherein the first set of parameter values of the second sound signal model is associated with the first object signal.
17. A system comprising the hearing device of claim 15, and an electronic device that comprises the recording unit.
18. A system comprising the hearing device of claim 15, and a second processing unit configured to determine the first set of parameter values of the second sound signal model for the first object signal.
19. A system comprising a hearing device configured to be worn by a user and an electronic device; wherein the hearing device comprises a first input transducer, a first processing unit coupled to the first input transducer and configured to perform signal processing according to a first sound signal model, and an acoustic output transducer coupled to an output of the first processing unit for conversion of an output signal from the first processing unit into an audio output signal; wherein the electronic device comprises a recording unit, and a second processing unit, wherein the electronic device is configured to record a first object signal by the recording unit, and wherein the second processing unit of the electronic device is configured to determine a first set of parameter values of a second sound signal model for the first object signal; wherein the hearing device is configured to obtain an input signal comprising a first signal part and a second signal part, the first signal part corresponding at least partly to the first object signal; and wherein the hearing device is also configured to apply the first set of parameter values of the second sound signal model to the first sound signal model, and process the input signal according to the first sound signal model; wherein the hearing device is configured to model the input signal using a generative probabilistic modelling approach, and/or wherein the first processing unit is configured to process the input signal according to the first sound signal model by estimating a set of average spectral power coefficients in each frequency band of a filter bank of the first sound signal model.
20. The method according to claim 9, further comprising: recording a second object signal by the recording unit; determining, by the second processing unit, a second set of parameter values of the second sound signal model for the second object signal; receiving, in the first processing unit of the hearing device, an additional input signal comprising a first signal part and a second signal part, the first signal part of the additional input signal corresponding at least partly to the second object signal; applying the determined second set of parameter values of the second sound signal model to the first sound signal model; and processing the additional input signal according to the first sound signal model.
21. The method according to claim 20, further comprising generating a library of sets of parameter values for the second sound signal model for respective object signals, the object signals comprising at least the first object signal and the second object signal, wherein the library of sets of parameter values comprises at least the first set of parameter values and the second set of parameter values.
22. The method according to claim 9, further comprising determining whether the input signal corresponds at least partly to the first object signal, wherein the act of applying the determined first set of parameter values to the first sound signal model is performed if the input signal corresponds at least partly to the first object signal.
23. The method according to claim 9, wherein the first set of parameter values of the second sound signal model is stored in a storage, and wherein the first set of parameter values of the second sound signal model is configured to be retrieved from the storage by the second processing unit.
24. The method according to claim 9, wherein the second processing unit is in an electronic device, and wherein the first set of parameter values of the second sound signal model is sent from the electronic device to the hearing device to be applied to the first sound signal model.
25. The method according to claim 9, wherein the recording unit comprises a second input transducer in an electronic device.
26. The method according to claim 9, further comprising modifying the first set of parameter values of the second sound signal model based on an interface output from a user interface.
27. The method according to claim 9, wherein the first object signal is a noise signal to be suppressed in the input signal.
28. The method according to claim 9, wherein the first object signal is a desired signal to be enhanced in the input signal.
29. The method according to claim 9, wherein the act of recording is initiated by the user of the hearing device.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The above and other features and advantages will become readily apparent to those skilled in the art by the following detailed description of exemplary embodiments thereof with reference to the attached drawings, in which:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
DETAILED DESCRIPTION
(12) Various embodiments are described hereinafter with reference to the figures. Like reference numerals refer to like elements throughout. Like elements will, thus, not be described in detail with respect to the description of each figure. It should also be noted that the figures are only intended to facilitate the description of the embodiments. They are not intended as an exhaustive description of the claimed invention or as a limitation on the scope of the claimed invention. In addition, an illustrated embodiment needs not have all the aspects or advantages shown. An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated, or if not so explicitly described.
(13) Throughout, the same reference numerals are used for identical or corresponding parts.
(14)
(15) The recording unit 22 may be an input transducer 48, such as a microphone, in the electronic device 46. The electronic device 46 may be a smartphone, a pc, a tablet etc. The recording is initiated by the user 4 of the hearing device 2. The method comprises determining, by a second processing unit 24, a first set of parameter values 26 of a second sound signal model 28 for the first object signal 20. The second processing unit 24 may be arranged in the electronic device 46. The method comprises subsequently receiving, in the first processing unit 10 of the hearing device 2, an input signal 8 comprising a first signal part 30, corresponding at least partly to the first object signal 20, and a second signal part 32. The method comprises, in the hearing device 2, applying the determined first set of parameter values 26 of the second sound signal model 28 to the first sound signal model 12. The method comprises, in the hearing device 2, processing the input signal 8 according to the first sound signal model 12.
(16) Thus, the electronic device 46 comprises a recording unit 22 and a second processing unit 24. The electronic device 46 is configured for recording the first object signal 20 by the recording unit 22, where the recording is initiated by the user 4 of the hearing device 2. The electronic device 46 is further configured for determining, by the second processing unit 24, the first set of parameter values 26 of the second sound signal model 28 for the first object signal 20.
(17) The electronic device may comprise the second processing unit 24. Thus the determined first set of parameter values 26 of the second sound signal model 28 for the first object signal 20 may be sent from the electronic device 46 to the hearing device 2 to be applied to the first sound signal model 12.
(18)
(19) The electronic device may comprise the second processing unit 24. Thus the determined second set of parameter values 36 of the second sound signal model 28 for the second object signal 34 may be sent from the electronic device 46 to the hearing device 2 to be applied to the first sound signal model 12.
(20) Further, the method comprises recording a respective object signal 44 by the recording unit 22, the recording being initiated by the user 4 of the hearing device 2. The respective object signal 44 may originate from or be transmitted from a respective sound source 56. The method comprises determining, by the second processing unit 24, a respective set of parameter values 42 of the second sound signal model 28 for the respective object signal 44. The method comprises subsequently receiving, in the first processing unit 10 of the hearing device 2, an input signal 8 comprising a first signal part 30, corresponding at least partly to the respective object signal 44, and a second signal part 32. The method comprises applying the determined respective set of parameter values 42 of the second sound signal model 28 to the first sound signal model 12. The method comprises processing the input signal 8 according to the first sound signal model 12.
(21) The electronic device may comprise the second processing unit 24. Thus the determined respective set of parameter values 42 of the second sound signal model 28 for the respective object signal 44 may be sent from the electronic device 46 to the hearing device 2 to be applied to the first sound signal model 12.
(22)
(23) The electronic device may comprise the second processing unit 24. Thus the determined set of parameter values of the second sound signal model 28 for the object signal may be sent from the electronic device 46 to the hearing device to be applied to the first sound signal model.
(24) The electronic device 46 may comprise a storage 38 for storing the determined first set of parameter values 26 of the second sound signal model 28. Thus, the determined first set of parameter values 26 of the second sound signal model 28 is configured to be retrieved from the storage 38 by the second processing unit 24.
(25) The electronic device may comprise a library 40. Thus the method may comprise generating the library 40. The library 40 may comprise determined respective sets of parameters values 42, see
(26) The electronic device 46 may comprise a recording unit 22. The recording unit may be an second input transducer 48, such as a microphone for recording the respective object signals 44, the respective object signal 44 may comprise the first object signal 20 and the second object signal 34.
(27) The electronic device may comprise a user interface 50, such as a graphical user interface. The user may, on the user interface 50, modify the respective set of parameter values 42 of the second sound signal model 28 for the respective object signal 44.
(28)
(29) The hearing device 2 is configured to be worn by a user (not shown). The hearing device 2 comprises a first input transducer 6 for providing an input signal 8. The hearing device 2 comprises a first processing unit 10 configured for processing the input signal 8 according to a first sound signal model 12. The hearing device 2 comprises an acoustic output transducer 14 coupled to an output of the first processing unit 10 for conversion of an output signal 16 from the first processing unit 10 into an audio output signal 18.
(30) The hearing device further comprises a recording unit 22. The recording unit may be a second input transducer 48, such as a microphone, for recording the respective object signals 44; the respective object signal 44 may comprise the first object signal 20 and the second object signal 34.
(31) The method may comprise recording a first object signal 20 by the recording unit 22. The first object signal 20 may originate from or be transmitted from a first sound source (not shown). The first object signal 20 may be a noise signal, which the user of the hearing device 2 wishes to suppress in the input signal 8. The first object signal 20 may be a desired signal, which the user of the hearing device 2 wishes to enhance in the input signal 8.
(32) The hearing device may furthermore comprise the second processing unit 24. Thus the determined set of parameter values of the second sound signal model 28 for the object signal may be processed in the hearing device to be applied to the first sound signal model. The second processing unit 24 may be the same as the first processing unit 10. The first processing unit 10 and second processing unit 24 may be different processing units.
(33) The first input transducer 6 may be the same as the second input transducer 22. The first input transducer 6 may be different from the second input transducer 22.
(34) The hearing device 2 may comprise a storage 38 for storing the determined first set of parameter values 26 of the second sound signal model 28. Thus, the determined first set of parameter values 26 of the second sound signal model 28 is configured to be retrieved from the storage 38 by the second processing unit 24 or the first processing unit 10. The hearing device may comprise a library 40. Thus the method may comprise generating the library 40. The library 40 may comprise determined respective sets of parameters values 42, see
(35) The hearing device may comprise a user interface 50, such as a graphical user interface, such as a mechanical user interface. The user may, via the user interface 50, modify the respective set of parameter values 42 of the second sound signal model 28 for the respective object signal 44.
(36)
(37)
(38) Below disclosed is an example of a technical realization of the system. In general, multiple approaches to the proposed system are available. A generative probabilistic modeling approach may be used.
(39) Model Specification
(40) We assume that audio signals are sums of constituent source signals. Some of these constituent signals are desired, e.g. speech or music, and we may want to amplify those signals. Some other constituent sources may be undesired, e.g. factory machinery, and we may want to suppress those signals. To simplify matters, we write
x.sub.t=s.sub.t+n.sub.t
(41) to indicate that an input signal or incoming audio signal x.sub.t is composed of a sum of a desired signal s.sub.t and an undesired (“noise”) signal n.sub.t. The subscript t holds the time index. As mentioned, there may be more than two sources present but we continue the exposition of the model for a mixture of one desired and one noise signal.
(42) We focus here on attenuation of the undesired signal. In that case, we are interested in producing the output signal
y.sub.t=s.sub.t+α.Math.n.sub.t
where 0≤α<1 is an attenuation factor.
(43) We may use a generative probabilistic modeling approach. This means that
p(x.sub.t|s.sub.t,n.sub.t)=δ(x.sub.t−s.sub.t−n.sub.t) and p(y.sub.t|s.sub.t,n.sub.t)=δ(y.sub.t−s.sub.t−α.Math.n.sub.t).
(44) Each source signal is modelled by a similar probabilistic Hierarchical Dynamic System (HDS). For a source signal s.sub.t, the model is given by
(45)
(46) In this model, we denote by s.sub.t the outcome (“observed”) signal at time step t, z.sub.t.sup.(k) is the hidden state signal at time step t in the k.sup.th layer, which is parameterized by θ.sup.(k). We denote the full set of parameters by θ={θ.sup.(1), . . . , θ.sup.(K)} and we collect all states in a similar manner in the variable s. In
(47) Many well-known models submit to the equations of the prescribed HDS, including (hierarchical) hidden Markov models and Kalman filters and deep neural networks such as convolutional and recurrent neural works.
(48) The generative model can be used to infer the constituent source signals from a received signal and subsequently we can adjust the amplification gains of individual signals so as to personalize the experiences of auditory scenes. Next, we discuss how to train the generative model, which is followed by a specification of the signal processing phase.
(49) Training
(50) We assume that the end user is situated in an environment where he has clean observations of either a desired signal class, e.g. speech or music, or an undesired signal class, e.g. noise sources such as factory machinery. For simplicity, we focus here on the case where he has clean observations of an undesired noise signal, corresponding to the object signal in the above. Let's denote a recorded sequence of a few seconds of this signal by D (i.e., the “data”). The training goal is to infer the parameters of a new source signal. Technically, this comes down to inferring p(θ|D) from the generative model and the recorded data.
(51) In a preferred realization, we implement the generative model in a factor graph framework. In that case, p(θ|D) can be inferred automatically by a message passing algorithm such as Variational Message Passing (Dauwels, 2007). For clarity, we have shown an appropriate message passing schedule in
(52) Signal Processing
(53)
(54) For Generative Model Figure
(55)
(56) For Learning Figure
(57)
(58) For Signal Processing Figure
(59)
REFERENCES
(60) H. A. Loeliger et al., The Factor Graph Approach to Model-Based Signal Processing, Proc. of the IEEE, 95-6, 2007.
(61) Sasha Korl, A Factor Graph Approach to Signal Modelling, System Identification and Filtering, Diss. ETH No. 16170, 2005. Justin Dauwels, On Variational Message Passing on Factor Graphs, ISIT conference, 2007.
(62) Although particular features have been shown and described, it will be understood that they are not intended to limit the claimed invention, and it will be made obvious to those skilled in the art that various changes and modifications may be made without departing from the scope of the claimed invention. The specification and drawings are, accordingly to be regarded in an illustrative rather than restrictive sense. The claimed invention is intended to cover all alternatives, modifications and equivalents.
LIST OF REFERENCES
(63) 2 hearing device
(64) 4 user
(65) 6 first input transducer
(66) 8 input signal
(67) 10 first processing unit
(68) 12 first sound signal model
(69) 14 acoustic output transducer
(70) 16 output signal
(71) 18 audio output signal
(72) 20 first object signal
(73) 22 recording unit
(74) 24 second processing unit
(75) 26 first set of parameter values
(76) 28 second sound signal model
(77) 30 first signal part corresponding at least partly to the first object signal 20
(78) 32 second signal part
(79) 34 second object signal
(80) 36 second set of parameter values
(81) 38 storage
(82) 40 library
(83) 42 respective set of parameter values
(84) 44 respective object signal
(85) 46 electronic device
(86) 48 second input transducer
(87) 52 first sound source
(88) 54 second sound source
(89) 56 respective sound source
(90) 58 system
(91) 601 step of recording a first object signal 20 by a recording unit 22;
(92) 602 step of determining, by a second processing unit 24, a first set of parameter values 26 of a second sound signal model 28 for the first object signal 20;
(93) 603 step of subsequently receiving, in a first processing unit 10 of the hearing device 2, an input signal 8 comprising a first signal part 30, corresponding at least partly to the first object signal 20, and a second signal part 32;
(94) 604 step of applying the determined first set of parameter values 26 of the second sound signal model 28 to the first sound signal model 12;
(95) 605 step of processing the input signal 8 according to the first sound signal model 12