METHOD AND APPARATUS FOR PROCESSING A STEREO SIGNAL

Abstract

The disclosure relates to a method for processing a stereo signal. The method can include obtaining a center channel signal by up-mixing the stereo signal. The method can also include generating a filtered center channel signal by applying one or more peak filters and one or more notch filters to the center channel signal. Furthermore, the method can include generating a binaural signal based on the filtered center channel signal.

Claims

1. A method for processing a stereo signal, the method comprising: obtaining a center channel signal by up-mixing the stereo signal; generating a filtered center channel signal by applying one or more peak filters and one or more notch filters to the center channel signal; and generating a binaural signal based on the filtered center channel signal.

2. The method of claim 1, wherein the method further comprises: obtaining a side channel signal by up-mixing the stereo signal; processing the side channel signal according to a first head related transfer function, to obtain a processed side channel signal; and processing the filtered center channel signal according to a second head related transfer function, to obtain a processed center channel signal; wherein the generating the binaural signal based on the filtered center channel signal comprises: generating the binaural signal based on the processed side channel signal and the processed center channel signal.

3. The method of claim 2, wherein the method further comprises: filtering the side channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated side signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated side signal and the decorrelated center signal.

4. The method of claim 1, wherein the method further comprises: obtaining a left channel signal and a right channel signal by up-mixing the stereo signal; processing the left channel signal and the right channel signal according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; and processing the filtered center channel signal according to a pair of head related transfer functions, to obtain a processed center channel signal; wherein the generating the binaural signal based on the filtered center channel signal comprises: generating a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, and generating a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

5. The method of claim 4, wherein the method further comprises: filtering the left channel signal, the right channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated n center signal.

6. An apparatus for processing a stereo signal, wherein the apparatus comprises processing circuitry configured to: obtain a center channel signal by up-mixing the stereo signal; obtain a filtered center channel signal by applying one or more peak filters and one or more notch filters to the center channel signal; and generating a binaural signal based on the filtered center channel signal.

7. The apparatus of claim 6, wherein the processing circuitry is further configured to obtain a side channel signal by up-mixing the stereo signal; process the side channel signal according to a first head related transfer function, to obtain a processed side channel signal; and process the filtered center channel signal according to a second head related transfer function, to obtain a processed center channel signal; wherein the binaural signal is generated based on the processed side channel signal and the processed center channel signal.

8. The apparatus of claim 7, wherein the processing circuitry is further configured to: filter the side channel signal and the center channel signal, to obtain a decorrelated side signal and a decorrelated center signal; and obtain a reflection signal based on the decorrelated side signal and the decorrelated center signal.

9. The apparatus of claim 6, wherein the processing circuitry is further configured to: obtain a left channel signal and a right channel signal by up-mixing the stereo signal; process the left channel signal and the right channel signal according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; and process the filtered center channel signal according to a pair of head related transfer functions, to obtain a processed center channel signal; and wherein a left signal of the binaural signal is generated based on the processed left channel signal and the processed center channel signal, and a right signal of the binaural signal is generated based on the processed right channel signal and the processed center channel signal.

10. The apparatus of claim 9, wherein the processing circuitry is further configured to: filter the left channel signal, the right channel signal and the center channel signal to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

11. The apparatus of claim 6, wherein the processing circuitry is configured to: obtain an initial audio signal, and decompose the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal.

12. The apparatus of claim 6, wherein the processing circuitry is configured to: obtain an initial audio signal, decompose the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal and an ambient signal; obtain a left channel signal and a right channel signal by up-mixing the stereo signal; add the ambient signal to the left channel signal, to obtain a left sum signal, add the ambient signal to the right channel signal, to obtain a right sum signal; process the left sum signal and the right sum signal according to two pairs of head related transfer functions to obtain a processed left channel signal and a processed right channel signal; process the filtered center channel signal according to a pair of head related transfer functions to obtain a processed center channel signal; generate a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal; and generate a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

13. The apparatus of claim 12, wherein the processing circuitry is further configured to: filter the left channel signal, the right channel signal and the center channel signal to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

14. The apparatus of claim 6, wherein the processing circuitry is further configured to: obtain a left channel signal and a right channel signal by up-mixing the stereo signal; convolve the stereo signal with a local reverberation to obtain a convolved stereo signal; add the convolved stereo signal with the left channel signal to obtain a left sum signal; add the convolved stereo signal with the right channel signal, to obtain a right sum signal; process the left sum signal and the right sum signal according to two pairs of head related transfer functions to obtain a processed left channel signal and a processed right channel signal; process the filtered center channel signal according to a pair of head related transfer functions to obtain a processed center channel signal; and generate a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, and generate a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

15. The apparatus of claim 14, wherein the processing circuitry is further configured to: filter the left channel signal, the right channel signal and the center channel signal to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

16. The apparatus of claim 6, wherein the processing circuitry is further configured to: obtain a left channel signal and a right channel signal by up-mixing the stereo signal: convolve the stereo signal with a local reverberation to obtain a convolved stereo signal; process the left channel signal and the right channel signal according to two pairs of head related transfer functions to obtain a processed left channel signal and a processed right channel signal; process the filtered center channel signal according to a pair of head related transfer functions, to obtain a processed center channel signal; generate a left signal of the binaural signal based on the processed left channel signal, the convolved stereo signal and the processed center channel signal; and generate a right signal of the binaural signal according to the processed right channel signal, the convolved stereo signal and the processed center channel signal.

17. The apparatus of claim 16, wherein the processing circuitry is further configured to: filter the left channel signal, the right channel signal and the center channel signal to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

18. The apparatus of claim 6, wherein the one or more peak filters comprises: a first peak filter centered at 4 kHz and having a ⅓-octave bandwidth; and a second peak filter centered at a frequency above 13 kHz and having a ¼-octave bandwidth; and wherein the one or more notch filters comprises: a notch filter centered at a frequency between 4 kHz and 8 kHz with 1-octave bandwidth.

19. The apparatus of claim 6, wherein the one or more peak filters comprise a first peak filter centered at 1 kHz and having a ⅓-octave bandwidth, and a second peak filter centered at a frequency between 10 kHz and 12 kHz and having a ¼-octave bandwidth, and wherein the one or more notch filters comprises: a first notch filter centered at 9 kHz and having a ¼-octave bandwidth, a second notch filter centered at 16 kHz and having a ¼-octave bandwidth.

20. A computer-readable storage medium storing program code which, when executed by a computer, causes the computer to carry out operations for processing a stereo signal, the operations comprising: obtaining a center channel signal by up-mixing the stereo signal; generating a filtered center channel signal by applying one or more peak filters and one or more notch filters to the center channel signal; and generating a binaural signal based on the filtered center channel signal.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0138] To illustrate the technical features of embodiments of the present disclosure more clearly, the accompanying drawings provided for describing the embodiments are introduced briefly in the following. The accompanying drawings in the following description are merely some embodiments of the present disclosure, but modifications on these embodiments are possible without departing from the scope of the present disclosure as defined in the claims.

[0139] FIG. 1 shows an example about a sound space is divided into three planes, the horizontal plane, the median plane and the frontal plane;

[0140] FIG. 2 shows a schematic diagram of a method of binaural rendering with externalization and localization enhancement method according to an embodiment;

[0141] FIG. 3 shows another schematic diagram of a method of binaural rendering with externalization and localization enhancement method according to an embodiment;

[0142] FIG. 4 shows a block diagram of a general method to simulate a virtual sound source according to an embodiment;

[0143] FIG. 5 shows another schematic diagram of a method of binaural rendering with externalization and localization enhancement method according to an embodiment;

[0144] FIG. 6 shows an example of magnitude spectra of peak notch filter for a frontal (left panel) and rear (right panel) sound source;

[0145] FIG. 7 shows an example of frontal and rear view direction in a rendering system;

[0146] FIG. 8 shows an example of the gain factor across different azimuth angles (θ) for the sound source located on the horizontal plane;

[0147] FIG. 9 shows a schematic diagram of a method to decorrelate the input audio signal according to an embodiment;

[0148] FIG. 10 shows a schematic diagram of a method to enhancement of externalization of a mono signal according to an embodiment;

[0149] FIG. 11 shows another schematic diagram of a method to enhancement of externalization of a mono signal according to an embodiment;

[0150] FIG. 12 shows another schematic diagram of a method to enhancement of externalization of a mono signal according to an embodiment;

[0151] FIG. 13 shows a schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0152] FIG. 14 shows another schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0153] FIG. 15 shows another schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0154] FIG. 16 shows another schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0155] FIG. 17 shows another schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0156] FIG. 18 shows another schematic diagram of a method to enhancement of externalization of a stereo signal according to an embodiment;

[0157] FIG. 19 shows a schematic diagram of a method for processing a stereo signal according to an embodiment;

[0158] FIG. 20 shows a schematic diagram illustrating an apparatus for processing a stereo signal according to an embodiment;

[0159] FIG. 21 shows a schematic diagram illustrating a device for processing a stereo signal according to an embodiment.

[0160] In the figures, identical reference signs will be used for identical or functionally equivalent features.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0161] In the following description, reference is made to the accompanying drawings, which form part of the disclosure, and in which are shown, by way of illustration, specific aspects in which the disclosure may be placed. It will be appreciated that the disclosure may be placed in other aspects and that structural or logical changes may be made without departing from the scope of the disclosure. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the disclosure is defined by the appended claims.

[0162] For instance, it will be appreciated that a disclosure in connection with a described method will generally also hold true for a corresponding device or system configured to perform the method and vice versa. For example, if a specific method step is described, a corresponding device may include a unit to perform the described method step, even if such unit is not explicitly described or illustrated in the figures.

[0163] Moreover, in the following detailed description as well as in the claims, embodiments with functional blocks or processing units are described, which are connected with each other or exchange signals. It will be appreciated that the disclosure also covers embodiments which include additional functional blocks or processing units, such as pre- or post-filtering and/or pre- or post-amplification units, that are arranged between the functional blocks or processing units of the embodiments described below.

[0164] Finally, it is understood that the features of the various exemplary aspects described herein may be combined with each other, unless specifically noted otherwise.

[0165] A channel is a pathway for passing on information, in this context sound information. Physically, it might, for example, be a tube you speak down, or a wire from a microphone to an earphone, or connections between electronic components inside an amplifier or a computer.

[0166] A track is a physical home for the contents of a channel when recorded on magnetic tape. There can be as many parallel tracks as technology allows, but for everyday purposes there are 1, 2 or 4. Two tracks can be used for two independent mono signals in one or both playing directions, or a stereo signal in one direction. Four tracks (such as a cassette recorder) are organized to work pairwise for a stereo signal in each direction; a mono signal is recorded on one track (same track as the left stereo channel) or on both simultaneously (depending on the tape recorder or on how the mono signal source is connected to the recorder).

[0167] A mono sound signal does not contain any directional information. In an example, there may be several loudspeakers along a railway platform and hundreds around an airport, but the signal remains mono. Directional information cannot be generated simply by sending a mono signal to two “stereo” channels. However, an illusion of direction can be conjured from a mono signal by panning it from channel to channel.

[0168] A stereo sound signal may contain synchronized directional information from the left and right aural fields. Consequently, it requires at least two channels, one for the left field and one for the right field. The left channel is fed by a mono microphone pointing at the left field and the right channel by a second mono microphone pointing at the right field (you will also find stereo microphones that have the two directional mono microphones built into one piece). In an example, Quadraphonic stereo uses four channels, surround stereo has at least additional channels for anterior and posterior directions apart from left and right. Public and home cinema stereo systems can have even more channels, dividing the sound fields into narrower sectors.

[0169] In an example, an audio signal processing arrangement includes a first filter for splitting off signal components from the left channel signal at least within one frequency band. Signal components are split off from the right channel signal by a second filter. The output signals of the filters are compared with the right channel signal and the left channel signal, respectively. The filter parameters of the filters are adjusted to values at which there is maximum correlation between the compared signals according to a given criterion. The center channel signal is derived in dependence on the filter adjustment. This can be effected by combining the output signals of the filters. In this manner, a center channel signal is obtained formed by the correlating left and right channel signal components, so that the stereo image is hardly disturbed by the addition of the center channel signal, whereas the perceived position of the virtual sources in the stereo image becomes less dependent on the listener's position with respect to the left and right loudspeakers.

[0170] It is important that the externalization and the localization accuracy can be enhanced by applying non-individual HRTFs/BRIRs for the binaural rendering system.

[0171] In an example, a sound space is divided into three specific planes: the horizontal plane, the median plane and the frontal plane, as shown in FIG. 1. The three planes are perpendicular to one another and intersect at the origin. This clockwise spherical coordinate system is also called head related coordinate system in some documents, in which the angle between the directional vector of the sound source and the horizontal plane is denoted by elevation angle φ with −90°≤φ≤90° and the angle between the horizontal projection of directional vector and the front is denoted by azimuth angle θ with −180°<θ≤180°. A sound source directly in front of the listening subject corresponds to 0° in Azimuth and Elevation.

[0172] There is another example to design some adjustment filters based on peak and notch filters to improve the sound localization in the median plane.

TABLE-US-00001 TABLE 1 Filter Type Center Frequency Band Width “Frontness” Peak  4 kHz 1/4 octave Notch 7.5 kHz    1 octave Peak 14 kHz 1/4 octave “Aboveness” Peak  4 kHz 1/4 octave Peak  8 kHz 1/4 octave “Behindness” Peak  4 kHz 1/4 octave Notch  9 kHz 1/4 octave Peak 11 kHz 1/4 octave Notch 16 kHz 1/4 octave

[0173] The positions of the peak and notch filters for frontal, above and rear sound sources are listed in Table 1. In this method, the design of peak and notch filters is based on the characteristic of HRTF itself and a little psychoacoustic experiments. Since some information of peaks and notches is already included in the HRTF, it is somehow like enlarge the spectral difference, which may introduce coloration problem. In addition, identical gain factors applied for different azimuth angles may introduce localization problem.

[0174] In another example, the input signals are divided into 5 sub-bands by a bandpass filter bank and configured to emphasize or deemphasize each band for maximum localization ability. However, this method requires fine-tuning the gains of all band-pass filters by the user which is not very practical. In addition, the bandwidth of the sub-bands is fixed, and there is no discussion about the choice of the bandwidth. Some psychoacoustic experiments indicated that the bandwidths of filters also play an important role in enhancement of sound source localization. Some methods tried to minimize the cone-of-confusion by spectral adjustments which simulate HRTF characteristics of subjects showing good performance in front-back localization (with large protrusion angle). One method is similar to emphasizing or deemphasizing the magnitude in some special frequencies. However, this method requires individual HRTF measurements, which is not practical. These methods may increase the peak or notch components of HRTF to enlarge the spectral difference of confusion direction. However, in these methods, larger spectral differences between rendered front and rear sound sources cannot guarantee better localization when only frontal or rear sound sources are rendered. These methods are only suitable on the horizontal plane. Also, loss of direction and bad sound quality may result.

[0175] In another example, a method is disclosed to enhance externalization of a mono audio signal. As shown in FIG. 2, a mono audio signal is first filtered by a pair of modeled HRTF, then the filtered signals are decorrelated to enhance the spaciousness of sound images. The image source method based reverberator is designed to simulate the reverberation. Finally, a pair of notch filters is designed based on averaged HRTFs at 0° from the center for image processing and integrated computing (CIPIC) database to enhance the sound localization. In this example, the decorrelator is applied to the direct part and thus the localization accuracy of a frontal sound source may be reduced (there is no separation between direct and early reflection in the processing). The notch filter is based on measured HRTFs and applied to binaural rendered signals. Any mismatch between the user's HRTF and the model used will cause bad quality.

[0176] In the case of a pair of virtual stereo signals (e.g., located at −30° and 30°), the generated phantom signal (0°) is difficult to be perceived as externalized. Some methods involving up-mixing stereo signals to center (i.e. center channel signal) and side signals are proposed. In these methods, the center and two side signals can be considered as three virtual sound sources. A method is disclosed to up-mix stereo signals to virtual surround sound to enhance the spaciousness of the rendered signals. However, the externalization and localization of rendered sound sources in the median plane are not enhanced. It is an object of one embodiment of the present disclosure to further enhance externalization based on an upmixed signal.

[0177] FIG. 19 shows a schematic diagram of a method for processing a stereo signal according to an embodiment. The method comprises:

[0178] S11: obtaining the stereo signal.

[0179] Stereophonic sound or, more commonly, stereo, is a method of sound reproduction that creates an illusion of multi-directional audible perspective. This is usually achieved by using two or more independent audio channels through a configuration of two or more loudspeakers (or stereo headphones) in such a way as to create the impression of sound heard from various directions, as in natural hearing.

[0180] A stereo signal may contain synchronized directional information from the left and right aural fields. Normally a stereo signal comprises at least two channels, one for the left field and one for the right field.

[0181] In an example, a stereo signal may be obtained by a receiver. For example, the receiver may obtain the stereo signal from another device or another system over a wired or wireless communication channel.

[0182] In another example, a stereo signal may be obtained according to a processor and at least two microphones. The at least two microphones are used to record information obtained from a sound source, and the processor is used to process information recorded by the microphones, to obtain the stereo signal.

[0183] In one embodiment, the obtaining the stereo signal comprises: obtaining an initial audio signal; and decomposing the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal.

[0184] S12: obtaining a center channel signal by up-mixing the stereo signal.

[0185] Up-mixing, in its most general sense, is the opposite of down-mixing. This means that up-mixing is a process that transforms a set of audio channels into a new set of audio channels which comprises more audio channels than the initial set. For example, up-mixing may transform 2 channels into 5.1 channels. Up-mixing is commonly used to better integrate legacy two-channel mono, stereo, or surround encoded content into 5.1 channel programs. Chosen properly, up-mixing further speeds the transition to 5.1 by helping out legacy content, and by assisting in the creation of new 5.1 channel material.

[0186] In an example, a strategy for up-mixing a stereo signal into a multi-channel signal is based on predicting or guessing the way in which the sound engineer would have proceeded if she or he were doing a multi-channel mix. For example, in the direct/ambient approach the ambience signals recorded at the back of the venue in the live recording could have been sent to the rear channels of the surround mix to achieve the immersion of the listener in the sound field. Or in the case of studio mix, a multi-channel reverberation unit could have been used to create this effect by assigning different reverberation levels to the front and rear channels. Also, the availability of a center channel could have helped the engineer to create a more stable frontal image for off-the-axis listening by panning the instruments among three channels instead of two. A series of techniques are disclosed for extracting and manipulating information in the stereo signals. Each signal in the stereo recording is analyzed by computing its Short-Time Fourier Transform (STFT) to obtain its time-frequency representation, and then comparing the two signals in this new domain using a variety of metrics. One or many mapping or transformation functions are then derived based on the particular metric and applied to modify the STFT's of the input signals.

[0187] In another example, in a stereo mix it is common that one featured vocalist or soloist is panned to the center. The intention of the sound engineer doing the mix is to create the auditory impression that the soloist is in the center of the stage. However, in a two-loudspeaker reproduction set up, the listener needs to be positioned exactly between the loudspeakers (e.g., the sweet spot) to perceive the intended auditory image. If the listener moves closer to one of the loudspeakers, the perception is destroyed by the precedence effect, and the image collapses towards the direction of the loudspeaker. For this reason (among others), a center channel containing the dialogue is used in movie theatres, so that the audience sitting towards either side of the room can still associate the dialogue with the image on the screen. In fact, most of the popular home multi-channel formats like 5.1 Surround now include a center channel to deal with this problem. If the sound engineer had had the option to use a center channel, he or she would have probably panned (or sent) the soloist or dialogue exclusively to this channel. Moreover, not only the center-panned signal collapses for off-axis listeners. Sources panned primarily toward on side (far from the listener) might appear to be panned toward the opposite side (closer to the listener). The sound engineer could have also avoided this by panning among the three channels, for example by panning between center and left-front channels all the sources with spatial locations on the left hemisphere, and panning between center and right-front channels all sources with locations toward the right.

[0188] S13: generating a filtered center channel signal.

[0189] A filtered center channel signal is generated by applying one or more peak filters and one or more notch filters to the center channel signal.

[0190] In one embodiment, the one or more peak filters and one or more notch filters, comprise: a notch filter centered at a frequency between 4 kHz and 8 kHz and having a 1-octave bandwidth, a first peak filter centered at 4 kHz and having a ⅓-octave bandwidth, and a second peak filter centered at a frequency above 13 kHz and having a ¼-octave bandwidth.

[0191] In an example, the typical center frequency for the notch filter is 7 kHz, and the typical center frequency for the second peak filter is 13 kHz.

[0192] In one embodiment, the one or more peak filters and one or more notch filters, comprises: a first notch filter centered at 9 kHz and having a ¼-octave bandwidth, a second notch filter centered at 16 kHz and having a ¼-octave bandwidth, a first peak filter centered at 1 kHz and having a ⅓-octave bandwidth, and a second peak filter centered at a frequency between 10 kHz and 12 kHz and having a ¼-octave bandwidth.

[0193] In an example, the typical center frequency for the second peak filter is 11 kHz.

[0194] In an example, the filtering process may be performed according to the following formula: [0195] Input signal: s(t) [0196] Peak and notch filter: p(t). [0197] This formula is a convolution in time domain, [0198] t denotes for time, τ is a variable which should is integrated from −∞ to ∞. dτ stands for an infinitesimal piece of the variable τ.

s′(t)=s(t)*p(t)=∫.sub.−∞.sup.∞p(t−τ)s(τ)dτ, [0199] * denotes convolution.
The input signal s(t) may be a mono signal or a center channel signal.

[0200] S14: generating a binaural signal based on the filtered center channel signal.

[0201] The method for processing a stereo signal improve the localization and externalization of stereo signal in the median plane.

[0202] In one embodiment, the method further comprises: obtaining a side channel signal by up-mixing the stereo signal; processing the side channel signal, according to a first head related transfer function, to obtain a processed side channel signal; processing the filtered center channel signal, according to a second head related transfer function, to obtain a processed center channel signal; and wherein the generating a binaural signal based on the filtered center channel signal comprises: generating the binaural signal based on the processed side channel signal and the processed center channel signal.

[0203] In one embodiment, a head related transfer function convolution is performed according to the formula:

d.sub.i(t)=s(t)*hrir.sub.i(t)=∫.sub.−∞.sup.∞hrir.sub.i(t−τ)s(τ)dτ,i∈{left,right}hrir.sub.i(t)=IFFT{HRTF.sub.i(f)} [0204] s(t) denotes a signal which is inputted to this process, * denotes convolution, s(t) is input signal, d.sub.i(t) is the output signal of this process. [0205] t denotes for time, τ is a variable, which should be integrated from −∞ to ∞. dτ stands for the smallest piece of the variable τ. IFFT is the backwards Fourier transformation. [0206] i∈{left,right} means, the symbol “i” can stand for the left or the right. For example, hrir.sub.i(t) means the hrir.sub.left(t) or hrir.sub.right(t).

[0207] In one embodiment, the method further comprises: obtaining a left channel signal and a right channel signal by up-mixing the stereo signal; processing the left channel signal and the right channel signal according to two pairs of head related transfer functions to obtain a processed left channel signal and a processed right channel signal; processing the filtered center channel signal according to a pair of head related transfer functions to obtain a processed center channel signal; and wherein the generating a binaural signal based on the filtered center channel signal comprises: generating a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, generating a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0208] In one embodiment, the method further comprises: filtering the side channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated side signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated side signal and the decorrelated center signal.

[0209] In an example, a decorrelated signal is generated in accordance with the following formula (which defines an example of a decorrelation filter):

[00001]$s\left(f_i,t\right)=IFFT\left\{FFT\left\{s\left(t\right)\right\}\times C\left(f_i,f\right)\right\}\},\mathrm{with}i=1,2,3,\mathrm{.Math.},24$$s_{\mathrm{left}}^{″}\left(t\right)=\underset{i=1}{\overset{24}{.Math.}}s\left(f_i,t\right)$$s_{\mathrm{right}}^{″}\left(t\right)=\underset{i=1}{\overset{24}{.Math.}}s\left(f_i,t-{\tau }_i\right)$

wherein τ.sub.i is randomized, f.sub.i is a center frequency, and the coefficients C(f.sub.i, f) represent a critical band filter bank. FFT means the Fourier transformation, transforming the signal from time domain to frequency domain. IFFT is the backwards Fourier transformation, transforming the signal from frequency domain to time domain. f means the frequency. f.sub.i is the center frequency. t is the time. Σ.sub.i=1.sup.24 s(f.sub.i, t) means the summation of s(f.sub.i,t), i.e., s(f.sub.1, t)+s (f.sub.2, t)+s (f.sub.3, t)+s(f.sub.4, t) . . . s(f.sub.24, t).

[0210] In audiology and psychoacoustics the concept of critical bands describes the frequency bandwidth of the “auditory filter” created by the cochlea, the sense organ of hearing within the inner ear.

[0211] In one embodiment, the method further comprises: filtering the left channel signal, the right channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0212] In one embodiment, the location of i.sup.th order image-sources along the x-, y- and z-coordinate {x.sub.i, y.sub.i, z.sub.i} can be expressed as:

[00002]$\left(\begin{array}{c}{x}_i\\ y_i\\ z_i\end{array}\right)=\left(\begin{array}{c}{\left(-1\right)}^i{x}_s+[i+\frac{1-{\left(-1\right)}^i}{2}{x}_r\\ {\left(-1\right)}^i{y}_s+[i+\frac{1-{\left(-1\right)}^i}{2}{y}_r\\ {\left(-1\right)}^i{z}_s+[i+\frac{1-{\left(-1\right)}^i}{2}{z}_r\end{array}\right)$

where {x.sub.s, y.sub.s, z.sub.5} and {x.sub.r, y.sub.r, z.sub.r} are the coordinate of the sound source and room, respectively.

[0213] The angle (θ.sub.i, φ.sub.i) between the each image source and the listener can be calculated as:

[00003]${\theta }_i=\arccos \frac{z_i-z_r}{\sqrt{{\left(x_i-x_r\right)}^2+{\left(y_i-y_r\right)}^2+{\left(z_i-z_r\right)}^2}}$${\phi }_i=\arccos \frac{y_i-y_r}{x_i-x_r}$

[0214] The attenuation of the early reflections is:

[00004]${\alpha }_i=\frac{1}{\sqrt{{\left(x_i-x_r\right)}^2+{\left(y_i-y_r\right)}^2+{\left(z_i-z_r\right)}^2}}$

[0215] The early reflection can be calculated as (N is the number of early reflections):

e.sub.left(t)=Σ.sub.i=1.sup.Nα.sub.is″.sub.left(t)*hrir.sub.left(t,θ.sub.i,φ.sub.i))

e.sub.right(t)=Σ.sub.i=1.sup.Nα.sub.is″.sub.right(t)*hrir.sub.right(t,θ.sub.i,φ.sub.i))

t is the time, θ.sub.i, φ.sub.i are azimuth and elevation angles, respectively. * denotes for convolution in time domain.

[0216] In one embodiment, the obtaining the stereo signal comprises: obtaining an initial audio signal; decomposing the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal and an ambient signal; wherein the method further comprises: obtaining a left channel signal and a right channel signal by up-mixing the stereo signal; adding the ambient signal with the left channel signal, to obtain a left sum signal; adding the ambient signal with the right channel signal, to obtain a right sum signal; processing the left sum signal and the right sum signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; processing the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and wherein the generating a binaural signal based on the filtered center channel signal comprises: generating a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, generating a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0217] In one embodiment, the method further comprises: filtering the left channel signal, the right channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0218] In one embodiment, the method further comprises: obtaining a left channel signal and a right channel signal by up-mixing the stereo signal; convolving the stereo signal with a local reverberation to obtain a convolved stereo signal; adding the convolved stereo signal with the left channel signal, to obtain a left sum signal; adding the convolved stereo signal with the right channel signal, to obtain a right sum signal; processing the left sum signal and the right sum signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; processing the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and wherein the generating a binaural signal based on the filtered center channel signal comprises: generating a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, generating a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0219] In one embodiment, the method further comprises: filtering the left channel signal, the right channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0220] In one embodiment, the method further comprises: obtaining a left channel signal and a right channel signal by up-mixing the stereo signal; convolving the stereo signal with a local reverberation to obtain a convolved stereo signal; processing the left channel signal and the right channel signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; processing the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and wherein the generating a binaural signal based on the filtered center channel signal comprises: generating a left signal of the binaural signal based on the processed left channel signal, the convolved stereo signal and the processed center channel signal, generating a right signal of the binaural signal based on the processed right channel signal, the convolved stereo signal and the processed center channel signal.

[0221] In one embodiment, the method further comprises: filtering the left channel signal, the right channel signal and the center channel signal, using one or more decorrelation filters, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and obtaining a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0222] In one embodiment, late reverberation e.g., calculated by convolution with late reverberation synthesized or recorded in the room (h.sub.late,left(t), h.sub.late,right(t)) is performed according to the following formula:

l.sub.left(t)=s(t)*h.sub.late,left(t)=∫.sub.−∞.sup.∞h.sub.late,left(t−τ)s(τ)dτ

l.sub.right(t)=s(t)*h.sub.late,right(t)=∫.sub.−∞.sup.∞h.sub.late,right(t−τ)s(τ)dτ

[0223] This is a convolution formula in time domain. t denotes for time. * denotes for convolution in time domain. t denotes for time, τ is a variable, which should be integrated from −∞ to ∞. dτ stands for the smallest piece of the variable τ. s(t) is the input signal in time domain.

[0224] In one embodiment, the binaural signals are the sum of direct sound, early reflections and late reverberation:

Left=d.sub.left(t)+e.sub.left(t)+l.sub.left(t)

Right=d.sub.right(t)+e.sub.right(t)+l.sub.right(t)

[0225] FIG. 20 shows a schematic diagram of an apparatus for processing a stereo signal according to an embodiment. The apparatus comprises: a stereo signal obtain unit configured to obtain the stereo signal; a up-mix unit configured to obtain a center channel signal by up-mixing the stereo signal; one or more peak filters and one or more notch filters configured to filter the center channel signal to obtain a filtered center channel signal; and a binaural signal generate unit (204) configured to generate a binaural signal based on the filtered center channel signal.

[0226] In one embodiment, the up-mix unit is further configured to obtain a side channel signal by up-mixing the stereo signal; the apparatus further comprises a head related transfer function, HRTF, unit, the HRTF unit is configured to process the side channel signal, according to a first head related transfer function, to obtain a processed side channel signal; the HRTF unit is further configured to process the filtered center channel signal, according to a second head related transfer function, to obtain a processed center channel signal; and the binaural signal generate unit is configured to generate the binaural signal based on the processed side channel signal and the processed center channel signal.

[0227] In one embodiment, the up-mix unit is further configured to obtain a left channel signal and a right channel signal by up-mixing the stereo signal; the apparatus further comprises a head related transfer function, HRTF, unit, the HRTF unit is configured to process the left channel signal and the right channel signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; the HRTF unit is further configured to process the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and the binaural signal generate unit is configured to generate a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, the binaural signal generate unit is configured to generate a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0228] In one embodiment, the apparatus further comprises: [0229] one or more decorrelation filters configured to filter the side channel signal and the center channel signal, to obtain a decorrelated side signal and a decorrelated center signal; and [0230] a reflection obtain unit configured to obtain a reflection signal based on the decorrelated side signal and the decorrelated center signal.

[0231] In one embodiment, the apparatus further comprises: [0232] one or more decorrelation filters configured to filter the left channel signal, the right channel signal and the center channel signal, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and [0233] a reflection obtain unit configured to obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0234] In one embodiment, the stereo signal obtain unit is configured to obtain an initial audio signal, and decompose the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal.

[0235] In one embodiment, the stereo signal obtain unit is configured to obtain an initial audio signal, decompose the initial audio signal, using one or any combination of the following methods: Ambient Phase Estimation, Principal Component Analysis or Least Squares Analysis, to obtain the stereo signal and an ambient signal; [0236] the up-mix unit is further configured to obtain a left channel signal and a right channel signal by up-mixing the stereo signal; [0237] the apparatus further comprises a head related transfer function, HRTF, unit, the HRTF unit is configured to add the ambient signal to the left channel signal, to obtain a left sum signal, [0238] add the ambient signal to the right channel signal, to obtain a right sum signal; [0239] the HRTF unit is further configured to process the left sum signal and the right sum signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal, and the HRTF unit is further configured to process the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and [0240] wherein the binaural signal generate unit is configured to generate a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, generate a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0241] In one embodiment, the apparatus further comprises: [0242] one or more decorrelation filters configured to filter the left channel signal, the right channel signal and the center channel signal, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and [0243] a reflection obtain unit configured to obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0244] In one embodiment, the up-mix unit is further configured to obtain a left channel signal and a right channel signal by up-mixing the stereo signal; [0245] the apparatus further comprises a convolve unit, the convolve unit is configured to convolve the stereo signal with a local reverberation to obtain a convolved stereo signal; [0246] the apparatus further comprises a head related transfer function, HRTF, unit, the HRTF unit is configured to add the convolved stereo signal with the left channel signal, to obtain a left sum signal, add the convolved stereo signal with the right channel signal, to obtain a right sum signal; [0247] the HRTF unit is further configured to process the left sum signal and the right sum signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal, and the HRTF unit is further configured to process the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and [0248] wherein the binaural signal generate unit is configured to generate a left signal of the binaural signal based on the processed left channel signal and the processed center channel signal, [0249] generate a right signal of the binaural signal based on the processed right channel signal and the processed center channel signal.

[0250] In one embodiment, the apparatus further comprises: [0251] one or more decorrelation filters configured to filter the left channel signal, the right channel signal and the center channel signal, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and [0252] a reflection obtain unit configured to obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0253] In one embodiment, the up-mix unit is further configured to obtain a left channel signal and a right channel signal by up-mixing the stereo signal; [0254] the apparatus further comprises a convolve unit, the convolve unit is configured to convolve the stereo signal with a local reverberation to obtain a convolved stereo signal; [0255] the apparatus further comprises a head related transfer function, HRTF, unit, the HRTF unit is configured to process the left channel signal and the right channel signal, according to two pairs of head related transfer functions, to obtain a processed left channel signal and a processed right channel signal; [0256] the HRTF unit is further configured to process the filtered center channel signal, according to a pair of head related transfer functions, to obtain a processed center channel signal; and [0257] wherein the binaural signal generate unit is configured to generate a left signal of the binaural signal based on the processed left channel signal, the convolved stereo signal and the processed center channel signal, generate a right signal of the binaural signal based on the processed right channel signal, the convolved stereo signal and the processed center channel signal.

[0258] In one embodiment, the apparatus further comprises: [0259] one or more decorrelation filters configured to filter the left channel signal, the right channel signal and the center channel signal, to obtain a decorrelated left signal, a decorrelated right signal and a decorrelated center signal; and [0260] a reflection obtain unit configured to obtain a reflection signal based on the decorrelated left signal, the decorrelated right signal and the decorrelated center signal.

[0261] In one embodiment, one or more peak filters and one or more notch filters, comprises: [0262] a notch filter centered at a frequency between 4 kHz and 8 kHz and having a 1-octave bandwidth, a first peak filterer centered at 4 kHz and having a ⅓-octave bandwidth, and a second peak filter centered at a frequency above 13 kHz and having a ¼-octave bandwidth.

[0263] In one embodiment, the one or more peak filters and one or more notch filters, comprises: [0264] a first notch filter centered at 9 kHz and having a ¼-octave bandwidth, a second notch filter centered at 16 kHz and having a ¼-octave bandwidth, a first peak filter centered at 1 kHz and having a ⅓-octave bandwidth, and a second peak filter centered at a frequency between 10 and 12 kHz and having a ¼-octave bandwidth.

[0265] The method according to the embodiments of the disclosure (for example, according to the embodiments disclosed in FIG. 19) can be performed by the apparatus 200 according to the embodiments of the disclosure. Further features of the method according to the embodiments of the disclosure result directly from the functionality of the apparatus 200 according to the embodiments of the disclosure and its different embodiments and/or implementation forms.

[0266] FIG. 21 shows a schematic diagram of a device 30 for processing a stereo signal according to an embodiment. The device 30 comprises a processor 31 and a computer-readable storage medium 32 storing program code. The program code comprises instructions for carrying out embodiments of the method for processing a stereo signal or one of its embodiments and/or implementations.

[0267] In an example, as shown in FIG. 2, externalization is enhanced and front-back confusion of binaurally rendered sound sources is reduced. In this embodiment, the input signals 21 may be a mono dry signal, a mono wet signal, stereo dry signals or stereo wet signals, or others. After the processing of the input signals by using methods disclosed herein (Binaural rendering with externalization & Localization enhancement method 22), a pair of binaural signals 23 for left and right ears is generated, which are then played pack over headphones.

[0268] In an example, a sound field can be divided into three parts: a direct part 221, an early reflection part 222 and a late reverberation part 223. The direct sound part 221 is essential for the sound source localization; the early reflection part 222 is still direction dependent, which provides spatial information, and is important for perception of externalization of sound sources. The late reverberation part 223 provides room information to listeners, and does not depend on the position of sound sources and listeners any more. These three parts should be simulated separately (see FIG. 3). To generate a virtual sound source in a free-field, there is no need to simulate the early reflections and the late reverberation. In contrast, the early reflections and the late reverberation are required to simulate a reverberant virtual sound source (with room information).

[0269] FIG. 4 shows the block diagram of a general method to simulate a virtual sound source. The direct sound part 221 is simulated by filtering the input signal through a pair of HRTFs. There are several methods to simulate the early reflection part 222, such as image source methods or ray tracing methods. Image source methods are commonly applied for real-time rendering of 3D audio. To simulate the early reflection part 222, some prerequisite, i.e., the positions of sound sources and listeners and the geometry of the room should be estimated or predefined. The late reverberation part 223 can be realized by using, for example, an artificial reverberator (e.g., based on a feedback delay network), or measured or synthesized late reverberation.

[0270] The embodiments of the present disclosure improve the externalization and reduce front-back confusion of binaurally rendered sound sources. Compared to the conventional method (for example, the method described with reference to FIG. 4), in the case of a mono sound source, the direct sound and the early reflections are additionally processed through peak and notch filters and decorrelation filters, respectively. In the case of stereo signals, the extracted phantom center signal is additionally filtered through peak and notch filters and together with the side signals to simulate the direct sound part. The early reflections are simulated by decorrelating the phantom center signal and the side signals and applying room geometric methods, e.g. image source method. Furthermore, for the augmented reality (AR) application, the ambient sound in the original signal is replaced by the reverberation in the current room.

[0271] In one embodiment, FIG. 5 shows a signal processing scheme according to an embodiment of the present disclosure in the case of a stereo signal scenario. The the input signal 51 is decomposed (in block 52, e.g., using an up-mix method) into a center signal 53 and one or more side signals 56. A peak and notch filter 54 is applied to the direct sound part (direct part 221) of the center signal 53 (i.e. the center channel signal). The peak and notch filter 54 may comprise (or be equivalent to) a filter chain comprising one or more peak filters and one or more notch filters. The decorrelation filters 57 are applied to the center signal 53 and to the one or more side signal(s) in order to simulate the early reflections (early reflection part 222) of the center signal 53 and the one or more side signals 56. The center signal 53 (after passing through by the peak and notch filter 54) and the one or more side signal 56 are each filtered with HRTFs 55 to generate a direct sound part 221. The early reflections are simulated by decorrelating 57 the center signal 53 and the side signals 56 and applying room geometric methods, e.g., an image source method 58. The late reverberation part 223 can be simulated using artificial reverberators, e.g. a feedback delay network, or using a measured or synthesized late reverberation part. The rendering process may be performed in mobile devices.

[0272] In an example, according to the psychoacoustic experiments, it can be observed that some special frequency components were correlated with the subjective impression on the sound source localization in the median plane. The experimental results may be summarized as: (1) Frontal localization is cued by a 1-octave notch having a lower cut-off frequency between 4 kHz and 8 kHz and increased energy above 13 kHz. (2) A sound source passing by a ¼-octave peak filter between 7 and 9 kHz is perceived as a sound located above. (3) A sound source filtered by a peak filter between 10 and 12 kHz is perceived as a sound located behind. The “directional band” indicated that 500 Hz and 4 kHz were related to the frontal localization, 1 kHz and 8 kHz were related to behind and above perception, respectively.

[0273] In an example, based on psychoacoustic experiments, a peak notch filter is designed to amplify the directional band information, thus to enhance the accuracy of sound source localization and reduce the front-back confusion for frontal and rear sound sources. The details of the peak and notch filter are: a notch filter centered at 7 kHz and having a 1-octave bandwidth, a peak filter centered at 4 kHz and having a ⅓-octave bandwidth and a peak filter centered at 14 kHz and having a ¼-octave bandwidth are designed for a frontal sound source; a peak filter centered at 1 kHz and having a ⅓-octave bandwidth, a notch filter centered at 9 kHz and having a ¼-octave bandwidth, a peak filter centered at 11 kHz and having a ¼-octave bandwidth and a notch filter centered at 16 kHz and having a ¼-octave bandwidth for a rear sound source. The audio quality and the localization performance both depend highly on the gain factors in the peak and notch filters. For example, +/−10 dB gain factors can be applied to achieve the trade-off between sound timbre coloration and the accuracy of sound localization. FIG. 6 shows an example of magnitude spectra of a peak notch filter designed for a frontal (left panel) and rear (right panel) sound source, respectively.

[0274] The peak and notch filters are only applied to the sound source in the frontal and rear regions, which is defined between, e.g., −20° and 20° in the horizontal and median plane around the frontal and rear view direction (see FIG. 7) in the rendering system. The frontal and rear regions are illustrated in FIG. 7.

[0275] In the case of a lateral sound source, the gain factor of the filters should be set to zero. To avoid the jump between frontal and lateral sound source, azimuth and elevation depending gain factors are considered. The gain factors G.sub.ff(θ, φ) and G.sub.rf(θ, φ) for the frontal and rear regions are expressed as:

[00005]$\begin{array}{cc}{g}_f\left(\theta ,\phi \right)=\{\begin{array}{cc}a.Math.\theta .Math.+b.Math.\phi .Math.+c.Math.\mathrm{\theta \phi }.Math.+d,& \mathrm{for}.Math.\theta .Math.\le {20}^{\circ },.Math.\phi .Math.\le {20}^{\circ }\\ 1,& \mathrm{otherwise}\end{array},g_r\left(\theta ,\phi \right)=\{\begin{array}{cc}\begin{array}{c}a({180}^{\circ }-.Math.\theta .Math.+b.Math.\phi .Math.+\\ c\left({180}^{\circ }-.Math.\theta .Math.\right).Math.\phi .Math.+d\end{array},& \mathrm{for}.Math.\theta .Math.\ge {160}^{\circ },.Math.\phi .Math.\le {20}^{\circ }\\ 1,& \mathrm{otherwise}\end{array},& \left(\mathrm{Eq}.1\right)\\ \mathrm{with}{G}_f\left(\theta ,\phi \right)=20{\log }_{10}\left(g_f\left(\theta ,\phi \right)\right),G_r\left(\theta ,\phi \right)=20{\log }_{10}\left(g_r\left(\theta ,\phi \right)\right),& \left(\mathrm{Eq}.2\right)\end{array}$

where θ and φ denote the azimuth and elevation angles, respectively. G.sub.f (θ, φ) and G.sub.r(θ, φ) represent the gain factors in the peak and notch filters for the frontal and rear sound sources, respectively. The parameters a, b, c and d are for example: −0.1081, −0.1081, 0.0054 and 3.1623, respectively. FIG. 8 shows an example of the gain factor across different azimuth angles (θ) for the sound source located on the horizontal plane (elevation angle φ=0°).

[0276] While the above mentioned peak and notch filter is considered for the frontal and rear sound sources to reduce front-back confusion, it should be noted that the peak and notch filter can also be designed for a virtual sound source located above the head to reduce up-down confusion.

[0277] The decorrelation filters, which simulate early reflections, have the effect of increasing the binaural reverberation cues, i.e. the fluctuations of Interaural-level difference (ILD) and the Interaural coherence (IC) between two ear signals in critical bands, and further to improve perceived externalization of 3D audio reproduction over headphones.

[0278] The input audio signal can be decorrelated by using a pair of static or dynamic FIR all-pass filters (see FIG. 9, left panel). One disadvantage of that method is, however, that a uniform magnitude spectrum cannot be guaranteed due to the phase variation in the filter. To avoid this problem, a decorrelation method based on a filter bank is disclosed. In this method, the input audio signal was divided into 24 critical bands by applying an equivalent rectangular band (ERB) filter bank. In each frequency band a random delay was applied (see FIG. 9, right panel). After that, the audio signals in each frequency bands are summed back together.

[0279] The pair of time varying decorrelation filters (random phase FIR filter or filter bank based decorrelation filters) is applied for the early reflections to improve the perceived externalization and spaciousness on the virtual sound source, especially for frontal and rear sound sources (based on our experiments).

Embodiment 1

[0280] Rendering of a Mono Dry Sound Source without Room Information.

[0281] FIG. 10 shows an embodiment of the enhancement of externalization of a mono dry signal without room information. A mono input signal 101 is filtered through a peak and notch filter 54 which depends on the azimuth and elevation angles of the sound source. The filtered signal is further filtered through a pair of HRTFs 55 of the desired azimuth and elevation angles to simulate a virtual sound source. For a dynamic binaural rendering system (binaural rendering coupled with head tracking devices), the HRTF and the gain factors of the peak and notch filter should be changed in real-time as a function of the relative position between the simulated virtual sound source and the listener's head.

Embodiment 2

[0282] Rendering of a Mono Dry Sound Source with Additional Room Information.

[0283] Embodiment 1 (FIG. 10) aims to simulate the virtual sound source in a free-field (without room information). FIG. 11 shows an example of a method of enhancing externalization of a mono dry signal with additional room information. The direct sound part 221 may be the same as in Embodiment 1, i.e. the input signal 101 is filtered through the peak and notch filter 54 and further filtered through a pair of HRTFs 55. To simulate the early reflections, some characteristics such as the positions of sound sources and listeners and the geometry of the room should be estimated or predefined. In this embodiment, the mono input signal 101 is first decorrelated by applying a pair of decorrelation filters 57. The decorrelated left and right signals are then used to generate the early reflection part 222, e.g., using an image source method 58. Late reverberation can be generated using a feedback delay network based artificial reverberator to measure or synthesize late reverberation. The direct sound 221, early reflections 222 and late reverberation 223 are summed up to yield left 231 and right 232 ear signals. The ears signals 231 and 232 can be rendered by headphones.

Embodiment 3

[0284] Rendering of a Mono Wet Sound Source with Local Room Information for the AR Application.

[0285] FIG. 12 shows an example of a method of enhancing externalization of a mono wet signal with additional local room information. This wet input signal 101 contains the original ambient sound 123 (e.g., the noise in an airport, strong reverberation in a church, etc.) which is not consistent with the acoustic of the local room (e.g., in a conference room, bedroom, etc.). Therefore, the mono wet input signal 101 received by the user is decomposed into primary and ambient sound using, e.g., an Ambient Phase Estimation (APE) method, Principal Component Analysis (PCA) or a least squares (LS) methods. The extracted primary sound is considered as a dry signal 122 and the ambient signal is discarded. The primary sound signal is filtered through the peak and notch filter 54 and further filtered through a pair of HRTFs 55 to simulate the direct part 221 of the virtual sound source. To simulate the early reflections, the primary sound is decorrelated by applying a pair of decorrelation filters 57, then the decorrelation left and right signals are processed by using e.g., image source method 58. The late reverberation can be generated using a feedback delay network based artificial reverberator, to measure or synthesize late reverberation 59. The room acoustic parameters (e.g., reverberation time and mixing time) used to simulate the late reverberation part 223 may be consistent with those in the local room. At last, the direct sound (direct part 221), early reflections (early reflection part 222) and late reverberation (late reverberation part 223) are summed up for left 231 and right 232 ear signals, and played back through headphones.

Embodiment 4

[0286] Rendering of Stereo Dry Sound Sources without Room Information.

[0287] FIG. 13 shows an example of a method of enhancing externalization of stereo dry signals without room information. The stereo dry signals 131 are up-mixed 132 to center (i.e. center channel) and side (left channel and right channel) signals. The center signal is filtered through the peak and notch filter 54, and further filtered by a pair of center HRTFs 55, e.g., HRTFs at 0°. The side (left and right) signals are filtered through two pairs of lateral HRTFs 133, e.g., HRTFs at +/−30° (position of the virtual loudspeakers).

Embodiment 5

[0288] Rendering of Stereo Dry Sound Sources with Additional Room Information.

[0289] FIG. 14 shows an example of a method of enhancing externalization of stereo dry signals with additional room information. The stereo dry signals 131 are up-mixed 132 to center and side (left and right) signals. The center signal is filtered through the peak and notch filter 54, and further filtered by a pair of center HRTFs 55, e.g. HRTFs at 0°. The side (left and right) signals are filtered through two pairs of lateral HRTFs 133, e.g., HRTFs at +/−30° (position of the virtual loudspeakers). The signals in these three channels are filtered through decorrelation filters 57, and further processed to simulate early reflections using e.g., the image source method 58. For that, a simple room model is needed, e.g., width, length, height of the room, the position of the listener and sound source. The late reverberation can be generated using a feedback delay network based artificial reverberator, to measure or synthesize late reverberation 59. In FIG. 14, the input stereo signals are directly used to generate the late reverberation. It is also possible to use the upmixed signals (center and side signals) to create the late reverberation.

Embodiment 6

[0290] Rendering of Stereo Wet Sound Sources without Room Information.

[0291] FIG. 15 shows an example of a method of enhancing externalization of stereo wet signals without room information. The primary and ambient signals from stereo wet signals 151 are extracted 152 using, e.g., an APE method, PCA, or a LS method, etc. The extracted primary sounds are considered as dry signals. Then the primary sounds are up-mixed 132 to the left, right and center signals. The center signal is filtered through the peak and notch filter 54, and is further filtered by a pair of center HRTF 55, e.g. HRTFs at 0°, resulting in left-ear center signal and a right-ear center signal. The side (left and right) signals and the ambient sound are summed up and filtered through two pairs of lateral HRTFs 133, e.g., HRTFs at +/−30° (position of the virtual loudspeakers), resulting in a left-ear “side plus ambient” signal and a right-ear “side plus ambient” signal. The left-ear center signal and the left-ear “side plus ambient” signal are summed up to produce a left-ear signal 231. Similarly, the right-ear center signal and the right-ear “side plus ambient” signal are summed up to produce a right-ear signal 232. Finally, the left-ear signal 231 and the right-ear signal 232 may be played back over headphones.

Embodiment 7

[0292] Rendering of Stereo Wet Sound Sources with Additional Room Information.

[0293] FIG. 16 shows an example of a method of enhancing externalization of stereo wet signals with additional room information. A pair of stereo signals 151 is first decomposed 152 into primary and ambient parts. The primary part (primary sound) is up-mixed 132 to center, side (left and right) channel signals. The center channel signal is filtered through the peak and notch filter 54 and further filtered through a pair of center HRTFs 55, e.g., HRTFs at 0°. The ambient sound, and side channel signals for the left and right ears are summed up and further filtered through two pairs of side HRTF 133, e.g., HRTFs at +/−30°. The three up-mixed signals (left, right and center signals) are decorrelated 57 for left and right ears, and further processed to simulate early reflections using the image source method 58. Furthermore, artificial reverberator, measured or synthesized late reverberation 59 is used to simulate the late reverberation part 223 for these three (left, right and center) virtual sound sources. Similar to FIG. 14, the extracted dry stereo signals are directly used to create the late reverberation in FIG. 16. It is also possible to use the upmixed signals (center and side signals) to create the late reverberation. Finally, the left and right ear signals are summed up and played back over headphones.

Embodiment 8

[0294] Rendering of Stereo Wet Sound Sources with Local Room Information for AR Application.

[0295] FIG. 17 shows an example of a method of enhancing externalization of stereo wet signals with room information for AR application. In this embodiment, the ambient sound is replaced with the local reverberation. A pair of stereo signals 151 is first decomposed 152 into primary and ambient parts. The extracted ambient sound is discard. Only the primary sounds (dry stereo signals) are further processed to virtualization. The primary part is up-mixed 132 to center, side (left and right) channel signals. The center channel signal is filtered through the peak and notch filter 54 to reduce the front-back confusion, and further filtered through a pair of center HRTFs 55, e.g., HRTFs at 0°. The primary sounds are convolved with measured or synthesized local late reverberation 171 and added to the side signals. These signals are further filtered through two pairs of side HRTF 133, e.g., HRTFs at +/−30° to create direct and late reverberation part. The three up-mixed signals (left, right and center signals) are decorrelated 57 for left and right ears, and further processed to simulate early reflections using the image source method 58. The resulting left-ear signal contributions are summed up to produce a left-ear signal 231. Similarly, the resulting right-ear signal contributions are summed up to produce a right-ear signal 232. Finally, the left-ear signal 231 and the right-ear signal 232 may be played back over headphones.

[0296] Instead of adding the synthesized reverberation part into the side signals, another alternative is to directly add the simulated reverberation part into the left and right ear signals, as shown in FIG. 18.

[0297] Applications of embodiments of the disclosure include any sound reproduction system or surround sound system using multiple loudspeakers.

[0298] In particular, embodiments of the presented disclosure can be applied to [0299] TV speaker systems, [0300] car entertaining systems, [0301] teleconference systems, and/or [0302] home cinema system,
where personal listening environments for one or multiple listeners is desirable.

[0303] The foregoing descriptions are only implementation manners and embodiments of the present disclosure, the protection of the scope of the present disclosure is not limited to this. Any variations or replacements can be easily made by a person skilled in the art. The scope of protection of the present application is defined by the attached claims.