AUDIO SCENE ENCODER, AUDIO SCENE DECODER AND RELATED METHODS USING HYBRID ENCODER-DECODER SPATIAL ANALYSIS

Abstract

An audio scene encoder for encoding an audio scene, the audio scene having at least two component signals, has: a core encoder for core encoding the at least two component signals, wherein the core encoder is configured to generate a first encoded representation for a first portion of the at least two component signals, and to generate a second encoded representation for a second portion of the at least two component signals, a spatial analyzer for analyzing the audio scene to derive one or more spatial parameters or one or more spatial parameter sets for the second portion; and an output interface for forming the encoded audio scene signal, the encoded audio scene signal having the first encoded representation, the second encoded representation, and the one or more spatial parameters or one or more spatial parameter sets for the second portion.

Claims

1. Audio scene encoder for encoding an audio scene, the audio scene comprising at least two component signals, the audio scene encoder comprising: a core encoder for core encoding the at least two component signals, wherein the core encoder is configured to generate a first encoded representation for a first portion of the at least two component signals, and to generate a second encoded representation for a second portion of the at least two component signals; a spatial analyzer for analyzing the audio scene comprising the at least two component signals to derive one or more spatial parameters or one or more spatial parameter sets for the second portion; and an output interface for forming an encoded audio scene signal, the encoded audio scene signal comprising the first encoded representation for the first portion, the second encoded representation for the second portion, and the one or more spatial parameters or one or more spatial parameter sets for the second portion.

2. Audio scene encoder of claim 1, wherein the core encoder is configured to generate the first encoded representation with a first frequency resolution and to generate the second encoded representation with a second frequency resolution, the second frequency resolution being lower than the first frequency resolution from subsequent time frames from the at least two component signals, wherein a first time frame of the at least two component signals is the first portion and a second time frame of the at least two component signals is the second portion, or wherein a border frequency between the first frequency subband of the time frame and the second frequency subband of the time frame coincides with a border between a scale factor band and an adjacent scale factor band or does not coincide with a border between the scale factor band and the adjacent scale factor band, wherein the scale factor band and the adjacent scale factor band are used by the core encoder, wherein the first frequency subband is the first portion and the second frequency subband is the second portion.

3. Audio scene encoder of claim 1, wherein the audio scene comprises, as a first component signal, an omnidirectional audio signal, and, as a second component signal, at least one directional audio signal, or wherein the audio scene comprises, as a first component signal, a signal captured by an omnidirectional microphone positioned at a first position, and, as a second component signal, at least one signal captured by an omnidirectional microphone positioned at a second position different from the first position, or wherein the audio scene comprises, as a first component signal, at least one signal captured by a directional microphone directed to a first direction, and, as a second component signal, at least one signal captured by a directional microphone directed to a second direction, the second direction being different from the first direction.

4. Audio scene encoder of claim 1, wherein the audio scene comprises A-format component signals, B-format component signals, First-Order Ambisonics component signals, Higher-Order Ambisonics component signals, or component signals captured by a microphone array with at least two microphone capsules or as determined by a virtual microphone calculation from an earlier recorded or synthesized sound scene.

5. Audio scene encoder of claim 1, wherein the output interface is configured to not include any spatial parameters from the same parameter kind as the one or more spatial parameters generated by the spatial analyzer for the second portion into the encoded audio scene signal, so that only the second portion has the parameter kind, and any parameters of the parameter kind are not included for the first portion in the encoded audio scene signal.

6. Audio scene encoder of claim 1, wherein the core encoder is configured to perform a parametric encoding operation for the second portion, and to perform a wave form preserving encoding operation for the first portion, or wherein a start band for the second portion is lower than a bandwidth extension start band, and wherein a core noise filling operation performed by the core encoder does not have any fixed crossover band and is gradually used for more parts of core spectra as a frequency increases.

7. Audio scene encoder of claim 1, wherein the core encoder is configured to perform a parametric processing for the second frequency subband of the time frame, the parametric processing comprising calculating an amplitude-related parameter for the second frequency subband and quantizing and entropy-coding the amplitude-related parameter instead of individual spectral lines in the second frequency subband, and wherein the core encoder is configured to quantize and entropy-encode individual spectral lines in the first subband of the time frame, or wherein the core encoder is configured to perform a parametric processing for a high frequency subband of the time frame corresponding to the second frequency subband of the at least two component signals, the parametric processing comprising calculating an amplitude-related parameter for the high frequency subband and quantizing and entropy-coding the amplitude-related parameter instead of a time domain signal in the high frequency subband, and wherein the core encoder is configured to quantize and entropy-encode the time domain audio signal in a low frequency subband of the time frame corresponding to the first portion of the at least two component signals, by a time domain coding operation such as LPC coding, LPC/TCX coding, or EVS coding or AMR Wideband coding or AMR Wideband+ coding.

8. Audio scene encoder of claim 7, wherein the parametric processing comprises a spectral band replication processing, and intelligent gap filling processing, or a noise filling processing.

9. Audio scene encoder of claim 1, wherein the first portion is a first subband of a time frame and the second portion is a second subband of the time frame, and wherein the core encoder is configured to use a predetermined border frequency between the first subband and the second subband, or wherein the core encoder comprises a dimension reducer for reducing a dimension of the audio scene to obtain a lower dimension audio scene, wherein the core encoder is configured to calculate the first encoded representation for the first portion of the at least two component signals from the lower dimension audio scene, and wherein the spatial analyzer is configured to derive the spatial parameters from the audio scene having a dimension being higher than the dimension of the lower dimension audio scene, or wherein the core encoder if configured to generate the first encoded representation for the first portion comprising M component signals, and to generate the second encoded representation for the second portion comprising N component signals, and wherein M is greater than N and N is greater than or equal to 1.

10. Audio scene encoder of claim 1, being configured to operate at different bitrates, wherein a predetermined border frequency between the first portion being a first frequency subband and the second portion being a frequency subband depends on a selected bitrate, and wherein the predetermined border frequency is lower for a lower bitrate, or wherein the predetermined border frequency is greater for a greater bitrate.

11. Audio scene encoder of claim 1, wherein the first portion is a first subband of the at least two component signals, and wherein the second portion is a second subband of the at least two component signals, and wherein the spatial analyzer is configured to calculate, for the second subband, as the one or more spatial parameters, at least one of a direction parameter and a non-directional parameter such as a diffuseness parameter.

12. Audio scene encoder of claim 1, wherein the core encoder comprises: a time-frequency converter for converting sequences of time frames comprising the time frame of the at least two component signals into sequences of spectral frames for the at least two component signals, a spectral encoder for quantizing and entropy-coding spectral values of a frame of the sequences of spectral frames within a first subband of the spectral frame corresponding to a first frequency subband; and a parametric encoder for parametrically encoding spectral values of the spectral frame within a second subband of the spectral frame corresponding to a second frequency subband, or wherein the core encoder comprises a time domain or mixed time domain frequency domain core encoder for performing a time domain or mixed time domain and frequency domain encoding operation of a lowband portion of the time frame, the lowband portion corresponding to the first portion, or wherein the spatial analyzer is configured to subdivide the second portion being a second frequency subband into analysis bands, wherein a bandwidth of an analysis band is greater than or equal to a bandwidth associated with two adjacent spectral values processed by the spectral encoder within the first portion being a first frequency subband, or is lower than a bandwidth of a lowband portion representing the first frequency subband, and wherein the spatial analyzer is configured to calculate at least one of a direction parameter and a diffuseness parameter for each analysis band of the second frequency subband, or wherein the core encoder and the spatial analyzer are configured to use a common filterbank or different filterbanks having different characteristics.

13. Audio scene encoder of claim 12, wherein the spatial analyzer is configured to use, for calculating the direction parameter, an analysis band being smaller than an analysis band used to calculate the diffuseness parameter.

14. Audio scene encoder of claim 1, wherein the core encoder comprises a multi-channel encoder for generating an encoded multi-channel signal for the at least two component signals, or wherein the core encoder comprises a multi-channel encoder for generating two or more encoded multi-channel signals, when a number of component signals of the at least two component signals is three or more, or wherein the core encoder is configured to generate the first encoded representation with a first resolution and to generate the second encoded representation with a second resolution, wherein the second resolution is lower than the first resolution, or wherein the core encoder is configured to generate the first encoded representation with a first time or first frequency resolution and to generate the second encoded representation with a second time or second frequency resolution, the second time or frequency resolution being lower than the first time or frequency resolution, or wherein the output interface is configured for not including any spatial parameters for the first portion or frequency subband into the encoded audio scene signal, or for including a smaller number of spatial parameters for the first frequency subband into the encoded audio scene signal compared to a number of the spatial parameters for the second frequency subband.

15. Method of encoding an audio scene, the audio scene comprising at least two component signals, the method comprising: core encoding the at least two component signals, wherein the core encoding comprises generating a first encoded representation for a first portion of the at least two component signals, and generating a second encoded representation for a second portion of the at least two component signals; analyzing the audio scene comprising the at least two component signals to derive one or more spatial parameters or one or more spatial parameter sets for the second portion; and forming the encoded audio scene signal, the encoded audio scene signal comprising the first encoded representation, the second encoded representation, and the one or more spatial parameters or the one or more spatial parameter sets for the second portion.

16. Non-transitory storage medium, having stored thereon a computer program for performing, when running on a computer or a processor, the method of claim 15.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Embodiments of the present invention are subsequently described with respect to the accompanying drawings, in which:

[0046] FIG. 1A is a block diagram of an embodiment of an audio scene encoder;

[0047] FIG. 1B is a block diagram of an embodiment of an audio scene decoder;

[0048] FIG. 2A is a DirAC analysis from an uncoded signal;

[0049] FIG. 2B is a DirAC analysis from a coded lower-dimensional signal;

[0050] FIG. 3 is a system overview of an encoder and a decoder combining DirAC spatial sound processing with an audio coder;

[0051] FIG. 4A is a DirAC analysis from an uncoded signal;

[0052] FIG. 4B is a DirAC analysis from an uncoded signal using grouping of parameters in the time-frequency domain and quantization of the parameters

[0053] FIG. 5A is a known DirAC analysis stage;

[0054] FIG. 5B is a known DirAC synthesis stage;

[0055] FIG. 6A illustrates different overlapping time frames as an example for different portions;

[0056] FIG. 6B illustrates different frequency bands as an example for different portions;

[0057] FIG. 7A illustrates a further embodiment of an audio scene encoder;

[0058] FIG. 7B illustrates an embodiment of an audio scene decoder;

[0059] FIG. 8A illustrates a further embodiment of an audio scene encoder;

[0060] FIG. 8B illustrates a further embodiment of an audio scene decoder;

[0061] FIG. 9A illustrates a further embodiment of an audio scene encoder with a frequency domain core encoder;

[0062] FIG. 9B illustrates a further embodiment of an audio scene encoder with a time domain core encoder;

[0063] FIG. 10A illustrates a further embodiment of an audio scene decoder with a frequency domain core decoder;

[0064] FIG. 10B illustrates a further embodiment of a time domain core decoder; and

[0065] FIG. 11 illustrates an embodiment of a spatial renderer.

DETAILED DESCRIPTION OF THE INVENTION

[0066] FIG. 1A illustrates an audio scene encoder for encoding an audio scene 110 that comprises at least two component signals. The audio scene encoder comprises a core encoder 100 for core encoding the at least two component signals. Specifically, the core encoder 100 is configured to generate a first encoded representation 310 for a first portion of the at least two component signals and to generate a second encoded representation 320 for a second portion of the at least two component signals. The audio scene encoder comprises a spatial analyzer for analyzing the audio scene to derive one or more spatial parameters or one or more spatial parameter sets for the second portion. The audio scene encoder comprises an output interface 300 for forming an encoded audio scene signal 340. The encoded audio scene signal 340 comprises the first encoded representation 310 representing the first portion of the at least two component signals, the second encoder representation 320 and parameters 330 for the second portion. The spatial analyzer 200 is configured to apply the spatial analysis for the first portion of the at least two component signals using the original audio scene 110. Alternatively, the spatial analysis can also be performed based on a reduced dimension representation of the audio scene. If, for example, the audio scene 110 comprises, for example, a recording of several microphones arranged in a microphone array, then the spatial analysis 200 can, of course, be performed based on this data. However, the core encoder 100 would then be configured to reduce the dimensionality of the audio scene to, for example, a first order Ambisonics representation or a higher order Ambisonics representation. In a basic version, the core encoder 100 would reduce the dimensionality to at least two components consisting of, for example, an omnidirectional component and at least one directional component such as X, Y, or Z of a B-format representation. However, other representations such as higher order representations or an A-format representations are useful as well. The first encoder representation for the first portion would then consist of at least two different components being decodable and will typically, consist of an encoded audio signal for each component.

[0067] The second encoder representation for the second portion can consist of the same number of components or can, alternatively, have a lower number such as only a single omnidirectional component that has been encoded by the core coder in a second portion. In case of the implementation where the core encoder 100 reduces the dimensionality of the original audio scene 110, the reduced dimensionality audio scene optionally can be forwarded to the spatial analyzer via line 120 instead of the original audio scene.

[0068] FIG. 1B illustrates an audio scene decoder comprising an input interface 400 for receiving an encoded audio scene signal 340. This encoded audio scene signal comprises the first encoded representation 410, the second encoded representation 420 and one or more spatial parameters for the second portion of the at least two component signals illustrated at 430. The encoded representation of the second portion can, once again, be an encoded single audio channel or can comprise two or more encoded audio channels, while the first encoded representation of the first portion comprises at least two different encoded audio signals. The different encoded audio signals in the first encoded representation or, if available, in the second encoded representation can be jointly coded signals such as a jointly coded stereo signal or are, alternatively, and even advantageously, individually encoded mono audio signals.

[0069] The encoded representation comprising the first encoded representation 410 for the first portion and the second encoded representation 420 for the second portion is input into a core decoder for decoding the first encoded representation and the second encoded representation to obtain a decoded representation of the at least two component signals representing an audio scene. The decoded representation comprises a first decoded representation for the first portion indicated at 810 and a second decoded representation for a second portion indicated at 820. The first decoded representation is forwarded to a spatial analyzer 600 for analyzing a portion of the decoded representation corresponding to the first portion of the at least two component signals to obtain one or more spatial parameters 840 for the first portion of the at least two component signals. The audio scene decoder also comprises a spatial rendered 800 for spatially rendering the decoded representation which comprises, in the FIG. 1B embodiment, the first decoded representation for the first portion 810 and the second decoded representation for the second portion 820. The spatial renderer 800 is configured to use, for the purpose of audio rendering, the parameters 840 derived from the spatial analyzer for the first portion and, for the second portion, parameters 830 that are derived from the encoded parameters via a parameter/metadata decoder 700. In case of a representation of the parameters in the encoded signal in a non-encoded form, the parameter/metadata decoder 700 is not necessary and the one or more spatial parameters for the second portion of the at least two component signals are directly forwarded from the input interface 400, subsequent to a demultiplex or a certain processing operation, to the spatial renderer 800 as data 830.

[0070] FIG. 6A illustrates a schematic representation of different typically overlapping time frames F.sub.1 to F.sub.4. The core encoder 100 of FIG. 1A can be configured to form such subsequent time frames from the at least two component signals. In such a situation, a first time frame could be the first portion and the second time frame could be the second portion. Thus, in accordance with an embodiment of the invention, the first portion could be the first time frame and the second portion could be another time frame, and switching between the first and the second portion could be performed over time. Although FIG. 6A illustrates overlapping time frames, non-overlapping time frames are useful as well. Although FIG. 6A illustrates time frames having equal lengths, the switching could be done with time frames that have different lengths. Thus, when the time frame F.sub.2 is, for example, smaller than the time frame F.sub.1, then this would result in an increased time resolution for the second time frame F.sub.2 with respect to the first time frame F.sub.1. Then, the second time frame F.sub.2 with the increased resolution would advantageously correspond to the first portion that is encoded with respect to its components, while the first time portion, i.e., the low resolution data would correspond to the second portion that is encoded with a lower resolution but the spatial parameters for the second portion would be calculated with any resolution entailed, since the whole audio scene is available at the encoder.

[0071] FIG. 6B illustrates an alternative implementation where the spectrum of the at least two component signals is illustrated as having a certain number of bands B1, B2, . . . , B6, . . . . Advantageously, the bands are separated in bands with different bandwidths that increase from lowest to highest center frequencies in order to have a perceptually motivated band division of the spectrum. The first portion of the at least two component signals could, for example, consist of the first four bands, for example, the second portion could consist of bands B5 and bands B6. This would match with a situation, where the core encoder performs a spectral band replication and where the crossover frequency between the non-parametrically encoded low frequency portion and the parametrically encoded high frequency portion would be the border between the band B4 and the band B5.

[0072] Alternatively, in case of intelligent gap filling (IGF) or noise filling (NF), the bands are arbitrarily selected in line with a signal analysis and, therefore, the first portion could, for example, consist of bands B1, B2, B4, B6 and the second portion could be B3, B5 and probably another higher frequency band. Thus, a very flexible separation of the audio signal into bands can be performed, irrespective of whether the bands are, as is of advantage and illustrated in FIG. 6B, typical scale factor bands that have an increasing bandwidth from lowest to highest frequencies, or whether the bands are equally sized bands. The borders between the first portion and the second portion do not necessarily have to coincide with scale factor bands that are typically used by a core encoder, but it is of advantage to have the coincidence between a border between the first portion and the second portion and a border between a scale factor band and an adjacent scale factor band.

[0073] FIG. 7A illustrates an implementation of an audio scene encoder. Particularly, the audio scene is input into a signal separator 140 that may be the portion of the core encoder 100 of FIG. 1A. The core encoder 100 of FIG. 1A comprises a dimension reducer 150a and 150b for both portions, i.e., the first portion of the audio scene and the second portion of the audio scene. At the output of the dimension reducer 150a, there does exist at least two component signals that are then encoded in an audio encoder 160a for the first portion. The dimension reducer 150b for the second portion of the audio scene can comprise the same constellation as the dimension reducer 150a. Alternatively, however, the reduced dimension obtained by the dimension reducer 150b can be a single transport channel that is then encoded by the audio encoder 160b in order to obtain the second encoded representation 320 of at least one transport/component signal.

[0074] The audio encoder 160a for the first encoded representation can comprise a wave form preserving or non-parametric or high time or high frequency resolution encoder while the audio encoder 160b can be a parametric encoder such as an SBR encoder, an IGF encoder, a noise filling encoder, or any low time or frequency resolution or so. Thus, the audio encoder 160b will typically result in a lower quality output representation compared to the audio encoder 160a. This “disadvantage” is addressed by performing a spatial analysis via the spatial data analyzer 210 of the original audio scene or, alternatively, a dimension reduced audio scene when the dimension reduced audio scene still comprises at least two component signals. The spatial data obtained by the spatial data analyzer 210 are then forwarded to a metadata encoder 220 that outputs an encoded low resolution spatial data. Both blocks 210, 220 may be included in the spatial analyzer block 200 of FIG. 1A.

[0075] Advantageously, the spatial data analyzer performs a spatial data analysis with a high resolution such as a high frequency resolution or a high time resolution and, the, in order to keep the used bitrate for the encoded metadata in a reasonable range, the high resolution spatial data may be grouped and entropy encoded by the metadata encoder in order to have an encoded low resolution spatial data. When, for example, a spatial data analysis is performed for, for example, eight time slots per frame and ten bands per time slot, one could group the spatial data into a single spatial parameter per frame and, for example, five bands per parameter.

[0076] It is of advantage to calculate directional data on the one hand and diffuseness data on the other hand. The metadata encoder 220 could then be configured to output the encoded data with different time/frequency resolutions for the directional and diffuseness data. Typically, directional data is used with a higher resolution than diffuseness data. As advantageous way in order to calculate the parametric data with different resolutions is to perform the spatial analysis with a high resolution for and typically an equal resolution for both parametric kinds and to then perform a grouping in time and/or frequency with the different parametric information for the different parameter kinds in different ways in order to then have an encoded low resolution spatial data output 330 that has, for example, a medium resolution with time and/or frequency for the directional data and a low resolution for the diffuseness data.

[0077] FIG. 7B illustrates a corresponding decoder-side implementation of the audio scene decoder.

[0078] The core decoder 500 of FIG. 1B comprises, in the FIG. 7B embodiment, a first audio decoder instance 510a and a second audio decoder instance 510b. Advantageously, the first audio decoder instance 510a is a non-parametric or wave form preserving or high resolution (in time and/or frequency) encoder that generates, at the output, a decoded first portion of the at least two component signals. This data 810 is, on the one hand, forwarded to the spatial renderer 800 of FIG. 1B and is, additionally input into a spatial analyzer 600. Advantageously, the spatial analyzer 600 is a high resolution spatial analyzer that may calculate high resolution spatial parameters for the first portion. Typically, the resolution of the spatial parameters for the first portion is higher than the resolution that is associated with the encoded parameters that are input into the parameter/metadata decoder 700. However, the entropy decoded low time or frequency resolution spatial parameters output by block 700 are input into a parameter de-grouper for resolution enhancement 710. Such a parameter de-grouping can be performed by copying a transmitted parameter to certain time/frequency tiles, where the de-grouping is performed in line with the corresponding grouping performed in the encoder-side metadata encoder 220 of FIG. 7A. Naturally, together with de-grouping, further processing or smoothing operations can be performed as needed.

[0079] The result of block 710 is then a collection of decoded advantageously high resolution parameters for the second portion that typically have the same resolution than the parameters 840 for the first portion. Also, the encoded representation of the second portion is decoded by the audio decoder 510b to obtain the decoded second portion 820 of typically at least one or of a signal having at least two components.

[0080] FIG. 8A illustrates an implementation of an encoder relying on the functionalities discussed with respect to FIG. 3. Particularly, multi-channel input data or first order Ambisonics or high order Ambisonics input data or object data is input into a B-format converter that converts and combines individual input data in order to generate, for example, typically four B-format components such as an omnidirectional audio signal and three directional audio signals such as X, Y and Z.

[0081] Alternatively, the signal input into the format converter or the core encoder could be a signal captured by an omnidirectional microphone positioned at the first portion and another signal captured by an omnidirectional microphone positioned at the second portion different from the first portion. Again, alternatively, the audio scene comprises, as a first component signal, a signal captured by a directional microphone directed to a first direction and, as a second component, at least one signal captured by another directional microphone directed to a second direction different from the first direction. These “directional microphones” do not necessarily have to be real microphones but can also be virtual microphones.

[0082] The audio input into block 900 or output by block 900 or generally used as the audio scene can comprise A-format component signals, B-format component signals, first order Ambisonics component signals, higher order Ambisonics component signals or component signals captured by a microphone array with at least two microphone capsules or component signals calculated from a virtual microphone processing.

[0083] The output interface 300 of FIG. 1A is configured to not include any spatial parameters from the same parameter kind as the one or more spatial parameters generated by the spatial analyzer for the second portion into the encoded audio scene signal.

[0084] Thus, when the parameters 330 for the second portion are direction of arrival data and diffuseness data, the first encoded representation for the first portion will not comprise directional of arrival data and diffuseness data but can, of course, comprise any other parameters that have been calculated by the core encoder such as scale factors, LPC coefficients, etc.

[0085] Moreover, the band separation performed by signal separator 140, when the different portions are different bands can be implemented in such a way that a start band for the second portion is lower than the bandwidth extension start band and, additionally, the core noise filling does not necessarily have to apply any fixed crossover band, but can be used gradually for more parts of the core spectra as the frequency increases.

[0086] Moreover, the parametric or largely parametric processing for the second frequency subband of a time frame comprises calculating an amplitude-related parameter for the second frequency band and the quantization and entropy coding of this amplitude-related parameter instead of individual spectral lines in the second frequency subband. Such an amplitude related parameter forming a low resolution representation of the second portion is, for example, given by a spectral envelope representation having only, for example, one scale factor or energy value for each scale factor band, while the high resolution first portion relies on individual MDCT or FFT or general, individual spectral lines.

[0087] Thus, a first portion of the at least two component signals is given by a certain frequency band for each component signal, and the certain frequency band for each component signal is encoded with a number of spectral lines to obtain the encoded representation of the first portion. With respect to the second portion, however, an amplitude-related measure such as the sum of the individual spectral lines for the second portion or a sum of squared spectral lines representing an energy in the second portion or the sum of spectral lines raised to the power of three representing a loudness measure for the spectral portion can be used as well for the parametric encoded representation of the second portion.

[0088] Again referring to FIG. 8A, the core encoder 160 comprising of the individual core encoder branches 160a, 160b may comprise a beamforming/signal selection procedure for the second portion. Thus, the core encoder indicated at 160a, 160b in FIG. 8B outputs, on the one hand, an encoded first portion of all four B-format components and an encoded second portion of a single transport channel and spatial metadata for the second portion that have been generated by a DirAC analysis 210 relying on the second portion and a subsequently connected spatial metadata encoder 220.

[0089] On the decoder-side, the encoded spatial metadata is input into the spatial metadata decoder 700 to generate the parameters for the second portion illustrated at 830. The core decoder which is an embodiment typically implemented as an EVS-based core decoder consisting of elements 510a, 510b outputs the decoded representation consisting of both portions where, however, both portions are not yet separated. The decoded representation is input into a frequency analyzing block 860 and the frequency analyzer 860 generates the component signals for the first portion and forwards same to a DirAC analyzer 600 to generate the parameters 840 for the first portion. The transport channel/component signals for the first and the second portions are forwarded from the frequency analyzer 860 to the DirAC synthesizer 800. Thus, the DirAC synthesizer operates, in an embodiment, as usual, since the DirAC synthesizer does not have any knowledge and actually does not require any specific knowledge, whether the parameters for the first portion and the second portion have been derived on the encoder side or on the decoder side. Instead, both parameters “do the same” for the DirAC synthesizer 800 and the DirAC synthesizer can then generate, based on the frequency representation of the decoded representation of the at least two component signals representing the audio scene indicated at 862 and the parameters for both portions, a loudspeaker output, a first order Ambisonics (FOA), a high order Ambisonics (HOA) or a binaural output.

[0090] FIG. 9A illustrates another embodiment of an audio scene encoder, where the core encoder 100 of FIG. 1A is implemented as a frequency domain encoder. In this implementation, the signal to be encoded by the core encoder is input into an analysis filter bank 164 advantageously applying a time-spectral conversion or decomposition with typically overlapping time frames. The core encoder comprises a wave form preserving encoder processor 160a and a parametric encoder processor 160b. The distribution of the spectral portions into the first portion and the second portion is controlled by a mode controller 166. The mode controller 166 can rely on a signal analysis, a bitrate control or can apply a fixed setting. Typically, the audio scene encoder can be configured to operate at different bitrates, wherein a predetermined border frequency between the first portion and the second portion depends on a selected bitrate, and wherein a predetermined border frequency is lower for a lower bitrate or greater for a greater bitrate.

[0091] Alternatively, the mode controller can comprise a tonality mask processing as known from intelligent gap filling that analyzes the spectrum of the input signal in order to determine bands that have to be encoded with a high spectral resolution that end up in the encoded first portion and to determine bands that can be encoded in a parametric way that will then end up in the second portion. The mode controller 166 is configured to also control the spatial analyzer 200 on the encoder-side and advantageously to control a band separator 230 of the spatial analyzer or a parameter separator 240 of the spatial analyzer. This makes sure that, in the end, only spatial parameters for the second portion, but not for the first portion are generated and output into the encoded scene signal.

[0092] Particularly, when the spatial analyzer 200 directly receives the audio scene signal either before being input into the analysis filter bank or subsequent to being input into the filter bank, the spatial analyzer 200 calculates a full analysis over the first and the second portion and, the parameter separator 240 then only selects for output into the encoded scene signal the parameters for the second portion. Alternatively, when the spatial analyzer 200 receives input data from a band separator, then the band separator 230 already forwards only the second portion and, then, a parameter separator 240 is not required anymore, since the spatial analyzer 200 anyway only receives the second portion and, therefore, only outputs the spatial data for the second portion.

[0093] Thus, a selection of the second portion can be performed before or after the spatial analysis and may be controlled by the mode controller 166 or can also be implemented in a fixed manner. The spatial analyzer 200 relies on an analysis filter bank of the encoder or uses his own separate filter bank that is not illustrated in FIG. 9A, but that is illustrated, for example, in FIG. 5A for the DirAC analysis stage implementation indicated at 1000.

[0094] FIG. 9B illustrates, in contrast to the frequency domain encoder of FIG. 9A, a time domain encoder. Instead of the analysis filter bank 164, a band separator 168 is provided that is either controlled by a mode controller 166 of FIG. 9A (not illustrated in FIG. 9B) or that is fixed. In case of a control, the control can be performed based on a bit rate, a signal analysis, or any other procedure useful for this purposed. The typically M components that are input into the band separator 168 are processed, on the one hand, by a low band time domain encoder 160a and, on the other hand, by a time domain bandwidth extension parameter calculator 160b. Advantageously, the low band time domain encoder 160a outputs the first encoded representation with the M individual components being in an encoded form. Contrary thereto, the second encoded representation generated by the time domain bandwidth extension parameter calculator 160b only has N components/transport signals, where the number N is smaller than the number M, and where N is greater than or equal to 1.

[0095] Depending on whether the spatial analyzer 200 relies on the band separator 168 of the core encoder, a separate band separator 230 is not required. When, however, the spatial analyzer 200 relies on the band separator 230, then the connection between block 168 and block 200 of FIG. 9B is not necessary. In case none of the band separators 168 or 230 are at the input of the spatial analyzer 200, the spatial analyzer performs a full band analysis and the parameter separator 240 then separates only the spatial parameters for the second portion that are then forwarded to the output interface or the encoded audio scene.

[0096] Thus, while FIG. 9A illustrates a wave form preserving encoder processor 160a or a spectral encoder for quantizing an entropy coding, the corresponding block 160a in FIG. 9B is any time domain encoder such as an EVS encoder, an ACELP encoder, an AMR encoder or a similar encoder. While block 160b illustrates a frequency domain parametric encoder or general parametric encoder, the block 160b in FIG. 9B is a time domain bandwidth extension parameter calculator that can, basically, calculate the same parameters as block 160 or different parameters as the case may be.

[0097] FIG. 10A illustrates a frequency domain decoder typically matching with the frequency domain encoder of FIG. 9A. The spectral decoder receiving the encoded first portion comprises, as illustrated at 160a, an entropy decoder, a dequantizer and any other elements that are, for example, known from AAC encoding or any other spectral domain encoding. The parametric decoder 160b that receives the parametric data such as energy per band as the second encoded representation for the second portion operates, typically, as an SBR decoder, an IGF decoder, a noise filling decoder or other parametric decoders. Both portions, i.e., the spectral values of the first portion and the spectral values of the second portion are input into a synthesis filter bank 169 in order to have the decoded representation that is, typically forwarded to the spatial renderer for the purpose of spatially rendering the decoded representation.

[0098] The first portion can be directly forwarded to the spatial analyzer 600 or the first portion can be derived from the decoded representation at the output of the synthesis filter bank 169 via a band separator 630. Depending on how the situation is, the parameter separator 640 is used or not. In case of the spatial analyzer 600 receiving the first portion only, then the band separator 630 and the parameter separator 640 are not required. In case of the spatial analyzer 600 receiving the decoded representation and the band separator is not there, then the parameter separator 640 is used. In case of the decoded representation is input into the band separator 630, then the spatial analyzer does not need to have the parameter separator 640, since the spatial analyzer 600 then only outputs the spatial parameters for the first portion.

[0099] FIG. 10B illustrates a time domain decoder that is matching with the time domain encoder of FIG. 9B. Particularly, the first encoded representation 410 is input into a low band time domain decoder 160a and the decoded first portion is input into a combiner 167. The bandwidth extension parameters 420 are input into a time domain bandwidth extension processor that outputs the second portion. The second portion is also input into the combiner 167. Depending on the implementation, the combiner can be implemented to combine spectral values, when the first and the second portion are spectral values or can combine time domain samples when the first and the second portion are already available as time domain samples. The output of the combiner 167 is the decoded representation that can be processed, similar to what has been discussed before with respect to FIG. 10A, by the spatial analyzer 600 either with or without the band separator 630 or with or without the parameter separator 640 as the case may be.

[0100] FIG. 11 illustrates an implementation of the spatial renderer although other implementations of a spatial rendered that rely on DirAC parameters or on other parameters than DirAC parameters, or produce a different representation of the rendered signal than the direct loudspeaker representation, like a HOA representation, can be applied as well. Typically, the data 862 input into the DirAC synthesizer 800 can consist of several components such as the B-format for the first and the second portion as indicated at the upper left corner of FIG. 11. Alternatively, the second portion is not available in several components but only has a single component. Then, the situation is as illustrated in the lower portion on the left of FIG. 11. Particularly, in the case of having the first and the second portion with all components, i.e., when the signal 862 of FIG. 8B has all components of the B-format, for example, a full spectrum of all components is available and the time-frequency decomposition allows to perform a processing for each individual time/frequency tile. This processing is done by a virtual microphone processor 870a for calculating, for each loudspeaker of a loudspeaker setup, a loudspeaker component from the decoded representation.

[0101] Alternatively, when the second portion is only available in a single component, then the time/frequency tiles for the first portion are input into the virtual microphone processor 870a, while the time/frequency portion for the single or lower number of components second portion is input into the processor 870b. The processor 870b, for example, only has to perform a copying operation, i.e., to copy the single transport channel into an output signal for each loudspeaker signal. Thus, the virtual microphone processing 870a of the first alternative is replaced by a simply copying operation.

[0102] Then, the output of blocks 870a in the first embodiment or 870a for the first portion and 870b for the second portion are input into a gain processor 872 for modifying the output component signal using the one or more spatial parameters. The data is also input into a weighter/decorrelator processor 874 for generating a decorrelated output component signal using the one or more spatial parameters. The output of block 872 and the output of block 874 is combined within a combiner 876 operating for each component so that, at the output of block 876 one obtains a frequency domain representation of each loudspeaker signal.

[0103] Then, by means of a synthesis filter bank 878, all frequency domain loudspeaker signals can be converted into a time domain representation and the generated time domain loudspeaker signals can be digital-to-analog converted and used to drive corresponding loudspeakers placed at the defined loudspeaker positions.

[0104] Typically, the gain processor 872 operates based on spatial parameters and advantageously, directional parameters such as the direction of arrival data and, optionally, based on diffuseness parameters. Additionally, the weighter/decorrelator processor operates based on spatial parameters as well, and, advantageously, based on the diffuseness parameters.

[0105] Thus, in an implementation, the gain processor 872 represents the generation of the non-diffuse stream in FIG. 5B illustrated at 1015, and the weighter/decorrelator processor 874 represents the generation of the diffuse stream as indicated by the upper branch 1014 of FIG. 5B, for example. However, other implementations that rely on different procedures, different parameters and different ways for generating direct and diffuse signals can be implemented as well.

[0106] Exemplary benefits and advantages of embodiments over the state of the art are: [0107] Embodiments of the present invention provide a better time-frequency-resolution for the parts of the signal chosen to have decoder-side-estimated spatial parameters over a system using encoder side estimated and coded parameters for the whole signal. [0108] Embodiments of the present invention provide better spatial parameter values for parts of the signal reconstructed using encoder side analysis of parameters and coding and transmitting said parameters to the decoder over a system where spatial parameters are estimated at the decoder using the decoded lower-dimension audio signal. [0109] Embodiments of the present invention allow for a more flexible trade-off between time-frequency resolution, transmission rate, and parameter accuracy than either a system using coded parameters for the whole signal or a system using decoder side estimated parameters for the whole signal can provide. [0110] Embodiments of the present invention provide a better parameter accuracy for signal portions mainly coded using parametric coding tools by choosing encoder side estimation and coding of some or all spatial parameters for those portions and a better time-frequency resolution for signal portions mainly coded using wave-form-preserving coding tools and relying on a decoder side estimation of the spatial parameters for those signal portions.

REFERENCES

[0111] [1] V. Pulkki, M-V Laitinen, J Vilkamo, J Ahonen, T Lokki and T Pihlajamäki, “Directional audio coding—perception-based reproduction of spatial sound”, International Workshop on the Principles and Application on Spatial Hearing, November 2009, Zao; Miyagi, Japan. [0112] [2] Ville Pulkki. “Virtual source positioning using vector base amplitude panning”. J. Audio Eng. Soc., 45(6):456{466, June 1997. [0113] [3] European patent application No. EP17202393.9, “EFFICIENT CODING SCHEMES OF DIRAC METADATA”. [0114] [4] European patent application No EP17194816.9 “Apparatus, method and computer program for encoding, decoding, scene processing and other procedures related to DirAC based spatial audio coding”.

[0115] An inventively encoded audio signal can be stored on a digital storage medium or a non-transitory storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0116] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0117] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

[0118] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0119] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0120] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0121] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0122] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0123] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

[0124] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0125] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0126] In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods may be performed by any hardware apparatus.

[0127] While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which will be apparent to others skilled in the art and which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.