APPARATUS, METHOD OR COMPUTER PROGRAM FOR GENERATING AN OUTPUT DOWNMIX REPRESENTATION

Abstract

An apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, includes: an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to obtain at least one upmixed portion; and a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme.

Claims

1. Apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the apparatus comprising: an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire at least one upmixed portion; and a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation.

2. Apparatus of claim 1, wherein only the portion of the input downmix representation is in accordance with the first downmixing scheme and a second portion of the input downmix representation is in accordance with the second downmixing scheme, wherein the downmixer is configured for downmixing the at least one upmixed portion in accordance with the second downmixing scheme to acquire the first downmixed portion; and further comprising a combiner for combining the first downmixed portion and the second portion of the input downmix representation or a downmixed portion derived from the second portion of the input downmix representation to acquire the output downmix representation comprising a first output representation for only the portion of the input downmix representation and a second output representation for the second portion of the input downmix representation, wherein the first output representation for only the portion of the input downmix representation and the second output representation for the second portion of the input downmix representation are based on the same downmixing scheme.

3. Apparatus of claim 1, wherein the at least the portion of the input downmix representation or only the portion of the input downmix representation is a first frequency band, wherein the first downmixing scheme is a downmixing scheme relying on a residual signal, and wherein the upmixer is configured to perform an upmix using the residual signal.

4. Apparatus of claim 1, wherein the second downmixing scheme is a fully parametric scheme, and wherein the downmixer is configured to apply the second downmixing scheme.

5. Apparatus of claim 2, wherein the second portion of the input downmix representation is a second frequency band, and wherein the combiner is configured to combine the first downmixed portion and the second portion of the input downmix representation to acquire the output downmix representation.

6. Apparatus of claim 1, further comprising an audio decoder for generating a decoded core signal for at least the portion of the input downmix representation or only the portion of the input downmix representation, and a decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation, wherein the upmixer is configured to use, in the upmixing scheme, the decoded core signal for at least the portion of the input downmix representation or only the portion of the input downmix representation and the decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation, wherein the downmixer is configured for receiving the at least one upmixed portion comprising more channels than the input downmix representation.

7. Apparatus of claim 6, wherein the second portion of the input downmix representation is in accordance with the second downmixing scheme, wherein the audio decoder is configured for generating a decoded core signal for the second portion of the input downmix representation and a decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation only, and wherein the combiner is configured to combine the first downmixed portion and the decoded core signal for the second portion of the input downmix representation.

8. Apparatus of claim 1, further comprising: a time-to-spectrum converter for converting a time domain input downmix representation of at least the portion of the input downmix representation or only the portion of the input downmix representation into a spectral domain; and a spectrum-to-time converter for converting an output signal into a time domain to acquire the output downmix representation, wherein the time-to-spectrum converter or the spectrum-to-time converter is configured to perform an overlap and add processing or to perform a crossover processing from an earlier time block to a later time block, or further comprising an output interface for outputting the output downmix representation to a rendering device or further comprising a rendering device for rendering the output downmix representation as a mono replay signal, or wherein the downmixer is configured to apply, as the second downmixing scheme, an active downmixing scheme, an energy conserving downmixing scheme, or a downmixing scheme, in which a target energy of the downmix signal is in a predetermined ratio to an energy of a mid-channel derived from a first channel and a second channel, wherein at least one of the first channel and the second channel is phase rotated before being added together to form the input downmix representation.

9. Apparatus of claim 8, wherein the second portion of the input downmix representation is in accordance with the second downmixing, wherein the time-to-spectrum converter is configured for converting a time domain input downmix representation of the second portion of the input downmix representation into the spectral domain, or wherein the predetermined ratio indicates an equality or a deviation range being 3 dB related to a higher energy of energies of a first original channel and a second original channel.

10. Apparatus of claim 1, wherein at least the portion of the input downmix representation is in accordance with the first downmixing scheme relying on a residual signal or on a residual signal and parametric information, wherein the upmixer is configured for upmixing the input downmix representation of at least the portion of the input downmix representation using the upmixing scheme corresponding to the first downmixing scheme and using the residual signal or the residual signal and the parametric information, respectively to acquire the at least one upmixed portion; and wherein the downmixer is configured for downmixing the at least one upmixed portion in accordance with the second downmixing scheme different from the first downmixing scheme, wherein the second downmixing scheme is an active downmixing scheme or a fully parametric downmixing scheme to acquire the output downmix representation comprising at least one downmixed portion.

11. Apparatus of claim 10, further comprising an output interface for outputting the output downmix representation to a rendering device or further comprising a rendering device for rendering the output downmix representation as a mono replay signal.

12. Apparatus of claim 10, wherein the downmixer is configured to apply, as the active downmixing scheme, an energy conserving downmixing scheme, or a downmixing scheme, in which a target energy of the downmix signal is in a predetermined ratio to an energy of a mid-channel derived from a first channel and a second channel, wherein at least one of the first channel and the second channel is phase rotated before being added together.

13. Apparatus of claim 10, wherein at least the portion of the input downmix representation comprises the full bandwidth of the input downmix representation.

14. Apparatus of claim 1, wherein the downmixer is configured to perform the second downmixing scheme, the second downmixing scheme comprising: calculating a first weight for a first channel and a second weight for a second channel for a spectral band of the at least one upmixed portion, the spectral band comprising a plurality of spectral lines, and applying the first weight to spectral lines of the spectral band of the first channel and applying the second weight to spectral lines of the spectral band of the second channel, and adding first weighted lines and second weighted lines to acquire downmixed spectral lines in the spectral band, and wherein the apparatus is configured to convert the downmixed spectral lines to a time domain to acquire time domain samples of the output downmix representation.

15. Apparatus of claim 14, wherein the calculation of the first weight and the second weight is performed band wise using energies of the first channel and the second channel and a target energy.

16. Apparatus of claim 15, wherein the target energy is equal to an energy of a phase-rotated mid-channel or is derived from the energies of the first channel, the second channel and from a correlation value between the first channel and the second channel.

17. Apparatus of claim 14, wherein calculating the first weight and the second weight comprises, for a spectral band: calculating an amplitude-related measure for the first channel in the spectral band; calculating an amplitude-related measure for the second channel in the spectral band: calculating an amplitude-related measure for a linear combination of the first channel and the second channel in the spectral band; calculating a cross-correlation measure between the first channel and the second channel in the spectral band; and calculating the first weight and the second weight using the amplitude-related measure for the first channel, the amplitude-related measure for the second channel, the amplitude-related measure for the linear combination and the cross-correlation measure.

18. Apparatus of claim 1, wherein the upmixer is configured to perform the upmixing scheme, the upmixing scheme comprising: calculating first channel spectral lines for a spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation from spectral lines of the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation using a prediction parameter for the spectral band and residual signal lines for the spectral band and a first calculation rule, and calculating second channel spectral lines for the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation from the spectral lines of the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation using the prediction parameter for the spectral band and the residual signal lines for the spectral band and a second calculation rule, wherein the first calculation rule is different from the second calculation rule.

19. Apparatus of claim 18, wherein the first calculation rule comprises one of an addition and a subtraction and the second calculation rule comprises the other one of the addition and the subtraction.

20. Multichannel decoder, comprising: an input interface for providing an input downmix representation and parametric data at least for a second portion of the input downmix representation; and the apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, said apparatus comprising: an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire at least one upmixed portion; and a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation, wherein the multichannel decoder is configured to upmix, with the upmixer, the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or to upmix the input downmix representation for the second portion and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and wherein a combiner is configured to combine the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal.

21. Multichannel decoder of claim 20, wherein the input interface comprises: a first time-spectrum converter for converting a first spectral representation of the at least the portion of the input downmix representation or only the portion of the input downmix representation and a second spectral representation of a second portion of the input downmix representation, the second portion of the input downmix representation comprising spectral values for higher frequencies than at least the portion of the input downmix representation or only the portion of the input downmix representation of the first spectral representation; a second time-spectrum-converter for generating a spectral representation of a residual signal for the at least the portion of the input downmix representation or only the portion of the input downmix representation, wherein the upmixer is configured to upmix the first spectral representation using the spectral representation of the residual signal to acquire the at least one upmixed portion in the spectral domain, wherein the downmixer is configured to downmix the at least one upmixed portion to acquire the first downmixed portion in the spectral domain, and wherein the combiner comprises a spectrum-time converter for combining the first downmixed portion and the spectral representation of the second portion of the input downmix representation and for converting into the time domain to acquire the output downmix representation.

22. Multichannel decoder of claim 20, further comprising: a second upmixer for upmixing the second portion of the input downmix representation to acquire the upmixed second portion, wherein, in a multichannel output mode, the combiner is configured to combine a first channel of the at least one upmixed portion and the first channel of the upmixed second portion and to convert into a time domain to acquire a first channel of a multichannel output, wherein the multichannel decoder further comprises a second combiner configured to combine, in the multichannel output mode, a second channel of the at least one upmixed portion and a second channel of the upmixed second portion and to convert into the time domain to acquire a second channel of the multichannel output.

23. Multichannel decoder of claim 21, further comprising: a second upmixer for upmixing the second portion of the input downmix representation to acquire the upmixed second portion, wherein, in a multichannel output mode, the combiner is configured to combine a first channel of the at least one upmixed portion and the first channel of the upmixed second portion and to convert into a time domain to acquire a first channel of a multichannel output, wherein the multichannel decoder further comprises a second combiner configured to combine, in the multichannel output mode, a second channel of the at least one upmixed portion and a second channel of the upmixed second portion and to convert into the time domain to acquire a second channel of the multichannel output, a switch connected between the first time-spectrum-converter and the second upmixer, and a controller, wherein the controller is configured to control, in a mono-output mode, the switch to connect an output of the first time-spectrum-converter to the combiner or to bypass the second upmixer and to connect an output of the upmixer to an input of the downmixer, or to control, in the multichannel output mode, the switch to connect an output of the first time-spectrum-converter to an input of the second upmixer.

24. Multichannel decoder of claim 22, further comprising a second switch connected between the upmixer and the downmixer; and a controller, wherein the controller is configured to control, in the mono-output mode, the second switch to connect an output of the upmixer to an input of the downmixer and to control, in the multichannel output mode, the second switch to connect an output of the upmixer to an input of the second combiner or to bypass the downmixer.

25. Method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method comprising: upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation.

26. Method of claim 25, wherein a second portion of the input downmix representation is in accordance with a second downmixing scheme, wherein the downmixing comprises downmixing the at least one upmixed portion in accordance with the second downmixing scheme to acquire the first downmixed portion; and wherein the method further comprises combining the first downmixed portion and the second portion or a downmixed portion derived from the second portion to acquire the output downmix representation, wherein the output downmix representation for at least the portion of the input downmix representation and the output representation for the second portion are based on the same downmixing scheme.

27. Method of claim 25, wherein at least the portion of the input downmix representation is in accordance with the first downmixing scheme relying on a residual signal or on a residual signal and parametric information, wherein the upmixing comprises upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme and using the residual signal or the residual signal and the parametric information, respectively to acquire the at least one upmixed portion; and wherein the downmixing comprises downmixing the at least one upmixed portion in accordance with the second downmixing scheme different from the first downmixing scheme, wherein the second downmixing scheme is an active downmixing scheme or a fully parametric downmixing scheme to acquire the output downmix representation for at least the portion of the input downmix representation.

28. Method of multichannel decoding, comprising: providing an input downmix representation and parametric data at least for a second portion of the input downmix representation; the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method for generating an output downmix representation comprising: upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation, wherein the method comprises the upmixing the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or upmixing the second portion of the input downmix representation and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and combining the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal.

29. Non-transitory digital storage medium having a computer program stored thereon to perform the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, said method comprising: upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation, when said computer program is run by a computer.

30. Non-transitory digital storage medium having a computer program stored thereon to perform the method of multichannel decoding, said method comprising: providing an input downmix representation and parametric data at least for a second portion of the input downmix representation; the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method for generating an output downmix representation comprising: upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation, wherein the method comprises the upmixing the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or upmixing the second portion of the input downmix representation and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and combining the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal, when said computer program is run by a computer.

31. Apparatus for generating an output downmix representation from an input downmix representation, wherein a first portion of the input downmix representation is in accordance with a first downmixing scheme and a second portion of the input downmix representation is in accordance with the second downmixing scheme, the apparatus comprising: an upmixer for upmixing the first portion of the input downmix representation using a first upmixing scheme corresponding to the first downmixing scheme to acquire a first upmixed portion and for upmixing the second portion of the input downmix representation using a second upmixing scheme corresponding to the second downmixing scheme to acquire a second upmixed portion; and a downmixer for downmixing the first upmixed portion and the second upmixed portion in accordance with a third downmixing scheme different from the first downmixing scheme and the second downmixing scheme to acquire the output downmix representation, wherein the output representation for the first portion of the input downmix representation and the output representation for the second portion of the input downmix representation are based on the same downmixing scheme of the input downmix representation.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Embodiments of the present invention will be detailed subsequently referring to the appended drawings, in which:

[0038] FIG. 1 illustrates an apparatus for generating an output downmix representation in an embodiment;

[0039] FIG. 2 illustrates an apparatus for generating an output downmix representation in a further embodiment, in which the downmixing scheme is based on a residual signal or a residual signal and parameters;

[0040] FIG. 3 illustrates a further embodiment, where different downmixing schemes are performed for different portions such as spectral portions of the input downmix representation;

[0041] FIG. 4 illustrates a further embodiment illustrating the usage of different downmixing schemes in different spectral portions for the input downmix representation and the procedure where the first downmixing scheme is based on residual data and the second downmixing scheme is an active downmixing scheme or a downmixing scheme without residual or parametric data;

[0042] FIG. 5 illustrates an advantageous implementation of the upmixing scheme corresponding to the first downmixing scheme in an embodiment;

[0043] FIG. 6 illustrates a multichannel decoder operating in a stereo output mode;

[0044] FIG. 7 illustrates a multichannel encoder in accordance with an embodiment that is switchable between the multichannel output mode or the mono output mode;

[0045] FIG. 8a illustrates an advantageous implementation for the second downmixing scheme;

[0046] FIG. 8b illustrates a further embodiment of the second downmixing scheme; and

[0047] FIG. 9 illustrates the separation of an input downmix representation into the portion of the input downmix representation in the first downmixing scheme indicated as the first portion and into the second portion of the input downmixing representation that relies on a downmixing scheme with weights.

DETAILED DESCRIPTION OF THE INVENTION

[0048] FIG. 1 illustrates an apparatus for generating an output downmix representation from an input downmix representation, where at least a portion of the input downmix representation is in accordance with a first downmixing scheme. The apparatus comprises an upmixer 200 for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to obtain at least one upmixed portion at the output of block 200. The apparatus furthermore comprises a downmixer 300 for downmixing the at least one upmixed portion in accordance with a second downmixing scheme being different from the first downmixing scheme.

[0049] Advantageously, the output of the downmixer 300 is forwarded to an output stage 500 for generating a mono output. The output stage is, for example, an output interface for outputting the output downmix representation to a rendering device or the output stage 500 actually comprises a rendering device for rendering the output downmix representation as a mono replay signal.

[0050] The apparatus illustrated in FIG. 1 provides a conversion from a downmix representation in a first “downmix domain” into another second downmix domain. As will be illustrated in other figures, the conversion can be valid only for a limited part of the spectrum such as the first portion illustrated, for example, in FIG. 9 for the exemplarily given lowest three bands b.sub.1, b.sub.2 and b.sub.3. Alternatively, the apparatus can also perform a conversion from one downmix domain to another downmix domain for the full band, i.e., for all bands b.sub.1 to b.sub.6 exemplarily illustrated in FIG. 9. The portion can be any portion of the signal such as a spectral portion, a time portion such as a time block or frame, or any other portion of the signal.

[0051] FIG. 2 illustrates an embodiment where the first downmixing scheme relies on a residual signal only or on a residual signal and parametric information. FIG. 2 comprises an input interface 10 where the input interface receives an encoded multichannel signal that comprises an encoded core signal and an encoded side information part. The core signal is decoded by a core decoder 20 to provide the input downmix representation without side information. Additionally, the side information part from the encoded multichannel signal is provided and processed by the side information decoder 30 within the input interface, and the side information decoder 30 provides the residual signal or the residual signal and parameters as indicated at 210 in FIG. 2. The data, i.e., the input downmix that corresponds to the decoded core signal and the residual data are both input the upmixer 200 and the upmixer 200 generates an upmix signal that has a first channel and a second channel and the first channel and the second channel data are high quality audio data, since the high quality audio data are generated not only by the core signal and some kind of passive upmix, but are generated additionally using the residual data or the residual data and the parameters, i.e., all data available from the encoded multichannel signal. The output of the upmixer 200 is downmixed by the downmixer 300 using, for example, an active downmix or, generally, a downmixing scheme that does not generate a residual signal or that does not generate any parameters but that generates a downmix or mono signal that is energy-compensated, i.e., that does not suffer from energy fluctuations that are normally a significant problem when only a passive downmix is performed as is, for example, the case with the core signal generated by the core decoder 20 of FIG. 2. The output of the downmixer 300 is forwarded, for example, to a renderer for rendering the mono signal or, for example, to the output stage 500 illustrated in FIG. 1.

[0052] FIG. 3 illustrates a further embodiment where, again referring to FIG. 9, the first portion is available in the first downmixing scheme such as a downmixing scheme with residual data and where there is a second spectral portion that is available, for example in a second downmixing scheme without any residuals, i.e., that has been generated by an active downmix using, for example, downmix weights derived based on energy considerations to combat any fluctuations that otherwise would occur if a passive downmix would be applied.

[0053] The first portion of the downmix representation is input into the upmixer 200 that upmixes corresponding to the first downmixing scheme and the first portion is forwarded, as discussed with respect to FIG. 1 or FIG. 2, into the downmixer 300 that now performs a downmix in the second downmixing scheme. The second portion illustrated in FIG. 3 can be, for example, in the second downmixing scheme but can also be in a third, i.e., any other downmixing scheme, from the downmixing scheme of the portion input into the upmixer 200 or the second downmixing scheme output by the downmixer 300. In case of the downmixing domain being the same for the second portion and the output of the downmixer 300, any second portion processor 600 is not required. Instead, the second portion can be forwarded into a combiner 400 for combining the first and the second portion that are now harmonized with respect to their downmixing schemes. However, when the second portion is in a downmixing domain, i.e., has an underlying downmixing scheme being different from the downmixing scheme in which the output of the downmixer 300 is available, the second portion processor 600 is provided. Generally, the second portion processor 600 also comprises an upmixer for upmixing the second portion being in a third downmixing scheme and the second portion processor 600 additionally comprises a downmixer for downmixing the upmixer representation into the same downmixing domain, i.e., using the same downmixing scheme, as is available from the downmixer 300. The second portion processor 600 can be implemented using the upmixer 200 and the subsequently connected downmixer 300 so that a full harmonization of the data input into the combiner 400 is obtained. The combiner 400 outputs, advantageously, a spectral representation of the mono output downmix representation which is converted into the time domain by means of a spectrum-time-converter such as a filterbank, an IDFT, an IMDCT, etc. Alternatively, the combiner 400 is configured for combining the individual inputs into individual time domain signals, and the time domain signals are combined in the time domain to obtain a time domain mono output downmix representation.

[0054] FIG. 4 comprises an input interface that may include a first time-to-spectrum converter 100 such as DFT block as illustrated in FIG. 4 and a second time-to-spectrum converter 120 such as the second DFT block in FIG. 4. The first block 100 is configured for converting the decoded core signal as, for example, output by the core decoder 20 of FIG. 2 into a spectral representation. Furthermore, the second time-to-spectral converter 120 is configured to convert the decoded residual signal as, for example, output by the side information decoder 30 of FIG. 2 into a spectral representation illustrated at 210a. Furthermore, line 210b illustrates optionally provided additional parametric data such as side gains that are also output by the side information decoder 30 of FIG. 2 for example. The upmixer 200 of FIG. 4 generates an upmixed left channel and an upmixed right channel for a lowband, i.e., exemplary for the first three band b.sub.1, b.sub.2, b.sub.3 of FIG. 9. Furthermore, the lowband upmix at the output of block 200 is input into the downmixer 300 advantageously performing an active downmix so that a lowband representation for the exemplarily illustrated three bands b.sub.1, b.sub.2, b.sub.3 of FIG. 9 is provided. This lowband downmix is now in the same domain as the highband downmix generated already by the DFT block 100. The output of block 100 for the highband would, in the example of FIG. 9, correspond to the downmix representation for bands b.sub.4, b.sub.5, b.sub.6. Now, at the input into the combiner 400, illustrated in FIG. 4 as an IDFT 400, the lowband representation and the highband representation of the downmix are in the same “downmix domain”, and have been generated with the same downmixing scheme. Now, the lowband and the highband of the harmonized downmix representation can be combined and advantageously converted into the time domain to provide the mono output signal at the output of block 400.

[0055] A mostly parametric stereo scheme as described in [8] is built around the idea of only transmitting a single downmixed channel and recreating the stereo image via side parameters. This downmix at the encoder side is done in an active manner by dynamically calculating weights for both channels in the DFT domain [7]. These weights are computed band-wise using the respective energies of the two channels and their cross-correlation. The target energy that has to be preserved by the downmix is equal to the energy of the phase-rotated mid-channel:

[00002] $E_{t a r g e t} = {.Math. \frac{L + R e^{- j φ}}{2} .Math.}^{2} = \frac{.Math. L, L .Math. + .Math. R, R .Math. + 2 .Math. .Math. L, R .Math. .Math.}{4} = \frac{{.Math. L .Math.}^{2} + {.Math. R .Math.}^{2} + 2 .Math. .Math. L, R .Math. .Math.}{4},$

[0056] where L and R represent the left and right channel. Based on this target energy the weights for the channels can be computed per band b as follows:

[00003] $w_{R b} = \frac{1}{2 \sqrt{2}} \frac{\sqrt{{.Math. L_{b} .Math.}^{2} + {.Math. R_{b} .Math.}^{2} + 2 .Math. .Math. L_{b}, R_{b} .Math. .Math.}}{.Math. L_{b} .Math. + .Math. R_{b} .Math.}$ $And$ $w_{L, b} = w_{R, b} + 1 - \frac{.Math. L_{b} .Math. + .Math. R_{b} .Math.}{.Math. L_{b} .Math. + .Math. R_{b} .Math.} .$

[0057] |L| and |R| are computed for each band b as

[00004] $.Math. L_{b} .Math. = \sqrt{\underset{i in b}{.Math.} (L_{r e a l, i, b^{2}} + L_{i m a g, i, b^{2}})}, .Math. R_{b} .Math. = \sqrt{\underset{i in b}{.Math.} (R_{r e a l, i, b^{2}} + R_{i m a g, i, b^{2}})}$

[0058] |L+R| is computed as

|L.sub.b+R.sub.b|=√{square root over (|L.sub.b|.sup.2+|R.sub.b|.sup.2+2dotprod.sub.real.sup.2)}

[0059] and |<L, R>| is computed as the absolute of the complex dot product

[00005] $.Math. .Math. L_{b}, R_{b} .Math. .Math. = \sqrt{d o t p r o d_{r eal, b}^{2} + d o t p r o d_{i m a g, b}^{2}}$ $with$ $dotpro d_{real, b} = \underset{i in b}{.Math.} (L_{real, i, b} R_{r e a l, i, b} + L_{i m a g, i, b} R_{i m a g, i, b})$ $and$ $dotpro d_{i m a g, b} = \underset{i in b}{.Math.} (L_{i m a g i, b} R_{r e a l, ib} - L_{r e a l, i, b} R_{i m a g, i, b})$

[0060] where i specifies the bin number inside spectral band b.

[0061] The downmixed spectrum is obtained for each band by adding the weighted spectral bins of left and right channel:

DMX.sub.real,i,b=W.sub.L,bL.sub.real,i,b+w.sub.R,bR.sub.real,i,b

and

DMX.sub.imag,i,b=W.sub.L,bL.sub.imag,i,b+W.sub.R,bR.sub.imag,i,b.

[0062] If all the stereo processing in such a system is entirely reliant on parameters and the described active downmix is done on the whole spectrum, a mono signal that satisfies the given quality requirements by avoiding the problems of a passive downmix is already available after the core decoding. This means that in most cases it suffices to skip all decoder stereo processing and output the signal without going into DFT domain.

[0063] However, for higher bitrates this kind of system also supports the coding of a residual signal for the lower spectral bands. The residual signal can be seen as the side-signal of an MS-transform of these lowest bands while the core signal is the complementary mid-signal, basically a passive downmix of left and right. To keep the side signal as small as possible, a compensation of the interaural level differences (ILDs) between the channels is applied to it using side gains that are computed per band.

[0064] The downmixed mid-channel is computed at the encoder side for every spectral bin i inside the residual coding spectrum as

[00006] $m i d_{i} = \frac{L_{i} + R_{i}}{2}$

[0065] while the complementary side channel is computed as

[00007] $s i d e_{i} = \frac{L_{i} - R_{i}}{2} .$

[0066] The residual signal is obtained by subtracting the predicted part due to an ILD between left and right:

res.sub.t=side.sub.i−g.sub.b*mid.sub.i

[0067] with side gain g.sub.b of the current spectral band b given as

[00008] $g_{b} = \frac{{.Math. L_{b} .Math.}^{2} - {.Math. R_{b} .Math.}^{2}}{{.Math. L_{b} + R_{b} .Math.}^{2}} .$

[0068] The full-band signal going into the core coder is a mixture of passive downmix in lower bands and active downmix in all higher bands. Listening tests have shown that there are perceptual issues when playing back such a mixed signal. A way of harmonizing the different signal parts is therefore useful.

[0069] FIG. 5 illustrates a representation of the upmixing scheme relying on residual data res, and parametric data illustrated by bandwise side gain indices g.sub.{circumflex over (b)}. i stands for spectral values and b stands for a certain band. FIG. 5 illustrates a situation, which is also illustrated in FIG. 9, where each band b.sub.i has several spectral lines. In particular, in order to calculate the spectral value L.sub.i, the mid-signal spectral value, i.e., the corresponding spectral value with index i of the output of the core decoder 20 or the output of DFT block 100 of FIG. 4 is used. Furthermore, the corresponding parameter g.sub.{circumflex over (b)} for the corresponding band, in which the spectral value i is located, may be used as illustrated in FIG. 4 by line 210b and the residual spectral value as generated by block 120 and as illustrated at line 210a for the certain spectral value with index i and for the respective band b may be used as well.

[0070] The L-R representations of the lowband signal with residual coding are thereby regained as follows:

L.sub.i=mid.sub.i+ custom-character *mid.sub.i+res.sub.t

and

R.sub.i=mid.sub.i− custom-character *mid.sub.i−res.sub.t.

[0071] Subsequently, the active downmix is applied as described above, only the weights are calculated from the upmixed decoded spectra L and R. The lowband is combined with the already actively downmixed highband to create a harmonized signal which is brought back to time domain via IDFT.

[0072] FIG. 6 illustrates an implementation of a multichannel decoder for a stereo output. The multichannel decoder comprises elements of FIG. 4 that are indicated with the same reference numbers. Additionally, the stereo multichannel decoder comprises a second upmixer 220 for upmixing the highband downmix, i.e., the second portion into a second upmix representation comprising, for example, a left channel and a right channel for a stereo output as one implementation of the multichannel decoder. For another implementation of the multichannel decoder, where there are more than two output channels, such as three or more output channels, the upmixer 220 as well as the upmixer 200 would generate a corresponding higher number of output channels rather than only the left channel and the right channel.

[0073] Furthermore, a second combiner 420 is illustrated in FIG. 6 for the multichannel decoder, i.e., for the illustrated stereo decoder. In case of more than two outputs, a further combiner would be there for the third output channel and another one for the fourth output channel and so on. In contrast to FIG. 6, however, the downmixer 300 of FIG. 4 is not necessary for the multichannel output.

[0074] FIG. 7 illustrates an advantageous implementation of a switchable multichannel decoder which is switchable by means of the actuation of a controller 700, between a mono mode or a stereo/multichannel output mode. Furthermore, in contrast to FIG. 6, the multichannel decoder additionally comprises the downmixer 300 already described with respect to FIG. 4 or the other figures. Furthermore, in the switchable implementation, one option is to provide two individual switches S1, S2. However, the switching functionalities illustrated at the bottom of FIG. 7 can also be implemented by other switching means such as combined switches or even more than two switches. Generally, switch 1 is configured to operate in the mono output mode, so that the second upmixer 220 also indicated as “upmix high” is bypassed. Furthermore, the second switch S2 is configured by the second control signal CTRL.sub.2 to feed the active downmix 300 with the output of the upmixer 200 indicated as “upmix low” in FIG. 7. Furthermore, in the mono output mode, the upmix high block 220 described with respect to FIG. 6 is inactive and, additionally, the second combiner 420 indicated as “IDFT.sub.R is inactive as well, since only a single combiner 400 for the generation of the single mono output signal may be used.

[0075] Contrary thereto, in the stereo output mode or, generally, in the multichannel output mode, the controller 700 is configured to activate, via control signal CTRL.sub.1 the first switch so that the output of the first time-to-frequency converter 100 is fed into the second upmixer 220 indicated as “upmix high” in FIG. 7. By means of the actuation of switch S1, the second combiner 220 is activated. Furthermore, the controller 700 is configured to control the second switch S2720 so that the output of block 200 is not input into the active downmixer 300, but the downmixer 300 is bypassed. The left channel (lowband) portion of the output of block 200 is forwarded as the lowband portion for the combiner 400 and the right channel lowband portion at the output of block 200 is forwarded to the lowband input of the second combiner 420 as illustrated in FIG. 7. Furthermore, in the stereo/multichannel output mode, the downmix 300 is inactive.

[0076] FIG. 8a illustrates a flow chart for an embodiment used in the downmix 300 for performing an active downmix. In a step 800, weights w.sub.R and w.sub.L are calculated based on a target energy. This is done per band such that a weight w.sub.R for the right channel and a weight w.sub.L for the left channel are obtained for each band.

[0077] In block 820, the weights are applied to the upmixed signal over the whole bandwidth of the signal under consideration or only in the corresponding portion per spectral bin. To this end, block 820 receives the spectral domain (complex) signals or bins or spectral values. Subsequent to the application of the weights and, particularly, an addition of the weighted values to obtain the downmix, a conversion 840 to the time domain is performed. Depending on whether only a portion or the full band is processed in block 820, the conversion to the time domain takes place without any other portion or takes place with the other portion particularly in the context of a harmonized downmix as, for example, illustrated and discussed with respect to FIG. 3 or FIG. 4.

[0078] FIG. 8b illustrates an advantageous implementation of the functionalities performed in block 800 of FIG. 8a. In particular, for the calculation of the weights w.sub.R and w.sub.L for each band, an amplitude-related measure for L is calculated for a band. To this end, the individual spectral lines for the left channel, i.e., for the left channel as output by block 200 of any of the FIGS. 1 to 7 are input. In block 804, the same procedure is performed for the second channel or right channel in the same band b. Furthermore, in block 806, another amplitude-related measure is calculated for a linear combination of L and R in the band b. In block 806, once again, the spectral values of the first channel L, the spectral values for the second channel R may be used for the band under consideration. In block 808, a cross-correlation measure is calculated between the left channel and the right channel or, generally, between the first channel and the second channel in the corresponding band b. To this end, once again, the spectral values at indices e for the first and the second channels may be used for the corresponding band.

[0079] As illustrated, the amplitude-related measure can be the square root over the squared magnitudes of the spectral values in a band. This is illustrated as |L.sub.b|. Another amplitude-related measure would, for example, be the sum over the magnitudes of the spectral lines in the band without any square root or with an exponent being different from ½ such as an exponent being between 0 and 1 but excluding 0 and 1. Furthermore, the amplitude-related measure could also refer to a sum over exponentiated magnitudes of spectral lines where the exponent is different from 2. For example, using an exponent of 3 would correspond to the loudness in psychoacoustic terms. However, other exponents being greater than 1 would be useful as well.

[0080] The same is true for the amplitude-related measure calculated in block 804 or the amplitude-related measure calculated in block 806.

[0081] Furthermore, with respect to the cross-correlation measure calculated in block 808, the corresponding mathematical equation illustrated before also relies on a squaring of the dot products and the calculation of a square root. However, other exponents for the dot products different from 2 such as exponents equal to 3 corresponding to a loudness domain or exponents greater than 1 can be used as well. At the same time, instead of the square root, other exponents different from ½ can be used such as ⅓ or, generally, any exponent being between 0 and 1.

[0082] Furthermore, block 810 indicates the calculation of w.sub.R and w.sub.L based on the three amplitude-related measures and the cross-correlation measure. Although it has been indicated that the target energy is preserved by the downmix and is equal to the energy of the phase-rotated mid-channel, it is not necessary, neither for the calculation of w.sub.R and w.sub.L nor for the calculation of the actual downmix signal that such a rotation with a rotation angle is actually performed. Instead, the only thing that is highly expedient when the actual rotation with the rotation angle ϕ is not performed is the calculation of the cross-correlation measure between L and R in the corresponding bands b. In the previously described embodiment, although it has been indicated that an energy of a phase-rotated mid-channel is used as the target energy, any other target energies can be used or any phase rotation has not to be performed at all. With respect to other target energies, these target energies are energies that make sure that an energy of the downmix signal generated by the downmix 300 is fluctuating for the same signal less than the energy of a passive downmix as, for example, underlying the decoded core signal input into block 100 of FIG. 4.

[0083] FIG. 9 illustrates a general representation of a spectrum indicating a lowband first portion that is provided, with respect to the input downmix representation, as a downmix with residual data and indicating a second portion that is provided, with respect to the input downmix representation, by a downmix generated with weights as discussed before with respect to FIG. 8a, 8b. Although FIG. 9 illustrates only six bands, where three bands are for the first portion and three bands are for the second portion, and although FIG. 9 illustrates certain bandwidths that increase from lower bands to higher bands, the specific numbers, the specific bandwidths and the separation of the spectrum into the first portion and into the second portion are only exemplary. In a real scenario, a significantly higher number of bands will be there and, additionally, the first portion that, additionally, has the residual signal will be less than 50% of the number of bands b.

[0084] Advantageously, the time-to-spectral converters 100, 120 of FIGS. 4, 6 and 7 and the combiner 400, 420 are implemented as DFT or IDFT blocks that advantageously implement an FFT or IFFT algorithm. For the processing of a continuous decoded signal input into blocks 100, 120, a block wise processing is performed where overlapping blocks are formed, analysis filtered, transformed into the spectral domain, processed and, in the combiners 400, 420 synthesis filtered, and combined, once again with a 50% overlap. The combination of a 50% overlap on the synthesis side will typically be performed by an overlap add operation with a cross fading from one block to the other where, advantageously, the cross fading weights are already included in the analysis/synthesis windows. However, when this is not the case, an actual cross fading is performed at the output of block 400, for example, or 420, for example, of FIG. 7 or FIG. 6, so that each time domain output sample of either the mono output signal or the left output signal or the right output signal is generated by an addition of two values of two different blocks. For an overlap of more than 50%, an overlap between three or corresponding even more blocks can be performed as well.

[0085] Alternatively, when the time-to-spectral conversion on the one hand and the spectral-time-conversion on the other hand are performed with, for example, a modified discrete cosine transform, an overlap processing is used as well. On the spectral-to-time conversion side, an overlap-add processing is performed so that, once again, each output time domain sample is obtained by summing corresponding time domain samples from two (or more) different IMDCT blocks.

[0086] Advantageously, the harmonization of the downmixing schemes is performed fully in the spectral domain as illustrated in FIGS. 4, 6 and 7. Any additional time-spectrum-transform or spectrum-time-transform is not required when switching from mono to stereo or from stereo to mono as illustrated in FIG. 7. Only manipulations of data in the spectral domain either by the downmixer 300 for the mono output mode or by the second upmixer 220 (upmix high) for the stereo output mode have to be done. The whole delay of the processing is the same either for mono or stereo output and this is also a significant advantage since any subsequent processing operations or preceding processing operations do not have to be aware of whether there is a mono or a stereo output signal.

[0087] Advantageous embodiments remove artifacts and spectral loudness imbalances that stem from having different downmix methods in different spectral bands in the decoded core signal of a system as described in [8] without the additional delay and significantly higher complexity that a dedicated post-processing stage would bring about.

[0088] Embodiments provide, in an aspect, an upmix and a subsequent downmix at the decoder of one (or more) spectral or time parts of a mono signal, that was downmixed using one or more than one downmix method, in order to harmonize all spectral or time parts of the signal.

[0089] The present invention provides, in an aspect, a harmonization of a stereo-to-mono downmix at the decoder side.

[0090] In an embodiment, the output downmix is for a replay device that receives the downmix included in the output representation and feeds this downmix of the output representation into a digital to analog converter and the analog downmix signal is rendered by one or more loudspeakers included in the replay device. The replay device may be a mono device such as a mobile phone, a tablet, a digital clock, a Bluetooth speaker etc.

[0091] It is to be mentioned here that all alternatives or aspects as discussed before and all aspects as defined by independent claims in the following claims can be used individually, i.e., without any other alternative or object than the contemplated alternative, object or independent claim. However, in other embodiments, two or more of the alternatives or the aspects or the independent claims can be combined with each other and, in other embodiments, all aspects, or alternatives and all independent claims can be combined to each other.

[0092] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method, where a block or device corresponds to a method step or a feature of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus.

[0093] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed.

[0094] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0095] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[0096] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier or a non-transitory storage medium.

[0097] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0098] A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein.

[0099] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

[0100] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0101] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0102] In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are advantageously performed by any hardware apparatus.

[0103] While this invention has been described in terms of several embodiments, there are alterations, permutations, and equivalents which fall within the scope of this invention. It should also be noted that there are many alternative ways of implementing the methods and compositions of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations and equivalents as fall within the true spirit and scope of the present invention.

REFERENCES

[0104] [1] ITU-R BS.775-2, Multichannel Stereophonic Sound System With And Without Accompanying Picture, 07/2006. [0105] [2] F. Baumgarte, C. Faller und P. Kroon, “Audio Coder Enhancement using Scalable Binaural Cue Coding with Equalized Mixing,” in 116th Convention of the AES, Berlin, 2004. [0106] [3] G. Stoll, J. Groh, M. Link, J. Deigmöller, B. Runow, M. Keil, R. Stoll, M. Stoll und C. Stoll, “Method for Generating a Downward-Compatible Sound Format”. USA Patent US 2012/0014526, 2012. [0107] [4] M. Kim, E. Oh und H. Shim, “Stereo audio coding improved by phase parameters,” in 129th Convention of the AES, San Francisco, 2010. [0108] [5] A. Adami, E. Habets und J. Herre, “Down-mixing using coherence suppression,” in IEEE International Conference on Acoustics, Speech and Signal Processing, Florence, 2014. [0109] [6] ISO/IEC 23008-3: Information technology—High efficiency coding and media delivery in heterogeneous environments—Part 3: 3D audio, 2019. [0110] [7] S. Bayer, C. Borß, J. Büthe, S. Disch, B. Edler, G. Fuchs, F. Ghido und M. Multrus, “DOWNMIXER AND METHOD FOR DOWNMIXING AT LEAST TWO CHANNELS AND MULTICHANNEL ENCODER AND MULTICHANNEL DECODER”. Patent WO18086946, 17052018. [0111] [8] S. Bayer, M. Dietz, S. Döhla, E. Fotopoulou, G. Fuchs, W. Jaegers, G. Markovic, M. Multrus, E. Ravelli und M. Schnell, “APPARATUS AND METHOD FOR ESTIMATING AN INTER-CHANNEL TIME DIFFERENCE”. Patent WO17125563, 27072017.

APPARATUS, METHOD OR COMPUTER PROGRAM FOR GENERATING AN OUTPUT DOWNMIX REPRESENTATION

Inventors

Cpc classification

Classification Explorer

H04S1/007

ELECTRICITY

Classification Explorer

H04S1/002

ELECTRICITY

Classification Explorer

H04S2420/07

ELECTRICITY

Classification Explorer

G10L21/04

PHYSICS

Classification Explorer

H04S2400/03

ELECTRICITY

Classification Explorer

H04S7/30

ELECTRICITY

Classification Explorer

G10L19/008

PHYSICS

Classification Explorer

H04S2400/05

ELECTRICITY

Classification Explorer

H04S3/002

ELECTRICITY

Classification Explorer

H04S2400/01

ELECTRICITY

Classification Explorer

G10L19/022

PHYSICS

International classification

Classification Explorer

G10L21/04

PHYSICS

Classification Explorer

G10L19/008

PHYSICS

Classification Explorer

G10L19/022

PHYSICS

Classification Explorer

H04S1/00

ELECTRICITY

Classification Explorer

H04S7/00

ELECTRICITY

Abstract

Claims

Description