METHOD AND SYSTEM FOR ENCODING A VIDEO DATA SIGNAL, ENCODED VIDEO DATA SIGNAL, METHOD AND SYSTEM FOR DECODING A VIDEO DATA SIGNAL

Abstract

Video data signals are encoded such that the encoded video data signal include at least a primary and at least a secondary video data signal. The primary and secondary video data signals are jointly compressed. The primary video data signal is compressed in a self-contained manner, and the secondary video data signal is compressed using data from the primary video data signal. The jointly compressed video data signal is split into separate bitstreams, at least a primary bitstream including data for the primary video data signal and at least a secondary bitstream including data for the secondary video data signal, whereafter the primary and secondary bitstreams are multiplexed into a multiplexed signal, and the primary and secondary signals are provided with separate codes.

Claims

1. A method for encoding a video signal including a first signal and a second signal, the method comprising: encoding, by an encoder, the video signal to form an encoded signal; splitting, by a splitter, the encoded signal into separate bitstreams including a first bitstream including data for the first signal and a second bitstream including data for the second signal; and assigning, by the multiplexer, a first code to the first bitstream to form a first coded bitstream, assigning a second code to the second bitstream to form a second coded bitstream, wherein the second code is different from the first code.

2. The method of claim 1, wherein the encoding step encodes the first signal in a self-contained manner without using information from the second signal and encodes the second signal using information from the first signal.

3. The method of claim 1, further comprising outputting the first coded bitstream and the second coded bitstream to one of a three dimensional (3D) decoder system for providing a 3D view and a two dimensional (2D) decoder system for providing a 2D view.

4. The method of claim 1, wherein the first code is decidable by the 2D decoder system to allow the 2D decoder system to extract and decode the first coded bitstream and discard the second coded bitstream.

5. The method of claim 1, wherein the encoding step encodes the first signal such that the first bitstream comprises data with a first set of frames having a first frequency, and encodes the second signal such that the second bitstream comprises data with a second set of frames having a second frequency, the second frequency being higher than the first frequency.

6. The method of claim 1, wherein the encoding step compresses the first signal with a lower quantization factor than the second signal.

7. The method of claim 1, further comprising interleaving frames of the first signal with frames of the second signal to form the video signal.

8. A system for encoding a video signal including a first signal and a second signal, the system comprising: an encoder configured to encode the video signal to form an encoded signal; a splitter configured to split the encoded signal into separate bitstreams including a first bitstream including data for the first signal and a second bitstream including data for the second signal; and a multiplexer configured to assign a first code to the first bitstream to form a first coded bitstream, and assign a second code to the second bitstream to form a second coded bitstream, wherein the second code is different from the first code.

9. The system of claim 8, wherein the encoder is configured to encode the first signal in a self-contained manner without using information from the second signal and encode the second signal using information from the first signal.

10. The system of claim 8, wherein the multiplexer is configured to transmit the first coded bitstream and the second coded bitstream to one of an enhanced two-dimensional (2D) decoder system for providing a three-dimensional (3D) view and a non-enhanced 2D decoder system for providing a 2D view.

11. The system of claim 10, wherein the first code is decidable by the non-enhanced 2D decoder system to allow the non-enhanced 2D decoder system to extract and decode the first coded bitstream and discard the second coded bitstream.

12. The system of claim 8, wherein the encoder is configured to encode the first signal such that the first bitstream comprises data with a first set of frames having a first frequency, and encode the second signal such that the second bitstream comprises data with a second set of frames having a second frequency, the second frequency being higher than the first frequency.

13. The system of claim 8, wherein the encoder is configured to compress the first signal with a lower quantization factor than the second signal.

14. The system of claim 8, further comprising a combiner configure to combine the first signal and the second signal to form a combined signal for encoding by the encoder.

15. The system of claim 8, further comprising an interleaver configure to interleave frames of the first signal with frames of the second signal to form a combined signal for encoding by the encoder.

16. A non-transitory computer readable medium comprising computer instructions which, when executed by a processor, configure the processor to perform the method of encoding of claim 1.

17. A method for decoding, by a 2D decoder system, a video signal including a first coded bitstream and a second coded bitstream, the method comprising: receiving the first coded bitstream having a first code and receiving the second coded bitstream having a second code, the second code being different from the first code; recognizing, by a demultiplexer, at least one of the first code and the second code and demultiplexing first coded bitstream and the second coded bitstream to form a first demultiplexed bitstream and a second demultiplexed bitstream; merging the first demultiplexed bitstream and the second demultiplexed bitstream to form a merged signal; and decompressing by a 2D decoder the merged signal to form a decompressed video signal.

18. The method of claim 17, wherein the first coded bitstream is self-contained, and wherein the decompressing step decompresses the first demultiplexed bitstream included in the merged signal without using information from second demultiplexed bitstream, and the decompresses second demultiplexed bitstream included in the merged signal using data from the first demultiplexed bitstream included in the merged signal.

19. The method of claim 17, wherein the first code is decidable by the 2D decoder system to perform one of (i) the merging step and (ii) extract and decode the first coded bitstream and discard the second coded bitstream.

20. The method of claim 17, further comprising a splitter configured to split the merged signal to provide a left view and a right view.

21. A system for decoding a video signal including a first coded bitstream and a second coded bitstream, the system comprising: a demultiplexer configured to: receive the first coded bitstream having a first code and receiving the second coded bitstream having a second code, the second code being different from the first code, recognize at least one of the first code and the second code, and demultiplex the first coded bitstream and the second coded bitstream to form a first demultiplexed bitstream and a second demultiplexed bitstream; a merger configured to merge the first demultiplexed bitstream and the second demultiplexed bitstream to form a merged signal; and a 2D decoder configured to decompress the merged signal to form a decompressed video signal.

22. The system of claim 21, wherein the first coded bitstream is self-contained such that predicted pictures associated with the first coded bitstream are temporally predicted only from pictures associated with the first coded bitstream, and predicted pictures associated with the second coded bitstream are temporally predicted from the second coded bitstream.

23. The system of claim 21, wherein the first code is decidable by the 2D decoder.

24. The system of claim 21, further comprising a splitter configured to split the merged signal to provide a left view and a right view.

25. A non-transitory computer readable medium comprising computer instructions which, when executed by a processor, configure the processor to perform the method of encoding of claim 16.

Description

[0042] FIG. 1 illustrates an example of a prior art encoding method for 3D video,

[0043] FIG. 2 illustrates a, from the point of view of coding efficiency, improved method.

[0044] FIGS. 3A and 3B illustrate the method for encoding and the method for decoding and an encoder and decoder of the invention.

[0045] FIG. 4 illustrates interleaving of two views, left (L) and right (R).

[0046] FIG. 5 illustrates a so-called dyadic temporal structure using hierarchical B frames.

[0047] FIG. 6 illustrates a diagram for a bit stream splitter.

[0048] FIG. 7 illustrates an example of a single encoded bit stream and its split version.

[0049] FIGS. 8, 9 and 10 illustrate the prior art techniques and the invention.

[0050] FIG. 11 illustrates a situation in which there are three interleaved view signals.

[0051] FIGS. 12, 13, 14A and 14B illustrate an embodiment of the invention for high frequency video.

[0052] FIGS. 15, 16A and 16 B illustrate an embodiment of the invention for high resolution video.

[0053] The Figures are not drawn to scale. Generally, identical components are denoted by the same reference numerals in the Figures.

[0054] FIG. 1 illustrates compression of video information wherein both views are separately treated and compressed. The left (L) and right (R) view video data signals 1 and 2 are each provide two substantially two dimensional images 2D(L) and 2D(R). Both of these signals are encoded and compressed in standard 2D video encoders 3 and 4. The two signals are transmitted, schematically indicated by T in FIG. 1, and received at the receiver side. The receiver decodes the two signals using 2D video decoders 3′ and 4′ and provides two 2D signals 1′ and 2′. These two signals are then ready to be combined in a 3D video display for providing a 3D image. Any 2D video display devices can simply operate using either the left or right view. Although this scheme works, encoding separately the left (L) and right (R) views of a stereo pair practically leads to doubling the bit-rate compared to a mono system (one single view) if one wants to guarantee the same quality. Thus this standard method, although ensuring that a 2D device can display an image, requires doubling of the bit-rate. Page 8 line 33 of original WO-701, where pg 9, line 1 is cited on page 170 of EP-FW

[0055] FIG. 2 illustrates a, form the point of view of coding efficiency, improved method. Prior to encoding the frames of the two views are interleaved in view interleaver VI to provide a combined L+R video data signal. The interleaving provides for a combined signal resembling 2 standard 2D signal. Said signal can be compressed using conventional 2D video compression in a 2S video encoder 5. This compressed signal can then be transmitted. At the receiver side the compressed signal is decompressed in a 2D video decoder 6 and de-interleaved using a deinterleaver VDI. For instance, the 2 views (L and R) can be interleaved as frame pictures before entering the video encoder 5. Hence depending on the temporal reference picture used to predict a given macroblock (MB), the prediction can be either “really” temporal (different times from the same view) or from the other view. This mechanism can be signaled through a so-called “Stereo Video Information” SEI message, to contain information on the performed interleaving so that a stereo decoder at the receiving end is able to distinguish one view from another and perform de-interleaving.

[0056] Although such a method does reduce the bit-rate (by about 25% compared to the method of FIG. 1) if the single encoded stream reaches a conventional 2D receiver (with a 2D decoder and a 2D screen) the displayed video sequence would look ugly as it results from the interleaving of a stereo one. The problem exists even in the case of AVC (Advanced Stereo Coding) with the “Stereo Video Information” SEI message. Indeed, the usage of SEI messages is not mandatory and a compliant decoder can perfectly ignore them. The single bit-stream from the method of FIG. 2 cannot be correctly played on a normal 2D display system, which is a big disadvantage. The method of FIG. 2 is thus not 2D backward compatible. A lack of 2D backward compatibility will greatly discourage people from using 3D video data signals and systems, since the overwhelming majority of displays will, at least during introduction of 3D video, be and remain standard 2D video devices. New methods are preferably backward compatible, i.e., the standard video devices should be able to cope with signals generated by new methods. Page 9 line 21 of original WO-701, cited on page 170 of EP-FW

[0057] The object of the invention is therefore to provide a method which, on the one hand reduces the bit rate compared to fully and separately encoding both views, while, on the other hand, still being having standard video backward compatibility.

[0058] To this end a method for encoding video data signals in accordance with the invention is a method wherein a video data signal is encoded, the encoded video data signal comprising at least a primary and at least a secondary video data signal, wherein the primary and secondary video data signal are jointly compressed, the primary video data signal being compressed in a self-contained manner, and the secondary video data signal being compressed using data from the primary video data signal, the jointly compressed video data signal is split into separate bitstreams, the bitstreams comprising at least a primary bitstream comprising data for the primary video data signal and at least a secondary bitstream comprising data for the secondary video data signal, whereafter the primary and secondary bitstreams are multiplexed into a multiplexed signal, and the primary and secondary signals are provided with separate codes.

[0059] The method of the invention combines the advantages of prior methods while avoiding their respective drawbacks. It comprises jointly compressing two or more video data signals, followed by splitting the single compressed bitstream into 2 or more (primary and secondary) separate bit-streams: a “primary” one that is self contained and is decidable by conventional video decoders, and one or more “secondary” set of frames (so called auxiliary-video- representation streams) that are dependent on the primary bitstream. The separate bitstreams are multiplexed wherein the primary and secondary bit-streams are separate bitstreams provided with separate codes and transmitted. Prima facie it may seem superfluous and a waste of effort to first jointly compress signals only to split them again after compression and provided them with separate codes. In all known techniques the compressed video data signal is given a single code in the multiplexer. Prima facie the invention seems to add an unnecessary complexity in the encoding of the video data signal.

[0060] The inventors have however realized that splitting and separately packaging (i.e., giving the primary and secondary bitstream separate codes in the multiplexer) of the primary and secondary bit stream in the multiplexed signal has the result that, on the one hand, a standard demultiplexer in a conventional video system will recognize the primary bit stream by its code and send it to the decoder so that the standard video decoder receives only the primary stream, the secondary stream not having passed the de-multiplexer, and the standard video decoder is thus able to correctly process it as a standard video data signal, for instance a standard 2D video data signal and/or a standard 50 Hz video data signal, or a signal of base resolution while on the other hand, a specialized system such as a 3D system or a 100 Hz display system or a high resolution video decoder can completely reverse the encoding process and re-create the original enhanced bit-stream before sending it to the for instance a stereo decoder or 100 Hz decoder or a HDTV decoder.

[0061] In an embodiment of the method of the invention a video data signal is encoded, the encoded video data signal comprising a first and at least a second view having frames, wherein the frames of the first and second view are interleaved to form an interleaved video sequence, whereafter the interleaved video sequence is compressed, wherein the frames of the first of the views are encoded and compressed without using frames of the second view, and the frames of the second view are encoded and compressed using frames of the first view, and where the compressed enhanced video data signal is split into a primary and a secondary bit stream each bit stream comprising frames, wherein the primary bit-stream comprises compressed frames for the first of the views, and the secondary bit-stream for the second of the views, the primary and secondary bit-stream forming separate bit-streams, whereafter the primary and secondary bit-stream are multiplexed into a multiplex signal, the primary and secondary bitstream being provided with separate codes.

[0062] FIGS. 3A and 3B illustrate the method and an encoding system and decoding system of the above embodiment of the invention. FIG. 3A illustrate a situation wherein the encoding system 7 of the invention and a decoding system 8 of the invention is used, features of the encoding and decoding system being schematically being shown by the elements in the dotted rectangles. FIG. 3B illustrates a situation wherein a standard decoder is used.

[0063] The frames of the left and right view are interleaved in VI to provide a combined signal. The combined signal resembles a 2D signal. The 2D video encoder 5 encodes and compresses the combined interleaved signal. A special feature of the compression is that the frames of one of the views form a self-contained system, i.e., in compression no information from the other view is used for the compression. The frames of the other view are compressed using information from frames of the first view. The invention departs from the natural tendency to treat two views on an equal footing. In fact, the two views are not treated equally during compression. One of the views becomes the primary view, for which during compression no information is used form the other view, the other view is secondary. The frames of the primary view and the frames of the secondary view are split into a primary bit-stream and a secondary bit stream by Bit Stream Splitter BSS. The coding system comprises a multiplexer MUX which assigns a code, e.g., 0×01 for MPEG or 0×1B for H.264, recognizable for standard video as a video bitstream, to the primary bitstream and a different code, e.g., 0×20, to the secondary stream. The multiplexed signal is transmitted (T). In FIG. 3A the signal is received by a decoding system in accordance with the invention. The demultiplexer recognizes the two bitstreams 0×01or 0×1B (for the primary stream) and 0×20 (for the secondary stream) and sends them both to a Bit Stream Merger (BSM) which merges the primary and secondary stream again and the combined video sequence is decoded by reversing the encoding method in a decoder. The decoding system 8 in combination with a 3D display device forms an example of a display device in accordance with the invention.

[0064] FIG. 3B illustrates the situation when the multiplexed signal is received by a standard video system 9. The standard demultiplexer recognizes the bitstream 0×01or 0×1B as a video data signal and transmits it to the standard decoder. The bitstream 0×20 is not recognizes and rejected, in FIG. 3B schematically indicated by sending the bitstream to the waste basket WB. The signal received by the decoder has a bitrate the standard decoder can handle and is in a form the standard decoder can manage.

[0065] Because the primary stream 0×1B is a fully self-contained signal, the problem associated with the method of FIG. 2 does not occur when a standard 2D video system displays the view of the primary signal. Thus the method of FIG. 3 is 2D backward compatible.

[0066] The method of encoding of the invention allows a reduction of bit rate compared to compressing the two views separately. Thus both a reduction in bitrate as well as 2D backward compatibility are achieved.

[0067] FIG. 4 illustrates interleaving of two views, left (L) and right (R), each comprised of frames 0 to 7 into an interleaved combined signal having frames 0 to 15. Interleaving of signal is a well known process in image processes.

Page 12 Line 15 of Original WO-701, Cited on Page 169 of EP-FW

[0068] FIG. 5 illustrates a so-called dyadic temporal structure using hierarchical B frames. In normal video processing uses is often made of different types of frames I frames B and P frames. Some frames need other frames for encoding. In the scheme depicted in FIG. 5 it is indicated by means of arrows which frames are used in encoding and compressing of other frames. In these schemes I.sub.0/P.sub.0 frames are used in encoding/compression Bi frames, B.sub.1 frames are used in encoding/compressing B.sub.2 frames and B.sub.2 frames are used in encoding/compressing B.sub.3 frames. Thus there is a hierarchy, wherein the lowest levels of hierarchy are the B.sub.3 frames. The B.sub.3 frames need higher level hierarchy frames (I.sub.0/P.sub.0, B.sub.1, B.sub.2) to be encoded (or at the receiver end be decoded) but are themselves not needed to encode or decode higher level hierarchy frames.

[0069] When the interleaving scheme of FIG. 4 is combined with the dependency scheme of FIG. 5, it becomes apparent that the frames of one of the views (either left or right view) correspond with the frames on the B.sub.3 level of hierarchy, and the frames of the other views with the higher level of hierarchy. This view does not need frames of the other view to be decoded and is thus a self-contained signal. In FIG. 5 this corresponds to the fact that no arrows go from a B.sub.3 frame to any B.sub.2, B.sub.1 or I.sub.0/P.sub.0 frame. The frames for the other view (B.sub.3) do not form a self-contained signal, said view needs information of the self-contained view to be decoded. The B.sub.3 frames may be interdependent.

Page 12 Line 33 of Original WO-701, Cited on Page 169 of EP-FW

[0070] By interleaving the frames of the left and right view and then compressing then with a compression scheme which provides for one self-contained signal for one of the views and then splitting the signal again in a primary bit stream (containing the self-contained signal) and a secondary bit-stream (containing the non-self-contained signal) a bit rate reduction is achieved while yet providing a fully operational 2D backward compatible signal. The bit stream splitter creates the primary stream (0×1B) by concatenating all the Access Units (an Access Units comprises the data for at least a frame) of the first view into a primary stream and creates a secondary stream by concatenating all the Access Units (AU) of the second view into the secondary bit stream. The multiplexer then assigns a different code to the primary and secondary stream.

[0071] FIG. 6 illustrates an exemplary diagram for a bit stream splitter. This provides a bit-stream splitting mechanism compatible with an AVC (Advanced Video Coding) coding scheme. In an AVC coding scheme SEI message can be given. A “Stereo Video Information” SEI message contain information on the performed interleaving so that a stereo decoder at the receiving end is able to distinguish one view from another and perform de-interleaving.

[0072] In this embodiment of the invention the SEI message is used inside the encoding system.

[0073] An access unit is taken in step 60.

[0074] In a first step 61 it is checked whether the current access unit comprises an SEI message.

[0075] If an access unit does not contain an SEI message, in this particular example the information on the set of frames to which the access unit belongs is deduced from previous received information. For instance, if the previous received information was: “if one access unit belongs to set A, the next belongs to set B”, it is not needed to provide each access unit with SEI information.

[0076] If the access unit does contain an SEI message the validity of the SEI message is checked with regards to a previous SEI message in step 62.

[0077] The SEI messages give information on interleaving which usually is a known sequence. If the SEI message is invalid there is an error 63.

[0078] If the SEI message is valid, the next step 64 is taken.

[0079] For each access unit the relevant interleaving information is now available, either by means of the fact that there was no SEI message, in which case there was no change in SEI message with respect to a previous access unit, or the access unit has a valid SEI message.

[0080] In the next step 64 it is checked whether the Access Unit forms part of the primary view, if so it is appended in step 65 to the primary view bit stream, if not, it is appended to the secondary view video bit-stream in step 66. It is obvious that this sequence could be reversed. Once an access unit is dealt with and appended to either the primary or secondary bit-stream, the next access unit is gotten in step 67 and the process is repeated. It is remarked that FIG. 6 is an exemplary embodiment of a splitter. In various ways the encoded signal may be split. The information on splitting could be implicit, for instance if a fixed sequence is used for encoding, or explicit, for instance by using SEI messages of NAL codes or a mixture thereof. The various abbreviations in the access unit 0 and 8, although not of relevance for the present invention, except for the SEI abbreviation, stand for: [0081] SPS: Sequence Parameter Set [0082] PPS: Picture Parameter Set [0083] IDR: Instantaneous Decoder Refresh [0084] Slice: Group of MB (macroblock 16*16 pixels) [0085] SEI: Supplemental Enhancement Information

[0086] FIG. 7 illustrates an example of a single encoded bit stream and its split version. It is to be noted that the access units 0 and 8 which comprise the video information are made part of the primary bit-stream. All access units for the primary bitstream (AU8, AU0, AU4, AU2, AU6) have an SEI message corresponding to one of the streams, the self-contained signal, the other access units have an SEI message pertaining to the secondary bitstream.

[0087] The two bit-streams are kept synchronous at systems level, for instance thanks to DTS (Decoding Time Stamp) flags in an MPEG-2 transport stream (broadcast application) or in an MPEG-4 File Format (file storage applications). A syntax element at systems level may be used to indicate that the secondary bit stream is dependent on the primary bit stream.

[0088] It is remarked that the secondary stream is no longer a valid stream by itself. Often this will not be a problem, Should problem occur, one can insert empty frames into the secondary stream which will hardly increase the bit rate. Before the merging process these empty frames will have to be removed first.

[0089] In embodiments regular changes of primary and secondary signal may be made. In the method the two views are not treated equally; the first view is a self-contained view, whereas the second view is derived from the first. This could lead to a small difference in quality between the left end right view which may, in time, lead to slightly different behavior of the left and right eye receiving the images. By regularly changing the primary view from left to right, for instance at scene changes, this can be avoided.

[0090] In embodiments the quantization factor of the compression may differ for the primary and secondary bit streams, in the 3D example the primary and secondary views. Especially when there are more secondary views, as will be explained below, it may be useful to assign more bandwidth to the primary view than to the secondary view.

[0091] FIGS. 8 to 10 illustrate schematically the prior art and the invention. In the prior art, see FIG. 8, a video encoder, in this example the AVC (Advanced Video coding) video encoder Venc encodes a video data signal, an audio encoder encodes an audio signal. The multiplexer MUX generates a multiplexed signal wherein the video bitstream gets a code e.g., 0×1B and the audio bitstream a code 0×03. The 2D demultiplexer 2D demux extracts from the multiplexed signal the two composing bitstreams and send the video bitstream to a video decoder and the audio bitstream to an audio decoder. The video decoder thus gets the whole of the video bitstream whether it can handle it or not.

[0092] FIG. 9 illustrates the situation when use is made of the method of encoding and the encoding system of the invention at the encoding side, and a standard video device at the decoding side.

[0093] After encoding the splitter splits the encoded stream into a primary stream prim and a secondary stream sec. The multiplexer mux generates a multiplexed signal comprising a bitstream 0×1B for the primary view, and a separate bitstream 0×20 for the secondary view and, as in the case of FIG. 8, an audio bitstream 0×03.

[0094] The standard device comprises a demultiplexer which extracts form the multiplexed signal the primary bitstream 0×1B since it recognizes this bitstream by its code; it rejects the bitstream 0×20. The video decoder receives the primary bitstream 0×1B. Since this is a self-contained bitstream with a “normal” bit rate, the video decoder is able to decode the bitstream without great difficulty. Thus the encoding method and system is backward compatible.

[0095] FIG. 10 illustrates a situation wherein use is made of a decoding system in accordance with the invention.

[0096] At the decoder side, the decoder comprises a 3D demultiplexer 3D demux. This demultiplexer sends the audio bitstream 0×03 to an audio decoder, extracts the two video bitstreams 0×1B (the primary bitstream) and 0×02 (the secondary bitstream) from the multiplexed signal and sends the two video bitstreams to their respective inputs at a Bit Stream Merger (BSM) which merges the primary and secondary stream again. The merged video stream is send to a decoder which decodes the merged bitstream using a reverse coding method providing a 3D video data signal.

[0097] Thus, a specialized 3D video decoding system is able to decode a 3D video data signal, while yet a standard 2D video decoding system is also able to provide a high quality image.

[0098] In the examples given above the enhanced video data signal was a 3D signal comprising two views, a left and a right view.

[0099] The invention is not restricted to this situation, although it is highly suitable for this situation.

[0100] The invention is also useful when, instead of two views, a multiview signal is generated.

[0101] FIG. 11 illustrates a situation in which there are three interleaved view signals, a first central view for which the bitstream is self-contained, comparable to the self-contained L view bitstream in FIG. 5, and two secondary view bit-streams B.sub.3 and B.sub.3′, each of which bit-streams is encoded and compressed in dependence on the central view data.

[0102] In this example three are three views, a primary view, for instance a central view, and a number of secondary views, for instance a left and a right view. For the central view a self-contained bitstream is generated. For the two secondary views, for instance a left and right view, secondary, not self-contained, bit streams are generated.

[0103] In this embodiment it may be useful to use in the compression a different quantization factor for the central view frames than for the secondary view frames especially if there are many secondary views.

[0104] This embodiment is useful for generation of multiview signal using a MVC (multi-View encoding) encoder. In the example of FIG. 11 three views, a single primary and two secondary views are generated. More than two secondary views may be generated depending on the multiplicity of views being generated. In complex embodiments more than one primary view and more secondary views may be generated.

[0105] A further embodiment of the invention is exemplified in FIG. 12.

[0106] Another category of enhanced video data signals in one in which a higher frequency (for instance 100 Hz) signal is to be generated. The problem described above for stereo signals play also for such video data signal. The majority of video display devices operates at standard frequencies and the decoders are designed for such frequencies.

[0107] FIG. 12 illustrates a signal that is produced by a high speed camera CAM 100 Hz comprises a large number of frames at a 100 Hz frame rate. Such a signal is not suitable for display in standard 50 Hz TV systems. It may be thought of comprising a normal frame distribution and a high speed frame distribution.

[0108] In a specialized video decoder the frames may be compressed in a scheme much the same as the scheme shown in FIG. 5. This is shown in FIG. 13. The signal originating from the high speed camera may be seen as providing an signal of two sets of interleaved frames (namely the odd and even numbered) and these two are compressed wherein the even numbered frames are compressed to form a self-contained set of frames at 50 Hz frequency, the odd numbered frames are compressed in dependence of the set of even numbered frames. This example is in so far different from the previous example in that the interleaving of the sets frames is inherent in the signal that is provided to the encoder. However, the sample principles apply. The even frames will form the primary bit-stream, the odd frames the secondary bitstream or vice versa. Of course when an even higher frequency signal is provided for instance because a 200 Hz camera is used, the number of sets could be made four, wherein a single primary 50 Hz bit-stream is generated and three secondary bit-streams.

[0109] FIGS. 14A and 14B show encoding and decoding schemes for the embodiment schematically shown in FIGS. 12 and 13. These schemes are in large part similar to the schemes of FIGS. 3A and 3B. In FIG. 14A a situation is schematically indicated wherein a specialized decoder is used. A 100 Hz camera CAM 100 Hz provides a signal as schematically shown in FIG. 12, in an SVC (Scalable Video Coding) encoder an encoding scheme as schematically shown in FIG. 13 is performed. The camera thus provides an interleaved (namely even and odd frames with 100 Hz) signal, which is received by the encoding system 7, the encoding system 7 comprising an encoder for compressing one of the sets comprised in the interleaved signal provided by the 100 Hz camera into a self-contained set of compressed frames. In the splitter BSS the frames are appended to either a primary bitstream 0×1B or a secondary bitstream 0×20 and in the multiplexer the two bitstreams are given a differing codes. The reverse is done at the decoder 8 side. STB stands for Set Top Box. The final decoded signal is fed to a 100 Hz TV. A display device according to this embodiment of the invention comprises decoding system 8 and a 100 Hz TV. A recording device in accordance with this embodiment of the invention comprises encoder 7 and a camera.

[0110] FIG. 14B illustrates the situation when a 50 Hz TV with a corresponding conventional decoding system is used. The standard demultiplexer will recognize and accept the primary bitstream and send it onward towards the 50 Hz TV. This bitstream is self-contained and has the proper format and frequency and will provide an adequate image quality on the 50 Hz TV. The secondary bitstream is rejected and does not interfere with the normal operation of the 50 Hz TV system.

[0111] In the above embodiment the SVC stream is split along the frequency (time axis). SVC also allows splitting frames along the resolution and/or the quantization axis (SNR, CGS (coarse Grain scalability), FGS (Fine grain scalability)) and/or color sampling axis (4:4:4,4:2:0,4:2:2). In such embodiments the problem described above i.e., the fact that a standard video decoder has problems handling the incoming bitstream also occur, in other words compatibility problems occur.

[0112] Within the framework of the invention, in its broadest sense, the video stream is split into at least two bit streams, a primary and secondary video stream (see FIG. 15), wherein the primary video stream is self-contained and the secondary video stream need the primary video stream for decoding, and in a multiplexer the primary video stream is provided with a standard code (0×1B) and the secondary video stream is provided with a differing code (0×20). The video data signal may be split along the resolution axis wherein the primary bit stream in the multiplexed signal comprises the data for a base layer, and the secondary bit stream in the multiplexed signal which is provided with a different code comprises the video data for an enhancement layer or enhancement layers.

[0113] The code provided to the primary stream is a standard code (e.g., 0×1B or 0×01) and is thus decidable by any normal standard non-scalable MPEG (for 0×01) or H.264 video decoder (for 0×1B) 9, whereas a specialized decoder 8 in accordance with the invention can draw full advantage of the scalability encoding. FIG. 16A and FIG. 16B provide an embodiment of the invention which can be applied to supply an SDTV video data signal to an SDTV (standard definition TV) as shown in FIG. 16B and an HDTV video data signal to a HDTV (High Definition TV) as shown in FIG. 16A.

[0114] The invention is also embodied in any computer program product for a method or device in accordance with the invention. Under computer program product should be understood any physical realization of a collection of commands enabling a processor -generic or special purpose-, after a series of loading steps (which may include intermediate conversion steps, like translation to an intermediate language, and a final processor language) to get the commands into the processor, to execute any of the characteristic functions of an invention. In particular, the computer program product may be realized as data on a carrier such as e.g,. a disk or tape, data present in a memory, data traveling over a network connection -wired or wireless-, or program code on paper. Apart from program code, characteristic data required for the program may also be embodied as a computer program product.

[0115] The invention also relates to devices comprising an encoding system in accordance with the invention, such as 3D video recording devices or high resolution video recording devices.

[0116] The invention also relates to display devices comprising a decoding system in accordance with the invention. Such devices may for instance be 3D video display devices or HDTV display device or display devices with increased resolution.

[0117] The invention furthermore relates to a multiplexed video data signal comprising at least two related video data signals with separate codes (0×01, 0×1B, 0×20), wherein a first video data signal (0×01, 0×1B) is a self-contained video data signal and at least a second video data signal (0×20) is not. Using a demultiplexer it is easy to treat the two related, but different, video data signals differently without having to use a decoder to make the distinction. For standard devices, such as standard 2D video display devices or SDTV device, the first selfcontained signal can be forwarded to the decoder, without overloading the decoder with the second signal. Specialized video system can make full use of the data in the two video data signals.

[0118] In short, the invention can be described as follows:

[0119] Video data signals are encoded such that the encoded video data signal comprises at least a primary and at least a secondary video data signal. The primary and secondary video data signal are jointly compressed. The primary video data signal is compressed in a self-contained manner, and the secondary video data signal is compressed using data from the primary video data signal. The jointly compressed video data signal is split into separate bitstreams, at least a primary bitstream comprising data for the primary video data signal and at least a secondary bitstream comprising data for the secondary video data signal, whereafter the primary and secondary bitstreams are multiplexed into a multiplexed signal, and the primary and secondary signals are provided with separate codes.

[0120] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

[0121] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim.

[0122] The word “comprising” does not exclude the presence of other elements or steps than those listed in a claim. The invention may be implemented by any combination of features of various different preferred embodiments as described above.