PROCESS AND SYSTEM FOR ENCODING AND PLAYBACK OF STEREOSCOPIC VIDEO SEQUENCES

Abstract

A method for decoding a compressed image stream, the image stream having a plurality of frames, each frame consisting of a merged image including pixels from a left image and pixels from a right image. The method involves the steps of receiving each merged image; changing a clock domain from the original input signal to an internal domain; for each merged image, placing at least two adjacent pixels into an input buffer and interpolating an intermediate pixel, for forming a reconstructed left frame and a reconstructed right frame according to provenance of the adjacent pixels; and reconstructing a stereoscopic image stream from the left and right image frames. The invention also teaches a system for decoding a compressed image stream.

Claims

1. A system for displaying stereoscopic video sequences, the system comprising: a dual input head mountable display; an output controller in communication with the dual input head mountable display, the output controller being operable to: add additional left pixels into each left image of a left image sequence, the additional left pixels being created using spatial interpolation based at least on a plurality of other left pixels in the respective left image, add additional right pixels into each right image of a right image sequence, the additional right pixels being created using spatial interpolation based at least on a plurality of other right pixels in the respective right image, adjust an output frame rate of the left image sequence to correspond with a display frame rate of the head mountable display by inserting, at certain locations in the left image sequence, a modified left image at a position in the left image sequence immediately after a selected left image of the left image sequence, the modified left image being created using movement anticipation based at least in part on the selected left image, adjust an output frame rate of the right image sequence to correspond with a display frame rate of the head mountable display by inserting, at certain locations in the right image sequence, a modified right image at a position in the right image sequence immediately after a selected right image of the right image sequence, the modified right image being created using movement anticipation based at least in part on the selected right image, generate an output left video signal comprised of the adjusted left image sequence, generate an output right video signal comprised of the adjusted right image sequence, and send the output left video signal and the output right video signal to the dual input head mountable display.

2. The system of claim 1, further comprising a rate controller that is operable to monitor clock signals to detect clock signal variations and to instruct the output controller to adjust the output frame rate based on the detected clock signal variations.

3. The system of claim 1, further comprising an anti-flicker filter that is operable to decrease colors of at least one pixel of at least one image of the left image sequence and the right image sequence when at least one color of the at least one pixel is above a certain value.

4. The system of claim 1, wherein the head mountable display is a first display operating in a stereoscopic 3D viewing mode, and wherein the output controller is operable to simultaneously drive one of the output left video signal and the output right video signal to a second display for display in a conventional 2D display mode.

5. The system of claim 1, further comprising a digital input connectable to the internet, the digital input being configured to receive a stereoscopic video sequence over an internet connection, wherein the output controller generates the left image sequence and the right image sequence based on the received stereoscopic video sequence.

6. The system of claim 1, wherein each left image in the left image sequence is generated from a left image mosaic and each right image in the right image sequence is generated from a right image mosaic, each left image having a larger number of pixels than the corresponding left image mosaic and each right image having a larger number of pixels than the corresponding right image mosaic.

7. The system of claim 6, wherein each left image includes a plurality of additional left pixels generated from the corresponding left image mosaic using interpolation and each right image includes a plurality of additional right pixels generated from the corresponding right image mosaic using interpolation.

8. The system of claim 1, wherein the head mountable display is also operable to display 2D format video.

9. A method for displaying stereoscopic video sequences on a dual input head mountable display, the method comprising the steps of: adding additional left pixels into each left image of a left image sequence, the additional left pixels being created using spatial interpolation based at least on a plurality of other left pixels in the respective left image, adding additional right pixels into each right image of a right image sequence, the additional right pixels being created using spatial interpolation based at least on a plurality of other right pixels in the respective right image, adjusting an output frame rate of the left image sequence to correspond with a display frame rate of the head mountable display by inserting, at certain locations in the left image sequence, a modified left image at a position in the left image sequence immediately after a selected left image of the left image sequence, the modified left image being created using movement anticipation based at least in part on the selected left image; adjusting an output frame rate of the right image sequence to correspond with a display frame rate of the head mountable display by inserting, at certain locations in the right image sequence, a modified right image at a position in the right image sequence immediately after a selected right image of the right image sequence, the modified right image being created using movement anticipation based at least in part on the selected right image; generating an output left video signal comprised of the adjusted left image sequence; generating an output right video signal comprised of the adjusted right image sequence; and sending the output left video signal and the output right video signal to the head mountable display.

10. The method of claim 9, further comprising the step of monitoring clock signals to detect clock signal variations and to adjust the output frame rate based on the detected clock signal variations.

11. The method of claim 9, further comprising the step of reducing flickering by decreasing colors of at least one pixel of at least one image of the left image sequence and the right image sequence when at least one color of the at least one pixel is above a certain value.

12. The method of claim 9, wherein the head mountable display is a first display operating in a stereoscopic 3D viewing mode, and further including the step of simultaneously sending one of the output left video signal and the output right video signal to a second display for display in a conventional 2D display mode.

13. The method of claim 9, further comprising the steps of: receiving, via a digital input connectable to the internet, a stereoscopic video sequence over an internet connection; and generating the left image sequence and the right image sequence based on the received stereoscopic video sequence.

14. The method of claim 9, wherein each left image in the left image sequence is generated from a left image mosaic and each right image in the right image sequence is generated from a right image mosaic, each left image having a larger number of pixels than the corresponding left image mosaic and each right image having a larger number of pixels than the corresponding right image mosaic.

15. The method of claim 14, wherein each left image includes a plurality of additional left pixels generated from the corresponding left image mosaic using interpolation and each right image includes a plurality of additional right pixels generated from the corresponding right image mosaic using interpolation.

16. The method of claim 9, wherein the head mountable display is also operable to display 2D format video.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG. 1a is a schematic representation of a system according to the present invention for the compression encoding of two planar image sequences into a stereoscopic image sequence to be recorded onto a data storage medium or broadcast on a single channel.

[0052] FIG. 1b is a schematic representation of a system according to the present invention, for expansion decoding and playback of a stereoscopic image sequence previously encoded with a system such as represented in FIG. 1a,

[0053] FIG. 2a is a schematic representation of a portion of a digitized image 60 topologically separated into two complementary mosaics of picture elements, forming fields A and B of a merged image 60.

[0054] FIG. 2b is a schematic representation of a portion of two digitized images 50 and 50′, topologically decimated into reduced mosaics respectively forming field A and field B of a merged image 60.

[0055] FIG. 2c is a schematic representation of a process for the spatial interpolation of a pair of decimated images comprised in a merged image 60, to rebuild two full-definition images 72 and 72′.

[0056] FIG. 2d is a schematic representation of a time-interpolation process for the creation of a new image 52 from two images 50.4 and 50.5 with a time delay.

[0057] FIG. 3a is a schematic representation of a first embodiment of a compression process according to the present invention, for compression encoding two planar image sequences into a stereoscopic combined image sequence.

[0058] FIG. 3b is a schematic representation of a first embodiment of a decompression process according to the present invention, for reconstruction and temporal expansion of a stereoscopic image sequence previously encoded according to a process such as represented in FIG. 3a.

[0059] FIG. 4a is a schematic representation of a second embodiment of a compression process according to the present invention, for compression encoding two planar image sequences into a stereoscopic image sequence.

[0060] FIG. 4b is a schematic representation of a second embodiment of a compression process according to the present invention, for reconstruction and temporal expansion of a stereoscopic image sequence previously encoded according to a process such as represented in FIG. 4a.

[0061] FIG. 5 is a schematic representation of a decoder according to a preferred embodiment of the invention.

[0062] Similar reference numerals refer to similar parts throughout the various Figures.

DETAILED DESCRIPTION OF THE DRAWINGS

[0063] Preferred embodiments of the method and associated systems for encoding and playback of stereoscopic video sequences according to the present invention will now be described in detail referring to the appended drawings.

[0064] Referring to FIG. 1, there is illustrated a typical system set-up according to the present invention, for the compression encoding of two planar image sequences into a stereoscopic image sequence. A first and a second source of image sequences represented by cameras 3 and 6 are stored into common or respective digital data storage media 4 and 7. Alternatively, image sequences may be provided from digitised movie films or any other source of digital picture files stored in a digital data storage medium or inputted in real time as a digital video signal suitable for reading by a microprocessor based system. Cameras 3 and 6 are shown in a position wherein their respective captured image sequences represent different views with a parallax of a scene 100, simulating the perception of a left eye and a right eye of a viewer, according to the concept of stereoscopy. Therefore, appropriate reproduction of the first and second captured image sequences would enable a viewer to perceive a three-dimensional view of scene 100.

[0065] Stored digital image sequences, typically available in a 24 fps digital Y U V format such as Betacam 4:2:2 (motion pictures), are then converted to an RGB format by processors such as 5 and 8 and fed to inputs 29 and 30 of moving image mixer unit 1, representing the main element of the encoding system of the present invention. It should be noted however the two image sequences can alternatively be converted on a time-sharing basis by a common processor, in order to reduce costs. Mixer 1 compresses the two planar RGB input signals into a 30 fps stereo RGB signal delivered at output 31 and then converted by processor 9 into a betacam 4:2:2 format at output 32 and in turn compressed into a standard MPEG2 bit stream format by a typical circuit 10. The resulting MPEG2 coded stereoscopic program can then be recorded on a conventional medium such as a Digital Video Disk (DVD) 11 or broadcasted on a single standard channel through, for example, transmitter 13 and antenna 14. Alternative program transport media could be for instance a cable distribution network or internet.

[0066] Turning now to FIG. 1b, there is illustrated a typical system according to the present invention for the decoding and playback of the stereoscopic program recorded or broadcasted using the system of FIG. 1. The stereo DVD 11 (3DVD) comprising the compressed information from the first and second images sequences, is played by a conventional player 15 of current manufacture, delivering a NTSC serial analog signal to the input 28 of the stereo image decoder 2, the main element of the decode/playback system of the present invention. Alternatively, any ATSC DTV signal in its analogue or digital format can be accepted.

[0067] Decoder 2 produces a synchronized pair of RGB signals at outputs 23 and 24, representative of the first an second image sequences, to drive a dual input stereoscopic progressive display device such as a head mounted display (HMI)) 16. Further, decoder 2 produces a time-sequenced stereo RGB signal at output 25, to supply a single input progressive display device such as projector 17, LCD display 22, CRT monitor or a SDTV or HDTV 21, whereby images from the first and second image sequences are presented in an alternating page flipping mode. Alternatively, the stereo RGB signal from output 25 may be converted into an interlaced NTSC signal to be reproduced by an analog CRT television set or in other stereoscopic formats (ex: column interleaved for autostereoscopic lenticular displays). Also, decoder 2 may be so internally configured to output the stereo RGB signal at one of RGB outputs 23 or 24, thus eliminating output 25.

[0068] Decoder 2 further produces a sync-timing signal at output 26 to drive an infrared shutter spectacle driver 20, driving spectacles 19, Shutter spectacles 19 can be worn by a viewer to view a three-dimensional program projected for instance on screen 18 by projector 17 fed by stereo output 25, by enabling the viewer to alternately see an image from the first image sequence with one eye and an image from the second image sequence with his second eye.

[0069] As stated in the foregoing description, the two original image sequences contain too much information to enable direct storage onto a conventional DVD or broadcast through a conventional channel using the MPEG2 or equivalent multiplexing protocol handling information at a rate of 30 fps. Therefore mixer 1 carries out a decimation process to reduce each picture's information by half.

[0070] The spatial decimation carried out by mixer 1 will now be described with reference to FIGS. 2a and 2h,

[0071] FIG. 2a illustrates a portion of an image 50 as defined by a RGB video signal processed by mixer 1 and decoder 2. As can be seen, image 50 is comprised of a plurality of pixels (alternating full and empty dots). In the RGB format, each pixel is defined by a vector of 3 digital numbers respectively indicative of the red, green and blue intensity. The present invention makes use of a presumption that three adjacent pixels have intensities that are not drastically dissimilar, either in the horizontal direction or the vertical direction. Consequently, the present invention advantageously provides for a compression, or decimation, or separation process which reduces the size of an image by 50% without unduly corrupting the image.

[0072] In a schematic representation, FIG. 2a illustrates how this is achieved. As mentioned above, image 50 is comprised of a plurality of pixels. The series of pixels (indicated by a solid dot) starting with the first pixel of the image (pixels one of line one), followed by the third one and so forth throughout the image, from the left to the right of each row and from the upper line to the last one are placed in one half of a frame buffer, in order to make mosaic A. The remaining pixels, viz. the even-numbered pixels, (indicated by the empty dots) are placed as mosaic B. In the example of FIG. 2a, the two complementary mosaics of image 50 are shown as being respectively stored in mosaic A and mosaic B of a common merged image 60. In practice however, this “separation” is preferably automatically done with appropriate hardware and software which could, for example, only “read” the odd-numbered or even-numbered pixels and directly placed them in a frame buffer.

[0073] As better illustrated in FIG. 2b, basically, images are spatially compressed by 50% by keeping only mosaic A of images of the first sequence (ex. left eye sequence) such as 50, and mosaic B of the images of the second sequence (ex. right eye sequence) such as 50′. Keeping mosaics of different types for each sequence promotes higher fidelity at playback when first and second sequences represent different views of a same scene. Alternatively, spatial compression could be carried out by saving mosaic A for even numbered images and mosaic B for odd numbered images, for both input sequences, so that two successive images of the same eye would be rebuilt from mosaics of different types and potentially stored in the same compressed frame.

[0074] The above operation is accomplished by inputting the data of one pixel at a time in a three-pixel input buffer 55 as shown in FIG. 2b, Pixel information is then transferred into the appropriate memory location of one or more frame buffer(s), each serving to build a different merged image. Mosaics from different input images are concatenated side by side by pair to form two adjacent fields (left field and right field) of a new series of merged frames of the original size such as 60. In the example illustrated at FIG. 2b, image 50′ is currently being processed, while processing of image 50 is completed, yielding a complete type A mosaic stored in the left field (A) of merged image 60. It should be pointed out however that the merged frames do not necessarily comprise an image from the first sequence and an image from the second sequence, or images captured at the same time, as will be apparent from the detailed description of preferred embodiments of the compressing/encoding (mixing) method. As a matter of fact, in the example of FIG. 2a, field A and field B of the merged image 60 are respectively filled with mosaic A and mosaic B from a same image 50. While that situation has been chosen to simplify the illustration and corresponds to an actual situation according to one of the embodiments of the invention contemplated herein, it shall be deemed that merged images such as 60 could comprise mosaics originating from any of the inputted images. That side-by-side compressed transport format is mostly transparent and unaffected by the compression/decompression processing characterizing the MPEG2 main viewprotocol downstream in the process.

[0075] Upon decoding of the merged images, reconstruction of the complete images is carried out by spatially interpolating missing pixels from the compressed half-size images (mosaics) located in the fields of the merged images such as 60. As illustrated in FIG. 2c, this is preferably accomplished in real time when each pixel of an input merged frame 60 decoded in decoder 2 is being transferred to or from memory. As mentioned above, the underlying premise of the system of the present invention is that values of adjacent pixels are not so dissimilar. Consequently, in order to reconstruct an image from a mosaic, adjacent pixels are weighted in order to interpolate a missing pixel.

[0076] In a preferred embodiment of the invention, data of one pixel at a time is stored into a three-pixel input buffer 65. As shown, the three pixels of the shadowed portion of input image 60 have been stored in input buffer 65, two adjacent pixels from the same mosaic being identified as P.sub.i and P.sub.i+1. Data of a third pixel P.sub.i is then calculated as being the arithmetic mean of each of the 3 components of the RGB vectors of adjacent pixels (P.sub.i and P.sub.i+1). For example, if pixel P.sub.i has an intensity vector of (10,0,30) and pixel P.sub.i+1 has an intensity vector of (20,0,60), then, pixel P.sub.j will be calculated as being (15,0,45), Therefore, the mean of two identical pixels is another identical pixel. That calculated (topologically interpolated) pixel replaces the missing pixel decimated upon creation of the mosaics from original image sequences such as 50.

[0077] The original pixels and the interpolated pixels are then stored in appropriate memory locations of a frame buffer where the corresponding image is to be reconstructed (image 72 in the present example). Passed the centre of each hue of the merged frame 60 (entering the right field), data is stored into a second frame buffer 72′, to rebuild the image from the mosaic stored in the right hand field of the stereo image. The process is followed line by line from left to right, until the two images are spatially reconstructed in their respective buffer.

[0078] Although the above embodiment interpolates a pixel as being the mean of two adjacent pixels of a mosaic, the invention provides for a weighting of more than two pixels. For example, if pixel P.sub.i is to be interpolated, then the two or three preceding and following pixels from the mosaic can be used with difference coefficients. More specifically, referring to FIG. 2c, P.sub.j can be interpolated as 0.6PET.sub.i+1+0.6P.sub.i−1−0.1P.sub.i+2−0.1P.sub.i−2. Of course a variety of different coefficients and formulae may be used according to preferred results. Furthermore, instead of performing horizontal interpolation, vertical interpolation may be performed following the same process, or a combination of both horizontal and vertical interpolation.

[0079] In order to assure flickerless viewing, the decoding method further comprises temporal expansion of image sequences as will be described in detail in the following description. When frame buffers are completely filled to provide complete rebuilt or temporally interpolated images (no more than four frame buffers are required in any embodiment of the reconstruction process and system), they may be read according to different modes to provide different types of desired output signals.

[0080] A first embodiment of the mixing method carried out by mixer 1 according to the present invention is schematically represented in FIG. 3a of the appended drawings and will now be described in detail.

[0081] A first sequence of images in RUB 24 fps format 50, identified as L1 to L4, is first time expanded by 25% to form a. 30 fps sequence of images such as 51, by the creation and insertion of a new image 52 after every fourth image of the original sequence 50. New image 52 is time-interpolated from the topological information of the immediately preceding and following images (#4 and #5 of original sequence 50). Each pixel of the new image 52 is calculated as the arithmetic mean of the corresponding pixel in the precedent and following image, in a manner similar to the spatial interpolation technique explained in the foregoing description. FIG. 2d provides a specific illustration of the time-interpolation process where a new image 52 is created from two time-successive images 50.4 and 50.5 of input image sequence 50. Creation of new images in the present invention is generally accomplished according to that technique to provide improved fluidity at playback, as compared for instance to simple repetition of frames, which would require less processing power. Alternatively, any known method, such as movement anticipation based methods, could be used for performing time-interpolation of images.

[0082] Images of the time-expanded sequence 51 are then spatially compressed according to the technique illustrated in FIG. 2b and described in detail in the foregoing description, to form mosaics represented by a new sequence 53. Similarly, the second input image sequence 50′ is time expanded into the 30 fps sequence 51′ and spatially compressed into mosaics represented by sequence 53′. In this specific embodiment, pairs of compressed images (mosaics) from the first sequence 53 and the second sequence 53′ respectively are then concatenated to form a left field and a right field of merged images of a 30 fps RUB sequence 60. This sequence 60 can then be encoded in stream 62 for transmission to a remote location.

[0083] It is worth mentioning that in spite of the schematic diagram of FIG. 3a, the processing of the image sequences is preferably not carried out in parallel and with long sequences of images. Actually, only a few images are being buffered at a given time to enable temporal interpolation, and images are alternatively imported from the first and the second (left and right) sequences and processed on a pixel by pixel basis more like the process steps represented in FIGS. 2b and 2d.

[0084] The decoding and reconstruction carried out by decoder 2 according to the first embodiment of the present invention will now be described by referring to FIGS. 3b, 2c and 2d.

[0085] In the example shown in FIG. 3b, stereo MPEG signal 62 is read from a DVD and is converted by the DVD player 15 (FIG. 1b) into an analog NTSC signal 70 inputted by the decoder 2. NTSC signal 70 is first converted into an RGB format to recuperate merged images such as in sequence 60. Reconstruction of first and second original sequences 50, 50′ can then be started by spatially interpolating and separating mosaics from the left and right fields of the merged images from sequence 60 on a pixel by pixel basis, as previously described with reference to FIG. 2c, to form 30 fps decompressed buffered images such as 72 and 72′. Therefore, spatial interpolation and separation are actually carried out simultaneously. Sequences of RGB images 72 and 72′ could be directly outputted and displayed on a dual input device to reproduce the original programs or stereoscopic program signals at a 60 fps (30 per eye) presentation rate. Further processing could also be performed to present the image sequences in an interlaced mode or in sequence (page flipping mode), anaglyphic, column interleaved, conventional 2D mode, etc. on a plurality of existing single input display devices.

[0086] However, in order to enable comfortable and fatigue free viewing, decoder 2 significantly reduces flicking by providing output signals at a typical rate of 36 full definition frames per eye per second, while satisfying results may be obtained at 30 fps per eye with high definition frames to match refresh rates of SDTV or HDTV for instance. On the other hand, output signals up to 120 fps (60 images per second per eye) can be provided by decoder 2 for a very high fidelity, reproduction, such an output being compatible however with display devices such as DLP projectors and a limited number of high end devices. By experience, a playback rate of 72 fps provides very good results, provided image quality is preserved throughout the coding/decoding process as contemplated herein, such a frequency being a standard for most display devices currently encountered in home theatre systems.

[0087] Therefore, the playback process carried out by decoder 2 preferably includes a further step to increase the presentation rate of sequences 72 and 72′. Additional images are inserted at regular intervals in the image sequence, using the temporal interpolation technique already explained in the foregoing description of the mixing process referring to FIGS. 3a and 2c, The position of insertion can be carefully controlled through the frame number information stored in blank lines of input sequence 60 at mixing. Alternatively, images from sequences 72 and 72′ can be repeated (read twice) to increase the rate at presentation. For instance every image of the sequences could be read twice to double the rate of presentation.

[0088] In the example illustrated in FIG. 3b, one new intermediate image pair 73, 73′, is time-interpolated using information from images #2 and #3 of sequences 72 and 72′ respectively and inserted between images #2 and #3 to thereby increase the rate of the resulting sequences 74 and 74′ to 36 fps (total of 72 fps for the stereoscopic program). The process is partly illustrated in FIG. 2c, where images 42 and 43 of sequence 72 are identified as 72.2 and 72.3. Alternatively, further images could be time-interpolated and inserted to provide a rate of say 48 frames per second per sequence for a total of 96 fps. A rate of 60 fps per sequence (eye) (total 120 fps for the stereoscopic program) is also an interesting case where no interpolation is required. All of the images of sequences 72 and 72′ are merely duplicated to double the number of images. At playback, shutter spectacles are driven at a rate of 120 Hz and all the images of a given sequence are presented twice to the corresponding eye in 1/30 s. An unsurpassed clarity is thereby provided, but presently only a very limited range of display devices can handle such a high refresh rate.

[0089] It should be noted that the foregoing description has been based on the fact that input sequences are supplied at a rate of 24 fps, which is common for motion picture movies. However, one can easily appreciate that the mixing process can be easily adapted to the case whereby two 30 fps sequences (ex. V programs) would be supplied, by merely skipping the preliminary step of temporal interpolation represented by time-expanded sequences 51 and 51′ of FIG. 3a. Obviously, since the decoding process always operates on a 30 fps input sequence, no substantial adaptation is required to that part of the process.

[0090] Alternatively, as illustrated in FIG. 4a, the encoding process of the present invention does not require temporal interpolation before creating the mosaics. In the example of FIG. 4a, a two 24 fps sequence is mixed to provide a 30 fps sequence by appropriately separating the frames. Since time-interpolated images may be inserted in the sequence (when input sequences comprise 24 fps), compressed sequence 80 becomes irregular. Therefore, the encoding (mixing) process according to the present embodiment of the invention further includes insertion of information in the compressed sequence 60 to enable identification of frame numbers as needed by the reconstruction process to identify image content and rebuild sequences with the proper sequential order (timing) at appropriate locations in the sequence. Such information may be stored in blank lines of merged images for instance. The usefulness of that procedure will become more apparent upon reading of the following description of the decoding process. This completes the mixing procedure per se carried out by mixer 1. Further in the process, as described in reference to FIG. 1a, RGB merged image sequence 60 (ex. AVI file) can be converted to a digital YUV format prior to being multiplexed into an MPEG2 bit stream format or be directly converted into an MPEG2 format identified by numeral 62 in FIG. 3a.

[0091] FIG. 4b illustrates how the compressed stereo sequence 80 can be decoded to provide two 36 fps streams, using a spatial and temporal interpolation.

[0092] A more specific representation of the decoder 2 of the present invention is shown in FIG. 5. However, it should be understood that variations are possible, depending, for example, on whether an all software, all hardware, or mixture of both, solution is chosen.

[0093] As can be seen, the decoder has two inputs: an analog and a digital. If the signal is analog, it is converted into a digital signal by ADC 101 FIFO buffer 103 changes the dock domain of the input signal into a clock domain used by the decoder. In practice, a broadcast signal or a DVD signal are clocked at a frequency different from the frequency used for RGB signals, hence the necessity of FIFO buffer 103. The signal is then passed through converter 105 which converts the signal from a Y C.sub.B C.sub.R signal into an RGB signal of 1×720×480 (pixels), This signal is then spatially interpolated according to the teachings of the present invention by spatial interpolator 107, resulting is a dual stream of 720×480 pixels. This dual stream is then scaled in scaler 109 to provide two 640×480 image streams (always in the RGB format). Alternatively, other resolutions can be supported by the system of the present invention. The frames are then placed in frame buffers 113, one for the right frames and the other for the left frames, the contents of which are controlled by input memory controller 111.

[0094] The output of the frame buffers is controlled by output memory controller 115 and, if necessary, time interpolator 117 to increase the frame rate.

[0095] A rate controller 119 is preferably provided. The purpose of rate controller is to accommodate variations in the clock signals, which variations, although minute, de-synchronise the system. Rate controller monitors the difference in rate and corrects the output frequency by adding or removing a certain number of pixels on inactive lines of the frame. For example, for a frame it may be necessary to add a few pixels to artificially slow the internal clock and to properly synchronise the clocks.

[0096] Another advantageous component of decoder 2 is the anti-flicker filter 121. Flicking occurs when wearing shutter spectacles, when there is a contrast between the image and the closing of the shutter of the head display. It has been surprisingly discovered that by evaluating the value of the green level in each RGB pixel, and by decreasing the corresponding pixel colours proportionally when the green level is above a certain value, flicking is greatly reduced.

[0097] The output is then directly digital, or converted into an analog signal by DAC 123. Sync module 125 synchronises the head display with the output signal in order to open and close the shutters at the appropriate times.

[0098] Further preferably, an adjuster 127 is further provided. This adjuster is useful when the display device includes its own frame buffer, which would otherwise result in a de-synchronisation between the sync signal for the shutters and the actual display. This is a manual adjustment that the user makes in order to reduce crosstalk/ghosting of the image.

[0099] A second embodiment of the mixing method carried out by mixer 1 according to the present invention will now be described in detail, by reference to FIG. 4a of the appended drawings. This second embodiment is particularly advantageous for addressing the problem of converting two image sequences available in a 24 fps format to produce a 30 fps MPG2 (main view profile) fully compatible sequence.

[0100] Full definition images from the two 24 fps sequences 50 and 50′ comprising mosaics A and B by definition are identified as L.sub.iAB and R.sub.iAB respectively (supposing two sequences of a stereoscopic program), index “i” representing the sequential number of a given image at time t. Dashed lines in FIGS. 4a and 4b indicate frame sequence. In a similar manner as for the first embodiment previously described, eight input images are spatially compressed and time-expanded to form five new merged images in a new 30 fps sequence 80. It should be noted that in this embodiment of the present invention, 25% more of the original image information is preserved to be recorded or broadcasted. Indeed, two out of the eight original images (images L1 and L2 in the example shown) have both of their mosaics A and B saved in fields of the compressed sequence 80 instead of one according to the first embodiment.

[0101] These fully saved images are nevertheless encoded in the form of two complementary mosaics stored in side-by-side merged images fields to ascertain homogeneity of the encoded sequence and compatibility with the MPEG2 compression/decompression protocol, by providing a certain temporal redundancy between successive images. Better definition and fidelity is thus generally obtained at playback with respect to the previously described embodiment, but at the expense of increased processing power requirement and system hardware cost. As for the above-described first embodiment, the encoding (mixing) process according to the present embodiment of the invention also further includes insertion of information in the compressed sequence 80 to enable identification of frame numbers as needed by the reconstruction process to identify image content and rebuild sequences with the proper sequential order and insert interpolated images at appropriate locations in the sequence. Again, such information may be stored in blank lines of merged images for instance.

[0102] The corresponding decoding process carried out by decoder 2 according to the present invention is schematically represented in FIG. 4b and operates as follows.

[0103] The five merged frames 81 to 85 representative of 30 fps RGB input sequence 80 are expanded to twelve images (six per channel) providing playback sequences 90 and 100 at 36 fps total (72 total, 36 per eye in the case of a three-dimensional stereoscopic program). In total, each group of twelve successive images of playback sequences 90 and 100, presented in a page flipping mode according to the frame sequence indicated by dashed lines 110, comprises two integral original images, six spatially interpolated images and four temporally interpolated images. Alternatively, sequences 90 and 100 could be outputted separately in parallel on two separate channels, as required by some display devices such as a head mounted or auto-stereoscopic devices. In the illustrated example:

1. Image 91 (L.sub.1AB) is totally rebuilt from mosaic L.sub.1A stored in the left field of frame 81 of sequence 80, and mosaic L.sub.1B stored in the right field thereof;
2. Image 101 (R.sub.1AX) is spatially interpolated from mosaic R.sub.1A, taken from the left field of frame 82 of sequence 80;
3. Image 103 (R.sub.2BX) is spatially interpolated from mosaic R.sub.2B, taken from the right field of frame 82 of sequence 80;
4. Image 102 is temporally interpolated from image 101 and image 103;
5. Image 93 (L.sub.2AB) is totally rebuilt from mosaic image L.sub.2A, stored in the left field of frame 83 of sequence 80, and mosaic L.sub.2B stored in the right field thereof;
6. Image 92 is temporally interpolated from image 91 (L.sub.1AB) and image 93 (L.sub.2AB);
7. Image 94 (L.sub.3AX) is spatially interpolated from mosaic L.sub.3A, stored in the left field of frame 84 of sequence 80;
8. Image 96 (L.sub.1BX) is spatially interpolated from mosaic 1,4B, stored in the right field of frame 84 of sequence 80;
9. Image 95 is temporally interpolated from images 94 and 96;
10. Image 104 (R.sub.3AX) is spatially interpolated from mosaic R.sub.3A, stored in the left field of frame 85 of sequence 80;
11. Image 106 (R.sub.4BX) is spatially interpolated from mosaic R.sub.1B, stored in the right field of frame 85 of sequence 80; and
12. Image 105 is temporally interpolated from image 104 and image 106.

[0104] Obviously, one may easily understand that such a reconstruction process requires proper identification of frame order in the 5 frame sequences constituting input sequence 80. Therefore, a frame recognition circuit is provided in decoder 2 to interpret frame number information stored by mixer 1 in merged image sequence 80.

[0105] It can be observed that in this latter embodiment as well as in the first one disclosed in the foregoing description, the first and second image sequences are being encoded and decoded totally independently, without any inference between each other, enabling processing of original video sequences referring to independent scenes.

[0106] The above described example of the second embodiment, processing sources at 24 fps to yield a presentation rate of 72 fps, is only illustrative of a more general process applicable to 24 or 30 fps sources to produce a stereo output at presentation rates such as 60, 72, 96 or 120 fps. The chart below provides additional exemplary arrangements for 24 or 30 fps sources and 60, 72, 96 or 120 fps presentation rates:

TABLE-US-00001 Spatially- Temporally- Source Output Original interpolated interpolated Repeated (fps) (fps) images images images images 24 + 24  60 12 36 12 or 0 0 or 12 24 + 24  77 12 36 24  0 24 + 24  96 12 36  0 48 24 + 24 120 12 36 12 60 30 + 30  60  0 60  0  0 30 + 30  72  0 60 12 or 0 0 or 12 30 + 30  96  0 60 36  0 30 + 30 120  0 60  0 60

[0107] As stated above, RGB sequences 90 and 100 obtained through the above described processing could be directly outputted and displayed on a dual input device to reproduce the original programs or stereoscopic program signals at a 72 fps (36 per eye) presentation rate. Further processing is however carried out by decoder 2 to provide a combined stereoscopic RGB output signal (not shown) comprising images of sequences 90 and 100 in a time sequenced arrangement as indicated by dashed arrows such as 110. Still referring to the example of FIG. 4b, images would be time sequenced by alternating left eye and right eye images in the following order 91, 101, 92, 102, 92, 103, 94, 104, 95, 105, 96, 106. This is accomplished through an appropriate read sequence of the complete images stored in memory buffers.

[0108] Presentation of the time sequenced combined signal with a standard projector or another display device is thus enabled to display the stereoscopic program in a page-flipping mode. Decoder 2 provides the necessary timing signals to a driver of shutter spectacles which can be worn by a viewer to view the displayed stereoscopic program in a three-dimensional mode, with high fidelity, negligible flicking and high comfort. As stated above, presentation rate can be increased up to 120 fps by inserting additional temporally interpolated image pairs or by repeating certain image pairs in the decoding process. It is also contemplated in the present invention that the RGB combined stereo output signal could be converted to another known standard presentation format such as an interlaced format or a conventional 2D format.

[0109] Therefore, one can easily appreciate that the above described embodiments of the present invention provide effective and practical solutions for the recording of two motion picture sequences on a conventional data storage medium, and playback with conventional videodisk player or broadcast source and display device, to enable viewing of stereoscopic 3D movies at home with unmatched performance and comfort, still at an affordable cost, in a plurality of output modes to match input signal requirement of a broad range of display devices. For example a universal set top box fed with a single input signal format as defined in the foregoing description, can be provided with selectable modes such as: page flipping, row interleaved, column interleaved, simultaneous dual presentation, anaglyphic, etc. The encoding/playback method and system of the present invention can thus be advantageously used in miscellaneous applications, including the processing of video sequences representing independent scenes, with numerous advantages over the solutions of the prior art.

[0110] It will thus be readily appreciated that the present invention presents advantages over the prior art. It provides a better quality of images, since no frequency filters are used low pass or band pass), decompression can be effected in real time with minimal resources, is compatible with progressive or interlaced systems, both at input and output, allows for pause, forward, reverse, slow, etc., and supports all stereoscopic displays presently, available.

[0111] Although the present invention has been described by means of preferred embodiments thereof, it is contemplated that various modifications may be made thereto without departing from the spirit and scope of the present invention. Accordingly, it is intended that the embodiment described be considered only as illustrative of the present invention and that the scope thereof should not be limited thereto but be determined by reference to the claims hereinafter provided and their equivalents.