Apparatus and method for decoding a panoramic video

Abstract

A system and method for improving quality of encoding and decoding a panoramic video. The panoramic video comprises a sequence of encoded picture frames. In panoramic video picture frames are mapped on a spherical viewing area. In the method the projection center of the encoded frame is transformed before encoding. In the decoding phase the projection center is transformed back accordingly after decoding the frame.

Claims

1. A picture decoding apparatus comprising processing circuitry, wherein the processing circuitry is configured to: receive an encoded panoramic picture; decode the received encoded panoramic picture; compute a geometrical transformation in accordance with a replacement vector, wherein the replacement vector is included with the encoded panoramic picture and the replacement vector determines a difference between a projection center of the received encoded panoramic picture and a projection center of a panoramic output picture; and construct the panoramic output picture, wherein the processing circuitry is further configured to transform the decoded panoramic picture in accordance with the computed geometrical transformation.

2. The picture decoding apparatus of claim 1, wherein the geometrical transformation is a backward transformation from a transformed projection center to an original projection center.

3. The picture decoding apparatus of claim 1, wherein the processing circuitry is further configured to use the projection center of the received encoded panoramic picture or the projection center of the panoramic output picture as a focal point.

4. The picture decoding apparatus of claim 1, wherein the processing circuitry is further configured to use a midway point between the projection center of the received encoded panoramic picture and the projection center of the panoramic output picture as a focal point.

5. A picture encoding apparatus comprising processing circuitry, wherein the processing circuitry is configured to: receive a panoramic picture; construct a transformed panoramic picture by computing a geometrical transformation of the received panoramic picture; and encode the transformed panoramic picture, wherein a replacement vector for a projection center is stored with the encoded transformed picture, and the replacement vector comprises a parameter set of the geometrical transformation between a projection center of the received panoramic picture and a projection center of the constructed transformed panoramic picture.

6. The picture encoding apparatus according to claim 5, wherein the geometrical transformation is a forward transformation from an original to a replaced projection center.

7. The picture encoding apparatus of claim 5, wherein the processing circuitry is further configured to use the projection center of the received panoramic picture or the projection center of the constructed panoramic picture as a focal point.

8. The picture encoding apparatus of claim 5, wherein the processing circuitry is further configured to use a midway point between the projection center of the received panoramic picture and the projection center of the constructed panoramic picture as a focal point.

9. A picture decoding method comprising: receiving an encoded panoramic picture; decoding the received encoded panoramic picture; computing a geometrical transformation in accordance with a replacement vector, wherein the replacement vector is included with the encoded panoramic picture and the replacement vector determines a difference between a projection center of the received encoded panoramic picture and a projection center of a panoramic output picture; constructing the panoramic output picture; and transforming the decoded panoramic picture in accordance with the computed geometrical transformation.

10. The picture decoding method of claim 9, wherein the geometrical transformation is a backward transformation from a transformed projection center to an original projection center.

11. The picture decoding method of claim 9, wherein the method further comprises using the projection center of the received encoded panoramic picture or the projection center of the panoramic output picture as a focal point.

12. The picture decoding method of claim 9, wherein the processing circuitry is further configured to use a midway point between the projection center of the received encoded panoramic picture and the projection center of the panoramic output picture as a focal point.

13. A picture encoding method comprising: receiving a panoramic picture; constructing a transformed panoramic picture by computing a geometrical transformation of the received panoramic picture; and encoding the transformed panoramic picture, wherein a replacement vector for a projection center is stored with the encoded transformed panoramic picture, and the replacement vector comprises a parameter set of the geometrical transformation between a projection center of the received panoramic picture and a projection center of the transformed panoramic picture.

14. The picture encoding method according to claim 13, wherein the geometrical transformation is a forward transformation from an original to a replaced projection center.

15. The picture encoding method of claim 13, wherein the method further comprises using the projection center of the received panoramic picture or the projection center of the constructed panoramic picture as a focal point.

16. The picture encoding method of claim 13, wherein the method further comprises using a midway point between the projection center of the received panoramic picture and the projection center of the transformed panoramic picture as a focal point.

Description

DESCRIPTION OF THE DRAWINGS

(1) The present description will be better understood from the following detailed description read in light of the accompanying drawings, wherein:

(2) FIG. 1 is a schematic illustration of an encoder;

(3) FIG. 2 is a schematic illustration of a decoder;

(4) FIG. 3 illustrates an example of encoding process;

(5) FIG. 4 illustrates an example of decoding process; and

(6) FIG. 5 illustrates an example of a geometrical transformation.

DETAILED DESCRIPTION

(7) The detailed description, provided below in connection with the appended drawings, is intended as a description of the embodiments and is not intended to represent the only forms in which the embodiment may be constructed or utilized. However, the same or equivalent functions and structures may be accomplished by different embodiments. In the following description decoding is discussed, however, a corresponding encoder includes a similar decoder part.

(8) In the following description a video coding apparatus, a method and a corresponding computer program are disclosed. The description relates to encoding and decoding of a panoramic video. The panoramic video may be, for example of 180 or 360 degrees, and stored into a video stream or other similar encoded sequence of pictures in the form of picture frames of any projection. The decoded picture frames are then mapped to a surface of a sphere forming the panoramic view. Thus, each of the mapped picture frames has a projection center, which can also be determined when the picture frame is stored as a two-dimensional picture frame in the video stream. In the following description the term “projection center” means the center of a sphere on which the panoramic video is projected.

(9) FIG. 1 shows an encoder 100, which comprises an input 102, a residual calculation unit 104, a transformation unit 106, a quantization unit 108, an inverse quantization unit 110, and inverse transformation unit 112, a reconstruction unit 114, a buffer 116, a loop filter 120, a frame buffer 130, an inter estimation unit 142, an inter prediction unit 144, an intra estimation unit 152, an intra prediction unit 154, a mode selection unit 160, an entropy encoding unit 170, and an output 172.

(10) The encoder 100 receives an input from a preprocessor that is not shown in FIG. 1. The preprocessor is configured to perform one or more preprocessing functions, wherein one of the preprocessing functions is transforming a received panoramic picture using forward transformation and storing a replacement vector according to which the transformation is performed. The encoder is then configured to perform the encoding using a picture block 101 comprising the second transformed picture that is received from the preprocessor at the input 102. The picture block further comprises the replacement vector. The encoder 100 is configured to encode the received picture

(11) At the encoding side the picture encoding is performed in conventional manner using the shown encoder and the respective components. However, the encoded picture data (encoded image data) 171 that is provided at the output 172 of the decoder is different also in that sense that the encoded picture data comprises the replacement vector, that may be also encoded, for example, in the entropy encoding block 170, or directly before or after the entropy encoding. As the introduction of the replacement vector does not directly relate to the encoding of the picture data of the picture block 101 it can be done in any suitable location of the encoder. Thus, because of the preprocessing, the encoded picture data 171 differs from the conventional encoding in that the projection center of the encoded picture is different and the encoding result is different. This provides better capability of improving quality as more details can be preserved. A further difference is the introduction of the replacement vector or other set of parameters that can be used for corresponding geometrical transformation.

(12) FIG. 2 shows an exemplary video decoder 200 configured to receive encoded picture data (bitstream) 171, e.g. encoded by encoder 100, to obtain a decoded picture 231. The decoder 200 comprises an input 202, an entropy decoding unit 204, an inverse quantization unit 210, an inverse transformation unit 212, a reconstruction unit 214, a buffer 216, a loop filter 220, a decoded picture buffer 230, an inter prediction unit 244, an intra prediction unit 254, a mode selection unit 260 and an output 232.

(13) In the decoder 200 the encoded picture data 171 comprises picture data of a picture and a replacement vector as explained with regard FIG. 1. The replacement vector may be extracted from the encoded picture data, for example, in conjunction with the entropy encoding 204, or at any other suitable location. The picture is decoded in the conventional manner and the output 232, which is provided to post processing, comprises a decoded picture that corresponds with the second panoramic picture of FIG. 1. Thus, the output is a panoramic picture that has been transformed in accordance with the replacement vector. At the post processing the output picture is transformed using backward transformation to correspond with the original picture which was received at the encoder of FIG. 1.

(14) The expression replacement vector is to be understood as a set of parameters that represents the geometrical difference between the projection centers of the received panoramic picture and the constructed second panoramic picture.

(15) In FIG. 3 an example encoding process is shown. In the process, panoramic pictures are received and encoded in order to form a panoramic video. The process starts by receiving a picture belonging to a panoramic video, step 300. In this example the received panoramic picture is nominated as the first panoramic picture. The picture comprises a projection center at an original location. When encoding a video, pictures are preprocessed before encoding. In the process of FIG. 3 the preprocessing step 301 includes receiving the first panoramic picture and a replacement vector for projection center. The replacement vector represents the difference between the projection centers of a received first panoramic picture and the second panoramic picture, which is the constructed panoramic output picture. The projection center of the second panoramic picture is chosen so that more details are preserved when the picture is encoded. Then the preprocessing function processes the received first panoramic picture and the replacement vector by performing a forward transformation for the projection center in accordance with replacement vector. The processing comprises constructing a second panoramic picture, wherein the second panoramic picture is a transformed panoramic Figure, 302. The construction is performed by computing a geometrical transformation from of the first panoramic picture in accordance with the replacement vector.

(16) The constructed second transformed panoramic picture is then encoded into a video stream, step 303. The encoding step includes encoding of the replacement vector. The replacement vector is associated into the video stream so that when the video stream is later being decoded the first panoramic picture that represents the original picture can be constructed using backward transformation in accordance with the replacement vector. As can be seen in Figure the encoded data 304 includes both the second transformed panoramic picture and the replacement vector.

(17) In FIG. 4 an example decoding process is shown. The process is explained in view of the encoding process of FIG. 3 for providing better understanding. In the process an encoded data is received, 400. The encoded data comprises a panoramic picture and a replacement vector, which corresponds with the encoded data 304 of FIG. 3. Then the received encoded data is decoded, 401. In the decoding process the panoramic picture is decoded and the replacement vector is extracted from the received encoded data. The result is a panoramic picture 402 that has been transformed using the extracted replacement vector. Thus, the transformed panoramic picture 402 corresponds to the second panoramic picture 302 of FIG. 3.

(18) At the post processing phase 403 a backward transformation is applied to the second panoramic picture 402 in accordance with the replacement vector. As an outcome a third panoramic picture 404 is achieved. The third panoramic picture 404 is a reconstructed copy of the original panoramic picture, which is in this example the first panoramic picture 300 of FIG. 3.

(19) Even if the result is a reconstructed copy of the first panoramic picture it is understood that it may not be necessarily identical. Deviations may be caused by, for example lossy compression algorithms, or other algorithms that irreversibly change the picture during pre-processing, encoding, decoding or post-processing.

(20) In FIG. 5 an example of a geometrical transformation is shown. In the following the principles of the geometrical transformation of FIG. 5 are explained. However, a person skilled in the art understands that the similar principles may be used in various geometrical transformations. Examples of transformations include zoom, perspective change, rotation and panning.

(21) A three-dimensional sphere centered at Si represents the initial panoramic picture or spherical picture. In the following 52 denotes the shifted center of the transformed spherical picture and R the scaling factor of the radius, i.e. the shifted sphere has a radius R, assuming a radius equal to 1 for the initial sphere. In the following, f represents the focal point of both pictures, so that any considered ray of light goes through f Without loss of generality, the f can be considered to represent the origin of the Cartesian coordinate system, i.e.
f=(0,0,0).sup.T

(22) In the following a=(ax, ay, az).sup.T denotes the local coordinates of a pixel at the surface of the initial sphere that is centered at coordinate Si. We denote by a′=(ax′, ay′, az′).sup.T the local coordinates of the transformed pixel at the surface of the normalized version of the shifted sphere that is centered at coordinate s.sub.2.

(23) Then, s.sub.2+Ra′ results in the intersection of the ray of light going through s.sub.1+a and focal point f Therefore, after some calculus, point a′ satisfies the following equation

(24) $T\left(a\right)=a^{\prime }=\frac{\rho }{R}\left(s_1+a\right)-\frac{1}{R}{s}_2$$\mathrm{with}\rho =\frac{s_2^T\left(s_1+a\right)\sqrt{{\left(s_2^T\left(s_1+a\right)\right)}^2-{.Math.s_1+a.Math.}^2\left({.Math.s_2.Math.}^2-R^2\right)}}{{.Math.s_1+a.Math.}^2}.$

(25) One can recover the initial picture from the transformed spherical picture through the following backward mapping:

(26) $T^{-1}\left(a^{\prime }\right)=a={\rho }^{\prime }\left(s_2+{\mathrm{Ra}}^{\prime }\right)-s_1$$\mathrm{with}{\rho }^{\prime }=\frac{s_1^T\left(s_2+\mathrm{Ra}\right)+\sqrt{{\left(s_1^T\left(s_2+{\mathrm{Ra}}^{\prime }\right)\right)}^2-{.Math.s_2+{\mathrm{Ra}}^{\prime }.Math.}^2\left({.Math.s_1.Math.}^2-1\right)}}{{.Math.s_2+{\mathrm{Ra}}^{\prime }.Math.}^2}.$

(27) Note that, if f=s.sub.2, the forward transformation simplifies into:

(28) $a^{\prime }=\frac{s_1+a}{.Math.s_1+a.Math.}.$

(29) Furthermore, if

(30) $f=\frac{s_1+s_2}{2}$
and R=1, the transformations are symmetrical, that is to say
T.sup.−1(a)=−T(−a).

(31) It is also possible to use the backward transformation when f=s.sub.1, i.e. the projection center of the current frame is used as focal point. This may be convenient in some applications.

(32) The video coding apparatus and method have been described in conjunction with various embodiments herein. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.