Panoramic video

Abstract

A panoramic video system obtains a digital video image having a width w.sub.1 pixels and a height of h pixels. A plurality of digital still images is also obtained, each still image comprising a ray having a width w.sub.2 comprising pixels, 1w.sub.250, a height substantially equal to h. A storage device records or a display device displays the digital video image and plurality of digital still images. The set of n rays on either or both sides of the first horizontal angle is obtained from the plurality of still images to form one or two still background images, adjacent horizontally to the digital video image, each of the background image(s) having a width of n.Math.w.sub.2 and a height substantially equal to h, and each ray forming one vertical stripe of the still background image.

Claims

1. An imaging system comprising a first imaging system operable to obtain a digital video image having a first width and a height, the first width comprising w.sub.1 pixels and a first horizontal viewing angle, the height comprising h pixels and a first vertical viewing angle; a second imaging system operable to obtain a plurality of digital still images, each still image comprising a ray, the ray having a second width and substantially the same height as the digital video image, the second width comprising w.sub.2 pixels, 1w.sub.250, and a second horizontal viewing angle; and a storage device operable to record said digital video image and said plurality of digital still images, or a display device operable to display said digital video image and said plurality of digital still images; wherein said second imaging system is aligned and operable to obtain a set of n rays on either or both sides of the first horizontal viewing angle, the set of rays forming one or two still background images adjacent horizontally to the digital video image, the background image(s) having substantially the same vertical viewing angle as the first vertical viewing angle, where the width of each still background image is n.Math.w.sub.2 and each ray forms one vertical stripe of the still background image.

2. The imaging system of claim 1 wherein the set of rays forms one background image extending from one side of the digital video image to the other side of the digital video image such that a 360 panoramic image is produced, the 360 panoramic image comprising the digital video image and the one background image covering all angles not included in the first horizontal viewing angle.

3. The imaging system of claim 1 wherein the second imaging system is the same as the first imaging system.

4. The imaging system of claim 1 wherein the second imaging system is operable to obtain the set of rays forming the background image(s) as follows: a) taking a first digital still image, horizontally adjacent to a first edge of the digital video image, b) selecting a first ray from the first still image, said first ray having a width w.sub.2, and corresponding to the area of the first still image which is immediately adjacent to said first edge of the digital video image, and out of the angle of vision covered by said digital video image; c) selecting a second ray from the second still image, said second ray having a width w.sub.2, and corresponding to the area of the second still image which is immediately adjacent to the first ray and out of the angle of vision already covered by the first ray; and d) repeating step c) with the remaining digital still images, until a last ray is selected from the last digital still image, said last ray having a width w.sub.2, and corresponding to the area of the last digital still image which is immediately adjacent to a second edge of the digital video image.

5. The imaging system of claim 1 further comprising a third imaging system substantially similar to the first imaging system but displaced horizontally by a distance d and a fourth imaging system substantially similar to the second imaging system but displaced horizontally by the same distance d such that stereoscopic panoramic images are recorded or displayed.

6. The imaging system of claim 1, wherein w.sub.2=1.

7. A method for capturing panoramic video comprising capturing a first digital video image with a first imaging system, the digital video image having a first width and a height, the first width comprising w.sub.1 pixels and a first horizontal viewing angle, the height comprising h pixels and a first vertical viewing angle; capturing a first plurality of digital still images with a second imaging system, each still image comprising a ray, the ray having a second width and substantially the same height as the first digital video image, the second width comprising w.sub.2 pixels, 1w.sub.250, and a second horizontal viewing angle; and recording said first digital video image and said plurality of digital still images on a storage device, or displaying said digital video image and said first plurality of digital still images on a display device; wherein said digital still images comprise a set of n rays on either or both sides of the first horizontal viewing angle, the set of rays forming one or two still background images adjacent horizontally to the digital video image, the background image(s) having substantially the same vertical viewing angle as the first vertical viewing angle, where the width of each still background image is n.Math.w.sub.2 and each ray forms one vertical stripe of the still background image.

8. The method of claim 7 wherein the set of rays forms one background image extending from one side of the first digital video image to the other side of the first digital video image such that a 360 panoramic image is produced, the 360 panoramic image comprising the first digital video image and the one background image covering all angles not included in the first horizontal viewing angle.

9. The method of claim 7 wherein the plurality of digital still images are captured by the first video camera.

10. The method of claim 7 wherein the rays forming the background image(s) are obtained following these steps: a) taking a first digital still image, horizontally adjacent to a first edge of the digital video image, b) selecting a first ray from the first still image, said first ray having a width w.sub.2, and corresponding to the area of the first still image which is immediately adjacent to said first edge of the digital video image, and out of the angle of vision covered by said digital video image; c) selecting a second ray from the second still image, said second ray having a width w.sub.2, and corresponding to the area of the second still image which is immediately adjacent to the first ray and out of the angle of vision already covered by the first ray; and d) repeating step c) with the remaining digital still images, until a last ray is selected from the last digital still image, said last ray having a width w.sub.2, and corresponding to the area of the last digital still image which is immediately adjacent to a second edge of the digital video image.

11. The method of claim 7 further comprising capturing a second digital video image with a third imaging system, substantially similar to the first digital video image but displaced horizontally by a distance d and capturing a second plurality of digital still images with a fourth imaging system substantially similar to the first plurality of digital still images but displaced horizontally by the same distance d such that stereoscopic panoramic images are recorded or displayed.

12. The method of claim 7, wherein w.sub.2=1.

13. The method of claim 7, wherein the first plurality of digital still images is captured before the first digital video image.

14. The method of claim 7, wherein the first plurality of digital still images is captured after the first digital video image.

15. The method of claim 7, wherein the panoramic video represents an equirectangular projection of a scene.

16. The method of claim 7, wherein the panoramic video represents a cylindrical projection of a scene.

Description

DESCRIPTION OF THE FIGURES

(1) FIG. 1 shows, schematically and in a simplified way, how the digital video image is obtained by an image system in a method according to the present invention;

(2) FIGS. 2a-n schematically show how the method according to the present invention obtains the set of adjacent fractional digital images and selects the appropriate rays thereof.

DETAILED DESCRIPTION OF THE INVENTION

Definitions

(3) As used herein, the term image refers to any representation of a scene as captured by an imaging system. If not otherwise identified, the image is assumed to consist of a single still picture.

(4) As used herein, the term digital image refers to an image recorded in digital form as a sequence of pixels. Typically, a digital image comprises a rectangular array of approximately square pixels. Thus, the size of a digital image can be measured as having a width and height defined as a number of pixels in each of two orthogonal directions.

(5) As used herein, the term fractional image refers to a portion of a larger image. For example, an image can comprise a set of horizontal portions each having a width less than that of the larger image.

(6) As used herein, the term video image refers to a representation of a moving picture comprising a set of frames captures at sequential and equally-space time intervals. The time interval may vary, but a digital video image is typically captured at, at least, 24 or 30 frames per second. A time interval of 60 frames per second is preferred for the purposes of the present invention. As used herein, the term panoramic image refers generally to any image having a width significantly greater than its height. The aspect ratio can vary, but the width is typically at least twice the height. While, in principle, a panoramic image can be captured all at once by a single camera, in practice, it is common to capture a plurality of images having smaller aspect ratios and then stitching the plurality of images together to form a single panoramic image.

(7) A panoramic image can be formed to represent, for example, an equirectangular, cylindrical or cubical projection. An equirectangular projection implies a forward projection, by which spherical coordinates are transformed into planar coordinates and a map projection, by which meridians are transformed into vertical straight lines of constant spacing (for meridional intervals of constant spacing) and circles of latitude are transformed into horizontal straight lines of constant spacing (for constant intervals of parallels). A cylindrical projection implies projecting the image onto a cylinder tangent to the equator as if were coming from a light source at the center of the scene. The cylinder is then cut along one of the projected meridians and unrolled into a flat image.

(8) As used herein, the terms viewing angle and angle of vision are used interchangeably to refer to fractions of a particular projection.

(9) As used herein, the terms stereo image and stereoscopic image are used interchangeably and refer to a set of two images where each image is intended to be viewed by one eye such that the brain perceives an approximation of a three-dimensional scene.

(10) As used herein, any of the above definitions can be combined in any sequence: for example, a digital panoramic video image is a video image that is also panoramic and digital.

(11) As used herein, the term ray refers to a fractional image comprising a vertical image stripe having a width of between 1 and 50 pixels. Generally, ray widths are chosen such that there are negligible parallax mismatches between adjacent rays.

(12) As used herein, the term substantially the same refers to quantities which are equal for practical purposes but may vary numerically by up to 5%.

DESCRIPTION OF THE EMBODIMENTS

Example 1

(13) FIGS. 1 and 2a to 2n illustrate this example. In this example, a 360 stereoscopic panoramic video is obtained, having a total horizontal width of 4,320 pixels.

(14) FIG. 1 schematically represents how an imaging system obtains a digital video image. Still digital images are obtained in a similar way. More particularly, FIG. 1 represents in a simplified way how the digital video image is obtained in a process according to the present invention.

(15) Two different imaging systems 10 are used, each one for obtaining a digital video image for one eye, for obtaining a 360 panoramic stereoscopic video. One single imaging system 10 has been illustrated in said FIG. 1, but two different imaging systems 10 are used and the operation of each imaging system 10 is exactly the same.

(16) As can be seen in FIG. 1, each imaging sensor 10 captures all rays 30 of light coming from a first horizontal angle of vision, to obtain the digital video image corresponding to one eye.

(17) In this particular embodiment of the present invention each imaging sensor 10 is a video camera set up to record 60 frames per second. Moreover, video cameras 10 are disposed horizontally, in parallel and 6.5 cm apart.

(18) The whole horizontal widths of the digital video images obtained by each video camera 10 are selected. In other words, all the angular area 20 captured by each video camera 10 is selected to form the corresponding digital stereoscopic panoramic video.

(19) In this particular example of the present invention, each video sensor 10 captures a digital video image covering an angular area 20 which has a horizontal width w.sub.1 of 1,080 pixels, and corresponding to a first horizontal angle of vision, of 90.

(20) The 3,240 remaining horizontal pixels (4,3201,080=3,240) required to form the 360 stereoscopic panoramic video, are selected from the plurality of digital still images.

(21) Given that the panoramic video to be obtained is (in this particular embodiment of the invention) a stereoscopic one, two different sets of digital still images are obtained for obtaining the panoramic stereoscopic video (one for each eye). One single imaging system 100 has been illustrated in the different steps of FIGS. 2a-n, but two different imaging systems 10 are used and the operation of each imaging system 100 is exactly the same.

(22) In the embodiment of the invention shown in FIGS. 2a-n, each imaging system 100 is a video camera moved with a motor, in order to obtain a plurality of digital still images. More particularly, each imaging system 100 is exactly the same video camera 10 used to obtain digital video image corresponding to one eye. Nevertheless, imaging systems 10 and imaging systems 100 used in the process according to the invention to obtain, respectively, the digital video image(s) and the digital still images could be of different types.

(23) In other words, in this particular example of the present invention, imaging systems 10 and imaging systems 100 are the same, a video camera. Two video cameras are used, one for the left eye and one for the right eye.

(24) Each imaging system 100 captures a second horizontal angle of vision which, in this case, is 90. In addition, each imaging system 100 is mounted on a piece which is, in turn, connected to a single and common motor that makes said imaging system 100 rotate about a central point between the two imaging systems 100. Said rotation center is not the optical center of any of the imaging systems 100, so the rotation introduces parallax to the digital still images obtained thereby.

(25) In order to improve the smoothness of the formed digital fractional video sequence as much as possible, one ray having a horizontal width of only 1 pixel can be selected from each of the set of adjacent fractional images. Consequently, in order to form the fixed background of the resulting 360 stereoscopic panoramic video, 3,240 different adjacent fractional images, are obtained. The rays selected from the whole set of adjacent fractional images taken with each image sensor 100, cover a total angle of 270 (36090=270).

(26) The process applied to each imaging system 100 to obtain the corresponding plurality of digital still images, as well as to obtain the rays that will form the still background image is exactly the same, and is schematically showed in FIGS. 2a-n.

(27) Imaging systems 100 are turned on, as well the motor, which rotates clockwise 360 in 54 seconds and stops exactly at the end of said 360 cycle (so each imaging system 100 is then at the initial position). However, the process of the present invention also contemplates the possibility of obtaining each plurality of digital still images by a counterclockwise rotation.

(28) After the conclusion of said 360 cycle, each of the image sensors 100 has obtained a set of 3,240 adjacent fractional images, which, in addition to the 90 of the digital video image cover a total angle of 360. One ray, having a horizontal width of 1 pixel, will be selected from each digital still image of each set to form a still background image having a horizontal width of 3,240 pixels and covering a horizontal angle of 270.

(29) The selection of rays is exactly the same for both sets (also described herein as pluralities) of digital still images. Consequently, this selection step will only be described once. Nevertheless, in this particular embodiment of the invention, the selection is actually performed twice, since two 360 panoramic videos should be formed, one for each eye.

(30) The first selected ray 300a is taken from the first digital still image (shown in FIG. 2a). This first selected ray 300a is the column (having a width of one pixel) of the first image, which is immediately adjacent to the right edge of the digital video image, and out of the angle of vision covered by digital video image. That is, the one at that is its most right column, since the selection of rays is performed clockwise.

(31) The first column of the still background image is then completed with the first selected ray 300a.

(32) The second selected ray 300b is taken from the second digital still image (shown in FIG. 2b). This second selected ray 300b is the column (having a width of one pixel) of the second still image, which is adjacent the first selected ray 300a, and out of the angle of vision covered by the fractional video sequence. That is, the one at the right, since the selection of rays is performed clockwise.

(33) The second column of the background part is then completed with the second selected ray 300b.

(34) The following rays 300c to 300m are selected using exactly the same method described above.

(35) The last ray 300n is selected from the last still image (i.e, the 3,240.sup.th image) and corresponds to the column (having a width of one pixel) of said last still image, that is its most left column, which is immediately adjacent to the left edge of digital video image, and out of the angle of vision covered by said digital video image.

(36) In this particular embodiment of the invention each digital still image has a width of 1,080 pixels. The columns of pixels forming each digital still image of are numbered from 1 (most left column) to 1,080 (most right column).

(37) Consequently, the first ray 300a is taken form the 1,080.sup.th column of the first digital still image and the last ray 300n is taken form the 1.sup.st column of the last digital still image. In other words, the selection of rays, in this particular embodiment of the invention, follows the following formula:
Ray n=Column(1,0801079*n/3,239) of n.sup.th still image,
where 0n3,239, (n=0 corresponds to the first ray 300a, n=1 corresponds to the second ray 300b, and so on), and in which the value of the column obtained is rounded to the nearest whole number.