Tridimensional rendering with adjustable disparity direction

Abstract

The disclosure pertains a method for determining a rendered tridimensional content intended to a viewer. The method includes inputting a reference content comprising at least a reference view, inputting at least one information related to the viewer's eyes orientation relatively to a reference axis of a display device, and determining a rendered view as a function of the reference view, and the information related to the viewer's eyes orientation relatively to the reference axis of the display device.

Claims

1. A method for determining a rendered tri-dimensional content intended to a viewer, comprising: inputting a reference content comprising at least a reference view corresponding to the view from the viewer's left or right eye, inputting at least one information related to the viewer's eyes axis orientation relatively to a reference axis of a display device, where said viewer's eyes axis orientation can include axes other than horizontal, determining a rendered view as a function of the reference view and the information related to the viewer's eyes axis orientation relatively to the reference axis of the display device, said rendered view corresponding to a view from the other of the viewer's left or right eye, generating the rendered tri-dimensional content by combining the reference view and the rendered view.

2. The method according to claim 1, wherein: the reference content comprises a depth map (Z.sub.L) depicting depth in the reference view, or information on the disparity between at least one given view and the reference view, or the at least one given view, determining the rendered view is also a function of the depth map (Z.sub.L) or the information on disparity, or the at least one given view.

3. The method according to claim 2, wherein: the reference content comprises occlusion information (V.sub.O) on a hidden portion of the reference view along at least one tilted axis forming an angle with the reference axis of the display device, determining the rendered view is also a function of the occlusion information (V.sub.O).

4. The method according to claim 3, wherein the occlusion information (V.sub.O) pertain to a hidden portion of the reference view along at least two tilted axis forming an angle with the reference axis of the display device.

5. The method according to claim 4, wherein determining consists in interpolating as a function of the reference view, the depth map (Z.sub.L) or the information on disparity, the occlusion information (V.sub.O), and the information related to the viewer's eyes axis orientation relatively to the reference axis of the display device.

6. The method according to claim 5, wherein interpolating comprises: determining a depth map (Z.sub.R) of the rendered view from the reference view depth map (Z.sub.L), determining a hidden portion of the reference view based on the occlusion map information (V.sub.O).

7. The method according to claim 6, wherein interpolating comprises determining the color of each pixel of the rendered view based on the color of the corresponding pixel in the reference view.

8. The method according to claim 3, wherein: the reference content comprises a plurality of given views, determining consists in selecting among the given views the rendered view, as a function of the reference content and the information related to the viewer's eyes axis orientation relatively to the reference axis of the display device.

9. The method according to claim 1, wherein a position (u.sub.R,v.sub.R) in the rendered view of an object located at a position (u.sub.L,v.sub.L) in the reference view is given by: $\begin{array}{c}{u}_R=u_L-\frac{f^u{B}_{L/R}^u}{Z_L\left(u_L,v_L\right)}\\ v_R=v_L-\frac{f^v{B}_{L/R}^v}{Z_L\left(u_L,v_L\right)}\end{array}$ where B.sub.L/R.sup.u and B.sub.L/R.sup.u are the horizontal and the vertical components of a baseline vector, B.sub.L/R.sup.u being equal to B*cos and B.sub.L/R.sup.v being equal to B*sin , B being the distance between the viewer's eyes, where Z.sub.L(u.sub.L,v.sub.L) corresponds to a depth value at the position (u.sub.L,v.sub.L), where is the at least one information related to the viewers eyes axis orientation relative to the reference axis of the display device, and where f.sup.u and f.sup.v are respectively the products of a focal length by the horizontal and the vertical density on a sensor adapted to sense the viewer's eyes axis orientation.

10. The method according to claim 9, wherein the reference content comprises light-field data.

11. A non-transitory computer-readable carrier medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for implementing the method according to claim 1.

12. An apparatus for determining a rendered tri-dimensional content intended to a viewer, comprising a module for determining a rendered view corresponding to a view from one of the viewer's eyes as a function of: a reference content comprising at least a reference view corresponding to a view from the other of the viewer's eyes, at least one information related to the viewer's eyes axis orientation relatively to a reference axis of a display device, wherein said viewer's eyes axis orientation can include axes other than horizontal.

13. The apparatus according to claim 12, comprising at least one sensor adapted to determine the viewer's eyes axis orientation relatively to the reference axis of the display device.

14. A light-field capture device, comprising an apparatus for determining a rendered tri-dimensional content, comprising a module for determining a rendered view corresponding to view from one of the viewer's eyes as a function of: a reference content comprising at least a reference view corresponding to a view from the other of the viewer's eyes, at least one information related to the viewer's eyes axis orientation relatively to a reference axis of a display device, wherein said viewer's eyes axis orientation can include axes other than horizontal.

Description

5. BRIEF DESCRIPTION OF THE DRAWINGS

(1) The invention can be better understood with reference to the following description and drawings, given by way of example and not limiting the scope of protection, and in which:

(2) FIG. 1 is a diagram illustrating a view synthesis for horizontal eyes, as known from the art,

(3) FIG. 2 is another diagram illustrating a view synthesis for horizontal eyes, as known from the art,

(4) FIG. 3 is a diagram illustrating a view reconstruction from LDV format, as known from the art,

(5) FIG. 4 is a flow chart of the successive steps implemented when performing a method for determining a rendered tridimensional content, according to one embodiment of the disclosure,

(6) FIG. 5 is a diagram illustrating the relationship between the perceived depth and the parallax between a viewer's left and right-eye images of a stereo pair,

(7) FIG. 6 is a diagram illustrating a set of left and right views depending on the axis direction of the viewer's eyes,

(8) FIG. 7 is a diagram illustrating three different cases for which the angle between the eyes axis and the horizontal axis is in the range [90,+90],

(9) FIG. 8 is a flow chart of the successive steps implemented when performing a method for determining a rendered tridimensional content, according to the first embodiment of the disclosure,

(10) FIG. 9 is a flow chart of the successive steps implemented when performing a method for determining a rendered tridimensional content, according to the second embodiment of the disclosure,

(11) FIG. 10 is a diagram illustrating a left view and a set of given views for which the angle between the disparity direction and the horizontal axis is in the range [90,+90],

(12) FIG. 11 is a schematic block diagram illustrating an example of an apparatus for determining a rendered tridimensional content, according to one embodiment of the present disclosure.

(13) The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.

6. DETAILED DESCRIPTION

(14) The present disclosure relates to a method 1 for determining a rendered tridimensional content as a function of a viewer's eyes orientation . Many specific details of certain embodiments of the disclosure are set forth in the following description and in FIGS. 1 to 9 to provide a thorough understanding of such embodiments. It is to be understood that the present disclosure may have additional embodiments, or may be practiced without several of the details described in the following description.

(15) As illustrated by FIG. 4, such a method 1 comprises: A step 2 of inputting a reference content Ref_C comprising at least a reference view 8, A step 4 of inputting at least one information related to the viewer's eyes orientation relatively to a reference axis X of a display device, A step 5 of determining a rendered view 6 as a function of the reference view 8, and the information related to the viewer's eyes orientation relatively to the reference axis X of the display device.

(16) In the following description, the reference view 8 corresponds to the view from the viewer's left eye, also referred to as left view, while the rendered view 6 to be determined corresponds to the view from the viewer's right eye, also referred to as right view. In another embodiment, such a correspondence may be reverted, the viewer's right eye then seeing the reference view. In another embodiment, such views may also be allocated to any object, for instance a sensor mounted on an unmanned vehicle and connected at distance with an operator through an interface unit.

(17) FIG. 5 illustrates the relationship between the perceived depth Z.sub.p and the parallax P between the viewer's left and right-eye images, where Z.sub.p is the perceived depth, t.sub.e is the interocular distance, Z.sub.s is distance from the viewer to the screen. In this way, the relationship between the perceived depth Z.sub.p, the parallax P and distance Z.sub.s to the screen is expressed as followed:

(18) $\{\begin{array}{c}{Z}_p=\frac{Z_S{t}_e}{t_e-P}\\ P=\frac{W_s}{N_{\mathrm{col}}}d\end{array}$

(19) Where d is the transmitted disparity information, W.sub.s is the width of the screen, and N.sub.col is the number of pixel columns of the screen.

(20) As a matter of illustration, three particular embodiments of the disclosure are described, each of them describing an alternative embodiment of the step 5 for determining the rendered view 6 either by a process of interpolation, or by a process of selection.

6.1. First Embodiment of the Disclosure

(21) In a first embodiment of the disclosure, the step 5 of determining a rendered view 6 of the method 1 for determining the rendered tridimensional content including both the left view 8 and right view 6 consists in a step of interpolation.

(22) FIG. 6 illustrates a set of left and right views (8; 6) depending on the orientation of the axis direction of the viewer's eyes, relatively to the reference axis X of the display device. A foreground object is symbolized by a square. In this embodiment, the right view 6 is interpolated and the left view 8 is considered as a fixed reference.

(23) The reference content Ref_C inputted in step 2 is composed of the left view 8, the corresponding depth map Z.sub.L and the occlusion map V.sub.O containing information masked in the left view but required to reconstruct the right view 6.

(24) Since the interpolation step 5.1 depends on the orientation of the eyes axis , the occlusion map V.sub.O preferentially contains occlusion information in all the directions, and not only along a single tilted axis. In one embodiment, the reference content Ref_C inputted in step 2 is a light-field data, for instance a demultiplexed light-field image, which comprises the left view 8, the corresponding depth map Z.sub.L and the occlusion map V.sub.O depicting hidden portions of the foreground object in all the directions. In an other embodiment, the occlusion information may be limited to a reduced number of directions, reducing, on one hand, the size of referenced content while affecting, in the other hand, the quality of the interpolated rendered view 6. Alternatively, the data, which are occluded in the reference view 8, may be extracted from the plurality of given views 9 having different disparity directions instead of being provided in an occlusion map Z.sub.O. In this case, the reference content Ref_C inputted in step 2 does not explicitly contain the occlusion map Z.sub.O, but this occlusion map or equivalent is extracted from the plurality of given views 9. The position (u.sub.R, v.sub.R) in the right view 6 of the foreground object, located at a position (u.sub.L, v.sub.L) in the left view 8, is given by the following equation (1):

(25) $\begin{array}{c}{u}_R=u_L-\frac{f^u{B}_{L/R}^u}{Z_L\left(u_L,v_L\right)}\\ v_R=v_L-\frac{f^v{B}_{L/R}^v}{Z_L\left(u_L,v_L\right)}\end{array}$

(26) Where B.sub.L/R.sup.u and B.sub.L/R.sup.v are the horizontal and the vertical components of a baseline, B.sub.L/R.sup.u being equal to B*cos and B.sub.L/R.sup.v being equal to B*sin , B being the distance between the viewer's eyes, where Z.sub.L(u.sub.L, v.sub.L) corresponds to the depth value at the position (u.sub.L, v.sub.L), and where f.sup.u and f.sup.v are respectively the products of the focal length by the horizontal and the vertical density on a sensor adapted to sense the viewer's eyes orientation.

(27) As illustrated by FIG. 7, the vector (B.sub.L/R.sup.u,B.sub.L/R.sup.v) is computed from the orientation of the eye axis with respect to the reference axis X of the display device. FIG. 6 illustrates four different cases for which the angle between the eyes axis and the reference axis X is in the range [90,+90]. A sensor mounted on the display device or on any other device may be configured to measure the angle or any other information that allows deducting this angle . For instance, dedicated glasses equipped with specific markers, gyroscopes, or camera-based systems detecting the viewer face as well as its orientation may be used as an eyes orientation sensor. Once measured, the angle is inputted in step 4 in the apparatus 7 implementing the method 1.

(28) As illustrated by FIG. 8, and in order to run the interpolation step 5, the apparatus 7 first determines, in step 5.1, the depth map Z.sub.R of the rendered view 6 from the reference view depth map Z.sub.L, or from the disparity map computed between the reference view 8 and at least a given view 9. In this way, for each pixel of the left view 8, the bidimensional (2D) location of the corresponding point is determined on the ground of the equation (1). Then, this depth value is allocated to the closest pixel in the right view based on the assumption that the depth value remains unchanged between two corresponding points, as expressed by the following equation:
Z.sub.R(u.sub.R,v.sub.R)=Z.sub.L(u.sub.L,v.sub.L)

(29) Then, for each pixel of the right view 6, the apparatus 7 determines the 2D location of the corresponding point in the left view 8 via the following equations:

(30) $\begin{array}{c}{u}_L=u_R+\frac{f^u{B}_{L/R}^u}{Z_L\left(u_L,v_L\right)}\\ v_L=v_R+\frac{f^v{B}_{L/R}^v}{Z_L\left(u_L,v_L\right)}\end{array}$

(31) Then, the apparatus 7 determines the color of each pixel of the right view 6 from the color of the corresponding point in the left view 8 in a step 5.2.

(32) Following the color determination step 5.2, the pixels that do not get a depth value at the end of the location determination step correspond to disoccluded areas 3, hidden in the left view. The missing depth data are then interpolated, in a step 5.3, from the neighboring available depth data from the location determination step output. The corresponding pixels in the occlusion map V.sub.O are then used, in a step 5.4, to fill these disoccluded areas 3 in the interpolated right view 6. Alternatively, when the reference content Ref_C does not comprise an occlusion map V.sub.O, filling the disoccluded areas 3 in the interpolated right view 6, in step 5.4, is conducted by interpolation of the neighboring available pixels.

(33) The combination of both the left view 8 and the right view 6 then forms a rendered tridimensional content adapted to the viewer's eyes orientation .

(34) According to this embodiment, a disparity map is provided into the reference content Ref_C. However, in another embodiment, this disparity map could be computed after the inputting step 2, based on the reference view 8 and at least one given view 9. In this case, the disparity map is not mandatory in the reference content Ref_C.

6.2. Second Embodiment of the Disclosure

(35) In a second embodiment of the disclosure, as illustrated by FIG. 9, the step 5 of determining a rendered view 6 comprises a step of selecting among a plurality of given views 9 a rendered right view 6, as a function of the viewer's eyes orientation relatively to the reference axis X of the display device. In this embodiment, the given views 9, as illustrated by FIG. 9, consist of a set of potential right views having different disparity directions compared to the left view 8. Each of these disparity directions forms an angle with the reference axis X of the display device.

(36) Following the input of the viewer's eyes orientation in step 4, the apparatus 7 is selecting one of the given views 9 so as to minimize the difference between the respective angles formed with the reference axis X by the disparity direction (angle ) and the eyes axis (angle ).

6.3. Third Embodiment of the Disclosure

(37) In a third embodiment of the disclosure, the content inputted in step 2 consists in a reference view 8 and in the viewer eyes axis orientation relatively to the reference axis X of the display device. The viewer eyes orientation corresponds to the disparity direction of a rendered view 6 to determine.

(38) According to this embodiment, a depth map Z.sub.L is first estimated as a function of the reference view 8 and the viewer eyes axis orientation , as described in the 2011 ICIP conference: VISUAL PERTINENT 2D-TO-3D VIDEO CONVERSION BY MULTI-CUE FUSION, by Z. Zhang.

(39) The rendered view 6 is then interpolated as a function of the reference view 8 and the depth map Z.sub.L, as described in the description of the first embodiment of the disclosure (see paragraph 5.1).

6.4 Apparatus According to One Embodiment of the Disclosure

(40) FIG. 10 is a schematic block diagram illustrating an example of an apparatus for determining a rendered tridimensional content, according to one embodiment of the present disclosure.

(41) An apparatus 7 illustrated in FIG. 10 includes a processor 10, a storage unit 11, an interface unit 12 and a sensor 13, which are connected by a bus 14. Of course, constituent elements of the computer apparatus 7 may be connected by a connection other than a bus connection using the bus 14.

(42) The processor 10 controls operations of the apparatus 7. The storage unit 11 stores at least one program to be executed by the processor 10, and various data, including data included in the reference content Ref_C, parameters used by computations performed by the processor 10, intermediate data of computations performed by the processor 10, and so on. The storage unit 11 may notably store the components of the reference content Ref_C and the viewer eyes axis orientation . The processor 10 may be formed by any known and suitable hardware, or software, or by a combination of hardware and software. For example, the processor 10 may be formed by dedicated hardware such as a processing circuit, or by a programmable processing unit such as a CPU (Central Processing Unit) that executes a program stored in a memory thereof.

(43) The storage unit 11 may be formed by any suitable storage or means capable of storing the program, data, or the like in a computer-readable manner. Examples of the storage unit 11 include non-transitory computer-readable storage media such as semiconductor memory devices, and magnetic, optical, or magneto-optical recording media loaded into a read and write unit. The program causes the processor 10 to perform a process for determining a rendered tridimensional content according to an embodiment of the present disclosure as described above with reference to FIG. 4.

(44) The interface unit 12 provides an interface between the apparatus 7 and an external apparatus. The interface unit 12 may be in communication with the external apparatus via cable or wireless communication. In this embodiment, the external apparatus may be a light-field capture device 15. In this case, light-field data can be inputted from the light-field capture device 15 to the apparatus 7 through the interface unit 12, and then stored in the storage unit 11.

(45) The apparatus 7 and the light-field capture device 15 may communicate with each other via cable or wireless communication.

(46) Although only one processor 10 is shown on FIG. 10, a skilled person will understand that such a processor may comprise different modules and units embodying the functions carried out by the apparatus 7 according to embodiments of the present disclosure, such as a module 16, which processes the different components of the reference content and the viewer's eyes orientation in order to determine 5 a rendered view 6 as a function of the viewer's eyes orientation relatively to the reference axis X of the display device. This module may also be embodied in several processors 10 communicating and co-operating with each other.

(47) As will be appreciated by one skilled in the art, aspects of the present principles can be embodied as a system, method or computer readable medium. Accordingly, aspects of the present principles can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, and so forth), or an embodiment combining software and hardware aspects.

(48) When the present principles are implemented by one or several hardware components, it can be noted that a hardware component comprises a processor that is an integrated circuit such as a central processing unit, and/or a microprocessor, and/or an Application-specific integrated circuit (ASIC), and/or an Application-specific instruction-set processor (ASIP), and/or a graphics processing unit (GPU), and/or a physics processing unit (PPU), and/or a digital signal processor (DSP), and/or an image processor, and/or a coprocessor, and/or a floating-point unit, and/or a network processor, and/or an audio processor, and/or a multi-core processor. Moreover, the hardware component can also comprise a baseband processor (comprising for example memory units, and a firmware) and/or radio electronic circuits (that can comprise antennas), which receive or transmit radio signals. In one embodiment, the hardware component is compliant with one or more standards such as ISO/IEC 18092/ECMA-340, ISO/IEC 21481/ECMA-352, GSMA, StoLPaN, ETSI/SCP (Smart Card Platform), GlobalPlatform (i.e. a secure element). In a variant, the hardware component is a Radio-frequency identification (RFID) tag. In one embodiment, a hardware component comprises circuits that enable Bluetooth communications, and/or Wi-fi communications, and/or Zigbee communications, and/or USB communications and/or Firewire communications and/or NFC (for Near Field) communications.

(49) Furthermore, aspects of the present principles can take the form of a computer readable storage medium. Any combination of one or more computer readable storage medium(s) may be utilized.

(50) Thus for example, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable storage media and so executed by a computer or a processor, whether or not such computer or processor is explicitly shown.

(51) Although the present disclosure has been described with reference to one or more examples, a skilled person will recognize that changes may be made in form and detail without departing from the scope of the disclosure and/or the appended claims.