METHOD FOR PROCESSING A HOLOGRAM, AND ASSOCIATED DEVICE, HOLOGRAPHIC DISPLAY SYSTEM AND COMPUTER PROGRAM

Abstract

Disclosed is a method for processing an input hologram HE associated with an input plane, to obtain an output hologram displayable on a holographic screen placed in a plane called the output plane of a display system, viewable from a viewing plane of the system. The method includes: receiving the input hologram and a position of the input plane; obtaining a first transfer matrix representative of a propagation between the input plane and the viewing plane; obtaining a second transfer matrix representative of a propagation between the viewing plane and the output plane; calculating an overall matrix of transfer of a light field emitted by the input hologram, between the input plane and the output plane, by taking the product of the two matrices; and converting the input hologram into the output hologram by applying an operator dependent of the input hologram and on the screen.

Claims

1. A method for processing at least one input hologram associated with at least one input plane in a World reference frame, in order to obtain an output hologram that is capable of being displayed on a holographic screen placed in a so-called output plane of a reproduction system and viewable by an observer from a viewing plane of said system, said system comprising at least one optical element arranged between the viewing plane and the output plane, wherein the method comprises the following steps: obtaining the input hologram and a pose of the associated input plane in the World reference frame; obtaining at least one first ray transfer matrix, called extrinsic matrix, depending on the position and the direction of the observer and representative of a propagation in free space between the input plane and the viewing plane of the reproduction system; obtaining at least one second ray transfer matrix, called intrinsic matrix, representative of a propagation between the viewing plane and the output plane of the reproduction system through said at least one optical element; calculating at least one overall ray transfer matrix of a light field emitted by the input hologram between the input plane and the output plane of the reproduction system, by a product of at least two of said extrinsic and intrinsic matrices; and transforming a light field of the input hologram into a light field of the output hologram by application of at least one linear integral operator determined from the at least one calculated overall ray transfer matrix, as a function of input sizes and input resolutions of the input hologram and of output sizes and output resolutions of the holographic screen.

2. The method according to claim 1, wherein, when the holographic reproduction system comprises a module for non-linear processing of the light field of the input hologram, the step of obtaining at least one intrinsic matrix comprises obtaining a first intrinsic matrix representative of a propagation of the light field through at least one linear optical element located upstream from the non-linear processing module and obtaining a second intrinsic matrix representative of a propagation of the processed light field through at least one other linear optical element located downstream from the non-linear processing module, the step of calculating at least one overall ray transfer matrix comprises calculating a first overall matrix by a product of the at least one extrinsic matrix and the first intrinsic matrix and a second overall matrix from the second intrinsic matrix, the step of transforming the light field of the input hologram comprises the successive application of a first linear integral operator determined from the first overall matrix, an operator of the non-linear processing and a second linear integral operator determined from the second overall matrix.

3. The method for processing at least one input hologram according to claim 2, further comprising: a step of obtaining a pose of the observer in the World reference frame, and a step of evaluating a visibility of at least one input hologram by the observer, wherein the steps of obtaining a first ray transfer matrix, obtaining a second ray transfer matrix and transforming the input hologram are implemented when the at least one input hologram is evaluated as visible by the observer.

4. The method for processing at least one input hologram according to claim 3, further comprising: after detection of a change of the observer's pose in the World reference frame, updating at least one extrinsic matrix and the at least one overall ray transfer matrix as a function of said change, and transforming the input hologram by application of at least one updated linear integral operator determined from the at least one updated overall ray transfer matrix.

5. The processing method according to claim 3, wherein, if the output plane of the reproduction system is not parallel to the input plane of the input hologram, the method further comprises: obtaining a rotation matrix representing rotation of the light field between the input plane and a corrected plane, parallel to the viewing plane, and determining a non-linear operator of transformation of the light field by composition of a Fourier transform, a rotation frequency operator determined from said rotation matrix and an inverse Fourier transform, wherein the transforming step further comprises the application of said non-linear operator previously to said at least one linear integral operator.

6. The processing method according to claim 5, wherein the step of calculating a first overall matrix implements the product of the inverse Fourier transform, the extrinsic matrix and the first intrinsic matrix.

7. The processing method according to claim 1, wherein the light field of the input hologram comprises a plurality of samples depending on the input sizes and input resolutions, and wherein the step of transforming the light field of the input hologram by application of at least one linear integral operator comprises the application to said samples of a discrete Fourier transform and a sampling of the light field transformed as a function of the output resolutions and output sizes of the output hologram.

8. The method for processing at least one input hologram according to claim 1, wherein, when a plurality of input holograms is associated with a plurality of input planes forming the faces of a geometric solid, the method further comprises: selecting input planes corresponding to faces visible from the output plane, wherein obtaining at least one extrinsic matrix, and calculating at least one overall ray transfer matrix are repeated for each of the selected input planes, and the transforming transforms the fields of the selected input planes and sums the transformed fields to obtain the output hologram.

9. The processing method according to claim 1, wherein the at least one obtained extrinsic matrix and the at least one obtained intrinsic matrix are stored into a memory.

10. A device for processing an input hologram associated with an input plane in a World reference frame, in order to obtain an output hologram that is capable of being displayed on a holographic screen placed in a plane, called output plane, of a reproduction system and viewable by an observer from a viewing plane of said system, said system comprising at least one optical element placed between the viewing plane and the output plane, wherein the device comprises a processor configured to: obtain the input hologram and a pose of the associated input plane in the World reference frame; obtain at least one first ray transfer matrix, called extrinsic matrix, depending on the position and the direction of the observer and representative of a propagation in free space between the input plane and the viewing plane of said system; obtain at least one second ray transfer matrix, called intrinsic matrix, representative of a propagation between the viewing plane and the output plane of the reproduction system through said at least one optical element; calculate at least one overall ray transfer matrix of a light field emitted by the input hologram between the input plane and the output plane of the reproduction system, by a product of at least two of said extrinsic and intrinsic matrices; transform a light field of the input hologram into a light field of the output hologram by application of at least one linear integral operator determined from the at least one calculated overall ray transfer matrix, discretized as a function of input sizes and input resolutions of the input hologram and resampled as a function of output sizes and output resolutions of the holographic screen.

11. A holographic reproduction system comprising a module for obtaining an input hologram associated with an input plane in a World reference frame, an output plane in which is placed a holographic screen intended to reproduce a hologram, called the output hologram, a viewing plane in which the output hologram is intended to be viewed by an observer, and at least one optical element, arranged between the viewing plane and the output plane, further comprising a device for processing the input hologram into the output hologram according to claim 10.

12. The holographic reproduction system according to claim 11, the holographic reproduction system being a head-mounted device type, capable of being worn on the head of the observer, wherein the viewing plane is placed in front of the observer's eyes, the output plane is placed out of a vision cone of the observer, the holographic reproduction system comprising: a module capable of receiving an input hologram having input sizes and input resolutions; a module capable of obtaining a pose of the observer in the World reference frame; the holographic screen in the output plane, having said output sizes and said output resolutions; an optical filtering system, of the 4F type, comprising a non-linear filtering module and two thin lenses; a field lens capable of enlarging a viewing field of the light field perceived by the observer on the viewing plane; and a beam splitter capable of redirecting the light field of the hologram in the direction of the viewing plane.

13. (canceled)

14. A non-transitory recording medium, readable by a computer, on which is recorded a computer program comprising program code instruction that, when executed by the computer, perform the method according to claim 1.

15. The method for processing at least one input hologram according to claim 1, wherein the method comprises a step of obtaining a pose of the observer in the World reference frame, a step of evaluating a visibility of at least one input hologram by the observer, and wherein, the steps of obtaining a first ray transfer matrix, obtaining a second ray transfer matrix and transforming the input hologram are implemented when the at least one input hologram is evaluated as visible by the observer.

16. (canceled)

17. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. A non-transitory recording medium, readable by a computer, on which is recorded a computer program comprising program code instruction that, when executed by the computer, perform the method according to claim 2.

Description

6. LIST OF FIGURES

[0066] Other advantages and features of the invention will appear more clearly upon reading of the following description of a particular embodiment of the invention, given by way of simple illustrative and non-limitative example, and of the appended drawings, among which:

[0067] FIG. 1 schematically illustrates the processing of an input hologram made by the invention to adapt it to a holographic reproduction system;

[0068] FIG. 2A schematically describes the arrangement of a first example of holographic reproduction system implementing the invention;

[0069] FIG. 2B describes in further detail the arrangement of the optical elements constituting this first holographic reproduction system;

[0070] FIG. 3 schematically shows the steps of processing an input hologram according to a first embodiment of the invention;

[0071] FIG. 4A schematically describes the arrangement of a second example of holographic reproduction system implementing the invention;

[0072] FIG. 4B describes in further detail the arrangement of the optical elements constituting this second holographic reproduction system;

[0073] FIG. 5A schematically illustrates the visibility cone of an input hologram;

[0074] FIG. 5B schematically illustrates the vision cone of an observer through a holographic reproduction system;

[0075] FIG. 6 illustrates an example of reproduction of a 3D scene based on a plurality of holograms displayed on the faces of a cube, according to a third exemplary embodiment of the invention;

[0076] FIG. 7 schematically describes the method for processing a plurality of holograms associated with the faces of a geometric solid according to this third embodiment of the invention;

[0077] FIG. 8 schematically illustrates a first example of hardware structure of an input hologram processing device according to the invention; and

[0078] FIG. 9 schematically illustrates a second example of hardware structure of an input hologram processing device according to the invention.

7. DESCRIPTION OF A PARTICULAR EMBODIMENT OF THE INVENTION

[0079] In relation with FIG. 1, a 3D scene is considered, from which an input hologram H.sub.E has been generated. It is associated with an input plane P.sub.E, or plane of the input hologram, in which it is intended to be displayed, with given sizes and resolution. In other words, the input data forming this hologram H.sub.E correspond to a plurality of discrete values or samples of a complex light field in a quadrilateral Q.sub.E of this plane, this plurality being defined by the sizes and resolution of the input hologram.

[0080] This input plane P.sub.E is associated with a pose, i.e. known position and direction in a reference frame, called the World reference frame RM. This pose is generally received with the input hologram, when the latter comes for example from a hologram library, and it allows reproducing the hologram in good conditions, i.e. viewing a reconstructed scene at the good scale. Of course, another pose of the input plane P.sub.E can be chosen by a particular application or use, for example virtual or augmented reality or also videoconference, which uses this input hologram.

[0081] For example, the pose of the input plane can be modified as a function of that of a holographic reproduction system (SRH) in such a manner that the face of the corresponding quadrilateral Q.sub.E at the output of the light field is visible by the latter.

[0082] Three examples of holographic reproduction systems SRH have been represented, which are a projector PRO, and body-worn display systems or HMD (for “Head-Mounted Display”), such as Augmented Reality Glasses LRA or a Virtual Reality Helmet VC.

[0083] The general principle of the invention is based on the rapid adaptation of an input hologram H.sub.E for the reproduction thereof by such a holographic system comprising at least one optical element arranged between the hologram display device, also called holographic screen, of the Spatial Light Modulator or SLM type, and a virtual viewing plane P.sub.V of the latter. This adaptation is obtained by transforming the light field of the input hologram in the input plane into an output hologram H.sub.S intended to be displayed by the SLM, by means of an overall linear operator, that compensates for the passing through of the holographic reproduction system optical elements arrangement by the light field.

[0084] Hereinafter, the image, formed by the eye of an observer Ob, of the light field that passes through a quadrilateral Q.sub.V of the viewing plane P.sub.V is called reconstructed scene SC.sub.R. This light field is initially formed on the output plane P.sub.S of the SLM by the product of a plane monochromatic wave of wavelength λ with the complex function representative of the hologram H.sub.S, called output hologram, then transformed by the arrangement of optical elements through which it passes up to the viewing plane P.sub.V. This complex function is recalculated by the processing method according to the invention so as to compensate for the transformation induced by the passing through the just-described optical element arrangement constituting the HMD. That way, the light field that passes through the viewing plane P.sub.V coincides with that of the input hologram that passes through the input plane.

[0085] In order to perform the rapid calculating of the output hologram H.sub.S, the invention performs two propagations: [0086] a direct propagation of the original hologram from the input plane H.sub.E to the viewing plane P.sub.V; then [0087] a so-called “reverse” propagation of the light field from the viewing plane P.sub.V to the output plane P.sub.S.

[0088] Then, to model these two propagations, the invention relies on the above-mentioned “Linear Canonical Transform” (LCT) theory. According to this theory, it is searched to represent a continuous succession of linear optical transformations due to the passing through optical elements, by a ray transfer matrix that is then associated with a single light propagation operator.

[0089] In relation with FIGS. 2A and 2B, a first exemplary embodiment of the invention is considered, when the holographic reproduction system SRH is of the projector PRO type PRO. The input hologram H.sub.E is associated with a plane P.sub.E, called input plane, and with a quadrilateral Q.sub.E in the plane P.sub.E. This plane is positioned in the World reference frame RM, like the holographic reproduction system SHR.

[0090] The projector PRO is fixed and comprises the following elements, arranged along an axis Ax and parallel to the input plane: [0091] Means (not shown) for receiving the input hologram H.sub.E. For example, the system SRH comprises wired or wireless means for the connection to a communication network. The input hologram H.sub.E is associated with an input plane P.sub.E whose pose is supposed to be fixed in the World reference frame RM. It is provided as a discretized form on a quadrilateral Q.sub.E of sizes Sx.sub.E and Sy.sub.E including Nx.sub.E×Ny.sub.E pixels, inscribed in the input plane P.sub.E. By way of example, the holograms available in publicly accessible data libraries are of the order of 8K×4K pixels for sizes Sx.sub.E and Sy.sub.E of the order of about one cm. The greatest holograms available may be close to 72K×36K pixels for sizes Sx.sub.E and Sy.sub.E of the order of about ten cm; [0092] A holographic screen of the SLM type placed in a so-called output plane P.sub.S; [0093] A field lens L.sub.C of focal length f intended to enlarge the view angle of the light field emitted by the SLM, located at a distance d.sub.4 from the plane P.sub.S. Typically, f and d.sub.4 are of the order of 5 cm; [0094] A reflective film FR placed at a distance d.sub.3 from the field lens whose purpose is to reflect the image coming from the holographic projector, as would do a cinema screen. Typically, the distance d.sub.3 is chosen equal to 3 metres; [0095] A virtual viewing plane P.sub.V placed at a distance d.sub.2 in front of the reflective film. The distance d.sub.2 corresponds to a minimum distance of the observer with respect to the reflective film. Typically, the distance d.sub.2 is chosen equal to 50 centimetres. The distance between the viewing plane P.sub.V and the input plane P.sub.E is given by d.sub.1, of the order of 50 centimetres.

[0096] In relation with FIG. 3, we will now describe the steps of a method for processing an input hologram H.sub.E according to this first embodiment of the invention. During a step E.sub.1, an input hologram H.sub.E is received by the holographic reproduction system SRH.

[0097] At this stage, a distinction is made between the following two successive propagations: [0098] a first propagation, called “extrinsic direct propagation”, in free space, of the light field from the input hologram H.sub.E to the viewing plane of the reproduction system SRH; [0099] a second propagation, called “intrinsic reverse propagation”, of the light field through the arrangement of optical elements constituting the SRH up to the plane P.sub.S of the holographic screen SLM.

[0100] At E4, an extrinsic ray transfer matrix M.sub.EX relating to the first propagation is obtained. It is not specific to the input hologram H.sub.E. It can hence have been calculated in a preliminary phase then stored into a local memory MEM. In the example of the projector PRO, it consists of a propagation in free space over a distance d.sub.1 between the input plane P.sub.E and the viewing plane P.sub.V of the projector.

[0101] It is hence simply expressed as:

[00005]$M_{\mathrm{EX}}=\left(\begin{array}{cc}I& \lambda d_{1}I\\ 0& I\end{array}\right)$

[0102] with I the identity matrix.

[0103] At E5, at least one intrinsic ray transfer matrix M.sub.IN relating to the intrinsic reverse propagation of the light field of the hologram H.sub.E, inside the SRH, is obtained, as a product of the ray transfer matrices of the optical elements that constitute it:

M.sub.IN=M.sub.6.Math.M.sub.5.Math.M.sub.4.Math.M.sub.3.Math.M.sub.2,

with:

[00006]$\text{-}{M}_2=\left(\begin{array}{cc}I& \lambda d_{2}I\\ 0& I\end{array}\right)$ [0104] for the propagation in free space over a distance d.sub.2 between the viewing plane P.sub.V and the reflective film FR;

[00007]$\text{-}{M}_3=\left(\begin{array}{cc}I& 0\\ 0& -I\end{array}\right)$ [0105] for the reflection or the rays by the reflective film FR;

[00008]$\text{-}{M}_4=\left(\begin{array}{cc}I& \lambda d_{3}I\\ 0& I\end{array}\right)$ [0106] for the propagation in free space over a distance d.sub.3 between the reflective film FR and the field lens L.sub.C;

[00009]$\text{-}{M}_5=\left(\begin{array}{cc}I& 0\\ -I/\lambda f& I\end{array}\right)$ [0107] for the propagation of the light field of the hologram H.sub.E through the field lens L.sub.C; and

[00010]$\text{-}{M}_6=\left(\begin{array}{cc}I& \lambda d_{4}I\\ 0& I\end{array}\right)$ [0108] for the propagation in free space over a distance d.sub.4 between the field lens L.sub.C and the output plane P.sub.S. It will be noted that such a ray transfer matrix makes it possible to model a so-called linear propagation, in the sense that the expression of the rays at the output of the optical element is linear as a function of the expression of the input rays, these rays being expressed by a point in space, respectively input and output ones, and angles, respectively input and output ones.

[0109] In the case where the system SRH would comprise a non-linear processing module interposed between two linear optical elements, not only it would be required to model the effect of this non-linear processing on the light field by means of a general light field transformation operator, but it would also be necessary to determine a first ray transfer matrix to model the effect of the optical elements located upstream from the non-linear transfer module on the light field of the hologram, and a second ray transfer matrix to model that of the optical elements located downstream from the non-linear processing module. An example will be described in more details in relation with FIGS. 4A and 4B.

[0110] It is herein supposed that the holographic reproduction system PRO comprises no non-linear processing module. Hence, a single intrinsic ray transfer matrix M.sub.IN is obtained.

[0111] At E6, an overall ray transfer matrix M.sub.G of a light field emitted by the input hologram H.sub.E from the input plane P.sub.E to the output plane P.sub.S of the reproduction system SRH is calculated, by a product of the extrinsic ray transfer matrix M.sub.EX and the intrinsic ray matrix M.sub.IN:

M.sub.G=M.sub.EX.Math.M.sub.IN

[0112] At E7, the input hologram H.sub.E is transformed by application of an integral transformation operator T obtained from the calculated overall matrix M.sub.G, in accordance with the above-described equations (1) to (3). This operator T performs the transformation of the light field of the input hologram H.sub.E through the arrangement of optical elements of the SRH up to the output plane P.sub.S of the holographic screen.

[0113] Hence, if the overall matrix M.sub.G is equal to

[00011]$\left(\begin{array}{cc}A& B\\ C& D\end{array}\right),$

with A, B, C and D 2×2 matrices, the operator T is defined in theory as the operator that transforms the field ƒ of the input hologram H.sub.E into the field ƒ′ of the output hologram

ƒ′(X′)=T(ƒ)(X′)=det(iB).sup.−1/2∫ƒ(X)e.sup.iπP(X,X′)dX,  (2)

with

P(X,X′)=B.sup.−1AX+DB.sup.−1X′−2X.sup.tB.sup.−1X′,  (3)

[0114] and X, X′ the vectors of spatial components of the light fields of the input plane P.sub.E and the output plane P.sub.S, respectively.

[0115] In practice, the operator T is discretized by transforming the integrals of the equation (2) into discrete sums to take into account the resolution Nx.sub.E, Ny.sub.E and the sizes Sx.sub.E, Sy.sub.E of the input hologram ƒ, and the resulting field ƒ′ is sampled according to the resolution Nx.sub.S, Ny.sub.S and the sizes Sx.sub.S, Sy.sub.S of the output hologram H.sub.S, as follows:

[00012]$\begin{array}{cc}{f}_{k,l}^{\prime }={\det \left(\mathrm{iB}\right)}^{-\frac{1}{2}}\underset{i=1}{\overset{\mathrm{NxE}}{.Math.}}\underset{j=1}{\overset{\mathrm{NyE}}{.Math.}}{f}_{i,j}{e}^{i\pi P\left(X_{i,j},X_{k,l}^{\prime }\right)}{\Delta }_X& \left(4\right)\end{array}$

[0116] where k, l are the indices of the sample ƒ′.sub.k,l, which corresponds to the discretized version of the light field ƒ′ transformed by the linear operator T;

[0117] with

ƒ.sub.i,j=ƒ(X.sub.i,j)  (5)

[00013]$\begin{array}{cc}{f^{\prime }}_{k,l}=f^{\prime }\left({X^{\prime }}_{k,l}\right)& \left(6\right)\\ X_{i,j}=\left(\begin{array}{c}{\mathrm{i\Delta }}_x\\ {\mathrm{j\Delta }}_y\end{array}\right)& \left(7\right)\\ {X^{\prime }}_{k,l}=\left(\begin{array}{c}{{\mathrm{k\Delta }}^{\prime }}_x\\ {{\mathrm{j\Delta }}^{\prime }}_y\end{array}\right)& \left(8\right)\\ {{\Delta }^{\prime }}_x=\left(\begin{array}{c}{{\mathrm{k\Delta }}^{\prime }}_x\\ {{\mathrm{j\Delta }}^{\prime }}_y\end{array}\right)& \left(9\right)\\ \mathrm{and}& \\ {\Delta }_X=\frac{S_x{S}_y}{\mathrm{NxE}.Math.\mathrm{NyE}}& \left(10\right)\end{array}$

[0118] Advantageously, the function ƒ′.sub.k,l is calculated according to a known method that consists in extracting therefrom a discrete Fourier transform:

[00014]$\begin{array}{cc}{f^{\prime }}_{k,l}={\det \left(\mathrm{iB}\right)}^{-\frac{1}{2}}{e}^{i\pi D{B}^{-1}{X^{\prime }}_{k,l}}{\Delta }_{X}TF{D}_B\left(G\right)\left({X^{\prime }}_{k,l}\right)& \left(11\right)\end{array}$

[0119] TFD.sub.B denotes the discrete Fourier transform staggered by a factor B.sup.−1, G being the following discrete function:

[00015]$\begin{array}{cc}{e}^{i\pi B^{-1}A{X}_{i,j}}{f}_{i,j}& \left(12\right)\end{array}$

[0120] A number of samples ƒ′.sub.k,l corresponding to the resolutions Nx.sub.S, Ny.sub.S and sizes Sx.sub.S, Sy.sub.S of the output holographic screen SLM is hence calculated.

[0121] The light field of the output hologram that is to be reproduced by the SLM for an observer Ob to visualize correctly the scene reconstructed by the input hologram H.sub.E in the viewing plane P.sub.V is hence obtained.

[0122] It is understood that the fact to group into a single linear operator all the transformations to be applied to the input light field makes it possible to limit the total number of Fourier transforms to be calculated and hence to reduce the complexity of the system.

[0123] In relation with FIGS. 4A and 4B, we will now describe a second exemplary embodiment of the processing method according to the invention, when the holographic reproduction system SRH is a device intended to be worn on the head of a user, such as a head-mounted display or augmented-reality glasses. It will be called hereinafter HMD.

[0124] In this example, the HMD comprises: [0125] a module (not shown) for receiving an input hologram H.sub.E, similar to the above-described one. The means for the connection to the communication network are herein preferentially of the wireless type; [0126] a module (not shown) for obtaining a pose T.sub.Ob, R.sub.Ob of the observer Ob in the reference frame RM. As already mentioned, this pose is generally expressed as 6 parameters representative of a translation T.sub.Ob and a rotation R.sub.Ob in the reference frame RM. It can be obtained with a position tracking device, of the IMU (“Inertial Measurement Unit”) type, known from the person skilled in the art, placed on the HMD; [0127] a display device of the holographic screen SLM type, placed in an output plane P.sub.S. The SLM has the sizes Sx.sub.S, Sy.sub.S and it is discretized into Nx.sub.S×Ny.sub.S pixels. The usual values of N×s and Ny.sub.S vary between 720 and 4096, for sizes Sx.sub.S and Sy.sub.S of the order of about one cm. It is generally placed on the side or on the top of the observer's head so as to be out of his viewing field; [0128] a filtering system of the 4F type, know from the person skilled in the art, intended to suppress frequencies associated with directions of emission of undesirable diffraction orders. It comprises a non-linear processing module MTNL and two thin lenses L.sub.1 and L.sub.2; [0129] a field lens L.sub.C intended to enlarge the light field emitted by the SLM; and [0130] a beam splitter SF intended to redirect the light field of the hologram in the direction of a viewing plane P.sub.V behind which the eye of an observer Ob is positioned. Indeed, the SLM is placed on the side or on the top of the observer's head, whereas the viewing plane of the hologram is located in front of the observer's eyes, as a glass of a pair of glasses. The viewing plane P.sub.V and the output plane P.sub.S are hence not parallel to each other. That way, the possibility for the observer Ob to see his real environment by transparency in superimposition on a 3D scene SC.sub.R reconstructed by the hologram is preserved.

[0131] The HMD/observer unit being mobile in the World reference frame RM, the input plane P.sub.E and viewing plane P.sub.V are this time not necessarily parallel to each other.

[0132] In relation with FIG. 4B, the propagation path of the light field of the input hologram from the input plane P.sub.E to the output plane of the SLM is decomposed as follows: [0133] a rotation EO.sub.1 between the input plane P.sub.E and a corrected plane P.sub.R parallel to the viewing plane P.sub.V; [0134] a propagation EO.sub.2 in free space from the corrected plane P.sub.R up to the viewing plane P.sub.V, over a length d.sub.1; [0135] a propagation EO.sub.3 in free space, over a length d.sub.2 from the viewing plane P.sub.V up to the beam splitter SF. The passing through the beam splitter SF is omitted because it acts as a mirror; [0136] a propagation EO.sub.4 in free space, from the beam splitter SF up to the field lens L.sub.C, over a length d.sub.3; [0137] a passing EO.sub.6 through the field lens L.sub.C; [0138] a propagation EO.sub.6 in free space between the field lens L.sub.C and the first lens L.sub.1 of the filtering system 4F, over a distance d.sub.4; [0139] a passing EO.sub.7 through the first lens L.sub.1 of the filtering system 4F; [0140] a propagation EO.sub.8 in free space between the first lens L.sub.1 of the filtering system 4F and the non-linear processing module MTNL, over a distance d.sub.5; [0141] a processing EO.sub.9 of non-linear filtering MTNL; [0142] a propagation EO.sub.10 in free space between the non-linear processing module MTNL and the second lens L.sub.2 of the system 4F, over a distance d.sub.6; [0143] a passing EO.sub.11 through the second lens L.sub.2 of the system 4F; and [0144] a propagation EO.sub.12 in free space between the second lens L.sub.2 of the filtering system 4F and the output plane P.sub.S of the SLM, over a distance d.sub.7.

[0145] Each of the optical elements or processing operations mentioned hereinabove is denoted EO.sub.1, EO.sub.2, EO.sub.3, EO.sub.4, EO.sub.5, EO.sub.6, EO.sub.7, EO.sub.8, EO.sub.9, EO.sub.10, EO.sub.11 and EO.sub.12 respectively, and

[0146] the propagation operator representing the passing through the optical element EO.sub.i, i.e. the operator that, with a light field arriving at the input of the optical element EO.sub.i, associates the light field that exits therefrom, is denoted T.sub.i. The propagation from the input hologram H.sub.E to the SLM on the output plane H.sub.S is hence represented by the operator:

T=T.sub.12∘T.sub.11∘T.sub.10∘T.sub.9∘ . . . ∘T.sub.1,

[0147] where ∘ denotes the composition of the operators.

[0148] This chain of operators is composed of a so-called extrinsic part, relating to the direct propagation of the light field, in free space between the input plane P.sub.E and the viewing plane P.sub.V, and of a so-called intrinsic part, relating to the reverse propagation of the light field, inside the augmented reality system SRH. The extrinsic part depends on the position and direction of the observer with respect to the input plane P.sub.E of the hologram H.sub.E and hence changes at each displacement of the observer Ob, whereas the intrinsic part is independent of the movements thereof and can be modelled once for all in a preliminary phase. The extrinsic part corresponds to the processing by the operators T.sub.1 and T.sub.2, the intrinsic part corresponds to the processing by the product of the operators T.sub.3 to T.sub.12.

[0149] In this example, two processing operations undergone by the light field on the path thereof between the input plane P.sub.E and the output plane P.sub.S cannot be represented by a ray transfer matrix. They are: [0150] 1) the rotation EO.sub.1 of the light field between the input plane P.sub.E and a corrected plane P.sub.R, parallel to the viewing plane P.sub.V corresponding to the relative direction between the HMD/observer system and the plane P.sub.E of the input hologram; [0151] 2) the filtering of the light field by the non-linear processing module MTNL, which corresponds to the optical element EO.sub.9.

[0152] It is understood that, in this second example, it is not possible to model the transformation of the light field by a single operator of the LCT type, due to the fact that these two non-linear processing operations segment the chain of propagation operators into several propagation operators, 3 of which are of the LCT type.

[0153] The steps of the method for processing a input hologram according to the invention will now be described in this particular exemplary embodiment.

[0154] At E1, a hologram H.sub.E is received.

[0155] At E2, a pose of the observer Ob in the World reference frame RM is obtained. It is used to evaluate, at E3, if the input hologram is visible by the observer Ob. It is herein considered that the observer is integral with the HMD system that he wears on his head. That way, the pose of the observer corresponds to the pose of the HMD system.

[0156] More precisely, in relation with FIGS. 5A and 5B, it is considered that the hologram is visible by the observer Ob if, on the one hand, the quadrilateral Q.sub.E of the input hologram H.sub.E is in his vision cone CV.sub.o, and if, on the other hand, the observer Ob himself is in the emission or visibility cone CV.sub.E of the hologram H.sub.E, i.e. inside an angular sector in which the hologram is capable of emitting a light signal.

[0157] In other words, the matter is to make a test of reciprocal visibility, which may, for example, be calculated as follows:

[0158] Four planes PH.sub.1, PH.sub.2, PH.sub.3 and PH.sub.4 inclined by an angle φ with respect to the quadrilateral Q.sub.E of the plane P.sub.E and respectively intersecting the four edges of Q.sub.E are defined. The angle φ may be chosen as the maximum diffraction angle given by φ=arcsin(X/(λ/(2Sx.sub.E/Nx.sub.E)). It is, for example, of 4.5 degrees for a hologram of resolution 72K with a size Sx.sub.E of the order of 30 cm. Likewise, four planes PO.sub.1, PO.sub.2, PO.sub.3 and PO.sub.4 inclined by an angle Υ with respect to the quadrilateral Q.sub.V, Υ being the maximum viewing angle of the holographic reproduction system HMD, are defined. For the existing HMDs, Υ is of the order of 30 degrees. The visibility cone CV.sub.E is delimited by the quadrilateral Q.sub.E and the planes PH.sub.1, PH.sub.2, PH.sub.3 and PH.sub.4, and the vision cone CV.sub.O of the observer is delimited by the quadrilateral Q.sub.V and the planes PO.sub.1, PO.sub.2, PO.sub.3 and PO.sub.4. The plane of the hologram P.sub.E and the viewing plane P.sub.V are then decreed mutually visible if at least one of the vertices of the quadrilateral Q.sub.E is inside CV.sub.O, and if one of the vertices of the quadrilateral Q.sub.V defining P.sub.V is inside CV.sub.E. We call inner side of a cone plane the side that is located inside the cone. For that purpose, it is verified that at least one vertex in question is located on the inner side of all the planes defining the concerned cone.

[0159] If the visibility test is positive, then the method continues with step E4. Otherwise, it is stopped until a new pose of the observer Ob is received.

[0160] At E4, the ray transfer matrices M.sub.12, M.sub.11, M.sub.10, M.sub.8, M.sub.7, M.sub.6, M.sub.5, M.sub.4, M.sub.3, M.sub.2, M.sub.f-1 and M.sub.f, respectively associated with the above-listed optical elements EO.sub.12 to EO.sub.10 and EO.sub.8 to EO.sub.2 are obtained.

[0161] At E5, a first overall ray transfer matrix M.sub.A corresponding to the path of the light field between the output of the filtering module EP.sub.9 and the output plane P.sub.S is calculated:

M.sub.A=M.sub.12.Math.M.sub.11.Math.M.sub.10 [0162] with

[00016]$M_{12}=\left(\begin{array}{cc}I& \lambda d_{7}I\\ 0& I\end{array}\right)$  for the propagation or the light field in free space over a distance d.sub.7. It is for example possible to choose d.sub.7=2 cm;

[00017]$M_{11}=\left(\begin{array}{cc}I& 0\\ -I/\lambda f_2& I\end{array}\right)$ [0163] for the propagation of the light field through the thin lens L.sub.2 of focal length f.sub.2. It is for example possible to place a lens of focal length f.sub.2=5 cm;

[00018]$M_{10}=\left(\begin{array}{cc}I& \lambda d_{6}I\\ 0& I\end{array}\right)$ [0164] for the propagation of the light field in free space over a distance d.sub.6. It is for example possible to choose d.sub.6=5 cm.

[0165] And at E6, a second overall ray transfer matrix M.sub.B corresponding to the path of the light field between the corrected input plane P.sub.R and the input of the non-linear filtering module EO.sub.9:

M.sub.B=M.sub.8M.sub.7M.sub.6M.sub.5M.sub.4M.sub.3M.sub.2M.sub.f-1 [0166] with

[00019]$M_{8=}\left(\begin{array}{cc}I& \lambda d_{5}I\\ 0& I\end{array}\right)$  for the propagation of the light field in free space over a distance d.sub.5. It is for example possible to choose d.sub.5=5 cm;

[00020]$M_7=\left(\begin{array}{cc}I& 0\\ -I/\lambda f_1& I\end{array}\right)$ [0167] for the propagation of the light field through the thin lens L.sub.1 of focal length f.sub.1. It is for example possible to place a lens of focal lens f.sub.1=5 cm;

[00021]$M_6=\left(\begin{array}{cc}I& \lambda d_{4}I\\ 0& I\end{array}\right)$ [0168] for the propagation or the light field in free space over a distance d.sub.4. It is for example possible to choose d.sub.4=1 cm;

[00022]$M_5=\left(\begin{array}{cc}I& 0\\ -I/\lambda f_C& I\end{array}\right)$ [0169] for the propagation of the light field through the thin lens L.sub.C of focal length f.sub.C. It is for example possible to place a lens of focal lens f.sub.C=2 cm;

[00023]$M_4=\left(\begin{array}{cc}I& \lambda d_{3}I\\ 0& I\end{array}\right)$ [0170] for the propagation of the light field in free space over a distance d.sub.3. It is for example possible to choose d.sub.3=1 cm;

[00024]$M_3=\left(\begin{array}{cc}I& \lambda d_{2}I\\ 0& I\end{array}\right)$ [0171] for the propagation of the light field in free space over a distance d.sub.2. It is for example possible to choose d.sub.2=1 cm;

[00025]$M_2=\left(\begin{array}{cc}I& \lambda d_{1}I\\ 0& I\end{array}\right)$ [0172] for the propagation of the light field in free space over a distance d.sub.1. d.sub.1 is of the order of about ten centimetres; [0173] M.sub.f.sup.−1 denotes an inverse Fourier transform.

[0174] M.sub.A is intrinsic and hence independent of the observer's displacements.

[0175] M.sub.B is partly extrinsic and intrinsic and depends on the observer's displacements, due to the matrix M.sub.2.

[0176] It must be recalculated at each displacement of the observer Ob.

[0177] A non-linear operator T.sub.1 for the transformation of the light field by the optical element EO.sub.1 is considered. It consists of a rotation of the direction of the input plane P.sub.E towards the direction of the viewing plane P.sub.V. The operator T.sub.1 can hence be decomposed as follows:

T.sub.1=U∘T.sub.f

[0178] where: [0179] T.sub.f denotes the 2D Fourier transform. This is a linear operator; [0180] U is the operator that, to a function ƒ, associates ƒ∘H, where H is the function that, to two spatial frequencies u and v, associates the two first components of the following product:

[00026]$R.Math.\left(\begin{array}{c}\begin{array}{c}u\\ v\end{array}\\ \sqrt{{\lambda }^{-2}-u^2-{\nu }^2}\end{array}\right),$

[0181] where R is a rotation matrix such that its application to the plane P.sub.E corrects the latter for it to be parallel to the viewing plane P.sub.V according to a method known by the person skilled in the art.

[0182] An operator LCT T.sub.A corresponding to the matrix M.sub.A and an operator LCT T.sub.B corresponding to the matrix M.sub.B are also considered, in accordance with the matching established by the equations (1) to (3).

[0183] We have T.sub.A=T.sub.12∘T.sub.11∘T.sub.10 and T.sub.B=T.sub.8∘ . . . T.sub.2∘T.sub.f-1.

[0184] It is noted that the inverse Fourier transform T.sub.f-1 can be herein integrated to the operator T.sub.B, because it is a linear operation.

[0185] An overall operator T for transforming the light field is deduced by means of the following expression:

T=T.sub.A∘T.sub.9∘T.sub.B∘U∘T.sub.f.

[0186] For example, the filtering T.sub.9 can be defined as the operator that, to a function ƒ, associates ƒ.Math.H (product of functions), where H is the function that, to a couple (u, v) of real numbers, associates 1 if v>0 and 0 otherwise.

[0187] For example, it is a directional filtering blocking the rays directed upward, u and v representing the spatial frequency variables of the light signal. As a variant, a filter of bass-band type could also be used.

[0188] At E7, the output hologram H.sub.S to be displayed on the screen SLM of the output plane so that the observer Ob can see, in the viewing plane P.sub.V of his HMD, the 3D scene SC.sub.R as reconstructed by the input hologram in the input plane, is now calculated.

[0189] For that purpose, the samples of the light field of the input hologram are transformed by successive applications to them of the operators T.sub.A, T.sub.9, T.sub.B, U and T.sub.f. As regards the linear operators, the successive transformations are made, as in the first exemplary embodiment, from the overall matrices M.sub.A and M.sub.B, by application to the plurality of samples of the input light field of a discrete Fourier transform according to the above-described equation (11). The transformed light field is resampled to obtain the number of samples f′.sub.k,l corresponding to the output resolutions Nx.sub.S, Ny.sub.S and output sizes Nx.sub.S, Ny.sub.S of the output hologram H.sub.S.

[0190] In a variant embodiment of the invention, we consider now the case in which the observer Ob wearing the system HMD moves in the World reference frame RM. It is supposed that the system HMD is equipped with a module for obtaining the position/direction of the observer wearing the HMD in the World reference frame, which allows it to obtain a regular updating of this position/direction and to detect a displacement of the observer Ob with respect to the plane of the input hologram P.sub.E. The existing HMDs allow for example an updating of this position/direction with a temporal frequency of the order of a few milliseconds. It is supposed that the just-described method according to the invention has been implemented at the instant t.sub.0, then that a displacement of the observer Ob has been detected at an instant t.sub.1 with respect to an instant t.sub.0.

[0191] This detection triggers the updating of the operator U of rotation of the input plane P.sub.E with respect to the viewing plane P.sub.V, of the matrix M.sub.2 corresponding to the propagation in free space over a distance d.sub.1(t.sub.1), and hence of the overall matrix M.sub.B.

[0192] A new operator T.sub.B is calculated from the updated version of the overall matrix M.sub.B(t.sub.1) at the instant t.sub.1.

[0193] The aggregation of the operators by multiplication of the ray transfer matrices reduces the number of Fourier transforms to be calculated, which contributes to reduce the overall calculation time of the transformed light field. Indeed, as explained hereinabove, the application of each operator T.sub.i to the complex light field of the input hologram H.sub.E is performed by calculating a discrete Fourier transform. Thanks to the invention, the fact to translate the transformations induced by the passing through a plurality of optical elements by means of a limited number of overall operators, reduces proportionally the number of Fourier transforms to be applied, and hence the complexity of the processing method.

[0194] In relation with FIGS. 6 and 7, a third exemplary embodiment of the invention will now be described. In this third exemplary embodiment, several input holograms H.sub.Ei, with i an integer comprised between 1 and N, are received, which are associated with the face planes of a geometric solid, such as for example a rectangular parallelepiped. In other words, there are as many reconstructed scenes as solid faces, which makes it possible to offer the observer a view of the scene from a wider range of points of view.

[0195] It is understood that this third embodiment is particular suited to a reproduction system of the HMD type, according to which the observer can move about the plane P.sub.E of the input hologram, and hence view certain faces of the geometric solid as a function his position/direction in the World Coordinate System RM.

[0196] In the example of FIG. 6, the geometric solid considered is a cube and the holographic reproduction system receives 6 holograms H.sub.Ei, with i an integer comprised between 1 and I=6, the hologram H.sub.Ei being associated with one of the six faces Fi of the cube. In relation with FIG. 7, the method for processing this plurality of input holograms {H.sub.Ei} according to this second embodiment of the invention will now be described.

[0197] During a step E1′, the plurality of holograms {H.sub.Ei} is received.

[0198] At E2′, a pose of the observer Ob is obtained. At E3′, at least one face F.sub.j, with j comprised between 1 and N, of the cube of which the observer is liable to view the associated hologram H.sub.Ej is selected. In other words, it searched if there exist at least one face Fi.sub.j for which the field of the associated hologram H.sub.Ei.sub.j is viewable by the observer Ob. Advantageously, the above-described visibility test is performed for each of the faces Fi.

[0199] Two cases are contemplated: [0200] A single visible face Fi.sub.j is selected and a single corresponding hologram H.sub.Ei.sub.j will be recalculated by the method according to the invention. This face will be for example the closest to the observer or that which passes the best the above-described visibility test; [0201] Several visible faces Fi.sub.1, Fi.sub.2 . . . Fi.sub.j are selected, with i an integer comprised between 1 and I, and the corresponding holograms are recalculated in order to compose an output hologram from the recalculated holograms. The contributions of each field are then summed into a total output hologram H.sub.S.

[0202] In the first case, it is supposed that the face Fi.sub.0 has been selected and then the method for processing the selected hologram H.sub.Ei.sub.0 is then executed as described in relation with FIG. 2, which corresponds to steps E4′ to E8′ of FIG. 7.

[0203] In the second case, steps E4′ to E8′ are repeated for each of the selected input holograms H.sub.Ei.sub.j, as long as all the selected faces have not been processed (E9′, E10′). Once all the output holograms H.sub.Si.sub.j calculated, an output hologram H.sub.S is composed from the J holograms H.sub.Si.sub.j, at E11′.

[0204] With the invention, it becomes possible for an observer wearing a system HMD to turn about the scene reconstructed by the plurality of holograms associated with the faces of the geometric solid.

[0205] It will be noted that the just-described invention can be implemented by means of software and/or hardware components. From this perspective, the terms “module” and “entity”, used in this document, can correspond either to a software component, or to a hardware component, or also to a set of hardware and/or software components, adapted to implement the function(s) described for the concerned module or entity.

[0206] In relation with FIGS. 8 and 9, we will now present the simplified structure of a processing device 100 adapted to implement the method for processing an input hologram according to any one of the particular embodiments of the invention that have just been described in relation with FIGS. 2 to 7.

[0207] The processing device is adapted to process at least one input hologram and to produce an output hologram.

[0208] The processing device 100 is in particular configured to: [0209] receive at least one input hologram and one associated pose in the World Coordinate System; [0210] obtain at least one first matrix, called extrinsic geometric matrix (Me), representative of a propagation in free space between the reference plane and the output plane of the reproduction system, said matrix being function of the reference position and the output position; [0211] obtain at least one second matrix, called intrinsic geometric matrix (Mi), representative of a propagation between the output plane and the input plane of the reproduction system through at least one said optical element; [0212] calculate at least one overall matrix M.sub.G for geometric transformation of a light field emitted by the input hologram between the reference plane and the input plane of the reproduction system, by a product of at least two of said matrices; [0213] transform the input hologram into the output hologram by application of at least one linear integral operator determined from the at least one calculated matrix, as a function of an input resolution Nx.sub.E, Ny.sub.E and of input sizes of the input hologram (P.sub.E) and of output resolutions Nx.sub.S, Ny.sub.S and of output sizes Sx.sub.S, Sy.sub.S of the holographic screen.

[0214] According to a particular embodiment of the invention illustrated by FIG. 8, the processing device 100 has the conventional architecture of a computer and comprises in particular a processing unit 110, equipped with a processor μ.sub.1, and piloted by a computer program Pg.sub.1 120, stored in a memory 130 and implementing the method according to the invention.

[0215] At the initialization, the code instructions of the computer program Pg.sub.1 120 are for example loaded in a memory RAM before being executed by the processor of the processing unit 110. The processor of the processing unit 110 implements the steps of the above-described method, according to the instructions of the computer program 120.

[0216] According to another particular embodiment of the invention illustrated by FIG. 9, the processing method is implemented by functional modules. For that purpose, the processing device 100 comprises at least the following functional modules: [0217] a module REC. for receiving one or several input holograms H.sub.E; [0218] a module OBT. M.sub.EX for obtaining at least one first, so-called extrinsic, ray transfer matrix, representative of a propagation in free space between the reference plane and the output plane of the reproduction system; [0219] a module OBT. M.sub.IN for obtaining at least one second, so-called intrinsic, ray transfer matrix, representative of a propagation between the output plane and the input plane of the reproduction system through at least one said optical element; [0220] a module CALC. M.sub.G for calculating at least one overall ray transfer matrix of a light field emitted by the input hologram between the reference plane and the input plane of the reproduction system, by a product of said matrices; and [0221] a module TRANS for transforming the light field of the input hologram H.sub.E by application of at least one linear integral operator determined from the at least one calculated matrix, as a function of an input resolution Nx.sub.E, Ny.sub.E and input sizes of the input hologram (P.sub.E) and of an output resolution Nx.sub.S, Ny.sub.S and output sizes Sx.sub.S, Sy.sub.S of the holographic screen.

[0222] Optionally, the device 100 comprises a module OBT. T.sub.Ob, R.sub.Ob for obtaining a pose of the observer in the World reference frame and a unit SEL H.sub.j for selecting at least one input hologram, visible by the observer.

[0223] According to an embodiment of the invention, the device 100 comprises a module COMPO for composing several output holograms obtained by transformation of input holograms selected as visible by the observer into a single output hologram.

[0224] The device 100 further comprises a module M1 for storing the ray transfer matrices calculated by the invention, as well as operators determined from these matrices.

[0225] The units are piloted by the processor μ1 of the processing unit 110.

[0226] The processing unit 110 cooperates with the different above-described functional modules and the memory MEM1 in order to implement the steps of the processing method. The different above-described functional modules may be in hardware and/or software form. In software form, such a functional module may include a processor, a memory and program code instructions to implement the function corresponding to the module when the code instructions are executed by the processor. In hardware form, such a functional module may be implemented by any type of adapted coding circuits, such as for example and non-limitatively, microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), a logical unit wiring.

[0227] Advantageously, such a device 100 may be integrated to a holographic reproduction system SRH. The device 100 is then arranged to cooperate at least with the following module of the system SRH: [0228] a data transmission/reception module E/R, through which the input hologram(s) H.sub.E are received via a telecommunication network, for example a wired network or a wireless network; and [0229] as the case may be, a module POS for tracking the position of the observer in the World reference frame RM.

[0230] It is obvious that the above-mentioned embodiments have been given for purely informative and non-limiting purposes, and that many modifications can be easily made by the person skilled in the art without thereby departing from the framework of the invention.