Iterative phase retrieval with parameter inheritance

Abstract

An iterative method of retrieving phase information in the Fourier domain representative of the n.sup.th frame of a sequence of 2D image frames. The method comprises using at least one parameter related to retrieval of phase information Fourier domain of the (n1) frame as a starting point for the iterative method.

Claims

1. A method of retrieving phase information in the Fourier domain, representative of the n.sup.th frame of a sequence of image frames, for holographic reconstruction of the image frames, the method comprising for each image frame: executing an algorithm comprising m iterations of the following sequence of steps, where m is an integer: establishing a first data set in the spatial domain having plural elements, each element having first phase information and first amplitude information to define respective phase and amplitude distributions, wherein the amplitude information of the first iteration is representative of the image frame; performing a Fourier transform on the first data set to produce a second data set in the frequency domain having plural elements, each element having second amplitude and second phase information to define respective phase and amplitude distributions; quantising the second phase information; forming a third data set having plural elements, each element having third amplitude and third phase information, the third phase information being the quantised phase information derived from the second data set; inverse Fourier transforming the third data set to produce a fourth data set having plural elements, each element having fourth amplitude and fourth phase information; modifying the amplitude information of the fourth data set by subtracting the amplitude information of the image frame from the amplitude information of the fourth data set, scaling the subtracted amplitude information by a gain element and subtracting the scaled amplitude information from the amplitude information of the image frame to produce a fifth data set having plural elements, each element having fifth amplitude and fifth phase information, wherein the phase information of the fifth data set is the phase information of the fourth data set; and using the fifth data set as the first data set of the next iteration, wherein the phase information of the third data set is usable to produce a holographic representation of the image frame at an image plane, the method further comprising the step of: receiving a control parameter from the algorithm related to retrieval of phase information in the Fourier domain of the (n1).sup.th frame and using the control parameter in the sequence of steps, wherein the control parameter is a parameter used to control gain or attenuation, wherein the control parameter is the gain element .

2. A method as claimed in claim 1 wherein the image frames are 2D or 3D image frames.

3. A method as claimed in claim 1 wherein the first amplitude information of the first data set is derived from intensity information of the nth frame.

4. A method as claimed in claim 1 wherein the fifth phase information of the fifth data set of the final iteration related to the (n1)th frame forms the phase information for the first data set of the first iteration related to the n.sup.th frame.

5. A method as claimed in claim 1, wherein the first phase information of the first data set for an m.sup.th iteration is a scaled form of the difference between the phase information of the fourth data set for the (m1) and (m2) iterations related to the n.sup.th frame.

6. A method as claimed in claim 1 further comprising stopping the iteration when the mean square error between the fifth amplitude information of the fifth data set and the intensity information of the image frame is less than a threshold.

7. A projector for forming a 2D video image, the projector comprising a spatial light modulator comprising a 2D array of phase modulating elements; processing means arranged to retrieve phase information in the Fourier domain representative of the nth frame of a sequence of 2D image frames in the spatial domain using the method of claim 1 and driving means arranged to drive the array of phase modulating elements of the spatial light modulator to display data representative of a Fourier transform of a nth frame of a sequence of 2D image frames.

8. A projector as claimed in claim 7 further comprising a light source.

9. A method as claimed in claim 1 wherein the amplitude information of the first data set for the first iteration is derived from intensity information of the image frame.

10. A method as claimed in claim 1 wherein the amplitude information of the first data set for the first iteration is based on amplitude information in the Fourier domain of the (n1).sup.th frame.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the present invention will now be described to the accompanying drawings in which:

(2) FIG. 1 is a schematic showing a reflective SLM, such as a LCOS, produce a holographic reconstruction at a replay field location;

(3) FIG. 2 is a chart showing the function of a modified Gerchberg-Saxton algorithm;

(4) FIG. 3 shows the mean squared error differences from one frame to the next for an example AVI file;

(5) FIG. 4 shows the mean squared error of a sequence of frames processed in accordance with an embodiment;

(6) FIG. 5 shows the effect of rapid changes upon the mean squared error of frames utilising inherited magnitude and phase;

(7) FIG. 6 shows quality driven dynamic allocation of iterations in accordance with embodiments;

(8) FIG. 7 shows an image projection system in accordance with embodiments; and

(9) FIG. 8 is a schematic of a LCOS SLM.

(10) In the figures like reference numerals referred to like parts.

DETAILED DESCRIPTION OF THE DRAWINGS

(11) Holographically generated 2D video images are known to possess significant advantages over their conventionally projected counterparts, especially in terms of definition and efficiency. However, the computational and hardware complexity of the current hologram generation algorithms previously precluded their use in real-time applications. Recently these problems have been solvedsee, for example, published PCT application WO 2005/059881 incorporated herein by reference.

(12) Modified algorithms based on Gerchberg-Saxton have been developedsee, for example, co-pending published PCT application WO 2007/131650 incorporated herein by reference.

(13) These improved techniques are able to calculate holograms at a sufficient speed that 2D video projection is realised. Embodiments described herein relate to 2D video projection using a computer-generated hologram calculated using such a modified Gerchberg-Saxton algorithm.

(14) FIG. 2 shows a modified algorithm which retrieves the phase information [x,y] of the Fourier transform of the data set which gives rise to a known amplitude information T[x,y] 362. Amplitude information T[x,y] 362 is representative of a target image (e.g. a photograph). The phase information [x,y] is used to produce a holographic representative of the target image at an image plane.

(15) Since the magnitude and phase are intrinsically combined in the Fourier transform, the transformed magnitude (as well as phase) contains useful information about the accuracy of the calculated data set. Thus, the algorithm may provided feedback on both the amplitude and the phase information.

(16) The algorithm shown in FIG. 2 can be considered as having a complex wave input (having amplitude information 301 and phase information 303) and a complex wave output (also having amplitude information 311 and phase information 313). For the purpose of this description, the amplitude and phase information are considered separately although they are intrinsically combined to form a data set. It should be remembered that both the amplitude and phase information are themselves functions of the spatial coordinates x and y and can be considered amplitude and phase distributions.

(17) Referring to FIG. 2, processing block 350 produces a Fourier transform from a first data set having magnitude information 301 and phase information 303. The result is a second data set, having magnitude information and phase information .sub.n[x,y] 305. The amplitude information from processing block 350 is set to a distribution representative of the light source but the phase information .sub.n[x,y] 305 is retained. Phase information 305 is quantised by processing block 354 and output as phase information [x,y] 309. Phase information 309 is passed to processing block 356 and combined with the new magnitude by processing block 352. The third data set 307, 309 is applied to processing block 356 which performs an inverse Fourier transform. This produces a fourth data set .sub.n[x,y] in the spatial domain having amplitude information |R.sub.n[x, y]| 311 and phase information R.sub.n[x, y] 313.

(18) Starting with the fourth data set, its phase information 313 forms the phase information of a fifth data set, applied as the first data set of the next iteration 303. Its amplitude information R.sub.n[x,y] 311 is modified by subtraction from amplitude information T[x,y] 362 from the target image to produce an amplitude information 315 set. Scaled amplitude information 315 (scaled by a) is subtracted from target amplitude information T[x,y] 362 to produce input amplitude information [x,y] 301 of the fifth data set for application as first data set to the next iteration. This is expressed mathematically in the following equations:
R.sub.n+1[x,y]=F{exp(i.sub.n[u,v])}
.sub.n[u,v]=F{.Math.exp(iR.sub.n[x,y])}
=T[x,y](|R.sub.n[x,y]|T[x,y])

(19) Where:

(20) F is the inverse Fourier transform.

(21) F if the forward Fourier transform.

(22) R is the replay field.

(23) T is the target image.

(24) is the angular information.

(25) is the quantized version of the angular information.

(26) is the new target magnitude, 0

(27) is a gain element 1

(28) The gain element may be predetermined based on the size and rate of the incoming target image data.

(29) In the absence of phase information from the preceding iteration, the first iteration of the algorithm uses a random phase generator to supply phase information as a starting point.

(30) In an alternative modification, the resultant amplitude information from processing block 350 is not discarded. The target amplitude information 362 is subtracted from amplitude information to produce anew amplitude information. A multiple of amplitude information is subtracted from amplitude information 362 to produce the input amplitude information for processing block 356.

(31) Further alternatively, the phase is not fed back in full and only a portion proportion to its change over the last two iterations is fed back.

(32) These modifications to the Gerchberg-Saxton algorithm relate to feeding back parameters within the phase retrieval calculation for a particular frame. This is distinct from inheriting parameters related to the phase retrieval calculation for the preceding frame.

(33) In summary, there is provided an improved method of phase retrieval. This method may be used in conjunction with existing phase retrieval algorithms, such as Gerchberg-Saxton or a modified version of Gerchberg-Saxton.

(34) There is provided a method of retrieving phase information in the Fourier domain representative of the n.sup.th frame of a sequence of 2D image frames, the method comprising using at least one parameter related to retrieval of phase information in the Fourier domain of the (n1) frame as a starting point for the iterative method.

(35) The at least one parameter may be phase information in the Fourier domain and/or amplitude information in the Fourier domain. In embodiments, the at least one parameter may be control parameters such as those used to control gain or attenuation.

(36) There is also provided a projector for forming a 2D video image, the projector comprising: a spatial light modulator comprising a 2D array of phase modulating elements; processing means arranged to retrieve phase information in the Fourier domain representative of the n.sup.th frame of a sequence of 2D image frames in the spatial domain including using at least one parameter related to retrieval of phase information in the Fourier domain of the (n1) frame as a starting point; and driving means arranged to drive the array of phase modulating elements of the spatial light modulator to display data representative of a Fourier transform of a nth frame of a sequence of 2D image frames.

(37) Embodiments relate to holographic projection that may be used as a generic display medium and therefore capable of showing moving pictures. This requires the system to be able to compute and display a minimum of 25 frames per second. One approach would be to simply execute the phase computation algorithm 25 times per second, however in most moving pictures the difference from one frame to the next is relatively small and this could provide a mechanism to reduce to computation time or improve the image quality.

(38) FIG. 3 shows the mean squared error differences from one frame to the next for an example AVI file containing an image of the Earth revolving. The spikes in the waveform are key frames.

(39) The aforementioned algorithms, based on Gerchberg-Saxton, require a number of iterations to be able to resolve the target image from a phase only modulation scheme. This is due to the initial conditions where the random phase seed has no statistical link to target image. After each iteration the statistical relationship between the hologram (Fourier) domain and far field (spatial domain) improves. If, after the first frame has been resolved, the phase and resultant image are retained the next frame could be treated as an error vector to be corrected. This idea forms the basis of frame inheritance. In embodiments, the calculated amplitude information in the Fourier domain and/or phase information in the Fourier domain form the magnitude information 301 and/or phase information 303, respectively, of the first data set of the next frame. In other embodiments, different parameters of the algorithm are used as at least a starting point for the calculation in relation to the next frame.

(40) The improved method in accordance with the present invention was implemented and tested against the first 10 frames of the AVI file of the Earth revolving. The mean squared error for each frame was monitored during the iterative process and is shown in FIG. 4. The advantages of inheriting the resultant magnitude and phase are very clear; the mean squared error of each frame continues to improve reaching a mean squared error of 14.85. This is perhaps best expressed as a peak signal to noise ratio (PSNR) of 36.4 dB, a 5.5 dB improvement over the standalone algorithm implementation.

(41) One interesting result is that after the algorithm has processed 5 frames there is very little improvement in the image quality beyond 3 or 4 iterations. This implies that it is possible to not only improve the quality of the image but do so with less iterations. This may in part due to the very similar nature of the frames in the chosen AVI file. To verify this hypothesis, the algorithm was tested against a second AVI file which contained a large number of rapidly changing scenes.

(42) The results shown in FIG. 5 indicate that after significant scene changes the algorithm is able to improve the image quality in successive frames provided that the scene is not changing too rapidly (as shown in region A). If the scene is constantly changing then the algorithm struggles to maintain the image quality (as shown in region B).

(43) The response of the algorithm indicates that it is better to use a dynamic system when deciding how many iterations are required, for example in scenes with a small amount of change a low number of iterations may be applied, whereas rapidly changing scenes may utilise more iterations. One way to implement such a system would be to target a quality level specified in terms of mean squared error.

(44) This system was implemented and tested against a mean squared error target of 25 (PSNR=34.1 dB) using second AVI video file. The results in FIG. 6 show there is a clear correlation between the number of iterations the algorithm needs to meet a specific target and the magnitude of the inter-frame changes. Using this approach the number of iterations needed to compute the 300 frames in this movie was reduced from 2400 to 1368 a reduction of 43%, or an average of 4.56 iterations per frame.

(45) The quality of the reconstructed hologram may be affect by the so-called zero order problem which is a consequence of the diffractive nature of the reconstruction. Such zero-order light can be regarded as noise and includes for example specularly reflected light, and other unwanted light from the SLM.

(46) This noise is generally focussed at the focal point of the Fourier lens, leading to a bright spot at the centre of a reconstructed hologram. Conventionally, the zero order light is simply blocked out however this would clearly mean replacing the bright spot with a dark spot.

(47) However as the hologram contains three dimensional information, it is possible to displace the reconstruction into a different plane in spacesee, for example, published PCT application WO 2007/131649 incorporated herein by reference.

(48) Whilst embodiments described herein relate to displaying one hologram per frame, the present disclosure is by no means limited in this respect and more than one hologram may be displayed on the SLM at any one time.

(49) For example, embodiments implement the technique of tiling, in which the surface area of the SLM is further divided up into a number of tiles, each of which is set in a phase distribution similar or identical to that of the original tile. Each tile is therefore of a smaller surface area than if the whole allocated area of the SLM were used as one large phase pattern. The smaller the number of frequency component in the tile, the further apart the reconstructed pixels are separated when the image is produced. The image is created within the zeroth diffraction order, and it is preferred that the first and subsequent orders are displaced far enough so as not to overlap with the image and may be blocked by way of a spatial filter.

(50) As mentioned above, the image produced by this method (whether with tiling or without) comprises spots that form image pixels. The higher the number of tiles used, the smaller these spots become. If one takes the example of a Fourier transform of an infinite sine wave, a single frequency is produced. This is the optimum output. In practice, if just one tile is used, this corresponds to an input of a single phase of a sine wave, with a zero values extending in the positive and negative directions from the end nodes of the sine wave to infinity. Instead of a single frequency being produced from its Fourier transform, the principle frequency component is produced with a series of adjacent frequency components on either side of it. The use of tiling reduces the magnitude of these adjacent frequency components and as a direct result of this, less interference (constructive or destructive) occurs between adjacent image pixels, thereby improving the image quality.

(51) Preferably, each tile is a whole tile, although it is possible to use fractions of a tile.

(52) Although embodiments relate to variants of the Gerchberg-Saxton algorithm, the skilled person will understand that other phase retrieval algorithms may implement the improved method disclosed herein.

(53) The skilled person will understand that the improved method disclosed herein is equally applicable to the calculation of a hologram used to form a three-dimensional reconstruction of an objection. The sequential image frames may be 1D, 2D and/or 3D image frames, or any sequence thereof.

(54) Equally, the present disclosure is not limited to projection of a monochromatic image.

(55) A colour 2D holographic reconstruction can be produced and there are two main methods of achieving this. One of these methods is known as frame-sequential colour (FSC). In an FSC system, three lasers are used (red, green and blue) and each laser is fired in succession at the SLM to produce each frame of the video. The colours are cycled (red, green, blue, red, green, blue, etc.) at a fast enough rate such that a human viewer sees a polychromatic image from a combination of the three lasers. Each hologram is therefore colour specific. For example, in a video at 25 frames per second, the first frame would be produced by firing the red laser for 1/75.sup.th of a second, then the green laser would be fired for 1/75.sup.th of a second, and finally the blue laser would be fired for 1/75.sup.th of a second. The next frame would then be produced, starting with the red laser, and so on.

(56) An alternative method, that will be referred to as spatially separated colours (SSC) involves all three lasers being fired at the same time, but taking different optical paths, e.g. each using a different SLM, and then combining to form the colour image.

(57) An advantage of the frame-sequential colour (FSC) method is that the whole SLM is used for each colour. This means that the quality of the three colour images produced will not be compromised because all pixels on the SLM are used for each of the colour images. However, a disadvantage of the FSC method is that the overall image produced will not be as bright as a corresponding image produced by the SSC method by a factor of about 3, because each laser is only used for a third of the time. This drawback could potentially be addressed by overdriving the lasers, or by using more powerful lasers, but this would require more power to be used, would involve higher costs and would make the system less compact.

(58) An advantage of the SSC (spatially separated colours) method is that the image is brighter due to all three lasers being fired at the same time. However, if due to space limitations it is required to use only one SLM, the surface area of the SLM can be divided into three equal parts, acting in effect as three separate SLMs. The drawback of this is that the quality of each single-colour image is decreased, due to the decrease of SLM surface area available for each monochromatic image. The quality of the polychromatic image is therefore decreased accordingly. The decrease of SLM surface area available means that fewer pixels on the SLM can be used, thus reducing the quality of the image. The quality of the image is reduced because its resolution is reduced.

(59) The skilled person will understand that a user may view a real image or virtual image of the holographic reconstruction. In particular, embodiments in accordance with the present disclosure may be implemented in a head-up display.

(60) FIG. 7 shows a head-up display (700) having an SLM based system (705) for providing a real image of a holographic reconstruction (710). The holographic reconstruction is formed at a so-called replay field.

(61) The display consists of an optical combiner (720) and a lens (730) disposed between the holographic reconstruction (710) and the combiner (720). The arrangement is such that a viewer (740) looking towards the combiner (720) will see a virtual image (750) of the holographic reconstruction (710) at a distance d from the viewer and behind the combiner (720). Such a system can be used for example in a head-up display or head-mounted display.

(62) In embodiments, the spatial light modulator is a Liquid Crystal over silicon (LCOS) device. LCOS devices are a hybrid of traditional transmissive liquid crystal display devices, where the front substrate is glass coated with Indium Tin Oxide to act as a common electrical conductor. The lower substrate is created using a silicon semiconductor process with an additional final aluminium evaporative process being used to create a mirrored surface, these mirrors then act as the pixel counter electrode.

(63) Compared with conventional glass substrates these devices have the advantage that the signal lines, gate lines and transistors are below the mirrored surface, which results in much higher fill factors (typically greater than 90%) and higher resolutions.

(64) LCOS devices are now available with pixels between 4.5 m and 12 m, this size is determined by the mode of operation and therefore amount of circuitry that is required at each pixel.

(65) The structure of an LCOS device is shown in FIG. 8.

(66) A LCOS device is formed using a single crystal silicon substrate (802). It has a 2D array of square planar aluminium electrodes (801), spaced apart by a gap (801a), arranged on the upper surface of the substrate. Each of the electrodes (801) can be addressed via circuitry (802a) buried in the substrate (802). Each of the electrodes forms a respective planar mirror. An alignment layer (803) is disposed on the array of electrodes, and a liquid crystal layer (804) is disposed on the alignment layer (803). A second alignment layer (805) is disposed on the liquid crystal layer (404) and a planar transparent layer (806), e.g. of glass, is disposed on the second alignment layer (805). A single transparent electrode (807) e.g. of ITO is disposed between the transparent layer (806) and the second alignment layer (805).

(67) Each of the square electrodes (801) defines, together with the overlying region of the transparent electrode (807) and the intervening liquid crystal material, a controllable phase-modulating element (808), often referred to as a pixel. The effective pixel area, or fill factor, is the percentage of the total pixel which is optically active, taking into account the space between pixels (801a). By control of the voltage applied to each electrode (801) with respect to the transparent electrode (807), the properties of the liquid crystal material of the respective phase modulating element may be varied, thereby to provide a variable delay to light incident thereon. The effect is to provide phase-only modulation to the wavefront, i.e. no amplitude effect occurs.

(68) A major advantage of using a reflective LCOS spatial light modulator is that the liquid crystal layer is half the thickness that it would be if a transmissive device were used. This greatly improves the switching speed of the liquid crystal (a key point for projection of moving video images). A LCOS device is also uniquely capable of displaying large arrays of phase only elements in a small aperture. Small elements (typically approximately 10 microns) result in a practical diffraction angle (a few degrees) so that the optical system does not require a very long optical path.

(69) It is easier to adequately illuminate the small aperture (a few square centimeters) of a LCOS SLM than it would be for the aperture of a larger liquid crystal device. LCOS SLMs also have a large aperture ratio, there is very little dead space between the pixels (as the circuitry to drive them is buried under the mirrors). This is an important issue to lowering the optical noise in the replay field.

(70) The above device typically operates within a temperature range of 10 C. to around 50 C., with the optimum device operating temperature being around 40 C. to 50 C.

(71) As a LCOS device has the control electronics embedded in the silicon backplane, the Fill factor of the pixels is higher, leading to less unscattered light leaving the device.

(72) Using a silicon backplane has the advantage that the pixels are optically flat, which is important for a phase modulating device.

(73) Whilst embodiments relate to a reflective LCOS SLM, the skilled person will understand that any SLM can be used including transmissive SLMs.

(74) The invention is not restricted to the described embodiments but extends to the full scope of the appended claims.