Holographic imaging device and method

Abstract

A holographic imaging device is disclosed. In one aspect, the holographic imaging device comprises an imaging unit comprising at least two light sources, wherein the imaging unit is configured to illuminate an object by emitting at least two light beams with the at least two light sources. A first and second light beams have different wave-vectors and wavelengths. The holographic imaging device further comprises a processing unit configured to obtain at least two holograms of the object by controlling the imaging unit to sequentially illuminate the object with respectively the first light beam and the second light beam, construct at least two 2D image slices based on the at least two holograms, wherein each 2D image slice is constructed at a determined depth within the object volume, and generate a three-dimensional image of the object based on a combination of the 2D image slices.

Claims

1. A holographic imaging device comprising: an imaging unit comprising at least two light sources and configured to: illuminate an object by emitting a first light beam with a first one of the at least two light sources and a second light beam with a second one of the at least two light sources, wherein the first light beam has a first wave-vector and a first wavelength and the second light beam has a second wave-vector that is different from the first wave-vector and a second wavelength that is different from the first wavelength; and a processing unit configured to: obtain at least two holograms of the object by controlling the imaging unit to sequentially illuminate the object with respectively the first light beam and the second light beam, obtain at least two phase-retrieved holograms based on estimating a phase of each of the at least two holograms, processing the at least two phase-retrieved holograms by: determining a signature for an out-of-focus plane signal originating from a ghost image of the object, based on a comparison of the at least two phase-retrieved holograms, and removing, from the at least two phase-retrieved holograms, at least one out-of-focus plane signal originating from the ghost image of the object, construct at least two two-dimensional (2D) image slices based on the at least two processed phase-retrieved holograms, wherein each 2D image slice is constructed at a determined depth within an object volume of the object, and generate a three-dimensional (3D) image of the object based on a combination of the 2D image slices.

2. The holographic imaging device according to claim 1, wherein: at least one of the first wave-vector or the second wave-vector are predetermined based on relative positions of the at least two light sources.

3. The holographic imaging device according to claim 1, wherein: the phase of a first hologram is estimated based on performing an iterative phase retrieval procedure by using a second hologram obtained with a different wavelength of illumination.

4. The holographic imaging device according claim 1, wherein: constructing the at least two 2D image slices comprises obtaining, based on the at least two processed phase-retrieved holograms, at least two 2D image slices each for a respective depth within the object volume.

5. The holographic imaging device according to claim 4, wherein the processing unit is further configured to: compare the at least two 2D image slices, identify spatially overlapped signals in the at least two 2D image slices indicating in-focus signals of the object, and identify substantially shifted signals in the at least two 2D image slices indicating out-of-focus signals originating from other real objects located in out-of-focus-planes.

6. The holographic imaging device according to claim 5, wherein the processing unit is further configured to: remove, from the at least two 2D image slices, at least one identified out-of-focus plane signal originating from another real object, and perform a noise removal procedure for minimizing a difference between the at least two 2D image slices.

7. The holographic imaging device according to claim 6, wherein performing the noise removal procedure comprises at least one of: removing out-of-focus plane signals that are spatially dislocated among the at least two 2D image slices, or determining and maintaining in-focus signals that are overlapped in the at least two 2D image slices.

8. The holographic imaging device according to claim 6, wherein the processing unit is further configured to: stack the constructed at least two 2D image slices, for generating the 3D image of the object, wherein each 2D image slice is used at its determined depth within the object volume.

9. The holographic imaging device according to claim 1, wherein the imaging unit is further comprising at least one imager and wherein each light source is arranged at a predefined position from the at least one imager.

10. The holographic imaging device according to claim 9, wherein the at least one imager is based on a complementary metal oxide semiconductor (CMOS) image sensor.

11. The holographic imaging device according to claim 1, wherein: at least one of the first wavelength or the second wavelength are within a range of 300 nm to 900 nm.

12. A method of using a holographic imaging device, the method comprises: illuminating, by an imaging unit comprising at least two light sources, an object by emitting a first light beam with a first of the at least two light sources and a second light beam with a second of the at least two light sources, wherein a first light beam has a first wave-vector and a first wavelength and a second light beam has a second wave-vector that is different from the first wave-vector and a second wavelength that is different from the first wavelength, obtaining, by a processing unit, at least two holograms of the object by controlling the imaging unit to sequentially illuminate the object with respectively the first light beam and the second light beam, obtaining, by the processing unit, at least two phase-retrieved holograms based on estimating a phase of each of the at least two holograms, processing, by the processing unit, the at least two phase-retrieved holograms by: determining a signature for an out-of-focus plane signal originating from a ghost image of the object, based on a comparison of the at least two phase-retrieved holograms, and removing, from the at least two phase-retrieved holograms, at least one out-of-focus plane signal originating from the ghost image of the object, constructing, by the processing unit, at least two two-dimensional (2D) image slices based on the at least two processed phase-retrieved holograms, wherein each 2D image slice is constructed at a determined depth within an object volume of the object, and generating, by the processing unit, a three-dimensional (3D) image of the object based on a combination of the 2D image slices.

13. A non-transitory computer readable medium storing a computer program which, when executed by a computer, causes the method of claim 12 to be performed.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The above described aspects and implementations are explained in the following description of embodiments with respect to the enclosed drawings:

(2) FIG. 1 shows a schematic view of a holographic imaging device for generating a 3D image of an object, according to an embodiment of the disclosed technology;

(3) FIG. 2 shows another schematic view of the holographic imaging device, according to an embodiment of the disclosed technology;

(4) FIG. 3 shows a flow chart of a method of generating a 3D image of an object;

(5) FIGS. 4A-B shows different obtained holograms;

(6) FIG. 5 shows a schematic view of a diagram illustrating obtaining a phase-retrieved hologram;

(7) FIGS. 6A-B shows exemplarily obtained holograms with the first wave-vector (FIG. 6A) and the second wave-vector (FIG. 6B);

(8) FIGS. 7A-B shows an image of planar samples obtained by, a conventional device (FIG. 7A) and the holographic imaging device of the invention (FIG. 7B);

(9) FIGS. 8A-B shows an image slice from a sample including micron beads dispersed in a 500-micron thick gel obtained by a conventional device (FIG. 8A) and the holographic imaging device of the disclosed technology (FIG. 8B);

(10) FIGS. 9A-B shows another image slice from the sample of FIGS. 8A-B obtained by, a conventional device (FIG. 9A) and the holographic imaging device of the disclosed technology (FIG. 9B);

(11) FIG. 10 shows a method for the holographic imaging device, according to an embodiment of the disclosed technology; and

(12) FIG. 11 shows problems of out-of-focus signals in 2D image slices obtained by a conventional holographic imaging device.

DETAILED DESCRIPTION OF CERTAIN ILLUSTRATIVE EMBODIMENTS

(13) FIG. 1 shows a schematic view of a holographic imaging device 10 for generating a 3D image of an object, according to an embodiment of the disclosed technology.

(14) The holographic imaging device 10 comprises an imaging unit 11 having at least two light sources 111, 112. The imaging unit 11 is configured to illuminate an object 12 by emitting at least two light beams with the at least two light sources 111, 112, wherein a first light beam has a first wave-vector and a first wavelength and a second light beam has a second wave-vector that is different from the first wave-vector and a second wavelength that is different from the first wavelength.

(15) The holographic imaging device 10 may thus use an illumination method entailing multiple light beams with both angular and spectral pluralities.

(16) The holographic imaging device 10 further comprises a processing unit 13 configured to obtain at least two holograms of the object 12 by controlling the imaging unit 11 to sequentially illuminate the object 12 with respectively the first light beam and the second light beam, construct at least two 2D image slices based on the at least two holograms, wherein each 2D image slice is constructed at a determined depth within the object volume, and generate a 3D image of the object 12 based on a combination of the 2D image slices.

(17) The processing unit 13 may comprise a circuitry. The circuitry may comprise hardware (e.g., the imaging unit, a display, the processing unit (e.g., a Central Processing Unit (CPU)), a memory, etc.) and software (e.g., a program for constructing 2D image slices, a program/algorithm for generating 3D image, etc.).

(18) For example, the processing unit 13 may obtain the holograms. Moreover, the holograms recorded with such pluralities may be reconstructed digitally based on the 2D diffraction theory and image processing algorithms.

(19) The holographic imaging device 10 (e.g., its processing unit 13, or a program such as an algorithm running on the processing unit 13) may use the differences in recorded holograms generated by the angular and spectral pluralities in an iterative procedure in which at each step the in-focus signal is identified and cleaned from the out-of-focus signal. The holographic imaging device 10 (e.g., its processing unit 13) may repeat this procedure several times at different depths.

(20) Moreover, the holographic imaging device 10 (e.g., its processing unit 13) may combine the 2D image slices to obtain a 3D image (i.e., a 3D representation of the sample).

(21) Reference is made to FIG. 2, which is another schematic view of the holographic imaging device 10, according to an embodiment of the invention.

(22) The imaging unit 11 of the holographic imaging device 10 comprises three light sources including the “light source 1” 111 illuminating with the first wavelength λ1, the “light source 2” 112 illuminating with the second wavelength λ2, the “light source 3” 211 illuminating with the third wavelength λ3.

(23) The holographic imaging device 10 further comprises an imager 22 and the processing unit 14 (e.g., a computer).

(24) The holographic imaging device 10 of FIG. 2 configured to illuminate multiple light beams with the wave-vector (angular) and wavelength (spectral) pluralities. For example, three light beams are generated from three light sources 111, 112, 211 which are in or at different locations with respect to the origin of the imaging system. Furthermore, each of the light sources 111, 112, 211 emits a corresponding light beam with a different wavelength. The pluralities of wave-vectors and wavelengths may lead (e.g., may be used) to suppressing two distinct sources of out-of-focus plane signals.

(25) Reference is made to FIG. 3, which is a flow chart of a method 100 for generating a 3D image of an object. The method 100 may be performed by the holographic imaging device 10 (of FIG. 1 and/or FIG. 2).

(26) Without limiting the present disclosure, in the following, the method 100 is discussed exemplarily being performed by the holographic imaging device 10.

(27) At step S101, the holographic imaging device 10 acquires series of holograms of object 12 illuminated sequentially with light sources 111, 112, 211 and transfer the data to the computer 14.

(28) At step S102, the holographic imaging device 10 performs iterative reconstruction of a 2D image slice at a chosen depth based on the developed algorithm.

(29) At step S103, the holographic imaging device 10 repeats the reconstruction process at different depths within the extent of the object 12.

(30) At step S104, the holographic imaging device 10 combines the 2D image slices to obtain a 3D representation of the object 12.

(31) Reference is made to FIG. 4A and FIG. 4B, in which FIG. 4A shows an example of recorded holograms (three holograms in sequence provided as a video) of a sample obtained by the holographic imaging device 10 with three light sources 111, 112, 211 with different wavelengths but the same wave-vector directions. The main image features on the hologram (the interference patterns) look very similar to each other except that the fine fringes on the interference pattern are observed to shift with different wavelength of illumination.

(32) The differences between the holograms are observed, since the optical path between the objects 12 in the sample and the imager 22 changes with different illumination beams despite the fixed physical distance. The difference in the optical paths can result in different ghost images of the same object 12 in the reconstructed hologram. Thus, the plurality of illumination wavelengths (illuminated by the holographic imaging device 10) results in a distinguishable signature for the out-of-focus plane signal originating from the ghost image of the object 12.

(33) The plurality of wavelengths in practice may be within a range of 300 nm to 900 nm (for example, it may be limited within visible spectrum λ=400 nm-700 nm, in which CMOS image sensors are typically sensitive and optical properties of objects do not vary widely). While such range is sufficient to create distinguishable signatures between the object 12 and its ghost in practice (typically located hundreds of microns away from the object in depth), this approach may not be useful for signals originating from other real objects located in out-of-focus planes that are relatively close to the in-focus plane slice.

(34) The holographic imaging device 10 is further configured to provide the plurality of wave-vectors of light beams to overcome the out-of-focus plane signals. FIG. 4B shows an example of recorded holograms of the sample with three light sources 111, 112, 211 with different wave-vectors (three holograms in sequence provided as a video in FIG.4B). Differently from FIG. 4A, the features in the holograms shift laterally according to the wave-vector of the illumination beam. The strength of the relative shift between the holograms depends on the distance of the objects 12 from the image sensor 22. The closer the object 12 is to the imager 22, the smaller the shift with varying wave-vector will be. The arrows in FIG. 4B indicate to three distinct features in the hologram sequence. These three features originate from objects located three different depths. The feature in the bottom side of the hologram hardly shifts while the other two features shift visibly in the hologram sequence. A careful inspection suggests that the L-shaped feature shifts less than the circular one indicating that the L-shaped object is closer to the imager in distance. Thus, the plurality of illumination wave-vectors (provided by the holographic imaging device 10) results in a distinguishable signature for the signals originating from objects located at different depths. The strength of the distinguishable signature is a function of illumination beam angle and the pixel size of the imager 22. The larger the angle of the illumination is and the smaller the imager pixel size is, the larger the signature distinguishing two objects closely spaced slices will be.

(35) It is important to note that the plurality of illumination wave-vectors alone may not be useful in creating a distinguishable signature of the out-of-focus plane originating from the ghost object image, since the ghost image shifts together with the object image and both always overlap with each other regardless of the wave-vector direction. Therefore, the holographic imaging device 10 provides both types of illumination pluralities (wave-vector and wavelength) in creating distinguishable signatures to effectively suppress different types of out-of-focus plane signals in the reconstructed images. A discussion of processing the multiple holograms to achieve the out-of-focus signal suppression is provided below.

(36) In the following, description of the 2D image construction and generating the 3D image is provided.

(37) Once the holographic imaging device 10 obtains the multiple holograms (e.g., acquired with the mentioned illumination method), the next step is digitally processing them to obtain 2D image slices of the object at different depths with suppressed out-of-focus signals. The steps of image reconstruction and processing may be split into three steps as follows.

(38) The goal of the first step is to identify the wave-vectors of the light sources using a sample suitable for the purpose. This may be a practical calibration procedure before taking a measurement of sample. For example, in some embodiments, accurate knowledge of the wave-vectors assists in the subsequent image reconstruction steps.

(39) Reference is made to FIG. 5, which shows a schematic view of a diagram 50 for determining the first wave-vector and the second wave-vector and obtaining a phase-retrieved hologram.

(40) Initially, the holographic imaging device 10 may determine the first wave-vector and the second wave-vector and may calibrate the position of the light sources 111, 112, 113 based thereon. The goals of the first step may be: Determining the relative source positions (x, y). Converting the source positions into wave-vectors with a known source displacement of z. The performed operations of the first step may be summarized as follows: Estimating the relative shift of the holograms with sub-pixel accuracy, for example, using an algorithm based on a fast implementation of cross correlation between the images. Reconstructing firstly the holograms in focus with an “initial guess of the positions”. Using the reconstructed object images for determining the sharp features and obtaining a better accuracy for registration. Applying an image registration procedure for the reconstructed images and determining the error in the initial guess. Identifying accurate x, y positions per source. Determining the wave-vectors.

(41) In the second step, the holographic imaging device 10 may further obtain the phase-retrieved holograms.

(42) For instance, the holograms acquired with the imager 22 may contain only the intensity of the hologram and the phase component may be lost. The holographic imaging device 10 may apply a phase retrieval procedure to estimate the phase of holograms using the plurality of the wavelengths of illumination. It is important to note that the second phase is repeated for all the holograms acquired from the imager. Therefore, a phase estimation is done on all the holograms. The output of the second step is the three phase-retrieved holograms, which will be passed to step three. The goals of the second step may be: Phase retrieval for the holograms using the wavelength plurality. Removal of an out of focus plane ghost image signal. The performed operations may be summarized as follow: Phase retrieval may be based on propagating complex valued holograms and replacing the phases at even steps (steps of a single iteration drawn for H1 on the left). Choosing a slice (z.sub.obj) in the object volume for reconstruction. After N iterations, repeating step one for obtaining ghost-free image of the object with wavevector-1. Repeating the process for all the holograms (e.g., H2 and H3). Obtaining as final output, three holograms with retrieved phases.

(43) The reconstruction of these holograms in the next step may result in a cleaned ghost image signal.

(44) At the third step, the object image reconstruction may be performed by using the phase retrieved holograms corresponding to three different wave-vector illumination beams. The first process is to reconstruct three 2D image slices within the object volume from the phase retrieved holograms. For example, the holographic imaging device 10 may use a 2D diffraction theory based on field propagation such as angular spectrum, etc. Note that the reconstructed slices are free-from out-of-focus signals originating from the ghost image but not the signals generated from the objects (for example, another object than the desired object) located near the slice depth. An example of such slices is shown in FIG. 6A and FIG. 6B. These two images reveal that in-focus signals overlap spatially in two images while the out-of-focus signals are shifted relatively.

(45) Once the reconstructed slices are obtained, the holographic imaging device 10 may suppress the out-of-focus signals using the three reconstructed slices and a noise removal algorithm designed to minimize differences between the reconstructed slices from three different holograms. The noise removal algorithm succeeds to reject out-of-focus plane signal since only the in-focus signal overlaps in all different slices while out-of-focus signal is spatially dislocated among the three slices.

(46) The holographic imaging device 10 may use different noise removal procedures. For example, the simplest approach may be averaging the three reconstructed slices as the overlapping in-focus signal will be enhanced while non-overlapping out-of-focus signal will be averaged out. More advanced approaches may be used to suppress the unwanted signals. One of such approaches is formulating the in-focus signal as an inverse problem and minimizing the error using regularization in an iterative optimization algorithm. L1, L2 and total variation (TV) normalization may be used in image processing to solve inverse problems depending on the nature of the problem. Such an approach can be used to minimize the error occurring in three distinct reconstructed slices enabled by the different wave-vectors.

(47) The holographic imaging device 10 may process the slices at other depths in a similar procedure as explained above. Once all the slices in the sample volume are processed, the slices can be stacked into a 3D data representation. The holographic imaging device 10 may enable enhanced depth resolution to create true 3D imaging of complex samples.

(48) Experimental validation of the proposed invention is done, and some initial results (obtained by the holographic imaging device 10) are presented in the following, without limiting the present disclosure.

(49) FIGS. 7A-B shows an image of planar samples of metal lines on a substrate obtained by, a conventional device (FIG. 7A) and the holographic imaging device 10 of the invention (FIG. 7B).

(50) As it can be derived from FIG. 7B, the holographic imaging device 10 removes the ghost image related out-of-focus signal.

(51) FIGS. 8A-B shows an image slice (slice #1) from a sample including micron beads dispersed in 500-micron thick gel obtained by, a conventional device (FIG. 8A) and the holographic imaging device of the invention (FIG. 8B).

(52) As it can be derived from FIG. 8B, the holographic imaging device 10 removes (from slice #1) the out-of-focus plane signals.

(53) FIGS. 9A-B shows another image slice (slice #2) from the sample of FIGS. 8A-B obtained by, a conventional device (FIG. 9A) and the holographic imaging device of the invention (FIG. 9B).

(54) As it can be derived from FIG. 9B, the holographic imaging device 10 removes (from slice #2) the out-of-focus plane signals.

(55) As it can be derived from the two image slices (e.g., FIG. 8B and FIG. 9B) that are obtained from the sample containing 1-micron sized beads distributed in a 3D volume, the holographic imaging device 10 effectively suppresses the out-of-focus signal originating from beads in other depths.

(56) FIG. 10 shows a method 200 according to an embodiment of the invention for a holographic imaging device 10. The method 200 may be carried out by the holographic imaging device 10, as it is described above.

(57) The method 200 comprises a step S201 of illuminating, by an imaging unit 11 comprising at least two light sources 111, 112, an object 12 by emitting at least two light beams with the at least two light sources, wherein a first light beam has a first wave-vector and a first wavelength and a second light beam has a second wave-vector that is different from the first wave-vector and a second wavelength that is different from the first wavelength.

(58) The method 200 further comprises a step S202 of obtaining, by a processing unit 13, at least two holograms of the object 12 by controlling the imaging unit 11 to sequentially illuminate the object 12 with respectively the first light beam and the second light beam.

(59) The method 200 further comprises a step S203 of constructing, by the processing unit 13, at least two 2D image slices based on the at least two holograms, wherein each 2D image slice is constructed at a determined depth within the object volume.

(60) The method 200 further comprises a step S204 of generating, by the processing unit 13, a 3D image of the object 12 based on a combination of the 2D image slices.