System, method and apparatus for ultra-resolved ultra-wide field-of-view multispectral and hyperspectral holographic microscopy
10088662 ยท 2018-10-02
Inventors
Cpc classification
G02B27/58
PHYSICS
G02B21/365
PHYSICS
G03H1/0866
PHYSICS
G03H2226/11
PHYSICS
G03H2001/005
PHYSICS
G02B21/0056
PHYSICS
G03H1/0443
PHYSICS
G02B21/0064
PHYSICS
International classification
G02B21/36
PHYSICS
G03H1/08
PHYSICS
Abstract
There is disclosed a novel system and method for achieving ultra-resolution, ultra-wide field-of-view multispectral and hyperspectral holographic microscopy and quantitative phase contrast microscopy. In an embodiment, the method comprises: providing a stationary illumination source; acquiring a plurality of low-resolution holograms of an image subject from different locations utilizing a subpixel sensor-scanning synthetic aperture mechanism whereby a detector scanning translationally, radially and/or rotationally; processing the acquired holograms utilizing a processing algorithm corresponding to the scanning motion of the detector used to acquire the holograms; and reconstructing a subpixel ultra-resolution image of the image subject based on the processed holograms; whereby, a desired synthetic aperture is achieved without loss of resolution. The multispectral and hyperspectral aspect is achieved in the novel system and method by use of different combination of illumination sources (i.e., LEDs, laser sources, broadband lamps, etc.) and wavelength selection mechanisms (i.e., bandpass spectral filters, acousto-optical and liquid crystal tunable filters, a dispersing element, etc.).
Claims
1. An apparatus for performing holographic microscopy and quantitative phase contrast microscopy, comprising: a spectral illumination source with capability of central wavelength and spectral bandwidth selection, the spectral illumination source adapted to illuminate a target through an illumination aperture or via a waveguide; a detector mounted on a movable scanning platform for acquiring a plurality of low-resolution holograms of an image subject from incremental locations utilizing a subpixel sensor-scanning synthetic aperture mechanism having one or more detector scanning patterns; a sample handling platform adapted to position: a sample along an optical axis of the system between the illumination source and the detector; and a processing unit adapted to reconstruct a subpixel ultra-resolution image of the image subject with an effective field-of-view (FOV) larger than the detector based on the plurality of processed low-resolution holograms; whereby, the apparatus adapted to position: loss of resolution.
2. The apparatus of claim 1, wherein the spectral illumination source comprises one or more of LEDs, laser sources, scanning grating monochrometers, a nano-photonic ring-resonator spectrograph wavelength selector, a broadband light source, and a tunable laser source.
3. The apparatus of claim 2, further comprising a light wavelength selection mechanism including one or more of a bandpass spectral filters, acousto-optical filters, liquid crystal tunable filters, and a dispersing element.
4. The apparatus of claim 1, wherein the spectral illumination source comprises one or more light-emitting diodes (LEDs), Halogen lamp, Halogen-Tungsten lamp, or a Xenon lamp, with or without narrow bandpass spectral transmission filters, which may be turned on simultaneously or sequentially.
5. The apparatus of claim 1, wherein the spectral illumination source comprises one or more laser sources that may be turned on simultaneously or sequentially.
6. The apparatus of claim 1, further comprising one or more scanning grating monochrometers for wavelength selection, in which the grating is placed on a rotational moveable platform in order to superpose a desired central wavelength on the illumination aperture.
7. The apparatus of claim 1, further comprising a nano-photonic ring-resonator spectrograph placed directly at the illumination aperture.
8. The apparatus of claim 1, wherein the scanning platform is adapted to move the detector translationally, radially, rotationally or randomly in plurality of incremental positions enabling synthetic aperture scanning.
9. The apparatus of claim 1, wherein the detector is a monochromatic or color charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) detector.
10. The apparatus of claim 1, wherein the processing unit is adapted to execute processing techniques and algorithms to reconstruct holograms, obtain amplitude and phase information from the holograms, reconstruct amplitude images, and reconstruct quantitative phase contrast microscopy images.
11. The apparatus of claim 1, further comprising a microscope objective lens placed above the detector to locate a focal point of the microscope objective lens at the surface of a microscope slide or on a screen which is displaying an interferometric diffraction pattern of the sample.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The present system, method and apparatus will be better understood and objects of the invention will become apparent when consideration is given to the following detailed description thereof. Such description makes reference to the annexed drawings, wherein:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)
(18) In the drawings, embodiments of the invention are illustrated by way of example. It is to be expressly understood that the description and drawings are only for the purpose of illustration and as an aid to understanding, and are not intended as describing the accurate performance and behavior of the embodiments and a definition of the limits of the invention.
DETAILED DESCRIPTION
(19) As noted above, the present system, method and apparatus relates generally to the field of micro-scale, and nano-scale muitispectral and hyperspectral holographic microscopic imaging and quantitative phase contrast microscopic imaging.
(20) More generally, the present system and method involves holographic microscopy and quantitative phase contrast microscopy (QPCM) utilizing subpixel scanning of the sensor to perform lensed or lens-free synthetic aperture imaging to achieve ultra-resolution imaging and a large, ultra-wide FOV.
(21) Holography, as first proposed by Dennis Gabor, was a result of efforts to improve the resolution of the electron microscope [1, 2]. By removing the lens used in such systems, the diffraction pattern of the beams can be observed which contain sufficient amplitude and phase information to numerically reproduce the object field.
(22) With the advent of digital sensors such as charge-coupled devices (CCDs) and complementary metal-oxide-semiconductor (CMOS) detectors, analogue holography has given way to digital holography in which interference patterns produced by holographic systems can now be registered and recorded using light detectors. Additionally, with the increase in computing power, processing of the holographic interference patterns are much more efficient and less costly to perform, and the results are much more representative using more advanced processing techniques and algorithms.
(23) Digital holography on a microscopic scale involves digitally acquiring the holographic information of micron-scale or nano-scale objects. The interference pattern arises from the superposition of two beams of lightone beam which travels unimpeded to the detector plane and second beam which arrives at the detector after some interaction with objects being probed. Unlike conventional photography, which only records the intensity as a function of spatial location, holography records both intensity and phase information of the light field, and these parameters can be extracted algorithmically. The combination of intensity and the phase of the light field can provide more complete information about the object being probed, and can therefore be processed and used for various different methods of interpretation.
(24) The performance of a holographic microscope is extremely dependent on many parameters such as: 1) a system's geometry; 2) central wavelength of the illumination source; 3) bandwidth of the illumination source; 4) size of the illumination aperture; 5) detector array size; 6) detector pixel pitch; and 7) detector bit depth.
(25) More particularly, the distance between the sample plane and the detector (z.sub.0) and the distance between the light source and the detector (z.sub.1) will affect the magnification of the fringes produced, MF=1+z.sub.0/z1. This fringe magnification will in turn pose limitations on the FOV of the microscope. In order to maximize the FOV of the device to be equivalent to the size of the active area of the detector array, the sample plane should be placed as close to the sensor plane as possible, resulting in MF1, and thereby giving rise to on-chip measurements. Large fringe magnification will also adversely affect the visibility of the interference patter of smaller objects if they are surrounded by larger objects.
(26) The central wavelength of the illumination source has a direct effect on the resolution of the diffraction or interference pattern and thereby affects the resolution of the reconstructed intensity and phase images. Light of shorter wavelength will produce higher resolution hologram than light of longer wavelength.
(27) Light coherence, temporal or spatial, is a necessary condition for interference and the operating principle of holographic microscopes. The bandwidth of the illumination source correlates directly with the temporal coherence of the light source, Narrowband light sources (<0.5 nm) such as lasers exhibit extreme temporal coherence such that self-interference becomes possible which gives rise to laser speckle noise. The proliferation of light-emitting diodes (LEDs) provided a solution to the speckle noise problem of laser sources if LEDs were to be used for illumination in conjunction with a small aperture (30-100 m). The purpose of the aperture is to impose spatial coherence on the LED light which is otherwise incoherent [3, 4]. With the now coherent light of the LED, holograms can be observed. There is an intrinsic tradeoff with the use of an aperture, spatial coherence is increased with smaller aperture size however illumination intensity decreases. High coherence is important for a holographic microscope since it increases the visibility of the interference pattern, V=(I.sub.maxI.sub.min)/(I.sub.max+I.sub.min). The increase in visibility directly correlates to the boost in SNR of the reconstructed intensity and phase map.
(28) Similar to any interferometric device the detector and its properties is the single most important component in a holographic microscope. The pixel pitch and the active area of the pixel array are crucial to the performance of the microscope. The active pixel array area determines the FOV pending the fringe magnification via FOV=array_area/(MF).sup.2. Therefore, having access to the detector sensor in close proximity will maximize the effective FOV by minimizing the fringe magnification. The pixel pitch determines the lateral resolution of a holographic microscope and places a theoretical lateral resolution limit on the device.
(29) More recently, superresolved holographic microscopes based around scanning illumination aperture techniques have been proposed and developed which can achieve sub-micron lateral resolution while maintaining a modest FOV, as mentioned before. The resolutions in these devices approach 250 nm per pixel which is rivaling the lateral resolution performance of a 40 optical microscope.
(30) A superresolution effect based around scanning illumination aperture techniques may achieved in different ways, including: 1) a series of LEDs are arranged side-by-side and are flashed on and off, and 2) a fiber-fed aperture is scanned systematically by translating it in two or three dimensions [5-17]. However, these superresolution techniques result in subpixel shifts in the hologram projected on the detector array and when contiguously amalgamated, simulate a pixel pitch much smaller than the actual detector pixel pitch. Consequently, the aforementioned superresolved holographic microscopy techniques require the accompaniment of powerful and costly reconstruction algorithms to register and back-project the acquired hologram information and solve for the location of the illumination aperture in order to use this information in the hologram reconstruction and processing techniques.
(31) Synthetic aperture sensing was originally introduced in the 1950s as a sensing modality for use in airborne radar measurements and has since seen many advances in order to be used on space borne platforms, such as synthetic aperture radar in which the motion of the aircraft or spacecraft simulates a larger (in size) radar antenna. Since then, the concept of synthetic aperture has found its way into other sensing modalities such as synthetic aperture sonar [18] and synthetic aperture magnetometry [19]. Synthetic aperture techniques have also been used in holographic microscopy to capture a higher range of spatial frequencies of the interferometric diffraction pattern of a given object. This is accomplished by moving the detector to capture a more complete interferometric pattern off of the detector's sensor plane. The detector is commonly displaced by a distance proportional to the full length of the active area to capture complementary adjacent holograms. The series of neighboring holograms are algorithmically stitched together to reproduce a more complete hologram and in turn a higher resolution image can be reconstructed [20]. This method is however limited in lateral resolution by the pitch of the detector pixel.
(32) An alternate method for synthetic aperture hologram capture can be used in order to produce a subpixel resolution. Using a scanning mirror or a collection of symmetric apertures, multiple holograms of the same FOV can be produced on a detector. Registration of these holograms results in artificial subpixel shifts in the holograms which make this approach comparable to the previously mentioned aperture shifting methods. Although high lateral resolution is achieved with the proposed synthetic aperture methods the FOV of these systems is limited since an objective lens is required.
(33) Advantageously, the present system, method and apparatus is able to achieve subpixel imaging resolution (ultra-resolution) beyond the pitch of the detector pixels while increasing the FOV at the same time by capturing a more complete interferometric pattern through the capturing of multiple holograms at different subpixel displacements of the detector. As such, the present system, method and apparatus is not limited by the imaging resolution and FOV restrictions of the previous synthetic aperture methods.
(34) Measuring the characteristics of the light reflected, scattered, emitted, re-emitted, and absorbed by an object can provide clues about the object's intrinsic properties. For example, these properties can include the physical state or the molecular composition of the object observed, along with many other derived properties. In holographic microscopy, capturing images at different wavelengths, not only affects the resolution of the reconstructed images but the combination of the correct wavelengths can produce color images. These images may contain information that would otherwise not have been observed. Additionally, color images can help with visualization of the different locations of the FOV, pertaining to different species and more fundamentally the difference in the index of refraction. Thus far color imaging in holographic microscopy have been limited to using a color CMOS sensor (a Bayer filter is placed on a group of four pixels [21]) in combination with 1) red, green, and blue laser illumination source [22]: 2) LEDs with central wavelength in the red, green, and blue range [22, 23]; 3) broadband light source, such as a Xenon or Halogen lamp, with a red, green, blue wavelength selector mechanism such as an acousto-optical tunable filer or a liquid crystal tunable filter[24]; 4) various pattern recognition calibration techniques of producing false-colors [25, 26].
(35) To the inventors' knowledge, no previous synthetic aperture color imaging holographic microscopy methods exist. The present system, method and apparatus utilizes synthetic aperture color imaging holographic microscopy methods to provide spectral information beyond the red, green, blue wavelengths achieved by previous methods, by sampling the optical wavelengths more systematically and at smaller wavelength intervals, utilizing wavelengths ranging from the ultraviolet wavelengths and the near Infra-red wavelengths. As such, the present system, method and apparatus achieves multispectral and hyperspectral imaging beyond the red, green, and blue wavelengths of previous methods.
(36) Lasers have historically been used for holographic metrology and are prevalent in other holographic microscope designs however for an in-line or Gabor design of the holographic microscope lasers are not preferred. The high temporal coherence of lasers gives rise to very strong speckle pattern which adversely affects the visibility of the hologram. Additionally, lasers produce undesired multiple reflection effects which further reduce the quality of the hologram.
(37) It will be appreciated that, for simplicity and clarity of illustration, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements or steps. In addition, numerous illustrative examples are set forth in order to provide a thorough understanding of the invention described herein. However, it will be understood that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail, so as not to obscure the key aspects of the invention as described herein. Furthermore, this description is not to be considered as limiting the scope of the embodiments described herein in any way, but rather as merely describing the implementation of the various embodiments described herein.
(38) Now referring to
(39) With reference to
(40) Referring now to
(41) Now referring to
(42) It will be appreciated that, using a combination of the proposed USAHM schemes, an ultra-wide FOV can be realized which can be many times larger than the active area of the detector used in the system, while at the same time preserving the ultra-resolution advantages of the proposed USAHM. For instance after performing USAHM using one of above mentioned methods a large translation of the detector would allow for observation and digitization of another region of the sample which would otherwise be off of the active sensor area of the detector. Additionally, this translation over a significantly larger area than the detector can be used to capture the continuation of the interferometric diffraction pattern of any of the object which can further improve resolution of the intensity and QPCM image as well as allowing for an ultra-wide FOV.
(43) An alternate method of performing ultra-resolution hologram capture while simultaneously capturing an ultra-wide FOV is to perform a so called off-axis PolUSAHM. With reference to
(44) The three aforementioned synthetic aperture mechanisms (TranUSAHM, PolUSAHM, RadUSAHM) may be used concurrently or individually, nonetheless enabling ultra-resolution holographic microscopy and ultra-resolution QPCM via hologram acquisitions at subpixel shifts of the detector. The three aforementioned synthetic aperture mechanisms may also be used concurrently in a random fashion to achieve random movements of the detector for synthetic aperture scanning.
(45) With reference to
(46) In these illustrative embodiments, the linear displacement platform 102 is used to translate the detector in the axial Z direction. This entity can be used for various calibration purposes such as removing the twin-image and reducing the interferometric noise as well as other aberrations. The rotational displacement platform is utilized to rotate the detector about the rotational axis of the system therefore enabling PolUSAHM and off-axis PolUSAHM.
(47) These moveable platforms can be controlled manually using analog or digital micrometers, or electronically by employing linear actuators pistons, picomotors, piezoelectric servos, etc. Alternatively, the locomotion mechanism can be embedded into the platform. As previously mentioned, with the extremely accurate and precise displacement mechanism, small displacement steps on the order of tens of nanometers can be achieved, enabling ultra-resolution hologram acquisition, and any number of subpixel displacements can be used. If the displacement mechanism does not have very high precision and accuracy, ultra-resolution hologram acquisition is nevertheless still possible since these systems will not have a very high rate of repeatability in displacement steps. Therefore, by stepping forward then backward, subpixel displacement can be achieved. To enable the ultra-wide FOV capability, the moveable platforms 100 and 101 would have to have a displacement range that is larger than the size of the active area of the detector 200.
(48) Again, with reference to
(49) With reference to
(50) With reference to
(51) Referring to
(52) With respect to the embodiments where a microscope objective lens is utilized,
(53) Referring back to
(54) In the illustrative embodiment depicted in
(55) With respect to
(56) With reference to
(57) Referring to
(58) In the illustrative embodiment depicted in
(59) With reference to
f=interp(h.sub.1,h.sub.2, . . . ,h.sub.n)
where interp(.) is an interpolation function that interpolates pixels in f using pixels in h.sub.1, h.sub.2, . . . , h.sub.n. Possible interpolation functions include nearest-neighbor, bilinear, and bicubic interpolation, but are not limited to these functions and other interpolation functions can be used in other embodiments. A numerical reconstruction algorithm 2002 is then used to perform aberration correction to reconstruct an ultra-resolved, aberration-corrected hologram (f) using the ultra-resolved hologram (f) as input:
f=A(f,L)
where A is an aberration correction function and L is the point spread function. A possible aberration correction function is the following Maximum A Posteriori optimization function:
f=arg max.sub.fP(f|f,L)
where P(f|f,L) is the posterior probability off given f and L. Note that aberration correction is not limited to this function, and other aberration correction functions can be used in other embodiments. A numerical reconstruction algorithm 2003 is then used to perform numerical diffraction to reconstruct ultra-resolved amplitude (intensity) and phase information 2004 at desired axial depths (z) (denoted by g={g.sub.amplitude, g.sub.phase}) using the ultra-resolved, aberration-corrected hologram:
g=D(f,z)
where D is a numerical diffraction function encompassing a numerical diffraction method and z is the desired depth. Possible numerical diffraction methods in the numerical diffraction function that can be used include Fresnel Transform, Huygens convolution, Angular Spectrum method, as well as Bayesian-based numerical diffraction as expressed by the following equation:
g=arg max.sub.gP(g|f,z)
where P(g|f,z) is the posterior probability of g given hologram f and depth z. Note that numerical diffraction is not limited to these methods, and other numerical diffraction methods can be used in other embodiments. A numerical reconstruction algorithm is then used to reconstruct 2D/3D ultra-resolved intensity images (g.sub.amplitude) and 2D/3D QPCM images (g.sub.QPCM) 2005 using the ultra-resolved amplitude and phase information from one or more axial depths z.sub.1, z.sub.2, . . . , z.sub.n 2004 (m denotes the number of depths):
g.sub.amplitude=U(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
g.sub.QPCM=V(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where g.sub.amplitude(z) denotes ultra-resolved amplitude information from depth z, g.sub.amplitude(z) denotes ultra-resolved phase information from depth z, U is a function for reconstructing 2D/3D ultra-resolved intensity images (g.sub.amplitude) from g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m), and U is a function for reconstructing 2D/3D QPCM images (g.sub.QPCM) from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). One possible function for U is to interpolate a 3D ultra-resolved intensity image g.sub.amplitude using pixels from g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m):
g.sub.amplitude=interp(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.amplitude using pixels in g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m). One possible function for V is to interpolate a 3D ultra-resolved QPCM image g.sub.QPCM using pixels from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m):
g.sub.QPCM=interp(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.QPCM using pixels in g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). Note that reconstruction is not limited to these methods, and other reconstruction methods can be used in other embodiments. This sequence of algorithms can be performed for each captured wavelength by the present invention to produce multispectral and hyperspectral 2D/3D ultra-resolved intensity images and 2D/3D QPCM images.
(60) With reference to
f=interp(h.sub.1,h.sub.2, . . . ,h.sub.n)
where interp(.) is an interpolation function that interpolates pixels in f using pixels in h.sub.1, h.sub.2, . . . , h.sub.n. Possible interpolation functions include nearest-neighbor, bilinear, and bicubic interpolation, but are not limited to these functions and other interpolation functions can be used in other embodiments. A numerical reconstruction algorithm 2100 is then used to perform joint aberration correction and numerical diffraction to reconstruct ultra-resolved amplitude (intensity) and phase information 2004 at desired axial depths (z) (denoted by g={g.sub.amplitude, g.sub.phase}) using the ultra-resolved hologram (f) as input:
g=AN(f,L,z)
where AN is a joint aberration correction and numerical diffraction function, L is the point spread function, and z is the desired depth. A possible joint aberration correction and numerical diffraction function is the following Maximum A Posteriori optimization function:
g=arg max.sub.gP(g|f,L,z)
where P(g|f,L) is the posterior probability of g given f, L, and z. Note that joint aberration correction and numerical diffraction is not limited to this function, and other joint aberration correction and numerical diffraction functions can be used in other embodiments. A numerical reconstruction algorithm is then used to reconstruct 2D/3D ultra-resolved intensity images (g.sub.amplitude) and 2D/3D QPCM images (g.sub.QPCM) 2005 using the ultra-resolved amplitude and phase information from one or more axial depths z.sub.1, z.sub.2, . . . , z.sub.m 2004 (m denotes the number of depths):
g.sub.amplitude=U(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
g.sub.QPCM=V(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where g.sub.amplitude(z) denotes ultra-resolved amplitude information from depth z, g.sub.amplitude(z) denotes ultra-resolved phase information from depth z, U is a function for reconstructing 2D/3D ultra-resolved intensity images (g.sub.amplitude) from g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m), and U is a function for reconstructing 2D/3D QPCM images (g.sub.QPCM) from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). One possible function for U is to interpolate a 3D ultra-resolved intensity image g.sub.amplitude using pixels from g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m):
g.sub.amplitude=interp(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.amplitude using pixels in g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m). One possible function for V is to interpolate a 3D ultra-resolved QPCM image g.sub.QPCM using pixels from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m):
g.sub.QPCM=interp(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.QPCM using pixels in g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). Note that reconstruction is not limited to these methods, and other reconstruction methods can be used in other embodiments. This sequence of algorithms can be performed for each captured wavelength by the present invention to produce multispectral and hyperspectral 2D/3D ultra-resolved intensity images and 2D/3D QPCM images.
(61) With reference to
g=HAN(h.sub.1,h.sub.2, . . . ,h.sub.n,L,z)
where HAN is a joint ultra-resolved hologram reconstruction, aberration correction, and numerical diffraction function, L is the point spread function, and z is the desired axial depth. A possible joint ultra-resolved hologram reconstruction, aberration correction and numerical diffraction function is the following Maximum A Posteriori optimization function:
g=arg max.sub.gP(g|h.sub.1,h.sub.2, . . . ,h.sub.n,L,z)
where P(g|h.sub.1, h.sub.2, . . . , h.sub.n, L, z) is the posterior probability of g given h.sub.1, h.sub.2, . . . , h.sub.n, L, and z. Note that joint ultra-resolved hologram reconstruction, aberration correction and numerical diffraction is not limited to this function, and other joint ultra-resolved hologram reconstruction, aberration correction and numerical diffraction functions can be used in other embodiments. A numerical reconstruction algorithm is then used to reconstruct 2D/3D ultra-resolved intensity images (g.sub.amplitude) and 2D/3D QPCM images (g.sub.QPCM) 2005 using the ultra-resolved amplitude and phase information from one or more axial depths z.sub.1, z.sub.2, . . . , z.sub.m 2004 (m denotes the number of depths):
g.sub.amplitude=U(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
g.sub.QPCM=V(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where g.sub.amplitude(z) denotes ultra-resolved amplitude information from depth z, g.sub.amplitude(z) denotes ultra-resolved phase information from depth z, U is a function for reconstructing 2D/3D ultra-resolved intensity images (g.sub.amplitude) from g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m), and U is a function for reconstructing 2D/3D QPCM images (g.sub.QPCM) from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). One possible function for U is to interpolate a 3D ultra-resolved intensity image g.sub.amplitude using pixels from
g.sub.amplitude=interp(g.sub.amplitude(z.sub.1),g.sub.amplitude(z.sub.2), . . . ,g.sub.amplitude(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.amplitude using pixels in g.sub.amplitude(z.sub.1), g.sub.amplitude(z.sub.2), . . . , g.sub.amplitude(z.sub.m). One possible function for V is to interpolate a 3D ultra-resolved QPCM image g.sub.QPCM using pixels from g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m):
g.sub.QPCM=interp(g.sub.phase(z.sub.1),g.sub.phase(z.sub.2), . . . ,g.sub.phase(z.sub.m))
where interp(.) is an interpolation function that interpolates pixels in g.sub.QPCM using pixels in g.sub.phase(z.sub.1), g.sub.phase(z.sub.2), . . . , g.sub.phase(z.sub.m). Note that reconstruction is not limited to these methods, and other reconstruction methods can be used in other embodiments. This sequence of algorithms can be performed for each captured wavelength by the present invention to produce multispectral and hyperspectral 2D/3D ultra-resolved intensity images and 2D/3D QPCM images.
(62) It will be appreciated that the present invention is not limited to the aforementioned embodiments of the algorithmic processing pipeline for producing ultra-resolved ultra-wide FOV intensity and QPCM images using the USAHM generated data. Furthermore, in all embodiments, the present invention may also produce 2D/3D visualizations of the multispectral and hyperspectral 2D/3D ultra-resolved ultra-wide FOV intensity images and 2D/3D QPCM images using data fusion methods including Principal Component Analysis, Factor Analysis, Independent Component Analysis, Linear Discriminant Analysis, Local Linear Embedding, Stochastic Neighbor Embedding, Large-Margin Nearest Neighbor, and their variants; note that the data fusion method used in the present invention for producing visualizations are not limited to these data fusion methods and other data fusion methods may be used.
(63) With reference to
(64) Now referring to
(65) Thus, in an aspect, there is provided a method of performing holographic microscopy and quantitative phase contrast microscopy, comprising: providing a stationary spectral illumination source with capability of central wavelength and spectral bandwidth selection for emission of light at desired wavelengths; acquiring with a detector a plurality of low-resolution holograms of an image subject from incremental locations utilizing a subpixel sensor-scanning synthetic aperture mechanism having one or more detector scanning patterns; processing the acquired holograms utilizing one or more processing algorithms corresponding to the one or more detector scanning patterns used to acquire the holograms; and reconstructing a subpixel ultra-resolution image of the image subject with an effective field-of-view (FOV) larger than the detector based on the plurality of processed low-resolution holograms; whereby, a desired synthetic aperture is achieved without loss of resolution.
(66) In an embodiment, the spectral illumination source comprises one or more of LEDs, laser sources, scanning grating monochrometers, a nano-photonic ring-resonator spectrograph wavelength selector, a broadband light source, and a tunable laser source.
(67) In another embodiment, the method comprises a light wavelength selection mechanism including bandpass spectral filters, acousto-optical and liquid crystal tunable filters, and a dispersing element.
(68) In another embodiment, the plurality of low-resolution holograms comprises algorithmically assembling the plurality of low-resolution holograms utilizing a numerical reconstruction algorithm, in dependence upon the detector scanning pattern used, thereby to create a sub-pixel ultra-resolution image.
(69) In another embodiment, the numerical reconstruction algorithm is adapted to perform aberration correction to reconstruct an ultra-resolved, aberration-corrected hologram.
(70) In another embodiment, the numerical reconstruction algorithm is further adapted to perform numerical diffraction to reconstruct ultra-resolved amplitude and phase information at desired axial depths (Z).
(71) In another embodiment, the numerical reconstruction algorithm is further adapted to reconstruct 2D/3D ultra-resolved intensity images and 2D/3D quantitative phase contrast microscopy (QPCM) images using the ultra-resolved amplitude and phase information from one or more axial depths.
(72) In another embodiment, the one or more detector scanning patterns includes one or more of translational, radial, polar rotational or random scanning of the image subject.
(73) In another embodiment, the subpixel sensor-scanning synthetic aperture mechanism is capable of motion in lateral X and Y directions, an axial Z direction, and rotation about a defined axis.
(74) In another embodiment, the subpixel sensor-scanning synthetic aperture mechanism is capable of concurrent motion in more than one direction.
(75) In another embodiment, the detector is adapted to capture images at different and selectable spectral bands comprising a central wavelength and spectral bandwidth of the spectral illumination source.
(76) In another embodiment, the method further comprises placing a microscope objective lens above the detector to locate a focal point of the microscope objective lens at a surface of a microscope slide or on a screen which is displaying an interferometric diffraction pattern of the sample.
(77) In another aspect, there is provided an apparatus for performing holographic microscopy and quantitative phase contrast microscopy, comprising: a spectral illumination source with capability of central wavelength and spectral bandwidth selection, the spectral illumination source adapted to illuminate a target through an illumination aperture or via a waveguide; a detector mounted on a movable scanning platform for acquiring a plurality of low-resolution holograms of an image subject from incremental locations utilizing a subpixel sensor-scanning synthetic aperture mechanism having one or more detector scanning patterns; a sample handling platform adapted to securely and accurately position a sample along an optical axis of the system between the illumination source and the detector; a processing unit adapted to reconstruct a subpixel ultra-resolution image of the image subject with an effective field-of-view (FOV) larger than the detector based on the plurality of processed low-resolution holograms; whereby, the apparatus achieves a desired synthetic aperture is achieved without loss of resolution.
(78) In an embodiment, the spectral illumination source comprises one or more of LEDs, laser sources, scanning grating monochrometers, a nano-photonic ring-resonator spectrograph wavelength selector, a broadband light source, and a tunable laser source.
(79) In another embodiment, the apparatus further comprises a light wavelength selection mechanism including bandpass spectral filters, acousto-optical and liquid crystal tunable filters, and a dispersing element.
(80) In another embodiment, the spectral illumination source comprises one or more light-emitting diodes (LEDs) or Halogen or Halogen-Tungsten or a Xenon lamp, with or without narrow bandpass spectral transmission filters, which may be turned on simultaneously or sequentially.
(81) In another embodiment, the spectral illumination source comprises one or more laser sources that may be turned on simultaneously or sequentially.
(82) In another embodiment, the apparatus further comprises one or more scanning grating monochrometers for wavelength selection, in which the grating is placed on a rotational moveable platform in order to superpose a desired central wavelength on the illumination aperture.
(83) In another embodiment, the apparatus further comprises a nano-photonic ring-resonator spectrograph placed directly at the illumination aperture.
(84) In another embodiment, the scanning platform is adapted to move the detector translationally, radially, rotationally or randomly in plurality of incremental positions enabling synthetic aperture scanning.
(85) In another embodiment, the detector is a monochromatic or color charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) detector.
(86) In another embodiment, the processing unit is adapted to execute processing techniques and algorithms to reconstruct holograms, obtain amplitude and phase information from the holograms, reconstruct amplitude images, and reconstruct quantitative phase contrast microscopy images.
(87) In another embodiment, the apparatus further comprises a microscope objective lens placed above the detector to locate a focal point of the microscope objective lens at the surface of a microscope slide or on a screen which is displaying an interferometric diffraction pattern of the sample.
(88) Although various examples have been described above, it will be appreciated that these illustrative embodiments are not meant to be limiting, and that the scope of the invention is defined by the following claims.
REFERENCES
(89) [1] Gabor D, A New Microscope Principle. Nature. 1948; 16: 777-778. [2] Gabor D, Microscopy by Reconstructed Wavefronts. Proc. Roy. Soc. 1949; A197: 454-487. [3] Kim M K, Digital Holographic Microscopy: Principles, Techniques, and Applications. New York, USA: Springer-Verlag; 2011. [4] Mudanyali O. Tseng D. Oh C, et al. Compact. Light-weight and Cost-effective Microscope based on Lensless Incoherent Holography for Telemedicine Applications. Lab on a chip. 2010, 10(11): 1417-1428. [5] Bishara W, Sikora U, Mudanyali O, Su T-W, Yaglidere O, Luckhart S, and Ozcan A. Holographic Pixel superresolution in Portable Lensless On-chip Microscopy Using a Fiber-optic Array. Lab Chip. 2011; 11: 1276-1279. [6] Mico V, Zlevsky Z, Garcia-Martinez P, Garcia J. Synthetic Aperture Superresolution with Multiple Off-axis Holograms. Journal of Optical Society of America. 2006; 23(12): 3162-3170. [7] Bishara W, Su T-W, Coskun A F, Ozcan A. Lensfree On-chip Microscopy Over a Wide Field-of-view Using Pixel Superresolution. Optics Express. 2010; 18: 11181-11191. [8] Paturzo M, Ferraro, P. Correct Self-assembling of Spatial Frequencies in Superresolution Synthetic Aperture Digital Holography. Optics Letters. 2009; 34: 3650-3652. [9] Pelagotti A, Paturzo M. Locatelli M. Geltrude A. Meucci R, Finizio A., Ferraro P. An Automatic Method for Assembling a Large Synthetic Aperture Digital Hologram. Optics Express. 2012; 20: 4830-4839. [10] Hardie R C, Barnard K J, Bognar J G, Armstrong E E, Watson E A. High-resolution Image Reconstruction from a Sequence of Rotated and Translated Frames and its Application to an Infrared Imaging System. Optical Engineering. 1998; 37: 247. [11] Luo W, Greenbaum A, Zhang Y, Ozcan A. Synthetic Aperture-based On-chip Microscopy. Light: Science & Application. 2015; accepted. [12] Mico V, Zalevsky Z. Superresolved Digital In-line Holographic Microscopy for High-resolution Lensless Biological Imaging. Journal of Biomedical Optics 2010; 15(4): 046027-1-5. [13] Lai X-J, Tu H-Y, Wu C-H, Lin Y-C, Cheng C-J. Resolution Enhancement of Spectrum Normalization in Synthetic Aperture Digital Holographic Microscopy. Applied Optics. 2015; 54(1): A51-A58. [14] Granero L, Mico V, Zalevsky Z, Garcia J. Synthetic Aperture Superresolved Microscopy in Digital Lensless Fourier Holography by Time and Angular Multiplexing of the Object Information. Applied Optics. 2010; 49(5): 845-857. [15] Cheng C-J, Lai X-J, Lin Y-C, Tu H-Y. Superresolution Imaging in Synthetic Aperture Digital Holographic Microscopy. IEEE 4.sup.th International Conference on Photonics. 2013; 215-217. [16] Ralston T S, Marks D L, Carnmey P S, Boppart S A. Interferometric Synthetic Aperture Microscopy. Nature Physics. 2007; 3: 129-134. [17] Hayes M P, Gough Pt. Synthetic Aperture Sonar: A Review of Current Status. IEEE Journal of Ocean. Engineering. 2009; 34(3): 207-224. [18] Robinson S E, Vrba J. Functional Neuroimaging by Synthetic Aperture Magnetometry (SAM). Recent Advances in Biomagnetism. Sendai: Tohoku University Press. 1998; 302-305. [19] Katz B, Rosen J. Superresolution in Coherent Optical Imaging Using Synthetic Aperture with Fresnel Elements. Optics Express. 2010; 18(2): 962-972. [20] Bayer B. Color imaging array. American Patent 555 477, 1976. [21] Yamaguchi I, Matsumura T, Kato J. Phase-shifting Color Digital Holography. Optics Letters. 2002; 27(13): 1108-1110. [22] Isikman S O, Sencan I, Mudanyali O, Bishara W, Oztoprak C, Ozcan A. Color and Monochrome Lensless On-chip Imaging of Caenorhabditiselegans Over a Wide Field-of-view. Lab Chip. 2010; 10: 1109-1112. [23] Kiss M Z, Nagy B J, Lakatos P, Grcs Z, TksS, Wittner B, Orzo L. Special Multicolor Illumination and Numerical Tilt Correction in Volumetric Digital Holographic Microscopy. Optics Express. 2014; 22(7): 7559-7573. [24] Isikman S O, Greenbaum A, Luo W, Coskun A F, Ozcan A. Giga-Pixel Lensfree Holographic Microscopy and Tomography Using Color Image Sensors. PloS ONE. 2012; 7(9): e45044. [25] Greenbaum A, Feizi A, Akbari N, Ozcan A. Wide-field Computational Color Imaging Using Pixel Superresolved On-chip Microscopy. Optics Express. 2013; 21(10): 12469-12483. [26] Greenbaum A, Zhang Y, Feizi A, Chung P-L, Luo W, Kandukuri S, Ozcan A. Wide-field Computational Imaging of Pathology Slides Using Lens-free On-chip Microscopy. Science Translational Medicine. 2014; 6(267): 267ra175. [27] Xia A, Eftekhar A A, Solatani M, Momeni B, Li Q, Chamanzar M, Yegnanarayanan S, Adibi A. High Resolution On-chip Spectroscopy Based on Miniaturized Microdonut Resonators. Optics Express. 2011; 19(13): 12356-12364.