High-illumination numerical aperture-based large field-of-view high-resolution microimaging device and a method for iterative reconstruction

Abstract

A high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, and a method for iterative reconstruction, the device comprising an LED array (1), a stage (2), a condenser (3), a microscopic objective (5), a tube lens (6), and a camera (7), the LED array (1) being arranged on the forward focal plane of the condenser (3). Light emitted by the i-th lit LED unit (8) of the LED array (1) passes through the condenser (3) and converges to become parallel light illuminating a specimen (4) to be examined, which is placed on the stage (2); part of the diffracted light passing through the specimen (4) is collected by the microscopic objective (5), converged by the tube lens (6), and reaches the imaging plane of the camera (7), forming an intensity image recorded by the camera (1). The present device and method ensure controllable programming of the illumination direction, while also ensuring an illumination-numerical-aperture up to 1.20 and thus achieving a reconstruction resolution up to 0.15 μm.

Claims

1. A high-illumination numerical aperture-based large field-of-view high-resolution micro-imaging iterative reconstruction method, by using a high-illumination numerical aperture-based large field-of-view high-resolution microimaging device comprising: an LED array; a stage; a condenser; a microscopic objective; a tube lens; and a camera, wherein the LED array is arranged on a forward focal plane of the condenser, and a center of the LED array is on an optical axis of the microscope objective, wherein a back focal plane of the microscope objective coincides with a forward focal plane of the tube lens, an imaging plane of the camera is placed on a back focal plane of the tube lens, and during imaging, a specimen to be examined on the stage is adjusted to a front focal plane position of the microscope objective to form an infinity-corrected imaging system, and wherein light emitted by an i-th lit LED unit of the LED array passes through the condenser and converges to become parallel light illuminating the specimen to be examined, which is placed on the stage, and part of diffracted light passing through the specimen is collected by the microscopic objective, converged by the tube lens, and reaches the imaging plane of the camera, forming an intensity image recorded by the camera, said method comprising the steps of: calibrating a brightness of the LED unit, by using the LED array as an illumination source of the high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, sequentially lighting up each LED unit of the LED array, and after illuminating a blank specimen, using high magnification objective lens to collect corresponding images and calculating a normalized brightness correction coefficient corresponding to each LED unit in three channels of red, green and blue; calibrating a position of the LED array, by using a resolution board as the specimen to be examined, using the LED array as the illumination source of the high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, sequentially lighting up each LED unit of the LED array, after illuminating the specimen to be examined, images in focus and images with defocus distance h are respectively collected, then calculating an illumination angle corresponding to each of the three channels of red, green and blue for each LED unit by using a sub-pixel registration algorithm, and then determining the position of the LED array by using nonlinear regression; capturing raw images, by using the LED array as the illumination source of the large field-of-view high-resolution micro-imaging device, sequentially lighting up each LED unit of the LED array, and after illuminating the specimen to be examined, collecting corresponding low-resolution raw images; pre-processing raw images, including threshold denoising and brightness correction, by firstly using an average value of a dark current noise of the camera as a threshold, performing threshold denoising on the captured low-resolution raw images, obtaining denoised low-resolution images, and then dividing each denoised low-resolution image by the normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit, thereby obtaining brightness-corrected low-resolution images; initializing high-resolution images, by adding and averaging all the low-resolution bright-field images among the brightness-corrected low-resolution images, and then initializing amplitude and phase of high-resolution images by up-sampling; and performing an iterative reconstruction, wherein all the brightness corrected low-resolution images are subjected to synthetic aperture calculation one by one in a frequency domain using a pixel binning based Fourier ptychographic iterative reconstruction method while gradually reducing an updating coefficient, a cost function value is used as a criterion, and when the cost function value is less than a given threshold, the iteration is stopped, and the amplitude and phase of the high-resolution image are determined to be a final reconstructed large field-of-view high-resolution microscopic image.

2. The method according to claim 1, wherein the microscope objective has a numerical aperture of NA.sub.obj and satisfies ${\mathrm{NA}}_{\mathrm{obj}}\frac{f}{d}>1,$ wherein f is a focal length of the condenser, d is a spacing between adjacent LED units, d<2 mm, and wherein an illumination wavelength of the LED array is λ, a magnification of the microscope objective is Mag, a pixel size is Δx.sub.cam, and the illumination wavelength satisfies $\lambda \frac{\mathrm{Mag}}{2\Delta x_{\mathrm{cam}}}\frac{f}{d}>1.$

3. The method according to claim 1, wherein the camera is a color or monochrome camera, and when the camera is a monochrome camera, when imaging, each LED unit of the LED array emits red or green or blue monochromatic light respectively, recording all monochrome images sequentially by the monochrome camera; when the camera is a color camera, when imaging, each of the LED units of the LED array simultaneously emits red, green and blue light, recording all color images by the color camera.

4. The method according to claim 1, further comprising a microscope oil disposed between the condenser and the specimen to be examined.

5. The method according to claim 1, wherein in the step of calibrating the brightness of the LED unit, the entire LED array comprises a total of N LED units, N>261, each LED unit emits monochromatic light of red, green and blue, respectively, a total of 3N low-resolution images are captured, and the image of the blank specimen lighted by the i-th LED unit with color c is captured and marked as I.sub.i,c.sup.kb(r), where i=1,2, . . . ,N, c=r,g,b, r is two-dimensional coordinates of real space r=(x,y), and then an average intensity of each image is calculated as $B_{i,c}^{kb}=\frac{1}{N_{\mathrm{pixel}}}\underset{r}{.Math.}{I}_{i,c}^{\mathrm{kb}}\left(r\right)$ and becomes the average brightness of each LED unit in three channels of red, green and blue, where N.sub.pixel is a total number of pixels in one image I.sub.i,c.sup.kb(r), an average brightness corresponding to the LED unit in a center of the LED array is B.sub.l,c.sup.kb, and then the normalized brightness correction coefficient R.sub.i,c.sup.bri of each LED unit corresponding to three channels of red, green and blue is R.sub.i,c.sup.bri=B.sub.i,c.sup.kb/B.sub.l,c.sup.kb.

6. The device according to claim 1, wherein the step of calibrating the position of the LED array further comprises the steps of: after the LED array illuminating the specimen to be examined, first respectively collecting in-focus image I.sub.i,c.sup.focus and defocus image I.sub.i,c.sup.defocus with defocus distance h; according to angular spectrum diffraction theory, numerically propagating in-focus image I.sub.l,c.sup.focus corresponding to the LED unit in a center of LED array along optical axis with numerical value h distance, thereby obtaining numerical defocus image I.sub.l,c.sup.pro; calculating an offset value (PY.sub.x,PY.sub.y) of each defocus image I.sub.i,c.sup.defocus relative to the numerical defocus image I.sub.l,c.sup.pro by using sub-pixel registration algorithm, wherein a spatial frequency vector of illumination light corresponding to the i-th LED unit is $u_i=\left(u_x,u_y\right)=\frac{2\pi }{\lambda }\left(\frac{P{Y}_x}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}},\frac{{\mathrm{PY}}_y}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}}\right),$ where (u.sub.x,u.sub.y) is a spatial frequency along the x,y direction, and λ is a wavelength of the illumination light; and finally determining the position of the LED array by nonlinear regression according to formulas of: $Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)=\underset{i}{.Math.}|u_i-u_i^0{|}^2,\left(\theta ,\Delta x,\Delta y,f\right)=\mathrm{��ℒ}\left[Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)\right],x_i=d\left[\cos \left(\theta \right)m_i+\sin \left(\theta \right)n_i\right]+\Delta x$ $y_i=d\left[-\sin \left(\theta \right)m_i+\cos \left(\theta \right)n_i\right]+\Delta y,u_i=\frac{2\pi }{\lambda }\frac{x_i}{f}$ $v_i=\frac{2\pi }{\lambda }\frac{y_i}{f},$ where Q ( . . . ) is an objective function of a nonlinear regression method, (θ, Δx, Δy, f) are updated four position parameters of the LED array, which are respectively a rotation error, a translation error in the x direction, a translation error in the y direction, and a focusing error, (θ.sup.0, Δx.sup.0, Δy.sup.0, f.sup.0) are initialized LED array position parameters, indicates that a nonlinear regression operation is performed, d is a spacing between two adjacent LED units of the LED array, (x.sub.i, y.sub.i) represents spatial position coordinates of the i-th LED unit, λ is a wavelength of the illumination light, and (m.sub.i,n.sub.i) is a row number and a column number corresponding to the i-th LED unit.

7. The method according to claim 1, wherein the step of pre-processing raw images further comprises the steps of: first sequentially lighting up the LED units and illuminating the specimen to be examined by using monochromatic light of red, green and blue, wherein the captured original low-resolution images are marked as I.sub.i,c; turning off all the LED units and the captured dark current noise image I.sub.dark; using the average value of the dark current noise of the camera as a threshold, and then performing threshold denoising on the captured original low-resolution images according to the formula of $I_{i,c}^{dn}\left(r\right)=\{\begin{array}{c}{I}_{i,c}\left(r\right)-\mathrm{mean}\left(I_{\mathrm{dark}}\right),I_{i,c}\left(r\right)>\mathrm{mean}\left(I_{\mathrm{dark}}\right)\\ 0,I_{i,c}\left(r\right)<=\mathrm{mean}\left(I_{\mathrm{dark}}\right)\end{array},$ where I.sub.i,c.sup.dn represents low-resolution images after the threshold denoising, and mean ( . . . ) indicates evaluating average gray value of the image; and then dividing each image by the normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit to accomplish an image brightness correction process according to the formula of:
I.sub.i,c.sup.uni=I.sub.i,c.sup.dn/R.sub.i,c.sup.bri, where I.sub.i,c.sup.uni is a low-resolution image after brightness correction, I.sub.i,c.sup.dn is a low-resolution image after performing threshold denoising, R.sub.i,c.sup.bri is a normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit.

8. The method according to claim 1, wherein in the step of performing the iterative reconstruction, formula based on the pixel binning based Fourier ptychographic iterative method is as follows: $O_i^k=F\left\{o_i^k\right\},{\Psi }_i^k=F\left\{\mathrm{UP}\left[\sqrt{\frac{I_{i,c}^{\mathrm{uni}}}{\mathrm{DOWN}.Math.|F^{-1}\left\{P_i^k{O}_i^k\right\}{|}^2.Math.}}\right]F^{-1}\left\{P_i^k{O}_i^k\right\}\right\},O_{i+1}^k=O_i^k-{\alpha }^k\frac{\left|P_i^k\right|}{|P_i^k{|}_{\max }\left(|P_i^k{|}^2+\gamma \right)}{P}_i^{k*}\left({\Psi }_i^k-P_i^k{O}_i^k\right),P_{i+1}^k=P_i^k-{\beta }^k\frac{\left|O_i^k\right|}{|O_i^k{|}_{\max }\left(|O_i^k{|}^2+\gamma \right)}{O}_i^{k*}\left({\Psi }_i^k-P_i^k{O}_i^k\right),{\alpha }^k=\frac{{\alpha }^1}{k},{\alpha }^1=\frac{1}{2},{\beta }^k=\frac{{\beta }^1}{k},{\beta }^1=\frac{1}{N},\mathrm{and}$ ${\mathrm{COST}}^k=\underset{i}{.Math.}\mathrm{abs}\left(I_{i,c}^{\mathrm{uni}}-\mathrm{DOWN}\left[|F^{-1}\left\{P_i^k{O}_i^k\right\}{|}^2\right]\right),$ where F { . . . } indicates that a Fourier transform is performed, F.sup.−1{ . . . } indicates that an inverse Fourier transform is performed, UP[ . . . ] indicates that the up-sampling nearest-neighbor interpolation is performed, DOWN[ . . . ] indicates a down-sampling pixel binning process; O.sub.i.sup.k is a high-resolution spectrum of the specimen to be examined, k represents the k-th iteration, P.sub.i.sup.k is the spectrum aperture function of the microscopic objective, and Ψ.sub.i.sup.k is an updated local spectrum of the specimen to be examined, γ is a constant to ensure that a denominator is not zero, and a typical value is 0.001; | . . . | represents a modulus of a two-dimensional complex matrix, | . . . |.sub.max represents a maximum value in the modulus of the two-dimensional complex matrix; α.sup.k is an updated coefficient of the spectrum of the specimen to be examined in the k-th iteration, β.sup.k is an updated coefficient of an aperture function of the microscopic objective in the k-th iteration, COST.sup.k is the cost function, and wherein when the cost function COST.sup.k is less than a certain fixed threshold ε at an end of the k-th iteration, the iteration is considered to be convergent and the iteration is stopped, the amplitude and the phase of the high-resolution image at the moment are determined to be the finally reconstructed large field-of-view high-resolution microscopic image.

9. A high-illumination numerical aperture-based large field-of-view high-resolution microimaging iterative reconstruction method by using a high-illumination numerical aperture-based large field-of-view high-resolution microimaging device comprising: an LED array; a stage; a condenser; a microscopic objective; a tube lens; and a camera, wherein the LED array is arranged on a forward focal plane of the condenser, and a center of the LED array is on an optical axis of the microscope objective, wherein a back focal plane of the microscope objective coincides with a forward focal plane of the tube lens, an imaging plane of the camera is placed on a back focal plane of the tube lens, and during imaging, a specimen to be examined on the stage is adjusted to a front focal plane position of the microscope objective to form an infinity-corrected imaging system, and wherein light emitted by an i-th lit LED unit of the LED array passes through the condenser and converges to become parallel light illuminating the specimen to be examined, which is placed on the stage, and part of diffracted light passing through the specimen is collected by the microscopic objective, converged by the tube lens, and reaches the imaging plane of the camera, forming an intensity image recorded by the camera, said method comprising the steps of: calibrating a brightness of the LED unit, by using the LED array as an illumination source of the high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, sequentially lighting up each LED unit of the LED array, and after illuminating a blank specimen, using high magnification objective lens to collect corresponding image and calculating a normalized brightness correction coefficient corresponding to each LED unit in three channels of red, green and blue; calibrating a position of the LED array, by using a resolution board as the specimen to be examined, using the LED array as the illumination source of the high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, sequentially lighting up each LED unit of the LED array, after illuminating the specimen to be examined, in-focus images and defocus images with defocus distance h are respectively collected, then calculating an illumination angle corresponding to each of the three channels of red, green and blue for each LED unit by using a sub-pixel registration algorithm, and then determining the position of the LED array by using nonlinear regression; capturing raw images, by using the LED array as the illumination source of the high-illumination numerical aperture-based large field-of-view high-resolution micro-imaging device, sequentially lighting up each LED unit of the LED array, and after illuminating the specimen to be examined, collecting corresponding low-resolution raw images; pre-processing raw images, including threshold denoising and brightness correction, by firstly using an average value of a dark current noise of the camera as a threshold, performing threshold denoising on the captured low-resolution raw images to obtain denoised low-resolution images, and then dividing each denoised low-resolution image by the normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit, thereby obtaining brightness-corrected low-resolution images; initializing high-resolution images, by adding and averaging all the low-resolution bright-field images among the brightness-corrected low-resolution images, and then initializing amplitude and phase of the high-resolution images by up-sampling; performing an iterative reconstruction, wherein each of the brightness corrected low-resolution images is subjected to synthetic aperture calculation one by one in a frequency domain by using a pixel binning based Fourier ptychographic recovery method and an updating coefficient is gradually reduced, a cost function value is used as a criterion, and when the cost function value is less than a given threshold, the iteration is stopped, and the amplitude and phase of the high-resolution images are determined to be a final reconstructed large field-of-view high-resolution microscopic images; and performing a color fusion by repeating the step of initializing the high-resolution images and the step of performing the iterative reconstruction, respectively reconstructing high-resolution images of red, green and blue channels, and then synthesizing three reconstructed high-resolution images as the red, green and blue channels respectively of a final true color image.

10. The method according to claim 9, wherein the microscope objective has a numerical aperture of NA.sub.obj and satisfies $N{A}_{\mathrm{obj}}\frac{f}{d}>1,$ wherein f is a focal length of the condenser, d is a spacing between adjacent LED units, d<2 mm, and wherein an illumination wavelength of the LED array is λ, a magnification of the microscope objective is Mag, a pixel size is Δx.sub.cam, and the illumination wavelength satisfies $\lambda \frac{Magf}{2\Delta x_{cam}d}>1.$

11. The method according to claim 9, wherein the camera is a color or monochrome camera, and when the camera is a monochrome camera, when imaging, each LED unit of the LED array emits red or green or blue monochromatic light respectively, recording all monochrome images sequentially by the monochrome camera; when the camera is a color camera, when imaging, each of the LED units of the LED array simultaneously emits red, green and blue light, recording all color images by the color camera.

12. The method according to claim 9, further comprising a microscope oil disposed between the condenser and the specimen to be examined.

13. The method according to claim 9, wherein in the step of calibrating the brightness of the LED unit, the entire LED array comprises a total of N LED units, N>261, each LED unit emits monochromatic light of red, green and blue, respectively, a total of 3N low-resolution images are captured, and the image of the blank specimen lighted by the i-th LED unit with color c is captured and marked as I.sub.i,c.sup.kb(r), where i=1,2, . . . ,N c=r,g,b, r is two-dimensional coordinates of real space r=(x,y), and then an average intensity of each image is calculated as $B_{i,c}^{kb}=\frac{1}{N_{\mathrm{pixel}}}\underset{r}{.Math.}{I}_{i,c}^{\mathrm{kb}}\left(r\right)$ and becomes the average brightness of each LED unit in three channels of red, green and blue, where N.sub.pixel is a total number of pixels in one image I.sub.i,c.sup.kb(r), an average brightness corresponding to the LED unit in a center of the LED array is B.sub.l,c.sup.kb, and then the normalized brightness correction coefficient R.sub.i,c.sup.bri of each LED unit corresponding to three channels of red, green and blue is R.sub.i,c.sup.bri=B.sub.i,c.sup.kb/B.sub.l,c.sup.kb.

14. The device according to claim 9, wherein the step of calibrating the position of the LED array further comprises the steps of: after the LED array illuminating the specimen to be examined, first respectively collecting in-focus image I.sub.i,c.sup.focus and defocus image I.sub.i,c.sup.defocus with defocus distance h; according to angular spectrum diffraction theory, numerically propagating in-focus image I.sub.l,c.sup.focus corresponding to the LED unit in a center of LED array along optical axis with numerical value h distance, thereby obtaining numerical defocus image I.sub.l,c.sup.pro; calculating an offset value (PY.sub.x,PY.sub.y) of each defocus image I.sub.i,c.sup.defocus relative to the numerical defocus image I.sub.l,c.sup.pro by using sub-pixel registration algorithm, wherein a spatial frequency vector of illumination light corresponding to the i-th LED unit is $u_i=\left(u_x,u_y\right)=\frac{2\pi }{\lambda }\left(\frac{P{Y}_x}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}},\frac{P{Y}_y}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}}\right),$ where (u.sub.x,u.sub.y) is a spatial frequency along the x,y direction, and λ is a wavelength of the illumination light; and finally determining the position of the LED array by nonlinear regression according to formulas of: $Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)=\underset{i}{.Math.}{.Math.u_i-u_i^0.Math.}^2,\left(\theta ,\Delta x,\Delta y,f\right)=\mathrm{��ℒ}\left[Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)\right],x_i=d[\cos \left(\theta \right)m_i+\sin \left(\theta \right)n_i]+\mathrm{\Delta x}$ $y_i=d\left[-\sin \left(\theta \right)m_i+\cos \left(\theta \right)n_i\right]+\Delta y^{\prime }{u}_i=\frac{2\pi }{\lambda }\frac{x_i}{f}{v}_i=\frac{2\pi }{\lambda }\frac{y_i}{f},$ where Q ( . . . ) is an objective function of a nonlinear regression method, (θ, Δx, Δy, f) are updated four position parameters of the LED array, which are respectively a rotation error, a translation error in the x direction, a translation error in the y direction, and a focusing error, (δ.sup.0, Δx.sup.0, Δy.sup.0, f.sup.0 are initialized LED array position parameters, [ . . . ] indicates that a nonlinear regression operation is performed, d is a spacing between two adjacent LED units of the LED array, (x.sub.i,y.sub.i) represents spatial position coordinates of the i-th LED unit, λ is a wavelength of the illumination light, and (m.sub.i, n.sub.i) is a row number and a column number corresponding to the i-th LED unit.

15. The method according to claim 9, wherein the step of pre-processing raw images further comprises the steps of: first sequentially lighting up the LED units and illuminating the specimen to be examined by using monochromatic light of red, green and blue, wherein the captured original low-resolution images are marked as I.sub.i,c; turning off all the LED units and the captured dark current noise image I.sub.dark; using the average value of the dark current noise of the camera as a threshold, and then performing threshold denoising on the captured original low-resolution images according to the formula of $I_{i,c}^{dn}\left(r\right)=\{\begin{array}{cc}{I}_{i,c}\left(r\right)­\mathrm{mean}\left(I_{dark}\right),& I_{i,c}\left(r\right)>\mathrm{mean}\left(I_{dark}\right)\\ 0,& I_{i,c}\left(r\right)<=\mathrm{mean}\left(I_{dark}\right)\end{array},$ where I.sub.i,c.sup.dn represents low-resolution images after the threshold denoising, and mean ( . . . ) indicates evaluating average gray value of the image; and then dividing each image by the normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit to accomplish an image brightness correction process according to the formula of:
I.sub.i,c.sup.uni=I.sub.i,c.sup.dn/R.sub.i,c.sup.bri where I.sub.i,c.sup.uni is a low-resolution image after brightness correction, I.sub.i,c.sup.dn is a low-resolution image after performing threshold denoising, R.sub.i,c.sup.bri is a normalized brightness correction coefficient obtained in the step of calibrating the brightness of the LED unit.

16. The method according to claim 9, wherein in the step of performing the iterative reconstruction, formula based on the pixel binning based Fourier ptychographic iterative method is as follows: $O_i^k=F\left\{o_i^k\right\},{\Psi }_i^k=F\left\{\mathrm{UP}\left[\sqrt{\frac{I_{i,c}^{\mathrm{uni}}}{\mathrm{DOWN}\left[{.Math.F^{-1}\left\{P_i^k{O}_i^k\right\}.Math.}^2\right]}}\right]F^{-1}\left\{P_i^k{O}_i^k\right\}\right\},O_{i+1}^k=O_i^k-{\alpha }^k\frac{.Math.P_i^k.Math.}{{.Math.P_i^k.Math.}_{\max }\left({.Math.P_i^k.Math.}^2+\gamma \right)}{P}_i^{k^{*}}\left({\Psi }_i^k-P_i^k{O}_i^k\right),P_{i+1}^k=P_i^k-{\beta }^k\frac{.Math.O_i^k.Math.}{{.Math.O_i^k.Math.}_{\max }\left({.Math.O_i^k.Math.}^2+\gamma \right)}{O}_i^{k^{*}}\left({\Psi }_i^k-P_i^k{O}_i^k\right),{\alpha }^k=\frac{{\alpha }^1}{k},{\alpha }^1=\frac{1}{2},{\beta }^k=\frac{{\beta }^1}{k},{\beta }^1=\frac{1}{N},\mathrm{and}$ ${\mathrm{COST}}^k=\underset{i}{.Math.}\mathrm{abs}\left(I_{i,c}^{\mathrm{uni}}-\mathrm{DOWN}\left[{.Math.F^{-1}\left\{P_i^k{O}_i^k\right\}.Math.}^2\right]\right),$ where F { . . . } indicates that a Fourier transform is performed, F.sup.−1{ . . . } indicates that an inverse Fourier transform is performed, UP[ . . . ] indicates that the up-sampling nearest-neighbor interpolation is performed, DOWN[ . . . ] indicates a down-sampling pixel binning process; O.sub.i.sup.k is a high-resolution spectrum of the specimen to be examined, k represents the k-th iteration, P.sub.i.sup.k is the spectrum aperture function of the microscopic objective, and Ψ.sub.i.sup.k is an updated local spectrum of the specimen to be examined, γ is a constant to ensure that a denominator is not zero, and a typical value is 0.001; | . . . | represents a modulus of a two-dimensional complex matrix, | . . . |.sub.max represents a maximum value in the modulus of the two-dimensional complex matrix; α.sup.k is an updated coefficient of the spectrum of the specimen to be examined in the k-th iteration, β.sup.k is an updated coefficient of an aperture function of the microscopic objective in the k-th iteration, COST.sup.k is the cost function, and wherein when the cost function COST.sup.k is less than a certain fixed threshold ε at an end of the k-th iteration, the iteration is considered to be convergent and the iteration is stopped, the amplitude and the phase of the high-resolution image at the moment are determined to be the finally reconstructed large field-of-view high-resolution microscopic image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is the schematic diagram of a microscope optical path based on a programmable LED array.

(2) FIG. 2 is the normalized brightness of each LED unit in the LED array, including FIG. 2(a), FIG. 2(b), and FIG. 2(c), wherein FIG. 2(a), FIG. 2(b), and FIG. 2(c) are normalized brightness corresponding to three channels of red, green and blue respectively.

(3) FIG. 3 is the schematic diagram of a reconstruction process of the high numerical aperture-based large field-of-view high-resolution iterative reconstruction of the present invention.

(4) FIG. 4 (a) is a low-resolution image taken by a 10× objective lens (0.4 numerical aperture) while simultaneously lighting up all bright field LED units in the LED array to illuminate the specimen of the 1951 USAF resolution test board; The images in the square frame of FIG. 4 are selected to be directly zoom in to obtain FIG. 4 (b1), FIG. 4 (b2) and FIG. 4 (b3), wherein FIG. 4 (b1), FIG. 4 (b2) and FIG. 4 (b3) are the image captured in three channels of red, green and blue respectively and then selecting smaller areas to be directly amplified, and FIG. 4 (c1), FIG. 4 (c2), FIG. 4 (C 3) are obtained, wherein FIG. 4 (c1), FIG. 4 (c2) and FIG. 4 (C3) are the images of three channels of red, green and blue respectively. The region in FIG. 4(c) is selected for iterative reconstruction, and the obtained high-resolution results are shown in FIG. 4 (d1), FIG. 4 (d2), and FIG. 4 (d3), wherein FIG. 4 (d1) and FIG. 4 (d2) and FIG. 4(d3) are images reconstructed in three bands of red, green, and blue respectively.

DESCRIPTION OF THE PREFERRED EMBODIMENT

(5) Referring to FIG. 1, the actual hardware platform utilized by the high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, and the method for iterative reconstruction is an LED array-based microscope. The invention of a high-illumination numerical aperture-based large field-of-view high-resolution microimaging device solves the technical problem of using a condenser to increase the resolution to 150 nm without sacrificing the imaging field-of-view, the device comprising an LED array (1), a stage (2), a condenser (3), a microscopic objective (5), a tube lens (6), and a camera (7), the LED array (1) being arranged on the forward focal plane of the condenser (3), and the center of the LED array (1) is on the optical axis of the microscope objective (5); The back focal plane of the microscope objective (5) coincides with the forward focal plane of the tube lens (6), and the imaging plane of the camera (7) is placed on the back focal plane of the tube lens (6), and during imaging, a specimen (4) to be examined on the stage (2) is adjusted to the front focal plane position of the microscope objective (5) to form an infinity-corrected imaging system.

(6) The LED array (1) includes a plurality of (at least 261) LED units (8) which are equally spaced to form a two-dimensional array. Each of the LED units is a three-color unit with the colors of red, green and blue and its typical wavelength is red light 633 nm, green light 525 nm and blue light 465 nm. The center-to-center spacing d between each LED unit is typically 1-4 mm. The LED array (1) does not need to be processed separately and is generally commercially available for purchase. Table 1 shows the product parameters of a commercially available LED array. In this LED array, the LED unit has 32 rows and 32 columns, 1024 in total, and the brightness of each LED unit is above 2000 cd/m.sup.2.

(7) Light emitted by the i-th lit LED unit (8) of the LED array (1) passes through the condenser (3) and converges to become parallel light illuminating a specimen (4) to be examined, which is placed on the stage (2); part of the diffracted light passing through the specimen (4) is collected by the microscopic objective (5), converged by the tube lens (6), and reaches the imaging plane of the camera (7), forming an intensity image recorded by the camera (7). Each LED unit in LED array (1) can be individually lighted up, and the specific method of lighting up LED units is a prior art, and the implementation circuit can be achieved by adopting (but is not limited to) a prior art such as microcontrollers, an ARM, or a programmable logic device and the like; for the specific implementation methods, the relevant references (Guo Baozeng, Deng Yumiao: FPGA-based LED display control system design [J]. LCD and Display, 2010, 25(3): 424-428) can be referred to.

(8) TABLE-US-00001 TABLE 1 Physical parameters of the LED array items parameters wavelength of LED unit red 633 nm, green 525 nm, blue 475 nm number of LED units 32 × 32 spacing of LED units 1.67 mm light emitting surface size 150 μm of LED units brightness of LED units 2000 cd/m.sup.2 array dimensions 55 mm × 55 mm × 17 mm weight 170 g cone angle of LED unit 150° power 5 v current the maximum 2A (all light up)

(9) In order to meet the minimum frequency domain sampling rate needed by the reconstruction method of the present invention, the microscope objective (5) has a numerical aperture of NA.sub.obj and satisfies

(10) ${\mathrm{NA}}_{\mathrm{obj}}\frac{f}{d}>1,$
wherein f is the focal length of the condenser (3), d is the spacing between adjacent LED units (8). In order to ensure the quality and accuracy of the reconstructed image, it is necessary to ensure that d is smaller than 2 mm, that is, an LED array with LED unit distance of less than 2 mm must be used. At the same time, in order to meet the minimum spatial sampling rate required by the reconstruction algorithm, the illumination wavelength of the LED array (1) is λ, the magnification of microscopic objective (5) is Mag, the size of pixel is Δx.sub.cam, which satisfies

(11) $\lambda \frac{\mathrm{Mag}}{2\Delta x_{\mathrm{cam}}}\frac{f}{d}>1.$

(12) The camera (7) is a color or monochrome camera, and if it is a monochrome camera, when imaging, each LED unit (8) of the LED array (1) emits red or green or blue monochromatic light respectively, recording all monochrome images sequentially by the monochrome camera; if it is a color camera, when imaging, each of the LED units (8) of the LED array (1) simultaneously emits red, green and blue light, recording all color images by the color camera.

(13) In the invention, microscopic oil can be added between the condenser (3) and the specimen (4) to be examined for obtaining a higher illumination numerical aperture. Generally, if no microscopic oil is added between the condenser (3) and the specimen (4) to be examined, the illumination numerical aperture is up to 0.95. After the addition of microscopic oil, the illumination numerical aperture can reach to more than 1.20.

(14) Referring to FIG. 3, the present invention utilizes the above-described device for a high—illumination numerical aperture—based large field-of-view high-resolution micrmaging iterative reconstruction method and the steps are as follows: step one, the brightness of the LED unit (8) is calibrated, using the LED array (1) as an illumination source of a high-illumination numerical aperture-based large field-of-view high-resolution microimaging device, sequentially lighting up each LED unit (8) of the LED array (1), and after illuminating the blank specimen (4), using the high magnification objective lens to collect the corresponding image and calculating the normalized brightness correction coefficient corresponding to each LED unit (8) in the three channels of red, green and blue. In step one, the LED array (1) serves as an illumination source of the microscope, sequentially lights up each of the LED units (8) in the LED array (1), and after illuminating the blank specimen (4), a high-magnification objective lens (a typical 40× microscopic objective with 0.95 numerical aperture) is used for collecting the corresponding images, the entire LED array (1) comprises a total of N LED units, N>261, each LED unit (8) emits monochromatic light of three colors of red, green and blue, respectively, and a total of 3N low-resolution images are captured, and the image of the blank specimen lighted by the i-th LED unit with color c is captured and marked as I.sub.i,c.sup.kb(r), where i=1, 2, . . . , N, c=r, g, b, r is two-dimensional coordinates of real space r=(x,y), and then the average intensity of each image is calculated

(15) $B_{i,c}^{kb}=\frac{1}{N_{\mathrm{pixel}}}\underset{r}{.Math.}{I}_{i,c}^{\mathrm{kb}}\left(r\right)$
and becomes the average brightness of each LED unit in three channels of red, green and blue, where N.sub.pixel is the total number of pixels in one image I.sub.i,c.sup.kb(r), and the average brightness corresponding to the LED unit (8) in the center of the LED array (1) is B.sub.l,c.sup.kb, then the normalized brightness correction coefficient R of each LED unit corresponding to three channels of red, green and blue is R.sub.i,c.sup.bri=B.sub.i,c.sup.kb/B.sub.l,c.sup.kb.

(16) Step two, the position of the LED array (1) is calibrated, using the resolution board as the specimen (4) to be examined, using the LED array (1) as the illumination source of a high illumination-numerical-aperture based large field-of-view high-resolution microscopic imaging device, sequentially lighting up each LED unit (8) of the LED array (1), after illuminating the specimen (4) to be examined, images in focus and images with defocus distance h are respectively collected, and then the illumination angle corresponding to each of the three channels of red, green and blue for each LED unit (8) is calculated by using a sub-pixel registration algorithm, then determining the position of the LED array by using nonlinear regression.

(17) In step two, using the resolution board as the specimen (4) to be examined, using the LED array (1) as the illumination source, sequentially lighting up each LED unit (8). First, after the LED array (1) illuminating the specimen (4) to be examined, in-focus image I.sub.i,c.sup.focus and defocus image I.sub.i,c.sup.defocus with defocus distance h (typical value of h ranges from 10 microns to 30 microns) are respectively collected; according to angular spectrum diffraction theory, numerically propagating in-focus image I.sub.l,c.sup.focus corresponding to LED unit (8) in the center of LED array (1) along optical axis with h distance, thereby obtaining numerical defocus image I.sub.l,c.sup.pro;

(18) and then calculating the offset value (PY.sub.x,PY.sub.y) of each defocus image I.sub.i,c.sup.defocus relative to the numerical defocus image I.sub.l,c.sup.pro by using sub-pixel registration algorithm, the spatial frequency vector of the illumination light corresponding to the i-th LED unit is

(19) $u_i=\left(u_x,u_y\right)=\frac{2\pi }{\lambda }\left(\frac{P{Y}_x}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}},\frac{{\mathrm{PY}}_y}{\sqrt{P{Y}_x^2+P{Y}_y^2+h^2}}\right)$
where (u.sub.x,u.sub.y) is the spatial frequency along the x,y direction, and λ is the wavelength of the illumination light; finally, the position of the LED array is determined by nonlinear regression, the formulas are:

(20) $Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)=\underset{i}{.Math.}|u_i-u_i^0{|}^2\left(\theta ,\Delta x,\Delta y,f\right)=\mathrm{��ℒ}\left[Q\left(u_i,{\theta }^0,\Delta x^0,\Delta y^0,f^0\right)\right]$$x_i=d\left[\cos \left(\theta \right)m_i+\sin \left(\theta \right)n_i\right]+\Delta x$$y_i=d\left[-\sin \left(\theta \right)m_i+\cos \left(\theta \right)n_i\right]+\Delta y$$u_i=\frac{2\pi }{\lambda }\frac{x_i}{f}$$v_i=\frac{2\pi }{\lambda }\frac{y_i}{f}$
where Q( . . . ) is the objective function of the nonlinear regression method, (θ, Δx, Δy, f) are the updated four position parameters of the LED array, which are respectively the rotation error, the translation error in the x direction, the translation error in the γ direction, and the focusing error, (θ.sup.0, Δx.sup.0, Δy.sup.0, f.sup.0) are the initialized LED array position parameters, [ . . . ] indicates that the nonlinear regression operation is performed, and d is the spacing between two adjacent LED units (8) of the LED array (1), (x.sub.i, y.sub.i) represents the spatial position coordinates of the i-th LED unit, λ is the wavelength of the illumination light, and (m.sub.i, n.sub.i) is the row number and the column number corresponding to the i-th LED unit.

(21) Step three, raw images are acquired, using the LED array (1) as an illumination source of a high-illumination numerical aperture-based large field-of-view high-resolution micro-imaging device, sequentially illuminating each LED unit (8) of the LED array (1), after irradiating the specimen (4) to be examined, the corresponding low-resolution raw images are collected.

(22) Step four, raw images are pre-processed, including threshold denoising and brightness correction, firstly using the average value of the dark current noise of the camera (7) as a threshold, performing threshold denoising on the captured low-resolution raw images, obtaining denoised low-resolution images (including low-resolution bright field images and low-resolution dark field images), and then dividing each denoised low-resolution image by the normalized brightness correction coefficient obtained in step one, brightness-corrected low-resolution images are obtained;

(23) In step four: first, sequentially lighting up the LED units and illuminating a specimen (4) to be examined by using the monochromatic light of red, green and blue, then the captured original low-resolution images are marked as I.sub.i,c; turning off all the LED units and the captured dark current noise image I.sub.dark; the average value of the dark current noise of the camera is used as a threshold, then performing threshold denoising on the captured original low-resolution images, the formula is

(24) $I_{i,c}^{dn}\left(r\right)=\{\begin{array}{c}{I}_{i,c}\left(r\right)-\mathrm{mean}\left(I_{\mathrm{dark}}\right),I_{i,c}\left(r\right)>\mathrm{mean}\left(I_{\mathrm{dark}}\right)\\ 0,I_{i,c}\left(r\right)<=\mathrm{mean}\left(I_{\mathrm{dark}}\right)\end{array}$
where I.sub.i,c.sup.dn represents the low-resolution images after the threshold denoising, mean ( . . . ) indicates evaluating average gray value of the image; then dividing each image by the normalized brightness correction coefficient obtained in step one to accomplish the image brightness correction process, wherein the formula is:
I.sub.i,c.sup.uni=I.sub.i,c.sup.dn/R.sub.i,c.sup.bri
where I.sub.i,c.sup.uni is the low-resolution image after brightness correction, I.sub.i,c.sup.dn is the low-resolution image after performing threshold denoising, R.sub.i,c.sup.bri is the normalized brightness correction coefficient obtained in step one.

(25) Step five, high-resolution images are initialized, adding and averaging all the low-resolution bright-field images among the brightness-corrected low-resolution images, and then initializing the amplitude and phase of high-resolution images by up-sampling. The formula for high-resolution image initialization is:

(26) $o_0^{\mathrm{ini}}=\sqrt{\mathrm{UP}\left[\frac{1}{N_b}\underset{i=1}{\overset{N_b}{.Math.}}{I}_{i,c}^{\mathrm{ini}}\right]}$
where o.sub.0.sup.ini is the initialized high-resolution complex amplitude image, UP[ . . . ] indicates the up-sampling nearest-neighbor interpolation and N.sub.b is the total number of bright-field images.

(27) Step six: iterative reconstruction, all the brightness corrected low-resolution images are subjected to synthetic aperture calculation one by one in the frequency domain using the pixel binning based Fourier ptychographic iterative reconstruction method and gradually reducing the updating coefficient; The cost function value is used as a criterion, and when the cost function value is less than a given threshold (the typical value of the threshold can be set as 0.01 and can also be adjusted empirically), the iteration is stopped, and at this moment, the amplitude and phase of the high-resolution image are the final reconstructed large field-of-view high-resolution microscopic image.

(28) In step six: the formula based on the pixel binning based Fourier ptychographic iterative method is as follows:

(29) $O_i^k=F\left\{o_i^k\right\}$${\Psi }_i^k=F\left\{\mathrm{UP}\left[\sqrt{\frac{I_{i,c}^{\mathrm{uni}}}{\mathrm{DOWN}.Math.|F^{-1}\left\{P_i^k{O}_i^k\right\}{|}^2.Math.}}\right]F^{-1}\left\{P_i^k{O}_i^k\right\}\right\}$$O_{i+1}^k=O_i^k-{\alpha }^k\frac{\left|P_i^k\right|}{|P_i^k{|}_{\max }\left(|P_i^k{|}^2+\gamma \right)}{P}_i^{k*}\left({\Psi }_i^k-P_i^k{O}_i^k\right)$$P_{i+1}^k=P_i^k-{\beta }^k\frac{\left|O_i^k\right|}{|O_i^k{|}_{\max }\left(|O_i^k{|}^2+\gamma \right)}{O}_i^{k*}\left({\Psi }_i^k-P_i^k{O}_i^k\right)$${\alpha }^k=\frac{{\alpha }^1}{k},{\alpha }^1=\frac{1}{2}$${\beta }^k=\frac{{\beta }^1}{k},{\beta }^1=\frac{1}{N}$${\mathrm{COST}}^k=\underset{i}{.Math.}\mathrm{abs}\left(I_{i,c}^{\mathrm{uni}}-\mathrm{DOWN}\left[|F^{-1}\left\{P_i^k{O}_i^k\right\}{|}^2\right]\right)$
where F{ . . . } indicates that the Fourier transform is performed, F.sup.−1{ . . . } indicates that the inverse Fourier transform is performed, UP[ . . . ] indicates that the up-sampling nearest-neighbor interpolation is performed, DOWN[ . . . ] indicates the down-sampling pixel binning process; O.sub.i.sup.k the high-resolution spectrum of the specimen to be examined, k represents the k-th iteration, P.sub.i.sup.k is the spectrum aperture function of the microscopic objective, and Ψ.sub.i.sup.k is the updated local spectrum of the specimen to be examined, γ is a constant to ensure that the denominator is not zero, the typical value is 0.001; | . . . | represents the modulus of the two-dimensional complex matrix, | . . . |.sub.max represents the maximum value in the modulus of the two-dimensional complex matrix; α.sup.k is the updated coefficient of the spectrum of the specimen to be examined in the k-th iteration, β.sup.k is the updated coefficient of the aperture function of the microscopic objective in the k-th iteration, COST.sup.k is the cost function. When the cost function COST.sup.k is less than a certain fixed threshold ε (the typical value of ε is 0.001) at the end of the k-th iteration, the iteration is considered to be convergent and the iteration is stopped, the amplitude and the phase of the high-resolution image at the moment are the finally reconstructed large field-of-view high-resolution microscopic image.

(30) The above reconstruction process is only applicable to reconstructing a monochrome image, and the present invention utilizes a high-illumination numerical aperture-based large field-of-view high-resolution microimaging iterative reconstruction method of the above device, the steps of reconstructing a true color image are as follows: step one, the brightness of the LED unit (8) is calibrated, using the LED array (1) as an illumination source of a high illumination-numerical-aperture based large field-of-view high-resolution microscopic imaging device, sequentially lighting up each LED unit (8) of the LED array (1), and after illuminating the blank specimen (4), using the high magnification objective lens to collect the corresponding image and calculating the normalized brightness correction coefficient corresponding to each LED unit (8) in the three channels of red, green and blue; step two, the position of the LED array (1) is calibrated, using the resolution board as the specimen (4) to be examined, using the LED array (1) as the illumination source of a high illumination-numerical-aperture based large field-of-view high-resolution microscopic imaging device, sequentially lighting up each LED unit (8) of the LED array (1), after illuminating the specimen (4) to be examined, in-focus images and defocus images with defocus distance h are respectively collected, and then the illumination angle corresponding to each of the three channels of red, green and blue for each LED unit (8) is calculated by using a sub-pixel registration algorithm, then determining the position of the LED array by using nonlinear regression; step three, raw images are captured, using the LED array (1) as an illumination source of a high illumination-numerical-aperture based large field-of-view high-resolution microscopic imaging device, sequentially lighting up each LED unit (8) of the LED array (1), after illuminating the specimen (4) to be examined, the corresponding low-resolution raw images are collected; step four: raw images are pre-processed, including threshold denoising and brightness correction, firstly using the average value of the dark current noise of the camera (7) as a threshold, performing threshold denoising on the captured low-resolution raw images to obtain denoised low-resolution images, and then dividing each denoised low-resolution image by the normalized brightness correction coefficient obtained in step one, then brightness-corrected low-resolution images are obtained; step five, high-resolution images are initialized, adding and averaging all the low-resolution bright-field images among the brightness-corrected low-resolution images, and then initializing the amplitude and phase of the high-resolution images by up-sampling; step six, iterative reconstruction; each of the brightness corrected low-resolution images is subjected to synthetic aperture calculation one by one in the frequency domain by using pixel binning based Fourier ptychographic recovery method and the updating coefficient is gradually reduced; the cost function value is used as a criterion, and when the cost function value is less than a given threshold, the iteration is stopped. and at this moment, the amplitude and phase of the high-resolution images are the final reconstructed large field-of-view high-resolution microscopic images; step seven, color fusion; repeating step five and step six, and respectively reconstructing high-resolution images of red, green and blue channels, then three reconstructed high-resolution images are synthesized as the red, green and blue channels respectively of the final true color image.

(31) In order to test the iterative reconstruction method of large field-of-view high-resolution microimaging, the present invention selects the 1951 USAF resolution test board for imaging test. In the experiment, the used LED array comprises 261 LED units, and these 261 LED units are used to generate 261 illumination light with different angles. The distance between LED units is 1.67 mm, the center wavelength of the emitted red light is 632.8 nm, and the spectral bandwidth is about 20 nm. The microscope objective used in the system has a numerical aperture of 0.4 and a magnification of 10×. At the same time, the low-resolution image captured when simultaneously lighting up all brightfield LED units in the LED array to illuminate the 1951 USAF resolution test board specimen is shown in FIG. 4(a). The images in the square frame of FIG. 4 are selected to be directly zoom in to obtain FIG. 4 (b1), FIG. 4 (b2) and FIG. 4 (b3), wherein FIG. 4 (b1), FIG. 4 (b2) and FIG. 4 (b3) are respectively the images captured in three channels of red, green and blue, then selecting a smaller area to directly amplify, and FIG. 4 (c1), FIG. 4 (c2), FIG. 4 (C3) are obtained, wherein FIG. 4 (c1), FIG. 4 (c2) and FIG. 4 (C3) are respectively the images of three channels of red, green and blue. As shown in the figures, the minimum distinguishable feature is the fifth elements in the ninth group, and according to the physical parameters of the 1951 USAF resolution test board (see Table 2), the original imaging resolution of the imaging system is about 0.62 um. This is in good agreement with the results of the Rayleigh diffraction limit formula of the imaging system. The high-resolution image reconstructed by the method of the present invention is shown in FIG. 4(d), wherein FIG. 4(d1), FIG. 4(d2), and FIG. 4(d3) are respectively reconstructed images of three channels of red, green and blue. As shown in the figure, the smallest distinguishable feature in the resolution board is the fifth elements in the eleventh group. As can be seen from Table 2, the synthesized resolution of the imaging system is superior to 0.154 um. Comparing FIG. 4(d) and FIG. 4(c), it can be clearly seen that the method of the invention can realize large field-of-view high-resolution imaging using low numerical aperture objective and the reconstructed image has a good signal to noise ratio.

(32) TABLE-US-00002 TABLE 2 Physical parameters of the 1951 USAF resolution test board group id lp/mm Elements −2 −1 0 1 2 3 4 5 6 7 8 9 10 11 1 0.250 0.500 1.00 2.00 4.00 8.00 16.0 32.0 64.0 128 256 512 1024 2048 2 0.280 0.561 1.12 2.24 4.49 8.98 18.0 36.0 71.8 144 287 575 1149 2299 3 0.315 0.630 1.26 2.52 5.04 10.1 20.2 40.3 80.6 261 323 645 1290 2580 4 0.353 0.707 1.41 2.83 5.66 11.3 22.6 45.3 90.5 181 362 724 1448 2896 5 0.397 0.793 1.59 3.17 6.35 12.7 25.4 50.8 102 203 406 813 1625 3251 6 0.445 0.891 1.78 3.56 7.13 14.3 28.5 57.0 114 228 456 912 1825 3649