SNAPSHOT HYPERSPECTRAL IMAGING METHOD WITH DE-BLURRING DISPERSED IMAGES

Abstract

A snapshot hyperspectral imaging method includes the steps of: S1, selecting a set of reference wavelengths for calibration, rectifying the shifted positions due to dispersion at each reference wavelength, and selecting a center wavelength; S2, estimating relative dispersion at each reconstructed wavelength with respect to the center wavelength; S3, generating a dispersion matrix describing the direction of dispersion, and generating a spectral response matrix using a spectral response curve of a sensor; S4, capturing images blurred with dispersion; S5, deblurring the dispersed images captured in S4 using the dispersion matrix and the spectral response matrix generated in S3 to obtain spectral data spatially aligned in all spectrums; and S6, projecting the aligned spectral data obtained in S5 into color space, extracting a foreground image by a threshold method, sampling the dispersed images obtained in S4 as strong prior constraints for the foreground image, and reconstructing accurate spatial hyperspectral data.

Claims

1. A snapshot hyperspectral imaging method with deblurring dispersed images using a device capable of dispersing incident light and a sensor configured to capture images dispersed by the device capable of dispersing incident light, wherein the method comprises following steps of: S1, selecting a set of reference wavelengths for calibration, rectifying the shifted positions due to dispersion at each reference wavelength, and selecting a center wavelength; S2, estimating relative dispersion at each reconstructed wavelength with respect to the center wavelength; S3, generating a dispersion matrix describing the direction of dispersion based on the dispersion results estimated in the S2, and generating a spectral response matrix using a spectral response curve of the sensor; S4, capturing images blurred with dispersion; S5, deblurring the dispersed images captured in the S4 using the dispersion matrix and the spectral response matrix generated in the S3 to obtain spectral data spatially aligned in all spectrum; and S6, projecting the aligned spectral data obtained in the S5 into color space, extracting a foreground image by a threshold method, sampling the dispersed images obtained in the S4 as strong prior constraints for the foreground image, and reconstructing accurate spatial hyperspectral data.

2. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein rectifying the shifted positions due to dispersion at each reference wavelength in the S1 comprises following steps of: when rectifying the shifted positions due to dispersion at a certain wavelength, putting a filter of that wavelength in front of an optical source so that only light beams of that wavelength are allowed to pass through the filter, and then marking the position of a reference object in image.

3. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein estimating relative dispersion at each reconstructed wavelength with respect to the center wavelength in the S2 comprises following steps of: acquiring the relative dispersion at each reference wavelength with respect to the center wavelength based on the measured shifted positions due to dispersion at all reference wavelengths and the selected center wavelength, and interpolating to obtain the relative dispersion at all other reconstructed wavelengths with respect to the center wavelength.

4. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein generating a dispersion matrix describing the direction of dispersion in the S3 comprises following steps of: setting the spatial hyperspectral data as i with dimensions of xyΛ×1, where x and y represent transverse and longitudinal dimensions of a 2D image, and Λ is the number of spectral channels, setting the dispersion matrix as Ω with dimensions of xyΛ×xyΛ to obtain a dispersed spectral matrix S=Ωi, where there is no translation in the center wavelength, and the relative translation in other wavelength is consistent with the data obtained in the S2, and constructing a dispersion matrix Ω based on the dispersed spectral matrix S and the spatial hyperspectral data i.

5. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein generating a spectral response matrix in the S3 comprises following steps of: setting the spectral response matrix as Φ with dimensions of xyN×xyΛ, wherein x and y represent transverse and longitudinal dimensions of a 2D image, Λ is the number of spectral channels, and N is 3 or 1, projecting the spatial hyperspectral data i into corresponding color space to obtain R=Ωi with dimensions of xyN×1, querying the response curve ϕ=N×Λ of the sensor and sampling to obtain response values of the sensor at the reconstructed spectral wavelengths, and constructing a spectral response matrix Φ based on i, R and ϕ.

6. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein the S5 specifically comprises a step of: solving $i_{\mathrm{aligned}}=\underset{i}{\arg \min }{.Math.\mathrm{\Omega \Phi }i-j.Math.}_2^2+{\alpha }_1{.Math.{\nabla }_{\mathrm{xy}}i.Math.}_1+{\beta }_1{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i.Math.}_1,$ where i.sub.aligned represents the obtained spectral data aligned in all spectrum, Ω is the dispersion matrix describing the direction of dispersion, Φ is the spectral response matrix, i represents the spatial hyperspectral data, j represents dispersed images actually captured, ∇.sub.xy is the spatial gradient, ∇.sub.λ is the gradient in the spectral dimension, and α.sub.1 and β.sub.1 are the coefficients of constraint terms, respectively.

7. The snapshot hyperspectral imaging method with deblurring dispersed images according to claim 1, wherein the S6 specifically comprises following steps of: obtaining deblurred image Φi.sub.aligned, and extracting a foreground image i.sub.front which is a binary matrix with dimensions of x×y using a proper threshold, where x and y represent transverse and longitudinal dimensions of a 2D image; sampling the blurred images captured in the S4 respectively in the dispersion direction with each pixel as a center, and taking each sampled vector as a prior of a spectral value at the corresponding pixel in each spectrum to finally obtain a spectral prior i.sub.prior with dimensions of x×y×Λ of all pixels; and solving $i_{\mathrm{recons}}=\underset{i}{\arg \min }{.Math.\mathrm{\Omega \Phi }i-j.Math.}_2^2+\gamma {.Math.W.Math.\left(i-i_{\mathrm{prior}}\right).Math.}_2^2+{\alpha }_2{.Math.{\nabla }_{\mathrm{xy}}i.Math.}_1+{\beta }_2{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i.Math.}_1,$ where i.sub.recons represents the final reconstructed hyperspectral data, W is a weight matrix with dimensions of x×y×Λ, which means that the foreground images in all spectrums are constrained, consisting of i.sub.front in each spectrum, and γ, α.sub.2 and β.sub.2 are adjustment coefficients for each term.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 is a structural diagram of a device adopted in the present invention, in which 1—sample, 2—device capable of dispersing incident light, and 3—sensor;

[0018] FIG. 2 is a flowchart of a method according to the present invention;

[0019] FIG. 3 shows hyperspectral results reconstructed with simulation data by the method according to the present invention, in which (a) an original image; (b) a simulated and dispersed image; (c) a reconstructed image; (d) comparison between an original spectral curve and a reconstructed spectral curve of a first pixel randomly selected, and (e) comparison between an original spectral curve and a reconstructed spectral curve of a second pixel randomly selected; and

[0020] FIG. 4 shows a dispersed image captured by the device according to the present invention and an image reconstructed by the method according to the present invention, in which (a) a dispersed image captured; (b) a reconstructed image; and (c) comparison between a reconstructed spectral curve of pixels in 22 channels and an original spectral curve, where the X-axis represents wavelengths in nm, and the Y-axis represents normalized pixel values.

DETAILED DESCRIPTION OF THE INVENTION

[0021] As shown in FIG. 1, an imaging device used in this embodiment includes a device 2 capable of dispersing incident light and a sensor 3. A sample 1 is put in front of the device 2 capable of dispersing incident light, and the sensor 3 is put behind the device 2 and captures images dispersed by the device of the sample. The device 2 capable of dispersing incident light includes, without limitation, meta-lens and a prism.

[0022] With reference to FIG. 2, a hyperspectral imaging method based on dispersion according to the present invention is provided, including following steps of acquiring aligned spatial spectral data using a proposed dispersion model, i.e., j=ΩΦi, where j represents dispersed images captured by the sensor, Ω is a spectral response matrix, Φ is a spatial dispersion matrix, and i represents spatial hyperspectral data, as well as a gradient sparsity constraint and constraints aligned with data of all channels, to deblur images after spatial dispersion, and then reconstructing accurate spectral data using the pixels with sampled strong prior constraints. The method includes following specific steps S1 to S6.

[0023] In the S1, reference wavelengths are selected, assuming that an optical source emits white beams with wavelengths of 450 nm to 650 nm, the reference wavelengths may be 450 nm, 500 nm, 550 nm, 600 nm and 650 nm, and the dispersion is rectified at these reference wavelengths respectively. To rectify the dispersion at a certain wavelength, a filter of that wavelength is put in front of the optical source so that only light beams of that wavelength are allowed to pass through the filter, and then position of a reference object in image is marked to obtain coordinates located approximately in a straight line of the reference object at all wavelengths. A center wavelength of 550 nm may be selected.

[0024] In the S2, relative dispersion at each reconstructed wavelengths with respect to the center wavelength is estimated based on the data obtained in the S1. When the center wavelength of 550 nm is selected, relative displacements at other reference wavelengths to an imaging center at 550 nm can be obtained, such as +13 pixels at the wavelength of 650 nm, +6 pixels at the wavelength of 600 nm, −4 pixels at the wavelength of 500 nm and −8 pixels w at the wavelength of 450 nm, and the relative dispersion at other reconstructed wavelengths with respect to the center wavelength can be obtained by interpolation, e.g., the relative displacement of 6 pixels is obtained at the wavelength of 550 nm to 600 nm, so the wavelengths represented by five median pixel displacements can be rectified, namely 550 nm, 550+50/6 nm, 550+50/6*2 nm, 550+50/6*3 nm, 550+50/6*4 nm, 550+50/6*5 nm and 550+50/6*6=600 nm. Of course, the more wavelengths are rectified, the more wavelength channels are reconstructed, and the more difficult it is to deblur images.

[0025] In the S3, a dispersion matrix is generated based on the data obtained in the S2: the spatial hyperspectral data are set as i with dimensions of xyΛ×1, x and y represent dimensions of a 2D image, Λ is the number of spectral channels, and the original data are x×y×Λ, which are reconstructed into column vectors with columns first followed by wavelength channels. The dispersion matrix is set as Ω with dimensions of xyΛ×xyΛ, and a spectral vector S=Ωi after dispersion is obtained by computing. There is no translation in a center wavelength channel, and the relative translation in other wavelength channels is consistent with the data obtained in the S2.

[0026] Taking one-dimensional data as an example (assuming an image size is 1×5), if the values of the center wavelength channel are a.sub.0, b.sub.0, c.sub.0, d.sub.0 and e.sub.0, the values of the channels with a relative displacement of −1 pixel are b.sub.−1, c.sub.−1, d.sub.−1, e.sub.−1 and 0, and the values of the channels with a relative displacement of +1 pixel are 0, a.sub.1, b.sub.1, c.sub.1, d.sub.1 and so on to obtain the spectral matrix after dispersion, which are reconstructed into a column vector with columns first followed by wavelength channels to obtain S. Ω can be constructed according to the relationship between S and i. The method of generating a response matrix includes following steps of: setting the spectral response matrix as Φ with dimensions of xyN×xyΛ, where N is 3 or 1 (depending on a color sensor or grayscale sensor), projecting the spatial hyperspectral data into corresponding color space to obtain R=Φi with dimensions of xyN×1, querying a response curve ϕ=N×Λ of the sensor, and constructing a response matrix Φ based on i, R and ϕ.

[0027] In the S4, an optical source the same as that in the S1 is adopted to capture images. With the device 2 capable of dispersing incident light, the images are dispersed in plane and blurred.

[0028] In the S5, the dispersed images captured in the S4 are de-blurred using the dispersion matrix and the spectral response matrix generated in the S3 to obtain spectral data aligned with images in all channels.

[00003]$i_{\mathrm{aligned}}=\underset{i}{\arg \min }{.Math.\mathrm{\Omega \Phi }i-j.Math.}_2^2+{\alpha }_1{.Math.{\nabla }_{\mathrm{xy}}i.Math.}_1+{\beta }_1{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i.Math.}_1$

is solved, where i.sub.aligned represents the obtained spectral data aligned in all spectrums, j represents the dispersed images actually captured, ∇.sub.xy is the spatial gradient, ∇.sub.A is the gradient in the spectral dimension, and α.sub.1 and β.sub.1 are the coefficient of constraint terms, respectively. The gradient computation can be expressed by matrix computation. The first item of the equation is a data term, which reduces the mean square error between results obtained by the model and the actual data. The last two terms are prior terms of which factors need to be adjusted as the case may be, where the second term is a variable differential operator commonly used, which reduces the spatial gradient artifact, and the third term ensures the alignment in all spectrums. Optimization can be realized by ADMM algorithm, which can be divided into three sub-problems:

ƒ(i)=∥ΩΦi−j∥.sub.2.sup.2,g(z.sub.1)=α.sub.1∥z.sub.1∥.sub.1, and h(z.sub.2)=β.sub.1∥z.sub.2∥.sub.1.

[0029] The objective optimization is transformed into:

[00004]$\underset{i,z_1,z_2}{\min }f\left(i\right)+ℊ\left(z_1\right)+h\left(z_2\right)$$\mathrm{subject}\mathrm{to}{\nabla }_{\mathrm{xy}}i-z_1=0,{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i-z_2=0$

[0030] The variables are iteratively optimized by the ADMM algorithm, namely:

[00005]$i^{k+1}=\underset{i}{\arg \min }f\left(i\right)+\frac{{\rho }_1}{2}{.Math.{\nabla }_{\mathrm{xy}}i-z_1^k+u_1^k.Math.}_2^2+\frac{{\rho }^2}{2}{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i-z_2^k+u_2^k.Math.}_2^2$$z_1^{k+1}=\underset{z_1}{\arg \min }ℊ\left(z_1\right)+\frac{{\rho }_1}{2}{.Math.{\nabla }_{\mathrm{xy}}{i}^{k+1}-z_1+u_1^k.Math.}_2^2$$z_2^{k+1}=\underset{z_2}{\arg \min }h\left(z_2\right)+\frac{{\rho }_2}{2}{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}{i}^{k+1}-z_2+u_2^k.Math.}_2^2$$u_1^{k+1}=u_1^k+{\nabla }_{\mathrm{xy}}{i}^{k+1}-z_1^{k+1}$$u_2^{k+1}=u_2^k+{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}{i}^{k+1}-z_2^{k+1}$

where u.sub.1 and u.sub.2 are Lagrange multipliers, only the term l.sub.2 of i.sup.k+1 is solved by a conjugate gradient method, and the introduced variables z.sub.1 and z.sub.2 are solved by a soft threshold operator, as follows:

[00006]$z_1^{k+1}=\mathrm{ST}\left(\frac{{\alpha }_1}{{\rho }_1},u_1^k+{\nabla }_{\mathrm{xy}}{i}^{k+1}\right)$$z_2^{k+1}=\mathrm{ST}\left(\frac{{\beta }_1}{{\rho }_2},u_2^k+{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}{i}^{k+1}\right)$$\mathrm{ST}\left(\theta ,X\right)=\{\begin{array}{cc}X-\theta ,& X>\theta \\ 0,& .Math.X.Math.\le \theta \\ X+\theta ,& X<-\theta \end{array}$

[0031] The Lagrange multipliers u.sub.1 and u.sub.2 are updated by a gradient ascent method.

[0032] In the S6, the aligned spectral data obtained in the S5 are projected into color space (RGB space or grayscale space depending on a color sensor or grayscale sensor), i.e., obtaining deblurred image Φi.sub.aligned, and extracting a foreground image i.sub.front which is a binary matrix with dimensions of x×y using a proper threshold. The blurred images after dispersion captured in the S4 are sampled respectively in the dispersion direction with each pixel as a center. Because each image sampled can be considered as a result of mixing fewer channels, it can be used as a prior of the spectral value at the corresponding pixel in each channel to finally obtain a spectral prior i.sub.prior with dimensions of x×y×Λ of all pixels. Finally,

[00007]$i_{\mathrm{recons}}=\underset{i}{\arg \min }{.Math.\mathrm{\Omega \Phi }i-j.Math.}_2^2+\gamma {.Math.W.Math.\left(i-i_{\mathrm{prior}}\right).Math.}_2^2+{\alpha }_2{.Math.{\nabla }_{\mathrm{xy}}i.Math.}_1+{\beta }_2{.Math.{\nabla }_{\lambda }{\nabla }_{\mathrm{xy}}i.Math.}_1$

is solved, where i.sub.recons represents the final hyperspectral data, W is a weight matrix with dimensions of x×y×Λ, which means that the foreground images in all spectrums are constrained, consisting of i.sub.front in each spectrum, and γ, α.sub.2 and β.sub.2 are adjustment coefficients for each term. In optimization, the first two terms can be considered as data terms, γ represents the credibility of strong priors, which may be a large value. The third and fourth terms are the same as above, and the value of β is generally 1e.sup.−3 to 1e.sup.−1, while the value of α is about 1e.sup.−5, which needs to be adjusted as the case may be.