HIGH-RESOLUTION HYPERSPECTRAL COMPUTATIONAL IMAGING METHOD AND SYSTEM AND MEDIUM

Abstract

The present invention discloses a high-resolution hyperspectral computational imaging method and system and a medium. The method of the present invention comprises: conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X.sub.0; and inputting the initial hyperspectral image X.sub.0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X. The present invention can effectively achieve reconstruction of the RGB image to the high-resolution hyperspectral image and has the advantages of high reconstruction precision, high computational efficiency, little memory consumption and strong generalization ability.

Claims

1. A high-resolution hyperspectral computational imaging method, comprising: 1) conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X.sub.0; and 2) inputting the initial hyperspectral image X.sub.0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X, wherein in the step 2), the deep convolutional neural network guided by an imaging model is formed by a plurality of modules of a same structure; the plurality of modules are in cascade connection; input of each module comprises the initial hyperspectral image X.sub.0 and an upsampling result of a previous module or the initial hyperspectral image X.sub.0; and the hyperspectral image X is obtained from output of the last module.

2. The high-resolution hyperspectral computational imaging method according to claim 1, wherein in the step 1), a function expression of the initial hyperspectral image X.sub.0 obtained by spectral upsampling on the input RGB image Y is:
X.sub.0=R.sup.†*Y  (1) in the above expression, R.sup.† is a generalized inverse of a spectral response function R.

3. The high-resolution hyperspectral computational imaging method according to claim 1, wherein each module is formed by a hyperspectral prior learning (HPL) module and an imaging model guidance (IMG) module; each hyperspectral prior learning (HPL) module is used for learning prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0; each imaging model guidance (IMG) module is used for optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model.

4. The high-resolution hyperspectral computational imaging method according to claim 3, wherein each hyperspectral prior learning (HPL) module is of a five-layer structure formed by sequentially connecting a 3×3 first convolutional layer, a nonlinear rectified linear unit, a channel attention mechanism, a 3×3 second convolutional layer and a spatial attention mechanism; each channel attention mechanism comprises a 1×1 convolutional operation, a nonlinear normalization unit, a linear arithmetic operation and a plurality of rectified linear units; and the 1×1 convolutional operation, the nonlinear normalization unit, the linear arithmetic operation and the plurality of rectified linear units are sequentially connected.

5. The high-resolution hyperspectral computational imaging method according to claim 3, wherein in the step 2), the step of conducting iteration computation to obtain the hyperspectral image X comprises: 2.1) initializing a number of iterations to be 1 and values of parameters in the deep convolutional neural network guided by the imaging model and a penalty factor μ.sub.t of a t.sup.th iteration; 2.2) learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0 by the hyperspectral prior learning (HPL) module in the t.sup.th module firstly, optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model by the imaging model guidance (IMG) module, and updating the hyperspectral image X obtained by the t*iteration; 2.3) judging whether the condition that the number t of iterations is equal to a preset threshold T is satisfied or not; if the condition is satisfied, outputting the hyperspectral image X obtained by the t.sup.th iteration as a final result; or otherwise, adding 1 to the number t of iterations, and jumping to execute the step 2.2) for continuous iteration.

6. The high-resolution hyperspectral computational imaging method according to claim 3, wherein the hyperspectral prior learning (HPL) module learns the prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0, which means that: a variable G is introduced and is updated by executing the expressions (3)-(4) to learn the upsampling result of the previous module or the initial hyperspectral image X.sub.0,
G.sub.t+1=argmin.sub.Gλø(G)+μ.sub.t∥G−X.sub.t∥.sub.F.sup.2  (2)
X.sub.t+1=argmin.sub.X∥Y−FX∥.sub.F.sup.2+μ.sub.t∥G.sub.t+1−X∥.sub.F.sup.2  (3)
μ.sub.t+1=γμ.sub.t  (4) in the expressions, G.sub.t+1 is a value of the introduced variable in t+1.sup.th iteration; G is the introduced variable; λ is a weighting parameter; ø (G) is a regularization term of the introduced variable G; μ.sub.t is a penalty factor of t.sup.th iteration; μ.sub.t+1 is a penalty factor of t+1.sup.th iteration; X.sub.t is a hyperspectral image obtained by the t.sup.th iteration; X.sub.t+1 is a hyperspectral image obtained by the t+1.sup.th iteration; Y represents an RGB image; F is a spectral response function; X represents a hyperspectral image; and γ is an update coefficient of the penalty factor.

7. The high-resolution hyperspectral computational imaging method according to claim 6, wherein a function expression that each hyperspectral prior learning (HPL) module is used for optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model is:
X.sub.t+1=(F.sup.TF+μ.sub.tI).sup.−1(F.sup.TY+μ.sub.tG.sub.t+1)  (5) in the expression, X.sub.t+1 is a result obtained by optimizing the learned features according to the imaging model; F is the spectral response function; I is a unit matrix, Y is the input RGB image; μ.sub.t is a penalty factor in the t.sup.th iteration; and G.sub.t+1 is a value of the introduced variable in the t+1.sup.th iteration.

8. A high-resolution hyperspectral computational imaging system, comprising a microprocessor and a memory which are connected to each other, wherein the microprocessor is programmed or configured to execute the steps of the high-resolution hyperspectral computational imaging method of claim 1; or the microprocessor is programmed or configured to execute the steps of the high-resolution hyperspectral computational imaging method of claim 1 with a neural network acceleration processor together.

9. A computer readable storage medium, wherein computer programs programmed or configured to execute the high-resolution hyperspectral computational imaging method of claim 1 is stored in the computer readable storage medium.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0028] FIG. 1 is a basic flow diagram of a method according to an embodiment of the present invention.

[0029] FIG. 2 is a principle diagram of a hyperspectral prior learning module proposed according to an embodiment of the present invention.

[0030] FIG. 3 is a result comparison diagram of a method according to an embodiment of the present invention and four imaging methods on a Harvard hyperspectral image.

[0031] FIG. 4 is a result comparison diagram of a method according to an embodiment of the present invention and four imaging methods on a CAVE hyperspectral image.

[0032] FIG. 5 is an objective performance index comparison diagram of a method according to an embodiment of the present invention and four imaging methods on a CAVE data set.

[0033] FIG. 6 is an objective performance index comparison diagram of a method according to an embodiment of the present invention and four imaging methods on a Harvard data set.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] As shown in FIG. 1, a high-resolution hyperspectral computational imaging method of this embodiment comprises:

[0035] 1) conducting spectral upsampling on an input RGB image Y to obtain an initial hyperspectral image X.sub.0; and

[0036] 2) inputting the initial hyperspectral image X.sub.0 into a pre-trained deep convolutional neural network guided by an imaging model, and conducting iteration computation to obtain a hyperspectral image X.

[0037] In this embodiment, generalized inverse upsampling is conducted on the input RGB image Y with a spectral response function to obtain the initial hyperspectral image X.sub.0; and a function expression of the initial hyperspectral image X.sub.0 obtained by generalized inverse upsampling on the input RGB image Y with the spectral response function is as follows:

X.sub.0=R.sup.†*Y  (1)

[0038] In the above expression, R.sup.† is a generalized inverse of the spectral response function R.

[0039] As shown in FIG. 1, in the step 2), the deep convolutional neural network guided by the imaging model is formed by a plurality of modules of a same structure; the plurality of modules are in cascade connection; input of each module comprises the initial hyperspectral image X.sub.0 and an upsampling result of a previous module or the initial hyperspectral image X.sub.0; and the hyperspectral image X is obtained from output of the last module. The deep convolutional neural network has the advantage of light model. Each module is formed by a hyperspectral prior learning (HPL) module and an imaging model guidance (IMG) module; each hyperspectral prior learning (HPL) module is used for learning prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0; each imaging model guidance (IMG) module is used for optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model.

[0040] Each hyperspectral prior learning (HPL) module is used for learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0. As shown in FIG. 2, each hyperspectral prior learning (HPL) module in this embodiment is of a five-layer structure formed by sequentially connecting a 3×3 first convolutional layer, a nonlinear rectified linear unit, a channel attention mechanism, a 3×3 second convolutional layer and a spatial attention mechanism; each channel attention mechanism comprises a 1×1 convolutional operation, a nonlinear normalization unit, a linear arithmetic operation and a plurality of rectified linear units; and the 1×1 convolutional operation, the nonlinear normalization unit, the linear arithmetic operation and the plurality of rectified linear units are sequentially connected, wherein the third layer is the channel attention mechanism used for learning the spectral characteristics of the hyperspectral image; and the last layer is the spatial attention mechanism used for learning the spatial characteristics of the hyperspectral image.

[0041] The foregoing deep convolutional neural network guided by the imaging model is a hyperspectral imaging model established in this embodiment, quantitatively describes a relationship between the RGB image and the hyperspectral image, uses the maximum a posteriori probability principle, decomposes the hyperspectral imaging problem into two subproblems to be alternately solved and solves the two subproblems by designing a hyperspectral prior learning module and an imaging model guidance module respectively, thereby effectively reconstructing the hyperspectral image from the RGB image and lowering the obtaining cost of the hyperspectral image. In this embodiment, in the step 2), the step of conducting iteration computation to obtain the hyperspectral image X comprises:

[0042] 2.1) Initializing a number of iterations to be 1 and values of parameters in the deep convolutional neural network guided by the imaging model and a penalty factor μ.sub.t of a t.sup.th iteration.

[0043] 2.2) Learning the prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0 by the hyperspectral prior learning (HPL) module in the t.sup.th module firstly, optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model by the imaging model guidance (IMG) module, and updating the hyperspectral image X obtained by the t.sup.th iteration.

[0044] 2.3) Judging whether the condition that the number t of iterations is equal to a preset threshold T is satisfied or not; if the condition is satisfied, outputting the hyperspectral image X obtained by the t.sup.th iteration as a final result; or otherwise, adding 1 to the number t of iterations, and jumping to execute the step 2.2) for continuous iteration.

[0045] In this embodiment, the hyperspectral prior learning (HPL) module learns the prior features of the upsampling result of the previous module or the initial hyperspectral image X.sub.0, which means that: a variable G is introduced and is updated by executing the expressions (3)-(4) to learn the upsampling result of the previous module or the initial hyperspectral image X.sub.0.

G.sub.t+1=argmin.sub.Gλø(G)+μ.sub.t∥G−X.sub.t∥.sub.F.sup.2  (2)

X.sub.t+1=argmin.sub.X∥Y−FX∥.sub.F.sup.2+μ.sub.t∥G.sub.t+1X∥.sub.F.sup.2  (3)

μ.sub.t+1=γμ.sub.t  (4)

[0046] In the expressions, G.sub.t+1 is a value of the introduced variable in t+1.sup.th iteration; G is the introduced variable; λ is a weighting parameter; ø (G) is a regularization term of the introduced variable G; μ.sub.t is a penalty factor of t.sup.th iteration; X.sub.t+1 is a penalty factor of t+1.sup.th iteration; X.sub.t is a hyperspectral image obtained by the t.sup.th iteration; X.sub.t+1 is a hyperspectral image obtained by the t+1.sup.th iteration; Y represents an RGB image; F is a spectral response function; X represents a hyperspectral image; and γ is an update coefficient of the penalty factor. Updating the variable G according to the expression (2) specifically refers to considering the expression (3) as an image denoising problem from regularization of image prior information. This embodiment uses the deep convolutional neural network to solve the problem; and the module is called as a hyperspectral prior learning (HPL) module. The hyperspectral image contains rich spectra and spatial information; and therefore, we propose a channel attention mechanism to learn the spectral information of the hyperspectral image and use the spatial attention mechanism to learn the spatial information of the hyperspectral image.

[0047] In this embodiment, a function expression that each hyperspectral prior learning (HPL) module is used for optimizing the learned features based on the input initial hyperspectral image X.sub.0 and the prior features output by the hyperspectral prior learning (HPL) module according to the imaging model is as follows:

X.sub.t+1=(F.sup.TF+μ.sub.tI).sup.−1(F.sup.TY+μ.sub.tG.sub.t+1)  (5)

[0048] In the expression, X.sub.t+1 is a result obtained by optimizing the learned features according to the imaging model; F is the spectral response function; I is a unit matrix, Y is the input RGB image; μ.sub.t is a penalty factor in the t.sup.th iteration; and G.sub.t+1 is a value of the introduced variable in the t+1.sup.th iteration. Updating X.sub.t+1 as the hyperspectral image obtained by the t+1.sup.th iteration according to the expression (3) specifically refers to considering the expression (3) to have a strongly-convex problem with an analytical solution and an analytical solution shown in the expression (5). In this embodiment, the imaging model guidance (IMG) module uses a matrix multiplication to execute the forgoing expression (5) to obtain the analytical solution of the strongly-convex problem.

[0049] This embodiment further comprises the steps of establishing a plurality of subproblem solution models shown as the expressions (3)-(4) in advance:

[0050] S1) establishing a linear mapping relationship between the hyperspectral image X and the traditional RGB image Y:

[0051] (6)

[0052] In the above expression, F is a spectral response matrix.

[0053] S2) According to a Bayes formula and maximum a posteriori, the estimation problem for the hyperspectral image is converted to a basic model in a following expression:

[00001]$\begin{array}{cc}X=\arg {\max }_{x}P\left(X.Math.Y\right)& \left(7\right)\end{array}$$\begin{array}{cc}X=\arg {\min }_x-\log \left\{P\left(X.Math.Y\right)\right\}-\log \left\{P\left(X\right)\right\}& \left(8\right)\end{array}$$\begin{array}{cc}-\log \left\{P\left(Y.Math.X\right)\right\}=\frac{1}{2{\sigma }^2}{.Math.\mathrm{FX}-Y.Math.}_F^2& \left(9\right)\end{array}$$\begin{array}{cc}-\log \left\{P\left(X\right)\right\}=\mathrm{\lambda \varphi }\left(X\right)& \left(10\right)\end{array}$

[0054] In the above expressions. P(X|Y) is a possibility that X occurs under the condition of occurrence of Y; P(Y|X) is a possibility that Y occurs under the condition of occurrence of X; σ is a variance of noise; λ is a weighting parameter (larger than 0); and φ(X) is a regularization term of a to-be-estimated hyperspectral image X.

[0055] S3) Introducing the variable G=X, wherein X is the to-be-estimated hyperspectral image; and establishing an exterior penalty function L(X,G) required to be optimized;

L(X,G)=∥Y−FX∥.sub.F.sup.2+λϕ(X)+μ∥G−X∥.sub.F.sup.2  11)

[0056] In the above expression, p is the penalty factor.

[0057] S4) Transforming and decomposing the exterior penalty function L(X,G) requiring to be optimized to obtain a multi-subproblem solution model shown as the expressions (3)-(4).

[0058] In order to verify the method according to this embodiment, in this embodiment, an image with a number of wavebands of 31 and a spatial size of 512×512 in a CAVE public data set and an image with a number of wavebands of 31 and a spatial size of 1392×1040 in a Harvard public data set are used for a verification experiment. In the experiment, the images serve as the high-resolution hyperspectral images, further serve as input images with the corresponding RGB images together and are obtained by downsampling through the spectral response function. In the actual process, in the CAVE data set, 20 hyperspectral images are randomly selected as a training set, and 12 hyperspectral images are randomly selected as a test set; and in the Harvard data set, 35 hyperspectral images are randomly selected as a training set, and 15 hyperspectral images are randomly selected as a test set. Also, 4 kinds of typical hyperspectral imaging methods are compared. FIG. 3 is a result comparison diagram of the method according to this embodiment of the present invention and four typical imaging methods on the Harvard hyperspectral images. FIG. 4 is a result comparison diagram of the method according to this embodiment of the present invention and four imaging methods on the CAVE hyperspectral images, wherein 4 kinds of evaluation indexes for fusion images are provided, including spectral angle mapping (SAM), root mean square error (RMSE), unified image quality index (UIQI) and structural similarity (SSIM); and wherein the larger the values of the UIQI and the SSIM are, the better the quality of each image is; and the larger the values of the SAM and RMSE are, showing that the poorer the quality of each high-resolution image is. A table shown in FIG. 5 shows objective evaluation indexes of 4 kinds of typical fusion methods (Arad, HSCNN-R, DFMN, AWAN+) and the method (SSRNet) proposed in this embodiment for imaging experiments on the CAVE data set. A table shown in FIG. 6 shows objective evaluation indexes of 4 kinds of typical fusion methods (Arad, HSCNN-R, DFMN, AWAN+) and the method (SSRNet) proposed in this embodiment for imaging experiments on the Harvard data set. From FIG. 5 and FIG. 6, all the objective evaluation indexes in the method (SSRNet) of this embodiment are superior to those in other methods, the reason for which is that the method (SSRNet) of this embodiment uses an image guidance model unit, which better optimizes parameters of the model. More importantly, the used deep convolutional neural network can well learn image prior knowledge and save spatial details of the images.

[0059] To sum up, the method in this embodiment uses the strong learning ability of the deep convolutional neural network and the spectral-superresolution imaging model and can improve the imaging precision and efficiency at the same time. Firstly, each RGB image is subjected to upsampling with a Moore-Penrose pseudo-inverse method; upsampled images and RGB images are overlaid by using a dense fusion strategy to serve as an input, so as to guide that due to high relevance between spectral bands and low rank characteristic of a spectral dimension, a channel attention block is firstly used to obtain the relevance between spectral bands of the hyperspectral images. Then, by using prior knowledge of space similarity of the spectral images, nonlocal spatial similarity of the hyperspectral images is obtained by using the nonlocal similarity, and then the hyperspectral images are reconstructed. The mentioned above in this embodiment is called as a spectral prior learning module. Then, based on the established spectral imaging model and the deep convolutional neural network, image features learned by the spectral prior learning module is optimized; and therefore, prior information learned from the images can be fully used. Whole estimation of the hyperspectral images uses a half quadratic splitting algorithm for continuous iterations, and finally, the high-resolution hyperspectral image is obtained. By comparing with other high-performance rapid hyperspectral imaging method, the rapid hyperspectral imaging method of this embodiment may obtain the hyperspectral image with higher quality, and the method in this paper is little in memory consumption and lower in requirement for hardware.

[0060] In addition, this embodiment further provides a high-resolution hyperspectral computational imaging system, comprising a microprocessor and a memory which are connected to each other; the microprocessor is programmed or configured to execute the steps of the forgoing high-resolution hyperspectral computational imaging method; or the microprocessor is programmed or configured to execute the steps of the forgoing high-resolution hyperspectral computational imaging method with a neural network acceleration processor together.

[0061] In addition, this embodiment further provides a computer readable storage medium, and computer programs programmed or configured to execute the high-resolution hyperspectral computational imaging method are stored in the computer readable storage medium.

[0062] Those skilled in the art should understand that the embodiments of this application may be provided as a method, a system or products of computer programs. Therefore, this application may be in the form of full hardware embodiments, full software embodiments, or a combination thereof. Moreover, this application may employ the form of the products of the computer programs implemented on one or more computer readable storage media containing computer available program codes (including, but not limited to, a disk memory, CD-ROM and an optical memory). This application is described with reference to flow charts and/or block diagrams of the method, equipment (system) and the products of the computer programs according to the embodiment of this application. It should be understood that instructions of the computer programs implement each flow and/or block in the flow charts and/or block diagrams and combination thereof. These instructions of the computer programs may be provided a general-purpose computer, a specialized computer, an embedded processor or a processor of other programmable data processing equipment, so as to give rise to a machine with the result that the instructions executed through the computer or processor of other programmable data processing equipment give rise to a device that is used for realizing functions designated by one or more processes in a flow chart and/or one or more blocks in a block diagram. These instructions of the computer program may also be stored in a computer-readable memory that can direct the computer or other programmable data processing equipment to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction devices for implementing the functions designated by one or more processes in a flow chart and/or one or more blocks in a block diagram. These instructions of the computer programs may also be loaded on the computer or other programmable data processing equipment so as to carry out a series of operation steps on the computer or other programmable equipment to generate processing to be achieved by the computer, so that the instructions to be executed by the computer or other programmable equipment provide the steps for implementing the functions designated by one or more processes in a flow chart and/or one or more blocks in a block diagram.

[0063] The above disclosure is preferred embodiments of the present invention only, the protection scope of the present invention is not limited to the above described embodiments only, and all the technical solutions within the spirit of the present invention should be within the protection scope of the present invention. For ordinary skilled in the art, it should be noted that a plurality of improvements and embellishments without departing from the principle of the present invention should all be regarded as within the protection scope of the present invention.