Compressive sensing-based multispectral video imager with double channels and imaging method

Abstract

A compressive sensing-based multispectral video imager comprises a beamsplitter, a first light channel, a second light channel, and an image reconstruction processor; the beamsplitter is configured to divide the beam of an underlying image into a first light beam and a second light beam; the first light beam enters the first light channel, processed and sampled in the first light channel, to obtain a first mixing spectral image which is transferred to the image reconstruction processor; the second light beam enters the second light channel, processed and sampled in the second light channel, to obtain a second mixing spectral image which is transferred to the image reconstruction processor; the image reconstruction processor is configured to reconstruct the underlying spectral image based on the first mixing spectral image and the second mixing spectral image by using non-linear optimization method.

Claims

1. A compressive sensing-based multispectral video imager with double channels, comprising: a beamsplitter, a first light channel, a second light channel, and an image reconstruction processor, wherein: the beamsplitter is configured to divide the light beam of an underlying image into a first light beam and a second light beam; wherein the first light beam enters the first light channel, and the second light beam enters the second light channel; the first light channel is configured to process and sample the first light beam to obtain a first mixing spectral image, and transfer it to the image reconstruction processor; the second light channel is configured to process and sample the second light beam to obtain a second mixing spectral image, and transfer it to the image reconstruction processor; and the image reconstruction processor is configured to reconstruct the underlying spectral image based on the first mixing spectral image and the second mixing spectral image by making use of non-linear optimization method, wherein: the first light channel comprises a first objective lens, a first coded aperture, a first bandpass filter, a first relay lens set, a first prism set, and a first sensor array which are sequentially arranged; the second light channel comprises a second objective lens, a second coded aperture, a second bandpass filter, a second relay lens set, a second prism set, and a second sensor array which are sequentially arranged; the first light beam entering the first light channel is focused and imaged on the first objective lens, the first coded aperture performs random coding for beams on positions of a spectral image; beams of the coded spectral image are filtered by the first bandpass filter, then are relayed by the first relay lens set to the first prism set to produce dispersion; and beams of the dispersed spectral image are sensed by the first sensor array to obtain the first mixing spectral image; and the second light beam is focused and imaged on the second objective lens, the second coded aperture performs random coding for beams on positions of a spectral image; beams of the coded spectral image are filtered by the second bandpass filter, then are relayed by the second relay lens set to the second prism set to produce dispersion; and beams of the dispersed spectral image are sensed by the second sensor array to obtain the second mixing spectral image.

2. The imager of claim 1, wherein: each of the first coded aperture and the second coded aperture is a rectangle plate having transparent and opaque blocks of same size; the transparent block represents code 1 and the opaque block represents code 0; the code of each block of the first coded aperture is randomly designated; the code of each block of the second coded aperture is opposite to that of the block of same position of the first coded aperture, to realize complementary coding of image information of each position of the spectral image and ensure the completeness of spectral image information sampling.

3. A compressive sensing-based multispectral video imaging method using double channels, comprising the following steps: (S1) dividing a light beam of a underlying spectral image into a first light beam and a second light beam; wherein the first light beam and the second light beam enter a first light channel and a second light channel respectively; (S2) obtaining a first mixing spectral image (X.sub.1) by the following steps: (S2a) focusing and imaging the first light beam in the first light channel to obtain a spectral image; (S2b) randomly coding the spectral image, comprising random blocking the light beam at each position of the spectral image; wherein the blocked position presents code 0, and the non-blocked position presents code 1; (S2c) filtering the coded spectral image, to filter out a light beam of the spectral image outside the bandwidth to be reconstructed; (S2d) shifting images of spectral dimensions of the spectral image in the direction of spatial dimension, to disperse the light beam of the spectral image and to change relative positions between all images of spectral dimension; (S2e) obtaining, by a sensor array, the light quantity of light beam at each position of the shifted spectral image, to obtain the first mixing spectral image (X.sub.1); (S3) obtaining a second mixing spectral image (X.sub.2) by the following steps: (S3a) focusing and imaging the second light beam in the second light channel to obtain a spectral image; (S3b) complementarily to the spectral image in the first light channel, coding the spectral image in the second light channel, comprising correspondingly blocking the light beam at each position of the spectral image in the second light channel, so that the blocking state of each position is complementary to that of the same position of the spectral image in the first light channel; (S3c) filtering the light beam of the coded spectral image, to filter out a light beam outside the bandwidth to be reconstructed; (S3d) shifting images of spectral dimensions of the spectral image in the direction of spatial dimension, to disperse the light beam of the spectral image and to change relative positions between all images of spectral dimension; (S3e) obtaining, by a sensor array, the light quantity of light beam at each position of the shifted spectral image, to obtain the second mixing spectral image (X.sub.2); (S4) based on the first mixing spectral image (X.sub.1) and the second mixing spectral image (X.sub.2), reconstructing the underlying spectral image (X) by using non-linear optimization method; and (S5) repeatedly imaging the same one scene by repeating the steps (S1) to (S4), to obtain multiple spectral images of the scene at different times, and to form the multispectral video.

4. The imaging method of claim 3, wherein the shifting steps (S2d) and (S3d) comprise: arranging a prism set on a transmission path of the light beam of the spectral image, so that light beams of different spectral dimension images pass the prism set to produce shifts of different distances, and relative positions between the images of spectral dimension are changed.

5. The imaging method of claim 3, wherein the reconstructing step comprises the following steps: (S4a) integrating the first mixing spectral image (X.sub.1) and the second mixing spectral image (X.sub.2) on the spatial plane, to obtain a general mixing spectral image (Y):
Y=[X.sub.1,X.sub.2]=[A.sub.1X,A.sub.2X]=AX where, A.sub.1 and A.sub.2 are two linear operators, A.sub.1X and A.sub.2X represent results obtained by the first light channel and the second light channel operating on the spectral image (X), respectively; A=[A.sub.1, A.sub.2] represents linear function operators of whole observation part, and AX represents the result obtained by the whole observation part operating on the spectral images; (S4b) assuming that Ψ is an orthogonal basis of the underlying spectral image (X) in a sparse domain, and the representing coefficient Φ of the underlying spectral image (X) on the basis Ψ is sparse, i.e., Φ=Ψ.sup.TX=Ψ.sup.−1X; (S4c) assuming that min ∥Ψ.sup.TX∥.sub.0 is an objective function of optimization solution and Y=AX is a constraint condition, solving the following equation by using non-linear optimization method, to obtain the approximate value {tilde over (X)} of the underlying spectral image (X):
{tilde over (X)}=arg min∥Ψ.sup.TX∥.sub.0s.Math.t Y=AX where ∥Ψ.sup.TX∥.sub.0 represents the norm l.sub.0 of the representing coefficient Φ of the underlying spectral image X on sparsity domain, arg min represents taking minimal value; and s.Math.t Y=AX represents that the constraint condition is Y=AX.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 show the structure block diagram of the multispectral video imager according to an embodiment of the present invention;

(2) FIG. 2 shows the structure block diagram of the first light channel in the imager according to an embodiment of the present invention;

(3) FIG. 3 shows the structure block diagram of the second light channel in the imager according to an embodiment of the present invention;

(4) FIG. 4 shows the coded aperture in the imager according to an embodiment of the present invention;

(5) FIG. 5 shows the flowchart of a compressive sensing-based multispectral video imaging method according to an embodiment of the present invention;

(6) FIGS. 6(a)-6(f) show spectral dimension image planes of multispectral images of a river as originally captured which are used in the simulation scheme;

(7) FIGS. 7(a)-7(f) show multispectral images reconstructed by using single channel and corresponding PSNR values of spectral dimension images; and

(8) FIGS. 8(a)-8(f) show multispectral images reconstructed by using double channels and corresponding PSNR values of spectral dimension images.

DETAILED DESCRIPTION OF EMBODIMENTS

(9) As shown in FIG. 1, an embodiment of the present invention provides a compressive sensing-based multispectral video imager with double channels, comprising: a beamsplitter (1), a first light channel (2), a second light channel (3) and an image reconstruction processor (4). FIG. 2 and FIG. 3 show structures of the first light channel (2) and the second light channel (3) respectively. The beam of the underlying image is divided, by the beamsplitter (1), into two light beams, which contain the same spectral information and enter two light channels at different directions. The first beam and the second beam are processed and sampled in the first light channel (2) and the second light channel (3), respectively, by various parts, to obtain a first mixing spectral image (X.sub.1) and a second mixing spectral image (X.sub.2); then the two mixing spectral images are transferred to the image reconstruction processor (4). The image reconstruction processor (4) reconstructs the underlying spectral image (X) based on the first mixing spectral image (X.sub.1) and the second mixing spectral image (X.sub.2) by making use of non-linear optimization method.

(10) As shown in FIG. 2, the first light channel (2) of the embodiment comprises a first objective lens (21), a first coded aperture (22), a first bandpass filter (23), a first relay lens set (24), a first prism set (25) and a first sensor array (26) which are sequentially arranged. The first objective lens (21) is located at the front end of the first light channel (2), and is configured to focus the light beam in the first light channel (2), to form a clear spectral image. The first coded aperture (22) is located in the image plane of the first objective lens (21). As shown in FIG. 4(a), the first coded aperture (22) is a rectangle plate having transparent and opaque blocks; and sizes of the blocks are the same. The white block is transparent and represents code “1”; and the black block is opaque and represents code “0”. The coding of each block is set randomly to ensure randomly coding of the spectral image information. The first bandpass filter (23) is located at the rear end of the first coded aperture (22), and comprises a set of lenses for filtering out an unnecessary beam of spectral dimension image. The first relay lens set (24) is located behind the first bandpass filter (23), and is configured to relay the beam of spectral dimension image to the first prism set (25) which is remained by the first bandpass filter (23). The first prism set (25) is located between the first relay lens set (24) and the first sensor array (26), and reveals different refraction ratios to different spectral dimension images, so that the spectral dimension images produce shifting of different distances when reaching the first sensor array (26) through the first prism set (25), to realize the shifting of different spectral dimension images. The first sensor array (26) is located at the rear end of the first light channel (2), obtains light quantity information of the beam at each position of the spectral image, and converts the information into digital form, i.e. the first mixing spectral image (X.sub.1).

(11) As shown in FIG. 3, the second light channel (3) of the embodiment comprises a second objective lens (31), a second coded aperture (32), a second bandpass filter (33), a second relay lens set (34), a second prism set (35) and a second sensor array (36) which are sequentially arranged. The second objective lens (31) is at the front end of the second light channel (3), and is configured to focus the light beam in the second light channel (3) to form a clear spectral image. The second coded aperture (32) is located in the image plane of the second objective lens. As shown in FIG. 4(b), similarly to the first coded aperture (22), the second coded aperture (32) is a rectangle plate containing transparent and opaque blocks of same size. The white block is transparent and represents code “1”; and the black block is opaque and represents code “0”. The code of every block of the second coded aperture (32) is not set randomly, and is complementary with the code of the block at the same position on the first coded aperture (22), so that the first coded aperture (22) and the second coded aperture (32) realize complementarily coding of spectral image information and ensure observation integrality of spectral image information. The second bandpass filter (33) is located at the rear end of the second coded aperture (32), and comprises a set of lens for filtering out an unnecessary beam of spectral dimension image. The second relay lens set (34) is located behind the second bandpass filter (33) and relays the beam of spectral dimension image remained by the second bandpass filter (33) to the second prism set (35). The second prism set (35) is located between the second relay lens set (34) and the second sensor array (36), and reveals different refraction ratios to different spectral dimension images, so that the spectral dimension images produce shifting of different distances when reaching the second sensor array (36), to realize the shifting of different spectral dimension images. The second sensor array (36) is located at the rear end of the second light channel (3), obtains the light quantity information of beam at each position on the spectral image and converts the information into digital form, i.e. the second mixing spectral image (X.sub.2).

(12) In the embodiment shown in FIG. 5, a compressive sensing-based multispectral video imaging method using double channels comprises the following steps:

(13) (Step 1) dividing the light beam of the underlying spectral image into two light beams containing the same spectral image information, i.e., first beam and second beam; the two light beams enter two light channels in two different directions respectively, i.e. a first light channel and a second light channel.

(14) (Step 2) obtaining a mixing spectral image of the first light channel, which comprises the following steps:

(15) (S2a) focusing and imaging the first beam in the first light channel, to obtain an initial spectral image (X.sub.11) of the first light channel;

(16) (S2b) randomly coding information at each position of the initial spectral image (X.sub.11), comprising: random blocking the beam at each position of the spectral image (X.sub.11) by using a shading plate. The beam cannot pass the blocked position, and can pass the non-blocked position through a quadrate aperture. The blocked position is coded as “0” and the non-blocked position is coded as “1”, so as to obtain the coded spectral image of the first light channel.

(17) (S2c) filtering the coded spectral image of the first light channel, to filter out an unnecessary beam of spectral dimension image outside the reconstruction bandwidth and maintain the beam of the spectral dimension image within the bandwidth, to obtain the filtered spectral image (X.sub.13) of the first light channel.

(18) (S2d) locating a prism set on the routine of the beam of the filtered spectral image (X.sub.13) of the first light channel to disperse the beam of the filtered spectral image. The spectral dimension images shift different distances along spatial dimension, to obtain the shifted spectral image (X.sub.14) of the first light channel.

(19) (S2e) capturing, by a sensor array, light quantity information of the shifted spectral image (X.sub.14) of the first light channel; the light quantity information captured at each position of the sensor array is sum of the light information at the same position of all shifted spectral dimension images of the spectral image (X.sub.14), so as to realize the information mixing of different spectral dimension images; the summed light quantity information is converted into digital form to obtain a mixing spectral image (X.sub.1) of the first light channel;

(20) (Step 3) obtaining a mixing spectral image of the second light channel, which comprise the following steps:

(21) (S3a) focusing and imaging the second beam in the second light channel, to obtain an initial spectral image (X.sub.21) of the second light channel;

(22) (S3b) complementarily to the spectral image in the first light channel, coding the initial spectral image (X.sub.21) of the second light channel, comprising: correspondingly blocking the beam at each position of the spectral image (X.sub.21) in the second light channel, so that the blocking state of each position is contrary to that of the same position of the spectral image (X.sub.11) in the first light channel, to ensure that information at any position of the underlying spectral image (X) is not missed, thereby obtaining the complementarily coded spectral image (X.sub.22) of the second light channel.

(23) (S3c) filtering the beam of the complementarily coded spectral image (X.sub.22) of the second light channel, to filter out an unnecessary beam of the spectral dimension image which is outside the reconstruction bandwidth, and maintain the beam of the spectral dimension image within the bandwidth, thereby obtaining the filtered spectral image (X.sub.23) of the second light channel.

(24) (S3d) locating a prism set at the routine of the beam of the filtered spectral image (X.sub.23) of the second light channel, to disperse the beam of the spectral image (X.sub.23). The spectral dimension images shift different distances along spatial dimension, to obtain the shifted spectral image (X.sub.24) of the second light channel.

(25) (S3e) capturing, by a sensor array, light quantity information of the shifted spectral image (X.sub.24) of the second light channel; the light quantity information captured at each position of the sensor array is sum of the light information at the same position of all shifted spectral dimension images of the spectral image (X.sub.24), so as to realize the information mixing of different spectral dimension images; the summed light quantity information is converted into digital form to obtain a mixing spectral image (X.sub.2) of the second light channel;

(26) (Step 4) based on the first mixing spectral image (X.sub.1) and the second mixing spectral image (X.sub.2), reconstructing the underlying spectral image (X) by using non-linear optimization method. The Step 4 comprises the following steps:

(27) (S4a) connecting the input first mixing spectral image (X.sub.1) with the second mixing spectral image (X.sub.2) on the spatial plane, to obtain a general mixing spectral image Y:
Y=[X.sub.1,X.sub.2]=[A.sub.1X,A.sub.2X]=AX

(28) where, A.sub.1 and A.sub.2 are two linear operators, A.sub.1X and A.sub.2X represent results obtained by the first light channel and the second light channel operating on the spectral image (X), respectively; A=[A.sub.1, A.sub.2] represents linear function operators of whole observation part, and AX represents the result obtained by the whole observation part operating on the spectral images;

(29) (S4b) assuming that Ψ is an orthogonal basis of the underlying spectral image (X) in a sparse domain, and the representing coefficient Φ of the underlying spectral image (X) on the basis Ψ is sparse, that is, the representing coefficient Φ contains many zero elements or those less than a preset threshold:
Φ=Ψ.sup.−1X=Ψ.sup.TX

(30) where, Ψ.sup.−1 and Ψ.sup.T represent the inverse and transposition of the matrix of the orthogonal basis Ψ, respectively.

(31) (S4c) assuming min ∥Ψ.sup.TX∥.sub.0 that is an objective function of optimization solution and Y=AX is a constraint condition, and solving the following equation by using non-linear optimization method, such as basis pursuit algorithm, two-step iteration algorithm or greedy algorithm, to obtain the approximate value {tilde over (X)} of the underlying spectral image (X):
{tilde over (X)}=arg min∥Ψ.sup.TX∥.sub.0s.Math.tY=AX

(32) where ∥Ψ.sup.TX∥.sub.0 represents the norm l.sub.0 of the representing coefficient Φ of the underlying spectral image X on sparsity domain, arg min represents taking minimal value; and s.Math.t Y=AX represents that the constraint condition is Y=AX;

(33) (Step 5) repeating the above stated steps for the same one scene, i.e., performing multiple steps of multiplespectral imaging the scene at different times, to form a multispectral video.

(34) To verify the feasibility and validity of the embodiments of the present invention, an instance is simulated by using MATLAB simulation software, and the procedure of one spectral imaging is as following:

(35) Simulation 1:

(36) (a) a multispectral image of a river that is originally captured is selected as a underlying spectral image, which contain 6 spectral bands; and each spectral band contains 512*512 pixels, as shown in FIG. 6;

(37) (b) according to the structure of existing single channel spectral imager, the cosine sparse domain and two-step iteration algorithm are utilized to reconstruct the underlying spectral image and calculate the PSNR value of reconstruction result of each spectral dimension image, as shown in FIG. 7; where FIGS. 7(a), 7(b), 7(c), 7(d), 7(e) and 7(f) show reconstruction results and corresponding PSNRs of multispectral images shown in FIGS. 6(a), 6(b), 6(c), 6(d), 6(e) and 6(f), respectively.

(38) Simulation 2:

(39) (a) a multispectral image of a river that is originally captured is selected as a underlying spectral image, which contain 6 spectral dimension images each of which contains 512*512 pixels, as shown in FIG. 6;

(40) (b) according to the structure of the double channel spectral imager of the embodiments of the present invention, the cosine sparse domain and two-step iteration algorithm are utilized to reconstruct the underlying spectral image and calculate the PSNR value of reconstruction result of each spectral dimension image, as shown in FIG. 8, where FIGS. 8(a), 8(b), 8(c), 8(d), 8(e) and 8(f) show reconstruction results and PSNRs of multispectral images shown in FIGS. 6(a), 6(b), 6(c), 6(d), 6(e) and 6(f), respectively.

(41) From the result of Simulation 1, it can be seen that over the range of the six spectral dimension images reconstructed by using single channel, the average PSNR value is 30.31 dB, the minimal PSNR value is 28.5167 dB, and the maximal PSNR value is 32.7970 dB.

(42) From the result of Simulation 2, it can be seen that over the range of the six spectral dimension images reconstructed by using double channels according to the embodiments of the present invention, the average PSNR value is 32.82 dB, the minimal PSNR value is 31.0005 dB, and the maximal PSNR value is 35.5981 dB.

(43) By comparing the result of Simulation 1 with that of Simulation 2, it can be seen that with respect to the prior art using single channel, the present invention using double channels can increase the average PSNR value by 2.51 dB, which demonstrates that the present invention can provide the better performance in multispectral image reconstruction.