Method and System for Generating High Resolution Worldview-3 Images

Abstract

The present invention presents four fusion approaches, which can be directly applied to Worldview-3 (WV-3) images. Moreover, they can also be applied to other current or future satellite images that have similar characteristics of WV-3. The present invention also presents data processing methods, including image fusion method, anomaly detection method, material classification method, and concentration estimation method that utilize the high-resolution images generated by the mentioned fusion methods. There are four fusion approaches disclosed in the present invention, e.g. Parallel one-step fusion approach; Sequential fusion of various bands; Sequential-Parallel fusion; and Parallel-Sequential fusion.

Claims

1. A system for generating high resolution super-spectral images, comprising: a panchromatic (PAN) module; a Visible Near Infrared (VNIR) module; a Short-Wave Infrared (SWIR) module; a first Super-Resolution Algorithm Module (SRAM); a second SRAM; and a merger having a Hybrid Color Mapping (HCM) module to process the outputs from the SRAMs.

2. A system for generating high resolution super-spectral images in accordance to claim 1, further comprising: an anomaly detection module for detecting high spatial resolution images regions that are different from the neighbors; and a sparsity based classification module for classification of surface materials.

3. A system for generating high resolution super-spectral images in accordance to claim 1, wherein: the PAN module generates a panchromatic band of 0.31 m resolution; the VNIR module generates eight VNIR bands of 1.2 m resolution; and the SWIR module generates eight SWIR bands of 7.5 m resolution.

4. A system for generating high resolution super-spectral images in accordance to claim 3, wherein: the first SRAM combines the panchromatic band and the eight VNIR bands to generate eight pan-sharpened VNIR bands; the second SRAM combines the panchromatic band and the eight SWIR bands to generate eight pan-sharpened SWIR bands; and the merger combines the eight pan-sharpened VNIR bands and the eight pan-sharpened SWIR bands to generate sixteen High-Resolution (HR) bands.

5. A system for generating high resolution super-spectral images in accordance to claim 4, wherein: the two SRAMs are executed in a parallel manner.

6. A system for generating high resolution super-spectral images in accordance to claim 4, wherein: the eight pan-sharpened VNIR bands are of 0.31 m resolution; the eight pan-sharpened SWIR bands are of 0.31 m resolution; and the sixteen HR bands are of 0.31 m resolution.

7. A system for generating high resolution super-spectral images in accordance to claim 3, wherein: the first SRAM combines the panchromatic band and the eight VNIR bands to generate eight pan-sharpened VNIR bands; the second SRAM combines the eight pan-sharpened VNIR bands and the eight SWIR bands to generate eight pan-sharpened SWIR bands; and the merger combines the eight pan-sharpened VNIR bands and the eight pan-sharpened SWIR bands to generate sixteen High-Resolution (HR) bands.

8. A system for generating high resolution super-spectral images in accordance to claim 7, wherein: the two SRAMs are executed in a sequential manner.

9. A system for generating high resolution super-spectral images in accordance to claim 3, further comprising: a third SRAM for receiving the eight VNIR bands and the eight SWIR bands from the VNIR and SWIR modules to generate eight pan-sharpened SWIR bands of 1.2 m resolution.

10. A system for generating high resolution super-spectral images in accordance to claim 9, wherein: the first SRAM combines the panchromatic band and the eight VNIR bands to generate eight pan-sharpened VNIR bands; the second SRAM combines the panchromatic band and the eight pan-sharpened SWIR bands from the third SRAM to generate eight pan-sharpened SWIR bands of 0.31 m resolution; and the merger combines the eight pan-sharpened VNIR bands from the first SRAM and the eight pan-sharpened SWIR bands from the second SRAM to generate sixteen HR bands.

11. A system for generating high resolution super-spectral images in accordance to claim 10, wherein: the three SRAMs are executed in a sequential-parallel manner.

12. A system for generating high resolution super-spectral images in accordance to claim 9, wherein: the first SRAM combines the panchromatic band and the eight VNIR bands to generate eight pan-sharpened VNIR bands; the second SRAM combines the eight pan-sharpened VNIR bands from the first SRAM and the eight pan-sharpened SWIR bands from the third SRAM to generate eight pan-sharpened SWIR bands of 0.31 m resolution; and the merger combines the eight pan-sharpened VNIR bands from the first SRAM and the eight pan-sharpened SWIR bands from the second SRAM to generate sixteen High-Resolution (HR) bands.

13. A system for generating high resolution super-spectral images in accordance to claim 12, wherein: the three SRAMs are executed in a parallel-sequential manner.

14. A method for generating high resolution super-spectral images, comprising the steps of: a. generating a panchromatic (PAN) band; b. generating eight Visible Near Infrared (VNIR) bands; c. generating eight Short-Wave Infrared (SWIR) bands; d. generating eight pan-sharpened VNIR bands and eight pan-sharpened SWIR bands from the PAN band, VNIR bands, and the SWIR bands; and e. merging the eight pan-sharpened VNIR bands and the eight pan-sharpened SWIR bands to generate sixteen HR bands.

15. A method for generating high resolution super-spectral images in accordance to claim 14, wherein: the eight pan-sharpened VNIR bands are generated by combining the PAN band and the eight VNIR bands; and the eight pan-sharpened SWIR bands are generated by combining the PAN band and the eight SWIR bands.

16. A method for generating high resolution super-spectral images in accordance to claim 14, wherein: the eight pan-sharpened VNIR bands are generated by combining the PAN band and the eight VNIR bands; and the eight pan-sharpened SWIR bands are generated by combining the eight pan-sharpened VNIR bands and the eight SWIR bands.

17. A method for generating high resolution super-spectral images, comprising the steps of: a. generating a panchromatic (PAN) band; b. generating eight Visible Near Infrared (VNIR) bands; c. generating eight Short-Wave Infrared (SWIR) bands; d. combining the Pan band and the eight VNIR bands to generate eight pan-sharpened VNIR bands; and e. combining the eight VNIR bands and the eight SWIR bands to generate eight pan-sharpened SWIR bands; f. combining the PAN band and the eight pan-sharpened SWIR bands to generate eight enhanced SWIR bands of higher resolution; and g. merging the eight pan-sharpened VNIR bands and the enhanced SWIR bands of higher resolution to generate sixteen High-Resolution HR bands.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] FIG. 1 illustrates an Approach 1: Parallel one-step fusion approach.

[0023] FIG. 2 illustrates an Approach 2: Sequential fusion of various bands.

[0024] FIG. 3 illustrates an Approach 3: Sequential-parallel fusion.

[0025] FIG. 4 illustrates an Approach 4: Parallel-sequential fusion.

[0026] FIG. 5 illustrates a system flow of color mapping. This approach can be used to generate high spatial resolution hyperspectral images.

[0027] FIG. 6 illustrates a hybrid color mapping, combining Multispectral bands with the addition of a white band.

[0028] FIG. 7 illustrates a local color mapping.

[0029] FIG. 8 illustrates a comparison of spatial resolution enhancement methods, showing the hybrid color mapping algorithm yielded much better classification accuracy.

[0030] FIG. 9 illustrates receiver operating characteristics (ROC) curves using anomaly detection results. The performance of Cluster Kernel Reed-Xiaoli (CKRX) and Kernel Reed-Xiaoli (KRX) reached almost an optimal performance as compared to others.

[0031] FIG. 10 illustrates the global version of the sparsity-driven face recognition method.

[0032] FIGS. 11(a) & (b) illustrate Face images of Subject-1 in 64 different lighting conditions in the Yale B face database.

[0033] FIG. 12 illustrates classification results of the algorithm of the present invention.

[0034] FIG. 13 illustrates preliminary results with the adaptation of Deep Belief Network (DBN) technique to chemical composition estimation in Laser Induced Breakdown Spectroscopy (LIBS).

DETAILED DESCRIPTION OF THE INVENTION

[0035] The present invention presents four fusion approaches, which can be directly applied to Worldview-3 images. Moreover, they can also be applied to other current or future satellite images that have similar characteristics of Worldview-3. The present invention also presents data processing methods, including image fusion method, anomaly detection method, material classification method, and concentration estimation method that utilize the high-resolution images generated by the fusion methods.

Approach 1: Parallel One-Step Fusion

[0036] As shown in FIG. 1, the first approach involves two parallel modules. Module 1 fuses the panchromatic and the eight VNIR bands to generate eight VNIR bands with 0.31 m resolution. Module 2 fuses the panchromatic and the eight SWIR bands to generate eight SWIR bands with 0.31 m resolution. The outputs of Modules 1 and 2 are merged to form a 16-band image cube. The two modules can be executed in parallel.

Approach 2: Sequential Fusion

[0037] FIG. 2 shows the sequential method. First, the pan and the eight VNIR bands are fused to generate 8 high resolution VNIR bands with 0.31 m resolution. Second, the eight pan-sharpened VNIR bands are fused with the eight SWIR bands to generate eight pan-sharpened SWIR bands. Finally, the high-resolution bands are merged into a single image cube.

Approach 3: Sequential-Parallel Fusion

[0038] FIG. 3 shows a sequential-parallel fusion approach. There are two parallel paths. The first path has only one step, which involves the fusion of the pan and the eight VNIR bands to yield eight pan-sharpened bands with 0.31 m resolution. The second path has two steps where Step 1 involves the fusion of eight VNIR and eight SWIR bands to generate eight pan-sharpened SWIR bands with 1.2 m resolution. Step 2 of the second path involves the fusion of the pan band with the outputs of Step 1 to generate high resolution SWIR bands with 0.31 m resolution. Finally, the outputs of the two paths are merged together to form a 16-band High-Resolution (HR) image cube with 0.31 m resolution.

Approach 4: Parallel-Sequential Fusion

[0039] FIG. 4 illustrates the parallel-sequential fusion approach. Module 1 is the fusion of the PAN band and the eight VNIR bands and Module 2 is the fusion of eight VNIR and eight SWIR bands. Both modules can be executed in parallel. Module 3 then fuses the outputs of Modules 1 and 2 to generate eight high resolution SWIR bands. Finally, the outputs of Module 1 and Module 3 are merged to form a 16-band HR image cube.

Color Mapping

[0040] The idea of color mapping is as the name suggests: mapping a multispectral pixel to a hyperspectral pixel. Here, multispectral images encompass color R-G-B images. This mapping is based on a transformation matrix T, i.e.

X=Tx,

where X is one (or more) hyperspectral pixels and x is one (or more) multispectral pixels. To get the transformation matrix, the present invention simulates a low resolution multispectral image and use the low resolution hyperspectral image to train the T. FIG. 5 shows the system flow.

[0041] Training is done by minimizing the mean square error:

[00001]$T^{*}=\underset{T}{\arg .Math..Math.\min }.Math.||H-\mathrm{TC}.Math.{||}_F,$

where H is the set of hyperspectral pixels and C is the set of multi-spectral pixels. With enough pixels, the optimal T can be determined with:

T=XC.sup.T(CC.sup.T).sup.1.

Hybrid Color Mapping

[0042] The present invention proposes a fusion algorithm, known as Hybrid Color Mapping (HCM) to perform the fusion in all four fusion approaches mentioned above. HCM is simple to implement, efficient, parallelizable and fast. The details can be found in the mentioned pending patents and papers by the present inventor. For completeness, the HCM algorithm is included in the following few paragraphs.

[0043] Extensive studies and results show that the method used in the present invention can generate highly accurate, high resolution reconstruction than the normal simple bicubic scaling and other state-of-the-art methods. In addition, the present invention conducted extensive classification study using reconstructed images. Results show that the method used in the present invention performs much better than other methods.

[0044] For many hyperspectral images, the band wavelengths range from 0.4 to 2.5 um. For color/multispectral images, the bands may include R-G-B, and some additional spectral bands. As shown in FIG. 6, the present invention also adds a white band, i.e. all the pixel values are 1. This white band can be used to compensate for atmospheric effect and other bias effects. Mathematically, adding a white band is equivalent to increasing one more column in the T matrix. Hence, the present invention has more parameters to adjust in the transformation.

Local Color Mapping

[0045] The present invention further enhances the method by applying color mapping patch by patch as shown in FIG. 7. A patch is a sub-image in the original image. Each patch will have a local transformation matrix. In this way, spatial correlation can be exploited. In addition, since the task is split into many small tasks, the process can be easily parallelized.

Experiment

[0046] The present invention used the NASA Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) hyperspectral data in this study. In each experiment, the image was downscaled by three times using Bicubic Interpolation (BI) method. The downscaled image was used as low resolution hyperspectral image. The R-G-B bands were picked from the original high resolution hyperspectral image for color mapping. The bicubic method in the following plots was implemented by upscaling the low-resolution image using BI. The results of the bicubic method were used as a baseline for comparison study. As shown in FIG. 8, it shows classification results using the endmembers extracted from ground truth AVIRIS hyperspectral image. The AVIRIS image has 213 bands with wavelengths range from 380 nm to 2500 nm. Also, the hybrid color mapping is significantly better than the bicubic method.

Material Classification Algorithm

[0047] The present invention proposes to apply the latest development in sparsity based classification algorithm to rock type classification. Similarly, the approach of the present invention requires some spectral signatures to be available as in other methods mentioned.

[0048] The present invention implemented a sparsity-driven recognition method in the articles and papers mentioned. In the sparsity-driven face recognition approach, the assumption is that a face image of subject i lies in the linear span of the existing face images for that same subject i in the training set. Suppose {v.sub.i1, v.sub.i2, . . . , v.sub.iD} are the vectorized D face images of subject i in the training set, and y is a new vectorized face image of subject i, which is not in the training set. Based on this assumption, y, can be expressed as:

[00002]$\begin{array}{cc}y={}_{i.Math..Math.1}.Math.v_{i.Math..Math.1}+{}_{i.Math..Math.2}.Math.v_{i.Math..Math.2}+\mathrm{.Math.}+{}_{\mathrm{iD}}.Math.v_{\mathrm{iD}}=\left[v_{i.Math..Math.1}.Math..Math.v_{i.Math..Math.2}.Math..Math.\mathrm{.Math.}.Math..Math.v_{\mathrm{iD}}\right]\left[\begin{array}{c}{}_{i.Math..Math.1}\\ {}_{i.Math..Math.2}\\ \mathrm{.Math.}\\ {}_{\mathrm{iD}}\end{array}\right]=A_i.Math.{}_i& \left(1\right)\end{array}$

[0049] Suppose there are C human subjects; the above expression can then be expanded as in (2) and this expression indicates that y is the sparse linear combination of face images in the training set.

[00003]$\begin{array}{cc}y=\left[A_1.Math..Math.A_2.Math..Math.\mathrm{.Math.}.Math..Math.A_C\right]\left[\begin{array}{c}{}_1\\ {}_2\\ \mathrm{.Math.}\\ {}_C\end{array}\right]=\mathrm{Ax}& \left(2\right)\end{array}$

[0050] The sparse representation, x.sub.0=[0 . . . 0 a.sub.i.sup.T 0 . . . 0], thus yields the membership of y to subject i. The above framework to small contact detection can be easily extended. Each contact image will be vectorized and put into the dictionary.

[0051] Referring to FIG. 10, it depicts an illustrative diagram of the global version. In the global version, the whole face image is vectorized and the sparsity coefficient vector estimations are done using these whole-face image vectors.

[0052] As shown in FIG. 11, the recognition performance of the global version (i.e. the use of the whole face) of the sparsity-driven face recognition method has been examined on the Extended Yale B face database. There are thirty-eight subjects and each subject has sixty-four faces with different illuminations. In addition to considering only the frontal images, the test face images are also rotated at four different rotational degrees to see how the performance of the sparsity-driven face recognition method will be affected. It should be noted that the down-sampled images of Extended Yale B face database of size 3228 are used in this work. Additionally, when rotational effects are introduced to the test face images, the original size image (192168) has been rotated and then down-sampling is applied to the rotated test image. It should be also noted that rotational effects are only applied to the face images in the testing set, but not to the face images in the training set. Specifically, the dictionary only contains the frontal face images.

[0053] As shown in FIG. 12, a maximum recognition rate of 100% has been achieved. This high recognition rate is quite remarkable, considering there are a number of poor illuminated images as shown in FIG. 10.

Concentration Estimation Algorithm

[0054] The present invention proposes to apply Deep Neural Network (DNN) techniques to further improve the chemical element classification and composition estimation performance in surface monitoring such as volcano monitoring. Possible applications include ash detection, composition estimation, and SO.sub.2 concentration estimation. The present invention adapts two of the DNN techniques, the Deep Belief Network (DBN) and Convolutional Neural Network (CNN), respectively, to the element classification and chemical composition estimation problem.

[0055] As shown in the website, https://github.com/rasmusbergpalm/DeepLearnToolbox, DNN techniques have the following advantages: [0056] i. Better capture of hierarchical feature representations; [0057] ii. Ability to learn more complex behaviors; [0058] iii. Better performance than conventional methods; [0059] iv. Use distributed representations to learn the interactions of many different factors on different levels; [0060] v. Can learn from unlabeled data such as using the Restricted Boltzmann Machines (RBM) pretraining method; and [0061] vi. Performance can scale up with the number of hidden layers and hidden nodes on fast GPUs.

[0062] One of the applications in which DNN techniques have proved themselves is the handwritten digit recognition application. The present invention applied the Deep Belief Network (DBN) technique to the Laser Induced Breakdown Spectroscopy (LIBS) spectrum database (sixty-six samples) based on preliminary investigation in the past. The total number of oxides is nine and these nine oxide compounds are: [0063] 1) SiO.sub.2, [0064] 2) TiO.sub.2; [0065] 3) Al.sub.2O.sub.3; [0066] 4) Fe.sub.2O.sub.3; [0067] 5) MnO; [0068] 6) MgO; [0069] 7) CaO; [0070] 8) Na.sub.2O; [0071] 9) K.sub.2O.

[0072] A Leave-One-Out (LOO) testing framework is applied to the LIES dataset of sixty-six samples to estimate oxide compositions. Two performance measures are computed: a) ERRORSUM, the sum of absolute error in the sample estimate and its ground truth, b) RMSEP, to assess the estimation accuracy for each of the nine oxide compounds. The initial results were quite encouraging for a DBN with 3-Level architecture. Level-1: RBM with 50 hidden units; Level-2: RBM with 5050 hidden units; and Level-3: connection to output with NN with 1000 epochs. Comparable results for DBN to the Partial Least Square (PLS) technique were observed. The resultant performance measures with PLS and DBN technique is shown in FIG. 13. It is quite likely to see further improvement in the composition estimation results after customization of the DBN architecture, and the number of hidden DBN units.

[0073] It will be apparent to those skilled in the art that various modifications and variations can be made to the system and method of the present disclosure without departing from the scope or spirit of the disclosure. It should be perceived that the illustrated embodiments are only preferred examples of describing the invention and should not be taken as limiting the scope of the invention.