OFF-BAND RESOLUTION EMHANCEMENT

Abstract

A method of enhancing an image includes increasing sampling rate of a first image to a target sampling rate to form an interpolated image. The method also includes processing a second image through a high pass filter to form a high pass features image, wherein the second image is at the target sampling rate. The method also includes extracting detail from the high pass features image relevant to the first image, merging the detail from the high pass features image with the interpolated image to form a prediction image at the target sampling rate, and outputting the prediction image.

Claims

1. A method of enhancing an image comprising: increasing sampling rate of a first image to a target sampling rate to form an interpolated image; processing a second image through a high pass filter to form a high pass features image, wherein the second image is at the target sampling rate; extracting detail from the high pass features image; merging the detail from the high pass features image with the interpolated image to form a prediction image at the target sampling rate; and outputting the prediction image.

2. The method as recited in claim 1, further comprising: processing at least one additional image through a high pass filter to form at least one respective additional high pass features image; and extracting detail from the at least one additional high pass features image, wherein merging the detail from the high pass features image with the interpolated image includes merging detail from the at least one additional high pass features image to form the prediction image.

3. The method as recited in claim 2, wherein the first image is from a first spectral band, the second image is from a second spectral band, and wherein the at least one additional image is from at least one respective additional spectral band.

4. The method as recited in claim 2, wherein the at least one additional image is at a higher resolution than the first image.

5. The method as recited in claim 1, wherein the first image is from a first spectral band, the second image is from a second spectral band, and wherein the first and second bands are overlapping.

6. The method as recited in claim 1, wherein the first image is from a first spectral band, the second image is from a second spectral band, and wherein the first and second bands are non-overlapping.

7. The method as recited in claim 1, wherein extracting detail from the high pass features image is done on a segment by segment basis from the high pass features image for use in merging with the interpolated image, wherein extracted detail for each segment is governed by individualized prediction coefficients.

8. The method as recited in claim 7, wherein extracting detail includes receiving an image segmentation map for segmenting the high pass features image and receiving prediction coefficients corresponding to respective segments of the image segmentation map for extracting detail on a segment by segment basis.

9. The method as recited in claim 8, wherein receiving the image segmentation map and receiving the prediction coefficients includes receiving the image segmentation map and prediction coefficients from a training operation, wherein the training operation includes: creating a reduced sampling rate copy of the second image; processing the reduced sampling rate copy of the second image and the first image through a high pass filter to form second and first feature images, respectively; processing the reduced sampling rate copy of the second image and the first image through a segmentation operation to produce the segmentation map; and processing the first and second feature images and the segmentation map through a fit process to generate the prediction coefficients.

10. The method as recited in claim 9, wherein creating a reduced copy of the second image includes creating a reduced sampling rate copy of at least one additional image, and wherein the segmentation map and prediction coefficients are based on the second image and the at least one additional image.

11. The method as recited in claim 8, wherein extracting detail from the high pass features image includes increasing the sampling rate of the segmentation map to the target sampling rate.

12. The method as recited in claim 7, wherein coefficients are constant within each segment, wherein the segments are sub-divided to only span a local region.

13. The method as recited in claim 1, wherein processing the second image through a high pass filter includes decimating the second image to a reduced sampling rate and then interpolating back to the target sampling rate and subtracting this result from the second image to the produce the high pass features image.

14. The method as recited in claim 1, wherein merging the detail from the high pass features image with the interpolated image to form a prediction image at the target sampling rate is governed by
Y.sub.s.sup.(e)=Y.sub.s.sup.(l)+X.sub.sC.sub.s, where the matrix Y.sub.s.sup.(l) denotes the data in segment s of the interpolated image and Y.sub.s.sup.(e) denotes the prediction image for segment s, X.sub.s is a matrix of detail data extracted from the high pass features image for segment s, C.sub.s is a matrix of prediction coefficients for segment s, and is a sharpening gain.

15. The method as recited in claim 14, wherein =1.

16. The method as recited in claim 1, wherein extracting detail from the high pass features image is done on a segment by segment basis from the high pass features image for use in merging with the interpolated image, wherein extracted detail for each segment is governed by individualized prediction coefficients, further comprising generating a segmentation map based on the interpolated image and the second image, wherein the second image and the interpolated image are processed through a segmentation algorithm to produce a segmentation map at the target sampling rate for use in extracting detail from the high pass features image on a segment by segment basis.

17. The method as recited in claim 1, wherein the first image is from a first modality, the second image is from a second modality, and wherein the first and second modalities are different from one another.

18. A system for enhancing images including: a module configured to implement machine readable instructions to perform the method recited in claim 1.

19. The system as recited in claim 18, wherein the machine readable instructions include instructions for: processing at least one additional image through a high pass filter to form at least one respective additional high pass features image; and extracting detail from the at least one additional high pass features image, wherein merging the detail from the high pass features image with the interpolated image includes merging detail from the at least one additional high pass features image to form the prediction image.

20. The system as recited in claim 18, wherein the machine readable instructions include instructions wherein extracting detail from the high pass features image is done on a segment by segment basis from the high pass features image for use in merging with the interpolated image, wherein extracted detail for each segment is governed by individualized prediction coefficients.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] So that those skilled in the art to which the subject disclosure appertains will readily understand how to make and use the devices and methods of the subject disclosure without undue experimentation, preferred embodiments thereof will be described in detail herein below with reference to certain figures, wherein:

[0016] FIG. 1 is a schematic view of an exemplary embodiment of a method of enhancing an image in accordance with the present disclosure, showing a process overview including training and application of the training;

[0017] FIG. 2 is a schematic view of the method of FIG. 1, showing the training process;

[0018] FIG. 3 is a schematic view of the method of FIG. 1, showing the application process; and

[0019] FIG. 4 is a schematic view of another exemplary embodiment of a process for generating segmentations maps in accordance with the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Reference will now be made to the drawings wherein like reference numerals identify similar structural features or aspects of the subject disclosure. For purposes of explanation and illustration, and not limitation, a partial view of an exemplary embodiment of a method in accordance with the disclosure is shown in FIG. 1 and is designated generally by reference character 100. Other embodiments of methods in accordance with the disclosure, or aspects thereof, are provided in FIGS. 2-4, as will be described. The systems and methods described herein can be used for enhancement of images from one band using one or more images from another band.

[0021] Method 100 of enhancing an image includes enhancing a first image 102 from a first spectral band/modality using detail extracted from one or more second images 104 from respective second spectral bands/modalities to produce an enhanced image 106. For example, the first image 102 can be an image from a low resolution sensor in a multi-spectral imaging system at a reduced sampling rate, and second images 104 can be images from higher resolution sensors in the same system at the target sampling rate. While FIGS. 1-3 show three images 104, those skilled in the art will readily appreciate that any suitable number of images 104, including only one image, can be used. The first and second spectral bands can be overlapping or non-overlapping. As shown in FIG. 1, method 100 includes using images 102 and 104 as inputs to a training operation or phase 108, as well as to an application operation or phase 110 which uses results from the training phase 108 to produce enhanced image 106. While shown and described herein in the exemplary context of having a single first image 102, e.g., a low-resolution band, those skilled in the art will readily appreciate that multiple low-resolution bands can be processed together using the systems and methods disclosed herein.

[0022] It should be noted that resolution in the existing literature can refer to the sampling rate (sampling interval) or the amount of high-frequency information present in an image. For example, there could be an image very finely sampled (high resolution) but also extremely blurry and lacking any high frequency detail (low resolution). For purposes of making this distinction clear in this disclosure, sampling rate specifies how finely an image is sampled and resolution specifies how much detail it has. Note that in this sense the interpolation operation increases the sampling rate but can end up decreasing the original resolution as the image can undergo a certain amount of blurring in the interpolation process.

[0023] With reference now to FIG. 2, training phase 108 includes creating a reduced sampling rate copy 128 of the one or more second images 104, as indicated by decimation operation 130. The decimation operation is well known to those skilled in the art. For example, if the sampling rate is reduced by an integer factor f, one can average blocks of ff pixels to produce the reduced rate image. The reduced sampling rate copies 128 of the one or more second images 104 together with the first image 102 make up a training set, which are used to produce the output of training phase 108. Training phase 108 also includes processing the reduced sampling rate copy 128 of each second image 104 and the first image 102 through a high pass filter, as indicated by box 132 in FIG. 2, to form second and first feature images 134 and 136, respectively.

[0024] In one embodiment, the training set of first and second images (e.g., 102, 128) are fed to a segmentor 126 that produces a segmentation map at the reduced rate as indicated by the reduced rate segmentation map 142. Segmentation operations are well known to those skilled in the art and any suitable technique or techniques such as Gaussian mixture models or k-means may be employed for this purpose. The segmentation criterion can be spectral similarity in the case of multi-spectral images. Any other appropriate similarity criterion may also be used in the case of images from different modalities. The segmentor 126 may further sub-divide the segments into smaller regions so that they span a local area to better model the changing characteristics of the image and improve the prediction of the enhanced image 106. The size of the local regions is controlled by a user specified parameter. In an embodiment, the Gaussian mixture model is used to perform the segmentation followed by a further sub-division of the segments into blocks that do not exceed 128128 pixels, for example.

[0025] Training phase 108 further includes processing each of the first and second feature images 134 and 136 and the reduced rate segmentation map 142 through a fit process 138 to generate the prediction coefficients 124. The fitting process 138 learns a different set of coefficients for each of the different regions in the segmentation map. In this manner, the segmentation map and prediction coefficients allow for object or context specific sharpening or enhancement that can vary spatially within the image space.

[0026] The fitting operation 138 and the generation of the coefficients 124 will now be described in detail. Note that all images in the fitting process of the training phase 108 are at the reduced sampling rate.

[0027] For each segment s, the second image data 134 can be arranged in a matrix X.sub.s of size M(s)N.sub.f where each row is a pixel in segment s and the columns are the N.sub.f bands in second feature image data 134. Here, M(s) denotes the number of pixels in segment s. The corresponding first feature image data 136 can be arranged similarly in a matrix Y.sub.s of size M(s)N.sub.p, where the rows correspond to the pixels in X.sub.s and the columns are the N.sub.p bands of the first image feature data 136. As shown in FIG. 2, N.sub.p=1, but those skilled in the art will readily appreciate that multiple bands can be enhanced at the same time.

[0028] Employing a linear prediction model .sub.s=X.sub.sC.sub.s to predict the first feature image from the second feature image, the optimization problem to compute the optimal coefficients is given as

[00001]$\begin{array}{cc}\underset{c_s}{\min }.Math..Math.\mathrm{diag}\left(E_s^{}.Math.E_s\right),& \left(1\right)\end{array}$

where diag( ) denotes an operator to extract all the diagonal entries of a matrix into a vector that are subsequently added up by the summation operator, E.sub.s is the prediction error matrix given as

E.sub.s=Y.sub.sX.sub.sC.sub.s,(2)

and C.sub.s is the N.sub.fN.sub.p prediction coefficient matrix for segment s. The closed form solution of eq. (1) for the optimal coefficients is given as

C.sub.s=(X.sub.s.sup.tX.sub.s).sup.1X.sub.s.sup.tX.sub.s,(3)

where the superscript t denotes matrix transposition.

[0029] If the number of pixels in a segment M(s) is small relative to the number of coefficients N.sub.f to be estimated for each prediction band, the optimization problem of eq. (1) may become ill-posed. It is desirable to include some a priori information to make the problem well-posed. Those skilled in the art will readily appreciate that such an approach is known as regularization. Since we are predicting the detail image, a reasonable approach is to reduce the magnitude of the prediction coefficient so the predicted detail image automatically goes towards zero when there is little or no information to be extracted from the data. Towards this end, we modify the original optimization problem given by eq. (1) to penalize the magnitude of the estimated prediction coefficients

[00002]$\begin{array}{cc}\underset{c_s}{\min }.Math.\left(.Math.\mathrm{diag}\left(E_s^{}.Math.E_s\right)+.Math..Math.\mathrm{diag}\left(C_s^{}.Math.C_s\right)\right).& \left(4\right)\end{array}$

Here is a user supplied parameter that controls the strength of our a priori information. The solution obtained from eq. (4) will be biased towards zero. The amount of bias depends on the amount of data present in the training for segment s. If the number of pixels M(s) is large, the bias will be negligible and the data will determine the unknown coefficients. When there is little or no information (M(s) is small), the a priori information takes over and the magnitude of the prediction coefficients will dial back to zero providing little or no enhancement to the low-resolution image.

[0030] The closed form solution of eq. (4) for the optimal coefficients is given as

[00003]$\begin{array}{cc}{C}_s={\left(A_s^{}.Math.A_s\right)}^{-1}.Math.A_s^{}.Math.{\stackrel{\_}{Y}}_s,& \left(5\right)\\ \mathrm{where}& \\ A_s=\left[\begin{array}{c}{X}_s\\ \sqrt{}.Math.I\\ \end{array}\right]& \left(6\right)\\ \mathrm{and}& \\ {\stackrel{\_}{Y}}_s=\left[\begin{array}{c}{Y}_s\\ Z\\ \end{array}\right].& \left(7\right)\end{array}$

Here I is an N.sub.fN.sub.f identity matrix and Z is an N.sub.fN.sub.p matrix of zeroes.

[0031] If the pixel values in the images are corrupted or the segment boundaries are such that some pixels do not fit the linear prediction model in that segment, the estimated prediction coefficients using eqs. (1) or (4) may be unduly biased. Even though the number of these outlying pixels may be small, their influence on the estimated prediction coefficients may be disproportionately large. It is desirable to remove these pixels in the fitting process so that the prediction of the good pixels are not affected. This can be done by introducing weights for each pixel in the optimization criterion of eqs. (1) and (4), and estimating the weights based on how well the pixel fits the model. Such a procedure is known as robust fitting and those skilled in the art will be familiar with the required modifications to the optimization criterion.

[0032] With reference now to FIG. 3, application phase 110 is shown in greater detail. Method 100 includes, in the application phase 110, increasing sampling rate of a first image to a target sampling rate to form an interpolated low-pass image 146, by the interpolation operation 112 in FIG. 3.

[0033] This increase in sampling rate can include any suitable interpolation operation on first image 102. In an embodiment, bi-cubic interpolation is employed. At least one of the one or more second images 104 is at the target sampling rate, and at a higher level of detail or resolution than that of the first image 102. Application phase 110 includes processing the one or more second images 104 through a high pass filter, as indicated in FIG. 3 by box 114, to form one or more respective high pass features images 116. Application phase 110 of method 100 also includes extracting detail 118 relevant to the first image 102 from the one or more high pass features images 116, merging the detail 118 with the interpolated low-pass image 146, as indicated by merger process 120 in FIG. 3, to form prediction image 106 at the target sampling rate in the first band. This prediction image 106 can be a resolution enhanced version of the original image 102. Method 100 can include outputting the prediction image 106 in any suitable format.

[0034] Extracting detail from the high pass features image relevant to the first image is done on a segment by segment basis from the high pass features images 116 for use in merging with the interpolated image, as indicated by box 122 in FIG. 2. Extracting detail for each segment is governed by individualized prediction coefficients 124, which are derived from the training phase 108 described above. Detail extraction includes receiving a target rate image segmentation map 144 and receiving prediction coefficients 124 corresponding to respective segments of the image segmentation map 144 for extracting detail on a segment by segment basis. The coefficients are constant within each segment, wherein the segments may be sub-divided to only span a local region.

[0035] The training phase 108 generates a reduced rate segmentation map 142 for learning the coefficients for each segment. However, this segmentation cannot be used directly in the application phase 110 as the images are now at the target sampling rate. For this purpose, a segment interpolator 140 is employed to increase the rate of the segmentation map 142 to the target rate segmentation 144. If the segmentation map 142 is stored as an image with pixel values set to the corresponding segment index it is assigned to, nearest neighbor interpolation can then be applied to generate the segmentation map 144 at the target rate. Nearest neighbor interpolation ensures that no new segment indices are created in the interpolation process. Since segmentation maps 142 and 144 have the same number of unique segment indices, the coefficients 124 learned in the training phase 108 can be directly applied in the application phase 110.

[0036] FIG. 4 shows another exemplary embodiment for generating the segmentation maps 142 and 144. In this case, the target sampling rate images 104 and 146 are fed to the segmentor 126 to produce the segmentation map 144 at the target sampling rate. The segment decimator 148 can then be used to generate the reduced sampling rate segmentation map 142. Note that reducing the sampling rate of the segmentation map 144 should not produce any new segment indices. To facilitate this process, the segmentation map 144 is stored as an image with pixel values set to the corresponding segment index it is assigned to, and the sampling rate is reduced by a normal decimation operation where instead of averaging ff blocks of pixels (f being the down-sampling factor) the mode (segment index that occurs the most number of times in the ff block of pixels) is taken to generate the reduced rate segmentation map 142.

[0037] The predictor 122 for extracting the detail image 118 and the generation of the enhanced image 106 will now be described in detail. For each segment s, the data in the high-pass feature images 116 is put into the matrix X.sub.s of size M(s)N.sub.f in the same manner as was done in the training phase 108. Note that M(s), the number of pixels in segment s, will be different in the application phase 110 than the training phase 108 since all the images in the application phase are at the target sampling rate whereas they were at the reduced sampling rate in the training phase. The coefficients 124 matrix for segment s learned in the training phase is then post multiplied with X.sub.s to obtain the detail image 118 X.sub.sC.sub.s for segment s. These details are relevant to the first image 102. The entire detail image can be constructed by repeating the process for all segments s. The detail image X.sub.sC.sub.s can optionally be multiplied by a user specified sharpening gain that controls the sharpness of the enhanced image. The value of =1 can provide optimal prediction. Values of >1 boost the high-frequency content whereas values of <1 subdue it. The parameter may be set based on personal preference.

[0038] In matrix notation, the enhanced data 106 for each segment s is obtained as

Y.sub.s.sup.(e)=Y.sub.s.sup.(l)+X.sub.sC.sub.s,(8)

where the matrix Y.sub.s.sup.(l) denotes the data in segment s for the low-pass image 146 obtained by interpolation of the first image 102 and Y.sub.s.sup.(e) denotes the prediction for the enhanced image 106 for segment s. Eq. (8) can be repeated for each segment s to reconstruct the complete enhanced image 106 at the target sampling rate.

[0039] Although low- and high-pass components of the images can be produced by filters either in the spatial or frequency domain as will be appreciated by those skilled in the art, a satisfactory approach is to use the interpolation and decimation operations to generate these components primarily because these operations are already applied elsewhere in the algorithm. Let X denote an image at a particular sampling rate. The low-pass component X.sup.(l) at the same sampling rate can be generated by first decimating the image to a reduced rate followed by interpolating back to the same rate

X.sup.(l)=IDX.(9)

where D is the image decimation operator to go from the target sampling rate to the reduced sampling rate and I is the image interpolation operator to go from the reduced sampling rate to the target sampling rate. The high-pass component X.sup.(h) can be produced by subtracting the low-pass component from the original image

X.sup.(h)=XX.sup.(l)=XIDX.(10)

One can use averaging of the super-pixel for the decimation and bi-cubic interpolation.

[0040] The bandwidth of the high- and low-pass images produced in this manner is controlled by the decimation/interpolation factor f and is a user specified parameter that can be chosen based on the desired bandwidth of detail information that needs to be transferred to the first image. The default decimation/interpolation factor is kept to be the same as the ratio between the sampling rates of the high- and low-resolution images. However, a larger factor can also be used if the high and/or the low-resolution images are over-sampled or blurred and there is little or no high-frequency content that can be transferred from the high-resolution images or learned from the low-resolution images at the default decimation factor.

[0041] If the number of high-resolution bands is large, N.sub.f>>1, it may be possible to improve the computational efficiency of the algorithm by leveraging redundancies between the bands. Principal component analysis is a well known method for compacting the available information in a fewer number of bands. The remaining bands can then be discarded with minimal loss of information. The number of transformed bands to retain can be chosen based on a user specified parameter that controls the amount of information to be retained. For example, one can choose 95% of the variance to be retained in the transformed space. The algorithm specified above is equally applicable in the transformed PCA space and no modifications are necessary. Other compression methods may also be used in conjunction with the proposed method.

[0042] While described herein in the exemplary context of a method, those skilled in the art will readily appreciate that the processes described herein can be implemented on any suitable system, such as a computing device with interconnected input/output interfaces, memory, and one or more processors, wherein a module of the system is configured to implement machine readable instructions to perform any embodiment of the operations described above.

[0043] Systems and methods disclosed herein transfer the high-resolution information from one or more images to another, lower resolution image using correlations that exist between the image or images with the high-resolution information and the image being enhanced. Since this correlation varies depending on what is being imaged, the segmentation algorithm described herein can be employed to divide the image into regions where the correlation remains constant, or nearly constant. Traditional methods only tend to work well when there is a spectral overlap between the higher-resolution band and the lower-resolution band, and they typically only utilize information from a single higher-resolution band. Moreover, in the traditional methods, the amount of sharpening cannot be controlled. In systems and methods as disclosed herein, these short comings are overcome since information from multiple higher-resolution bands can be transferred to multiple lower-resolution bands, no spectral overlap is required, and the amount of sharpening can be controlled on a spectral properties context or object specific basis that can vary throughout an image. Methods and systems disclosed herein are guaranteed to produce images equal or better in sharpness than interpolation methods, wherein the increase in sharpness over interpolation methods depends on registration and correlation between the bands used, and the amount of high frequency information in the higher-resolution band or bands.

[0044] Those skilled in the art will readily appreciate that although methods have been described herein for images of different spectral bands acquired at differing resolutions, the techniques can be equally applicable to different resolution images acquired using differing modalities. The segmentation criterion can be modified from spectral similarity to some other suitable measure of similarity that makes the prediction model more applicable across modalities.

[0045] The methods and systems of the present disclosure, as described above and shown in the drawings, provide for image enhancement with superior properties including superior enhancement even when there is no overlap between the band being enhanced and the band or bands used for the enhancement. While the apparatus and methods of the subject disclosure have been shown and described with reference to preferred embodiments, those skilled in the art will readily appreciate that changes and/or modifications may be made thereto without departing from the scope of the subject disclosure.