Robust image registration for multiple rigid transformed images

Abstract

Systems and methods for multiple image registration of images of a scene or an object. Receiving image data, the image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene or the object. Estimating registration parameters, using pairs of images, each pair of images includes a reference image and a floating image. Generating parameter matrices corresponding to registration parameters using an image registration process for all pairs of images. Decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration. Using the updated registration parameters to form a transformation matrix to register the images with at least one reference image, resulting in robust registration of the images.

Claims

1. A computer-implemented method for multiple image registration of images a scene or an object, comprising: receiving image data of the scene or the object, the image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene; estimating registration parameters, through an image processor, using pairs of images from the received image data, wherein each pair of images includes a reference image and a floating image; generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using an image registration process for all pairs of images; decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration; and using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images.

2. The computer-implemented method of claim 1, wherein the image registration process is one of a mutual information (MI) process, a cross-correlation process or a feature-based process.

3. The computer-implemented method of claim 1, wherein each pair of images correspond to the registration parameter between the floating image and the reference image.

4. The computer-implemented method of claim 1, wherein the image data includes a single modality or different modalities, including one or a combination of multispectral (MS) images, panchromatic (PAN) images, color images, of the scene.

5. The computer-implemented method of claim 1, wherein the image data includes different modalities, including one of CT images or MRI images, or both, of the object.

6. The computer-implemented method of claim 1, wherein the image data includes radar images of the scene.

7. The computer-implemented method of claim 1, wherein the parameter matrices are generated in parallel by the imaging processor, resulting in a less computational time, when compared to a total computation time of generating each parameter matrix of the parameter matrices separately.

8. The computer-implemented method of claim 1, wherein the low-rank matrix of updated registration parameters is imposed with a strict rank 2 structure on the low-rank matrix.

9. The computer-implemented method of claim 1, wherein the registering of the images using the updated registration parameters to form the transformation matrix continues until a predetermined convergence threshold is met, such that the registering ends, resulting in the robust registration of the images.

10. The computer-implemented method of claim 9, wherein the predetermined convergence threshold is a positive number.

11. The computer-implemented method of claim 1, further comprising: receiving the image data via a transceiver; and storing the image data in a memory, wherein the memory and the transceiver are in communication with the image processor, such that previously stored in the memory is data, the data includes image historical data of the scene, image historical data of the object, or image historical data of both the scene and the object, the image registration process, along with stored instructions executable by the image processor.

12. A system for multiple image registration of images of a scene or an object, comprising: a transceiver for receiving image data of the scene or the object, the image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene; an image processor in communication with the transceiver, is configured to estimate registration parameters, including an image rotation angle, an image horizontal shift and an image vertical shift, using pairs of images from the received image data, wherein each pair of images includes a reference image and a floating image; generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using an image registration process for all pairs of images; decompose each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration; use the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images; and display the robust registered images on a display device, a user reviews the displayed robust registered images, and based upon the users review, the user conducts an image management action.

13. The system of claim 12, wherein the image registration process is one of a mutual information (MI) process, a cross-correlation process or a feature-based process.

14. The system of claim 12, wherein the image data includes one or combination of multispectral (MS) images, panchromatic (PAN) images, color images, CT images, MRI images or radar images.

15. The system of claim 12, wherein the low-rank matrix of updated registration parameters for the image rotation angle matrix is imposed with a strict rank 2 structure on the low-rank matrix.

16. A non-transitory computer readable storage medium embodied thereon a program executable by a computer for performing a method, the method is for image processing of images of a scene or an object, comprising: acquiring image data of the scene or the object, the image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene; estimating registration parameters, through the computer, using pairs of images from the acquired image data, wherein each pair of images includes a reference image and a floating image; generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using an image registration process for all pairs of images, wherein each entry of each matrix corresponds to a registration parameter between the floating image and the reference image pair; decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration; and using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images.

17. The non-transitory computer readable storage medium of claim 16, wherein the image registration process is one of a mutual information (MI) process, a cross-correlation process or a feature-based process.

18. The non-transitory computer readable storage medium of claim 16, wherein the low-rank matrix of updated registration parameters for the image rotation angle matrix is imposed with a strict rank 2 structure on the low-rank matrix.

19. The non-transitory computer readable storage medium of claim 16, further comprising: using an input interface to acquire the image data; and storing the image data in an image database of the storage medium, wherein the storage medium also includes an image processing database having previously stored data, the data includes image historical data of the scene, image historical data of the object, or both the image historical data of the scene and the object, the image registration process, along with stored instructions executable by the computer, wherein an amount of stored image data in the image database is larger than all amounts of data in the image processing database, such that the storage medium requires a lesser amount of computer processing time for operating the method due to processing less data by separately accessing the image processing database, when compared to accessing data from a combined databased including all data from the image database and data from the imaging processing database.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The presently disclosed embodiments will be further explained with reference to the attached drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments.

(2) FIG. 1A is a block diagram illustrating a method for multiple image registration of images a scene or an object, according to some embodiments of the present disclosure;

(3) FIG. 1B is a schematic illustrating a method that includes some components that may be used for implementing some methods of the present disclosure, according to some embodiments of the present disclosure;

(4) FIG. 1C is a block diagram illustrating a flow diagram of the method of FIG. 1A, according to some embodiments of the present disclosure;

(5) FIG. 1D is a schematic illustrating how a method can collect image data, using different modality sensors, or a single modality sensor to collect images multiple times, according to some embodiments of the present disclosure;

(6) FIG. 1E is a schematic illustrating how another method can collect image data, i.e. X-ray images and CT images, according to some embodiments of the present disclosure;

(7) FIG. 1F is a schematic illustrating how another method can collect image data, i.e. camera images, can include black & white images and color images, etc., according to some embodiments of the present disclosure;

(8) FIG. 2 is a schematic illustrating some methods receiving image data and using mutual information based registration and sparsity-driven post processing to produce images with an improved registration, according to some embodiments of the present disclosure;

(9) FIG. 3A is a registration reference image illustrating a reference, for example, high resolution Pan image from experimentation, according to some experimentation of the present disclosure;

(10) FIG. 3B are un-registered images illustrating a mosaic of 16 un-registered images (including images such as RGB, infra-red, near infra re and short wave infra-red, etc.) from experimentation, according to some experimentation of the present disclosure;

(11) FIG. 3C are registered images illustrating images registered using mutual information, according to some embodiments of the present disclosure;

(12) FIG. 3D are registered images illustrating images registered using mutual information and sparsity-driven post process, according to some embodiments of the present disclosure;

(13) FIG. 4 are images before registration and after registration using different registration methods, according to some embodiments of the present disclosure;

(14) FIG. 5A is a graph illustrating a registration parameter matrix of rotation angle using the mutual information process from experimentation, according to some experimentation of the present disclosure;

(15) FIG. 5B is a graph illustrating a low-rank registration parameter matrix of robust registration parameter of rotation angle using robust decomposition analysis from experimentation, according to some experimentation of the present disclosure;

(16) FIG. 5C is a graph illustrating a sparse registration parameter matrix of rotation angle error from experimentation, according to some experimentation of the present disclosure;

(17) FIG. 6A is a graph illustrating a registration parameter matrix of horizontal shift using the mutual information process from experimentation, according to some experimentation of the present disclosure;

(18) FIG. 6B is a graph illustrating a low-rank registration matrix of robust registration parameter of horizontal shift using robust decomposition analysis from experimentation, according to some experimentation of the present disclosure;

(19) FIG. 6C is a graph illustrating a sparse registration parameter matrix of horizontal shift error from experimentation, according to some experimentation of the present disclosure;

(20) FIG. 7A is a graph illustrating a registration parameter matrix of vertical shift using the mutual information process from experimentation, according to some experimentation of the present disclosure;

(21) FIG. 7B is a graph illustrating a low-rank registration matrix of vertical shift using robust decomposition analysis from experimentation, according to some experimentation of the present disclosure;

(22) FIG. 7C is a graph illustrating a registration parameter matrix of vertical shift error from experimentation, according to some experimentation of the present disclosure;

(23) FIG. 8A is a graph illustrating a comparison of registration errors for rotation angles, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure;

(24) FIG. 8B is a graph illustrating a comparison of registration errors for horizontal shifts, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure;

(25) FIG. 8C is a graph illustrating a comparison of registration errors for vertical shifts, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure; and

(26) FIG. 9 is a block diagram of illustrating the method of FIG. 1A, that can be implemented using an alternate computer or processor, according to some embodiments of the present disclosure.

(27) While the above-identified drawings set forth presently disclosed embodiments, other embodiments are also contemplated, as noted in the discussion. This disclosure presents illustrative embodiments by way of representation and not limitation. Numerous other modifications and embodiments can be devised by those skilled in the art, which fall within the scope and spirit of the principles of the presently disclosed embodiments.

DETAILED DESCRIPTION

(28) FIG. 1A is a block diagram illustrating a flow chart of a method for image processing for multiple image registration of images of a scene or an object, according to embodiments of the present disclosure. Method 100A includes step 110A of receiving image data of the scene or the object, the image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts.

(29) Step 115A of FIG. 1A can include initially estimating registration parameters, including an image rotation angle, an image horizontal shift and an image vertical shift, using pairs of images from the received image data, using an image registration process such as mutual information (MI)-based registration Wherein each pair of images includes a reference image and a floating image.

(30) Step 120A of FIG. 1A can include generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using the image registration process for all pairs of images.

(31) Step 125A of FIG. 1A can include decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration.

(32) Step 130A of FIG. 1A can include using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images.

(33) FIG. 1B is a schematic illustrating a method that includes some components that may be used for implementing some methods of the present disclosure, according to embodiments of the present disclosure. For example, method 100B can include the hardware processor 12B in communication with an external sensor 14B or sensors, that collects data including data such as image data. Further, the sensor 14B can generate image data that may be converted to an image signal. The hardware processor 12B is in communication with a computer storage memory, i.e. memory 10B, such that the memory 10B can include stored data, including algorithms, instructions and other data, that can be implemented by the hardware processor 12B.

(34) Optionally, the hardware processor 12B can be connected to a network 7B, that is in communication with a data source(s) 3B, computer device 4B, a mobile phone device 5B and a storage device 6B. Also optionally, the hardware processor 12B can be connected to a network-enabled server 39B connected to a client device 18B. The hardware processor 12B can optionally be connected to an external memory device 19B, a transmitter 16B. The transceiver 13B can include capabilities as an input interface/output interface, and may receive data including image data such as unregistered images. Also the received image data can include multiple modalities of an image that can be considered to include images such as Pan, RGB, infra-red, near infrared, short-wave infrared, CT images, X-ray images, radar images, etc.

(35) The image data can be gathered by the external sensors 14B as well as acquired by the transceiver 13B or from an external memory device 15B, or some other means of communication either wired or wireless. For example, the image data or other data can be acquired from the network 7B, which is connected Data sources 3B, computer 4B, telephone like device 5B or storage 6B. Contemplated is that the processor 12B can acquire the image data or other data directly or indirectly, e.g., a memory transfer device or a wireless communication like device (not shown). It is possible, that the user interface 17B can have a keyboard (not shown), such that upon receiving an input from a surface of the keyboard of the user interface 17B by a user, data can be obtained/received.

(36) Still referring to FIG. 1B, the processor 12B can be in communication with the network-enabled server 39B, that can be connected to a controller (not shown) to direct the client device(s) 18B. Further, the processor 12B can be connected to a first controller 26B, and/or a second controller 28B, such that the second controller 28B can be connected to device(s) 29B. Wherein the controllers 26B, 28B can be configured to control devices associated with devices related with image processing technology, or like devices, so as to assist in implementing at least one action of some or all embodiments of the present disclosure.

(37) FIG. 1C is a block diagram illustrating a flow diagram of the method FIG. 1A, that is for multiple image registration of images a scene or an object, according to some embodiments of the present disclosure. Method 100C includes step 1, 110C for acquiring data including image data of the scene or the object. The image data includes images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts. A transceiver can acquire the data and image data of the scene or the object. The image data includes images of a single modality or different modalities, collected from different measurements, at different rotation angles, horizontal shifts, or vertical shifts. Contemplated is that the data can be historical image data of the scene and/or object and other historical data.

(38) Step 2, 115C of FIG. 1C can include can include estimating registration parameters, including an image rotation angle, an image horizontal shift and an image vertical shift, using pairs of images from the received image data, using an image registration process that can be one of a mutual information (MI) process, a cross-correlation process or a feature-based process. Wherein each pair of images includes a reference image and a floating image.

(39) Step 3, 120C of FIG. 1C can include generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using the image registration process for all pairs of images.

(40) Step 4, 125C of FIG. 1C is a post processing step or an extra step that is used for adding robustness for registration, which can include decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration.

(41) Step 5, 130C of FIG. 1C can include using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images. Further, discovered from experimentation is that the image registration based methods of the present disclosure demonstrated effective registration of both multi-spectral (MS), multi-modal images, color images, CT & MRI images, as well as radar images. Wherein the image registration based methods of the present disclosure resulted in overcoming the conventional image registration method problems by converging in searching registration parameters, which resulted in registration, whereas the conventional image registration methods failed to converge in searching registration parameters, resulting mis-registration. For example, learned from experimentation is that in order to achieve high spatial and spectral resolution images, an accurate registration between the Pan image and the multi-spectral images can be required. Realized if there is a single pixel or even sub-pixel error in the registration, then the error can lead to significant color or spectral distortion.

(42) Still referring to FIG. 1C, contemplated is that the robust registration of the images can be displayed on a display device, wherein a user reviews the displayed robust registered images, and based upon the user's review, the user conducts an image management action that assists in providing a high fusion performance such as reduced spectral distortion and improved spatial resolution, among other benefits.

(43) FIG. 1D is a schematic illustrating how a method 100D can collect data including image data from a multi-spectral (MS) image device 121D, a panchromatic (Pan) image device 122D and a camera device 123D, of the scene 109D, according to some embodiments of the present disclosure. Embodiment 100D illustrates the sensors 103D capable of multiple sensing features including capturing or collecting data over a wide frequency range beyond the optical bands, including color pictures 103D via the camera 123D, MS image data 101D via the MS device 121D, Pan data 102D via the Pan device 122D of the scene 109D. The images can be taken while the imaging device(s) is stationary or moving. Contemplated is that the imaging devices 103D could be taking pictures of one or more object(s) including a living or non-living object, metal or non-metal object, etc. Exemplary collected image data can include images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene.

(44) In regard to remote sensing, image fusion can be executed between images of different spectra such as Pan image and multispectral or hyper-spectral images, of different modalities such as radar and optical, or of different view-angles. Since the registration problem is a non-convex problem, there is no guarantee that a single image registration method will always succeed in searching optimal registration parameters, especially for multi-spectral or multi-modal images. Thus, to improve the robustness of multiple image registration, one realization included taking each one of the images as a reference and try to register the others. Wherein a robust registration plan of the present disclosure was further realized by combining all the registration information. Further, discovered is that a sparsity-driven method is capable of extracting parameters for robust multi-image registration. In particular, a first step included generating multiple matrices of registration parameters, where each matrix corresponds to a parameter of registration, such as rotation angle, horizontal shift, and vertical shift, etc. Such that, each column of the parameter matrix corresponds to a reference image and each row corresponds to a floating image to be registered. Therefore, each entry of the matrix corresponds to the registration parameter between the floating-reference image pair. Then, a next step included decomposing each parameter matrix into a low-rank matrix of robust registration parameters and a sparse matrix corresponding to parameter errors. Wherein upon testing the method(s) and system(s) by registering multi-spectral images and the panchromatic image with random rigid transformations, experiments demonstrated that the method(s) and system(s) significantly improved performance for registering multi-spectral/modality images viewed from different viewing-angles, when compared to convention image registration methods.

(45) FIG. 1E is a schematic illustrating how another method can collect image data, i.e. X-ray images and CT images, of an object(s) living or not living, according to some embodiments of the present disclosure. Embodiment 100E illustrates the X-ray images and CT images can be collected by an imaging system that can include X-ray sources 103E and 103EE and X-ray detectors (imagers) 104E and 104EE. The two x-ray sources 103E and 103EE may be mounted in fixed positions on the ceiling of an operating room and may be aligned to project imaging x-ray beams from two different angular positions (e.g., separated by 90 degrees) to intersect at a machine center 105E (which provides a reference point for positioning the patient on a treatment couch 106E during treatment) and to illuminate imaging planes of respective detectors 104E and 104EE after passing through the patient. In other embodiments, system 100E may include more or less than two X-ray sources and more or less than two detectors, and any of the detectors may be movable rather than fixed. In yet other embodiments, the positions of the x-ray sources and the detectors may be interchanged. The detectors 104E and 104EE may be fabricated from a scintillating material that converts the X-rays to visible light (e.g., amorphous silicon), and an array of CMOS (complementary metal oxide silicon) or CCD (charge-coupled device) imaging cells that convert the light to a digital image that can be compared with the reference images during the registration process. Exemplary collected image data can include images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the object.

(46) FIG. 1F is a schematic illustrating how another method can collect image data, i.e. camera images, that can include black & white images and color images, etc., according to embodiments of the present disclosure. Embodiment 100F illustrates an imaging device 133F including a camera device 134F. It is contemplated more than one camera 134F can be in the imaging device 133F, or one or more different types of camera devices. Wherein the subject of the image data can include body parts of a human, such as a face, a head 135F, etc. Other subjects of the image data can include non-living objects, living objects, stationary or moving objects, etc.

(47) FIG. 2 is a schematic illustrating some methods receiving image data and using mutual information based registration and sparsity-driven post processing to produce images with an improved registration, according to some embodiments of the present disclosure. Embodiment 200 includes receiving image data 210 of the scene or an objection. The image data 210 can include images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene

(48) Step 215 of FIG. 2 can include estimating registration parameters, through an image processor, using pairs of images from the received image data, wherein each pair of images includes a reference image and a floating image.

(49) Step 220 of FIG. 2 can include generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using an image registration process for all pairs of images.

(50) Mutual Information Based Registration

(51) When registering two images, one image is treated as the reference and the other one as the floating image. Pixel samples of the floating image are then transformed to the reference image such that both images are in the same coordinate system. Let f(s) denote the image intensity in the floating image at position s and r(T.sub.s) the intensity at the transformed position T.sub.s in the reference image, where T.sub. represents the transformation matrix with parameter . The mutual information based registration process determines through the following processes.

(52) The joint image intensity histogram h.sub.(f,r) of the overlapping volume of both images at position a is computed by binning the image intensity pairs)) (f(s), r(T.sub.s)) for all sS.sub., where S.sub. is the set of grid pixels for which T.sub.s falls inside the domain of the reference image. The joint marginal and joint image intensity distributions are obtained by normalization of h.sub.(f,r):

(53) $\begin{array}{cc}{p}_{\mathrm{FR},}\left(f,r\right)=\frac{h_{}\left(f,r\right)}{{}_{f,r}{h}_{}\left(f,r\right)},& \left(1\right)\\ p_{F,}\left(f\right)=\underset{r}{.Math.}{p}_{\mathrm{FR},\left(f,r\right)},& \left(2\right)\\ p_{R,}\left(r\right)=\underset{f}{.Math.}{p}_{\mathrm{FR},\left(f,r\right)}.& \left(3\right)\end{array}$

(54) The Powell algorithm is typically utilized to search the optimal registration parameter * which maximizes the mutual information between f(s) and r(T.sub.s), i.e.,

(55) $\begin{array}{cc}{}^{*}={\mathrm{argmax}}_{}\underset{f,r}{.Math.}{p}_{\mathrm{FR},}\left(f,r\right){\log }_2\frac{p_{\mathrm{FR},}\left(f,r\right)}{p_{F,}\left(f\right)p_{R,}\left(r\right)}.& \left(4\right)\end{array}$

(56) For rigid transformations of 2D images, we have three-degrees of freedom
*={*,x*,y*},(5)
where *, x*, and y* represents rotation angle, horizontal shift, and vertical shift respectively. Once the parameters are determined, image registration can be executed with image transformation.

(57) Step 225 of FIG. 2 can include a post processing step which can include decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration.

(58) Robust Registration by Matrix Analysis

(59) Problem Formulation:

(60) Assume there are N images including a reference image and (N1) unregistered images with objective to register all (N1) images with the reference image. To improve the robustness of registration, then consider all possible pairs of N images for registration and jointly analyze the registration parameters. Let .sub.i,j={.sub.i,j, x.sub.i,j, y.sub.i,j} be the true registration parameter corresponding to the i.sup.th image as a floating image with the j.sup.th image as a reference image. For all possible image pairs, a set of matrices of registration parameters can be formed as follows
A={=[.sub.i,j],X=[x.sub.i,j],Y=[y.sub.i,j]}.(6)

(61) In particular, if the first image is taken as the reference, i.e., j=1, then .sub.i,1={.sub.i,1, x.sub.i,1, y.sub.i,1} is the parameter of the transform matrix of the i.sup.th(i=1, 2, . . . , N) floating image. We define =[.sub.1,1, . . . , .sub.n,1].sup.N, x=[x.sub.1,1, . . . , x.sub.n,1].sup.N, and y=[y.sub.1,1, . . . , y.sub.n,1].sup.N. For rigid transformations, we have .sub.i,j=.sub.i,1.sub.j,1. It is straightforward to verify that
=1.sup.T1.sup.T,(7)
where 1 is a N-dimensional vector with all entries being 1. Similarly, we have X=x1.sup.T1x.sup.T and Y=y1.sup.T1y.sup.T for rigid image transformations. Note that (7) shows that the true registration matrices have rank ()rank(1.sup.T)+rank (1.sup.T)=1+1=2, meaning is a low-rank matrix of rank not greater than 2. We will rely on this property to denoise the registration matrices when registration errors occur.

(62) In practice, the registration parameter matrices A*={*, X*, Y*} acquired by MI-based methods are generally noisy. To achieve robust image registration, one approach is to solve a least-squares problem with an explicit rank-2 constraint to extract the registration parameter. For example, for the rotational angle, we solve
=argmin.sub.*L().sub.F.sup.2,(8)
where
L()=1.sup.T1.sup.T.(9)

(63) The underlying assumption of the least-squares method is that the parameter error is random Gaussian noise, which is however not true in our problem.

(64) Inspired by the robust principal component analysis (RPCA), alternative to the least-squares method we propose a sparsity-driven method to achieve robust registration parameters. It is realized by solving the following problem:

(65) $\begin{array}{cc}\underset{,S}{\min }\frac{}{2}{.Math.{}^{*}-L\left(\right)-S.Math.}_F^2+{.Math.\mathrm{vec}\left\{S\right\}.Math.}_1,& \left(10\right)\end{array}$
where L represents a low-rank matrix and S denotes a sparse outlier matrix. The difference between (10) and RPCA is that here we impose a strict rank 2 structure on L whereas RPCA looks for a general low rank matrix. The low-rankness is satisfied automatically by its definition in (9). Other registration parameters such as the horizontal shift and the vertical shift can be achieved in a similar way.

(66) Step 230 of FIG. 2 can include using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images. Wherein the resulting robust registration of the images provides improved fusion performance such as increased spatial resolution, among other aspects and benefits.

(67) Algorithm:

(68) To solve (10), an alternating minimization method is used. First a step is used to initialize S as S.sup.0=0, then and step is used to update S and sequentially as follows:

(69) For k=1, . . . , K

(70) $\begin{array}{cc}{}^k={\mathrm{argmin}}_{}\frac{}{2}{.Math.{}^{*}-L\left(\right)-S^{k-1}.Math.}_F^2,& \left(11\right)\\ S^k={\mathrm{argmin}}_S\frac{}{2}{.Math.{}^{*}-L\left({}^k\right)-S.Math.}_F^2+{.Math.\mathrm{vec}\left\{S\right\}.Math.}_1.& \left(12\right)\end{array}$

(71) The first updating process in (11) is a standard least-squares problem, which can be, solve using the pseudo-inverse of the project matrix of . The second updating process in (12) is a simplified LASSO problem for which the solution is given by

(72) $\begin{array}{cc}{S}^k=\left({}^{*}-L\left({}^k\right)\right)\max \left(0,1-\frac{1}{.Math.{}^{*}-L\left({}^k\right).Math.}\right),& \left(13\right)\end{array}$
where represents element-wised product.

(73) The iterative algorithm is terminated until a convergence criterion meets such as

(74) $\begin{array}{cc}\frac{{.Math.{}^{k+1}-{}^k.Math.}_2}{{.Math.{}^{k+1}.Math.}_2}<,& \left(14\right)\end{array}$
where E1 is a preset small positive number.

EXPERIMENTATION/SIMULATION

(75) FIG. 3A is a registration reference image illustrating a reference image from experimentation, according to some experimentation of the present disclosure. For example, FIG. 3A is a photo of a high resolution Pan image that is used as a reference image, such that in order to validate at least one method of the present disclosure, the problem of registering MS images with the corresponding Pan image is examined using the mutual information based method.

(76) FIG. 3B are un-registered images illustrating a mosaic of 16 un-registered images from experimentation, according to some experimentation of the present disclosure. For example, FIG. 3B, can have 16 multi-spectral images (including RGB, infrared, near-infra-red, and short wave infra-red, etc.) are considered to register with the reference Pan image for further fusion process. For example, in order to simulate un-registered images, rigid transformations are performed on a well-registered image set, wherein each band image with a random transformation of parameters {tilde over ()}={{tilde over ()}, {tilde over (x)}, {tilde over (y)}} i.i.d drawn from uniform distributions ({tilde over ()}[3, 3] in degree, {tilde over (x)}[50, 50] and {tilde over (y)}[50, 50] in pixel).

(77) FIG. 3C are registered images illustrating images registered using initial estimated registration parameters using mutual information based method, according to embodiments of the present disclosure. As can be viewed, several bands totally failed to register with the panchromatic image due to the divergence of optimization algorithm.

(78) FIG. 3D are registered images illustrating images registered using updated registration parameters using mutual information based method and sparsity-driven post process, according to embodiments of the present disclosure. As can be viewed, all images are well registered with the reference panchromatic image with robust registration performance.

(79) FIG. 4 are images illustrating images before registration and after registration using different registration methods, according to embodiments of the present disclosure. For example, FIG. 4 shows columns 1, 2, and 3, wherein each column corresponds to a spectral band, and Rows A of FIG. 4 shows unregistered MS images.

(80) Row B of FIG. 4 shows the registered images respectively using the MI method. Observed from Row B is that some image registration in the images are not well registered.

(81) Row C of FIG. 4 shows all the registered images using registration parameters estimated by the MI method combined using the least-squares method. Noticed in Row C is that some image registration is getting a little better, but still with a small rotation angle, however, the registration results are not to a level of quality admissible for meeting standards of the present disclosure and many of today's imaging processing technological needs within the image process industry.

(82) Row D of FIG. 4 illustrate registration using the MI-based plus sparsity-driven method, according to some systems and methods of the present disclosure. According to the method of the present disclosure, we set =50, K=500, and =110.sup.4, as can be viewed in Row D, all the MS images are very well registered with the Pan image visually.

(83) FIG. 5A is a graph illustrating registration parameter matrix of rotation angle estimated by the mutual information based method from experimentation, and to be decomposed into a low-rank matrix and a sparse matrix, according to embodiments of the present disclosure.

(84) FIG. 5B is a graph illustrating a low-rank registration parameter matrix of updated rotation angle parameters from experimentation, according to some experimentation of the present disclosure.

(85) FIG. 5C is a graph illustrating a sparse registration parameter matrix of rotation angle errors in a post process from experimentation, according to some experimentation of the present disclosure.

(86) FIG. 6A is a graph illustrating a registration parameter matrix of horizontal shift estimated by the mutual information based method from experimentation, and to be decomposed into a low-rank matrix and a sparse matrix, according to some experimentation of the present disclosure.

(87) FIG. 6B is a graph illustrating a low-rank registration parameter matrix of updated horizontal shift parameters from experimentation, according to some experimentation of the present disclosure.

(88) FIG. 6C is a graph illustrating a sparse registration parameter matrix of horizontal shift errors from experimentation, according to some experimentation of the present disclosure.

(89) FIG. 7A is a graph illustrating a registration parameter matrix of vertical shift estimated by the mutual information based method from experimentation, according to some experimentation of the present disclosure.

(90) FIG. 7B is a graph illustrating a low-rank registration parameter matrix of updated vertical shift parameters from experimentation, according to some experimentation of the present disclosure.

(91) FIG. 7C is a graph illustrating a sparse registration parameter matrix of vertical shift errors from experimentation, according to some experimentation of the present disclosure.

(92) Referring to FIGS. 8A, 8B and 8C, the errors between the registration parameter and the true image transformation parameter are compared.

(93) FIG. 8A is a graph illustrating a comparison of registration errors for rotation angles, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure. Observed is that for the MI based method, some spectral images are not well registered with relative large rotation angle errors. Further, by combining all parameters using the least-squares analysis (MI+L2), the errors are significantly reduced, but still lie in the range of [0, 4] degrees.

(94) However, if the robust method of the present disclosure is used (MI+L2+L1), the rotation angle errors reduce to nearly zeros for all 16 multi-spectral images, leading to accurate and robust image registration.

(95) FIG. 8B is a graph illustrating a comparison of registration errors for horizontal shifts, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure.

(96) FIG. 8C is a graph illustrating a comparison of registration errors for vertical shifts, for mutual information (MI), mutual information plus least-squares (MI+L2) method, and mutual information plus least-squares plus sparsity-driven (MI+L2+L1) method, from experimentation, according to some experimentation of the present disclosure.

(97) As can be understood from FIGS. 8A, 8B and 8C, the systems and methods of the present disclosure provide a robust sparsity-driven image registration method for multiple image registration that solves the conventional divergence problems of mutual information based image registration.

(98) Further, the systems and methods of the present disclosure are examined on registering high-resolution panchromatic image and low-resolution multi-spectral images under rigid transformations with random parameters. Wherein the results show that, the systems and methods of the present disclosure significantly improve the accuracy and robustness for multiple image registration when the mutual information based registration fails to register some images correctly.

(99) Features

(100) According to another embodiment of the present disclosure, computer-implemented method for multiple image registration of images a scene or an object. The computer-implemented method including receiving image data of the scene or the object. The image data can include images collected from different measurements of a single modality or multiple modalities, either at different rotation angles, horizontal shifts, or vertical shifts, of the scene. Estimating registration parameters, through an image processor, using pairs of images from the received image data, wherein each pair of images includes a reference image and a floating image. Generating parameter matrices corresponding to registration parameters including an image rotation angle matrix, an image horizontal shift matrix and an image vertical shift matrix, using an image registration process for all pairs of images. Decomposing each parameter matrix into a low-rank matrix of updated registration parameters and a sparse matrix corresponding to the registration parameter errors for each low-rank matrix, to obtain updated registration parameters for robust registration. Using the updated registration parameters to form a transformation matrix to register the images of the image data with at least one reference image, resulting in robust registration of the images. Wherein the following aspects below are contemplated as configuring one or a combination of modified embodiments of the above embodiment.

(101) According to an aspect of the present disclosure, in view of the above embodiment, the image registration process is one of a mutual information (MI) process, a cross-correlation process or a feature-based process. Another aspect can include each pair of images entered in a parameter matrix correspond to the registration parameter between the floating image and the reference image.

(102) According to an aspect of the present disclosure, in view of the above embodiment, the image data includes multispectral (MS) images, panchromatic (PAN) images, color images, or some combination, of the scene. Further another aspect may include the image data of multiple modalities including one of CT images or MRI images, or both, of the object, and/or the image data includes radar images of the scene. Further still, the image data can include a single modality or different modalities, including one or a combination of multispectral (MS) images, panchromatic (PAN) images, color images, of the scene.

(103) Another aspect can be that the parameter matrices are generated in parallel by the imaging processor, resulting in a less computational time, when compared to a total computation time of generating each parameter matrix of the parameter matrices separately.

(104) According to an aspect of the present disclosure, the low-rank matrix of updated registration parameters is imposed with a strict rank 2 structure on the low-rank matrix. Further, another aspect can be that the registering of the images using the updated registration parameters to form the transformation matrix continues until a predetermined convergence threshold is met, such that the registering ends, resulting in the robust registration of the images. Wherein the predetermined convergence threshold is a positive number.

(105) According to an aspect of the present disclosure, in view of the above embodiment, further comprising, receiving the image data via a transceiver, and storing the image data in a memory. Wherein the memory and the transceiver are in communication with the image processor, such that previously stored in the memory is data, the data includes image historical data of the scene, image historical data of the object, or image historical data of both the scene and the object, the image registration process, along with stored instructions executable by the image processor. Another aspect of the above embodiment can include further comprising, using an input interface to acquire the image data; and storing the image data in an image database of the storage medium. Wherein the storage medium also includes an image processing database having previously stored data, the data includes image historical data of the scene, image historical data of the object, or both the image historical data of the scene and the object, the image registration process, along with stored instructions executable by the computer. Wherein an amount of stored image data in the image database is larger than all amounts of data in the image processing database. Such that the storage medium requires a lesser amount of computer processing time for operating the method due to processing less data by separately accessing the image processing database, when compared to accessing data from a combined databased including all data from the image database and data from the imaging processing database.

Definitions

(106) CT Images:

(107) According to the systems and methods can be understand as computed tomography (CT) which is referred to as X-ray CT, and can be used in the medical technology fields and in industrial imaging applications. For example, medical imaging can include uses as part of part of biological imaging and incorporates radiology which uses the imaging technologies of X-ray radiography, magnetic resonance imaging, medical ultrasonography or ultrasound, endoscopy, elastography, tactile imaging, thermography, medical photography and nuclear medicine functional imaging techniques as positron emission tomography (PET) and Single-photon emission computed tomography (SPECT). Just as in medical imaging, industrial imaging applications can include both nontomographic radiography (industrial radiography) and computed tomographic radiography (computed tomography). But, many other types of CT exist, such as positron emission tomography (PET) and single-photon emission computed tomography (SPECT). X-ray tomography, a predecessor of CT, is one form of radiography, along with many other forms of tomographic and nontomographic radiography. In regard to industrial applications of X-ray CT, some use irradiation to produce three-dimensional internal and external representations of a scanned object, which is used for internal inspection of components. Some key uses for industrial CT scanning include flaw detection, failure analysis, metrology, assembly analysis and reverse engineering applications and non-destructive material analysis.

(108) FIG. 9 is a block diagram of illustrating the method of FIG. 1A, that can be implemented using an alternate computer or processor, according to embodiments of the present disclosure. The computer 911 includes a processor 940, computer readable memory 912, storage 958 and user interface 949 with display 952 and keyboard 951, which are connected through bus 956. For example, the user interface 964 in communication with the processor 940 and the computer readable memory 912 acquires and stores the image data in the computer readable memory 912 upon receiving an input from a surface, keyboard 953, of the user interface 957 by a user.

(109) The computer 911 can include a power source 954; depending upon the application the power source 954 may be optionally located outside of the computer 911. Linked through bus 956 can be a user input interface 957 adapted to connect to a display device 948, wherein the display device 948 can include a computer monitor, camera, television, projector, or mobile device, among others. A printer interface 959 can also be connected through bus 956 and adapted to connect to a printing device 932, wherein the printing device 932 can include a liquid inkjet printer, solid ink printer, large-scale commercial printer, thermal printer, UV printer, or dye-sublimation printer, among others. A network interface controller (NIC) 934 is adapted to connect through the bus 956 to a network 936, wherein image data or other data, among other things, can be rendered on a third party display device, third party imaging device, and/or third party printing device outside of the computer 911. The computer/processor 911 can include a GPS 901 connected to bus 956. Further,

(110) Still referring to FIG. 9, the image data or other data, among other things, can be transmitted over a communication channel of the network 936, and/or stored within the storage system 958 for storage and/or further processing. Further, the time series data or other data may be received wirelessly or hard wired from a receiver 946 (or external receiver 938) or transmitted via a transmitter 947 (or external transmitter 939) wirelessly or hard wired, the receiver 946 and transmitter 947 are both connected through the bus 956. The computer 911 may be connected via an input interface 908 to external sensing devices 944 and external input/output devices 941. The input interface 908 can be connected to one or more input/output devices 941, external memory 906, external sensors 904, which may be connected to a machine like device 902. A controller(s) 942 can be connected to device(s) 943. Further, other computer(s) 945 can be connected to the bus 956). For example, the external sensing devices 944 may include sensors gathering data before-during-after of the collected time-series data of the machine. The computer 911 may be connected to other external computers 942. An output interface 909 may be used to output the processed data from the processor 940. It is noted that a user interface 949 in communication with the processor 940 and the non-transitory computer readable storage medium 912, acquires and stores the region data in the non-transitory computer readable storage medium 912 upon receiving an input from a surface 952 of the user interface 949 by a user. Further, a controller 961 can be connected to the bus 956, to control devices associated with the embodiments of the systems and methods of the present disclosure.

(111) The description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

(112) Specific details are given in the following description to provide a thorough understanding of the embodiments. However, understood by one of ordinary skill in the art can be that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like reference numbers and designations in the various drawings indicated like elements.

(113) In addition, individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the function's termination can correspond to a return of the function to the calling function or the main function.

(114) Furthermore, embodiments of the subject matter disclosed may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, with machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. A processor(s) may perform the necessary tasks.

(115) The above-described embodiments of the present disclosure can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component. Though, a processor may be implemented using circuitry in any suitable format.

(116) Also, the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and/or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

(117) Also, the embodiments of the present disclosure may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts concurrently, even though shown as sequential acts in illustrative embodiments. Further, use of ordinal terms such as first, second, in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.

(118) Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure. Therefore, it is the aspect of the append claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.