Joint imaging system based on unmanned aerial vehicle platform and image enhancement fusion method

Abstract

A joint imaging system based on unmanned aerial vehicle platform and an image enhancement fusion method are provided. The system includes a flying unit as a load platform, a shutter control system for controlling the operation of a load camera, a posture control system for recording the movement track and POS data of the flying unit, an airborne image transmission system for communicating with ground equipment, and an onboard computing unit with an image processing module for receiving the output images and performing image processing and fusion.

Claims

1. An image enhancement fusion method based on an unmanned aerial vehicle platform, wherein the method is realized based on a joint imaging system of the unmanned aerial vehicle platform, and the system comprises a flying unit, an airborne computing unit, a shutter control unit, a posture control unit, an airborne image transmission unit, a ten-lens imaging unit and a load pan-tilt interface; wherein the flying unit is used for carrying load equipment to fly according to a predetermined route; the shutter control unit is used for controlling shooting parameters of a load camera; the airborne image transmission unit is used for transmitting thermal infrared images, visible light images and thermal infrared enhanced images after image enhancement and fusion to ground equipment in a wireless way; the posture control unit comprises an inertial measurement unit IMU module and a global positioning system GPS module, wherein the inertial measurement unit IMU module is used for measuring and recording posture parameters of the flying unit and positions of the load camera, and the global positioning system GPS module is used for measuring and recording an accurate geographic information position of the flying unit; the airborne computing unit comprises an image processing module, and the image processing module is used for receiving and storing image data of the load camera and the positions and postures of the load camera recorded by the posture control unit, and processing and fusing images in real time; the ten-lens imaging unit is connected with the shutter control unit, the posture control unit and the airborne image transmission unit through the load pan-tilt interface, and the ten-lens imaging unit is used for collecting image data of color imaging and thermal infrared imaging of a target object, and transmitting the image data to the airborne computing unit; one visible light lens and one thermal infrared lens form a lens group, totally five lens groups are formed, and a distance between the visible light lens and the thermal infrared lens in each lens group is equal; wherein one lens group is arranged at a center position of an unmanned aerial vehicle, and the other four lens groups are arranged in a cross layout, respectively arranged at four corner positions of the unmanned aerial vehicle equidistant from the center position; a lens in the center position of the unmanned aerial vehicle is arranged vertically downward, and the lenses in the four corner positions are inclined to a center at a same angle, and an inclination angle is 30-50; the visible light lens and the thermal infrared lens are all overlapped in terms of imaging ranges in five visual angles of down view, forward view, back view, left view and right view, and an overlapping rate of the imaging ranges in the five visual angles is over 60%, and the imaging ranges of the thermal infrared lenses are comprised in the imaging ranges of the visible light lenses; the method comprises following steps: step 1, collecting thermal infrared images and visible light images of a same object in a same time domain and a same space domain by the ten-lens imaging unit; step 2, performing monocular correction on the visible light lens and the thermal infrared lens respectively: inputting an internal parameter matrix and a distortion coefficient of the visible light lens and the thermal infrared lens into the image processing module respectively, and correcting radial and tangential distortion of the visible light lens and the thermal infrared lens according to following formulas: $\mathrm{Zc}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left(\begin{array}{ccc}{f}_x& 0& u_o\\ 0& f_y& v_o\\ 0& 0& 1\end{array}\right)*\left(\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right)\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]\underset{}{\underset{}{\mathrm{def}}}{\mathrm{KCP}}_W$ $r^2=x^2+y^2$ $\{\begin{array}{c}{x}_{\mathrm{distortion}}=x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+2p_1\mathrm{xy}+p_2\left(r^2+2x^2\right)\\ y_{\mathrm{distortion}}=y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+2p_2\mathrm{xy}+p_1\left(r^2+2y^2\right)\end{array},$ wherein f.sub.x, f.sub.y, u.sub.0 and v.sub.0 are internal parameter coefficients; R is an external parameter rotation matrix; T is an external parameter translation vector; K is an internal parameter matrix; Z.sub.C is a Z-axis coordinate of a calibration point in a camera coordinate system; C is a spatial transformation matrix from a world coordinate system to the camera coordinate system; u and v respectively represent an abscissa and an ordinate of the calibration point in a pixel coordinate system; x and y are an X-axis coordinate and an Y-axis coordinate of the calibration point on a normalized plane of a camera, respectively; X.sub.W, Y.sub.W and Z.sub.W respectively represent an X-axis coordinate, a Y-axis coordinate and a Z-axis coordinate of the calibration point in the world coordinate system; r is a polar coordinate form of the calibration point in the normalized plane; k.sub.1, k.sub.2, k.sub.3, p.sub.1, and p.sub.2 are distortion coefficients; P.sub.W is a coordinate of the calibration point in the world coordinate system; and x.sub.distortion, y.sub.distortion are distortion values of the camera in directions of x, y; def represents an equation of definition; step 3, respectively identifying a same feature point in the visible light images and the thermal infrared images, measuring coordinate values of the feature point in a visible light pixel coordinate system and a thermal infrared pixel coordinate system, according to solved transformation matrices K and C of the visible light pixel coordinate system, the thermal infrared pixel coordinate system and the world coordinate system, performing linear transformation to obtain a rotation matrix and a translation matrix of pixel coordinates of the visible light images and pixel coordinates of the thermal infrared images, so as to complete a spatial geometric registration; wherein a transformation matrix between a visible light camera and a thermal infrared camera satisfies a following formula: $C_I^{-1}\left(\begin{array}{ccc}{f}_{\mathrm{x\_I}}^{-1}& 0& u_1{f}_{\mathrm{x\_I}}^{-1}\\ 0& f_{\mathrm{y\_I}}^{-1}& v_I{f}_{\mathrm{y\_I}}^{-1}\\ 0& 0& 1\end{array}\right)P_{\mathrm{pixel\_I}}=C_V^{-1}\left(\begin{array}{ccc}{f}_{\mathrm{x\_V}}^{-1}& 0& u_V{f}_{\mathrm{x\_V}}^{-1}\\ 0& f_{\mathrm{y\_V}}^{-1}& v_V{f}_{\mathrm{y\_V}}^{-1}\\ 0& 0& 1\end{array}\right)P_{\mathrm{pixel\_V}},$ wherein C.sub.I.sup.1 and C.sub.V.sup.1 are an inverse matrix of the internal parameter matrix of the thermal infrared camera and an inverse matrix of the internal parameter matrix of the visible light camera respectively, and P.sub.pixel_I and P.sub.pixel_V are an abscissa and an ordinate of the calibration point in the thermal infrared pixel coordinate system and the visible light pixel coordinate system respectively; f.sub.x_I.sup.1 and f.sub.y_I.sup.1 are reciprocals of the internal parameter coefficients of the thermal infrared camera; f.sub.x_V.sup.1 and f.sub.y_V.sup.1 are reciprocals of the internal parameter coefficients of the visible light camera; u.sub.I and v.sub.I are coordinates of the thermal infrared camera in the pixel coordinate system; and u.sub.V and v.sub.V are coordinates of visible light in the pixel coordinate system; calculating difference values between abscissas and ordinates of pixels of two adjacent calibration points in the visible light images and the thermal infrared images respectively, wherein the image scale factor .sub.Scalefactor is a ratio of a distance between the two adjacent calibration points in visible light images and a distance between the two adjacent calibration points in thermal infrared images; ${}_{\mathrm{Scalefactor}}=\frac{X_{\mathrm{infrared}}^n-X_{\mathrm{infrared}}^{n-1}}{X_{\mathrm{visible}}^n-X_{\mathrm{visible}}^{n-1}},$ wherein n is a number of checkerboard calibration points; X.sub.infrared.sup.n is a coordinate value of the pixel coordinate system of an n-th calibration point in the thermal infrared camera; X.sub.visible.sup.n is a coordinate value of the pixel coordinate system of the n-th calibration point in the visible light camera; offset vectors x.sub.shift, y.sub.shift are obtained by subtracting a column vector of the same calibration point in the visible light pixel coordinate system from a column vector in the thermal infrared pixel coordinate system of a same calibration point: $\left[\begin{array}{c}{x}_{\mathrm{shift}}\\ y_{\mathrm{shift}}\end{array}\right]=\left[\begin{array}{c}{x}_{v\mathrm{isible}}^n\\ y_{\mathrm{visible}}^n\end{array}\right]-\left[\begin{array}{c}{x}_{\mathrm{infrared}}^n\\ y_{\mathrm{infrared}}^n\end{array}\right],$ step 4, down-sampling the visible light images, unifying image parameters and field of view of the thermal infrared images and the visible light images: unifying resolution of the images, unifying the imaging ranges to be thermal infrared image imaging ranges and unifying spatial scale to be scale parameters of the thermal infrared images, and taking the thermal infrared images as interest areas or target areas, and according to a solved image scale factor .sub.Scalefactor and offset vector $\left[\begin{array}{c}{x}_{\mathrm{shift}}\\ y_{\mathrm{shift}}\end{array}\right],$ correcting and matching the visible light images, and reserving the interest areas or the target areas; multiplying a visible light image V(x, y) by the image scale factor, and scaling to a spatial scale L(u, v) of the thermal infrared images, and realizing a spatial information registration of the thermal infrared images and the visible light images by combining the offset vector:
L(u,v)=V(x,y)*.sub.Scalefactor; step 5, performing a convolution calculation on down-sampled visible light images by using an edge detection operator, and performing an edge detection on the down-sampled visible light images, so as to extracting edge skeleton maps of the visible light images and performing binarization as an edge feature detail matrix; and step 6, adopting a pixel-level fusion method, converting the thermal infrared images into a form of numerical matrix by using an image processing function, and adding the numerical matrix of the thermal infrared images and an edge feature binary image matrix to improve a number of feature points and recognition capability of the thermal infrared images.

2. The method according to claim 1, wherein the ten-lens imaging unit comprises five visible light lenses with same specifications and five thermal infrared lenses with same specifications.

3. The method according to claim 2, wherein the visible light lenses are RGB color cameras and the thermal infrared lenses are uncooled medium-wave infrared cameras.

4. The method according to claim 3, wherein the five lens groups are arranged adjacent to each other at the center position and the four corner positions of the unmanned aerial vehicle, and adjacent visible light lenses and the thermal infrared lenses are arranged in a parallel optical axis structure with a same inclination direction and imaging visual angles.

5. The method according to claim 4, wherein the method is built into the airborne computing unit as the image processing module, and received thermal infrared images and visible light images in the same time domain and space domain are processed and fused in real time.

6. The method according to claim 5, wherein in the step 2, monocular calibration is performed on the visible light lens and the thermal infrared lens to obtain the internal parameter matrix K and the transformation matrix C between the camera coordinate system and the world coordinate system.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The advantages of the above and/or other aspects of the present disclosure will become clearer when the present disclosure is further described in detail with reference to the attached drawings and specific embodiments.

(2) FIG. 1 is a schematic diagram of the system structure provided by the present disclosure.

(3) FIG. 2 is a ten-lens thermal infrared and visible light imaging unit provided by the present disclosure.

(4) FIG. 3 is a schematic diagram of a thermal infrared camera and a visible light camera arranged parallel to the optical axis.

(5) FIG. 4 is a flowchart of an image enhancement fusion method provided by the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

(6) The disclosure adopts a joint imaging system based on an unmanned aerial vehicle platform, as shown in FIG. 1, which includes a flying unit 1, a shutter control unit 2, a posture control unit 3, an airborne image transmission unit 4, a load pan-tilt interface 5, a ten-lens imaging unit 6 and an airborne computing unit 7.

(7) The flying unit 1 is used for carrying load equipment to fly according to a predetermined route; the shutter control unit 2 is used for controlling shooting parameters such as shutter time and exposure time of the load camera; the posture control unit 3 includes an IMU module and a GPS module, where the IMU module is used for measuring and recording the posture parameters of the flying unit and the position of the load camera, and the GPS module is used for measuring and recording the accurate geographic information position of the flying unit; the airborne image transmission unit 4 is used for transmitting thermal infrared images and visible light images to ground equipment in a wireless transmission mode; the airborne computing unit 7 includes an image processing module, which is used for receiving the image data of the load camera and the camera position and posture recorded by the posture control unit, and processing and fusing the received images in real time; the ten-lens imaging unit 6 is connected with the shutter control unit 2, the posture control unit 3 and the airborne computing unit 7 through the load pan-tilt interface 5. The ten-lens imaging unit 6 is used for color imaging and thermal infrared imaging of the target object, collecting image data and transmitting the image data to the airborne computing unit 7.

(8) In order to further illustrate the above-mentioned system of the present disclosure, it will be explained by the following specific embodiments.

(9) As shown in FIG. 2, the structure of the ten-lens imaging unit 6 includes visible light lenses 601 and thermal infrared lenses 602.

(10) The ten-lens imaging unit 6 includes five visible light lenses with the same specifications and five thermal infrared lenses with the same specifications.

(11) The visible light lenses are RGB color cameras and the thermal infrared lenses are uncooled medium-wave infrared cameras.

(12) The visible light lens and the thermal infrared lens are arranged in an equidistant cross layout, and are arranged at the center position and four corners equidistant from the center; the lens at the center position is arranged vertically downwards, and the lenses at the four corners are inclined to the center at the same angle, and the inclination angles are between 30 degrees and 50 degrees.

(13) The imaging ranges of the visible light lens and the thermal infrared lens overlap in five directions, and the overlapping rate of the imaging ranges of the visible light lens and the thermal infrared lens in five visual angles is above 60%, and the imaging ranges of the thermal infrared lens are included in the imaging ranges of the visible light lens.

(14) The visible light lens and the thermal infrared lens are adjacently arranged at the center position and four corners respectively, and the adjacent visible light lenses and the thermal infrared lens have the same inclination direction and imaging visual angle, and are arranged in a parallel optical axis structure, as shown in FIG. 3.

(15) As shown in FIG. 4, the method for realizing image enhancement fusion by using the system include the following steps.

(16) Step 1, collecting thermal infrared images and visible light images of the same object in the same time domain and the same space domain.

(17) Step 2, inputting the internal parameter matrix and distortion coefficient of the thermal infrared camera and the visible light camera into the image processing module respectively, and correcting the radial and lateral distortion of the thermal infrared camera and the visible light camera according to the following formulas:

(18) $\mathrm{Zc}\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left(\begin{array}{ccc}{f}_x& 0& u_o\\ 0& f_y& v_o\\ 0& 0& 1\end{array}\right)*\left(\begin{array}{cc}R& T\\ 0^T& 1\end{array}\right)\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]\underset{}{\underset{}{\mathrm{def}}}{\mathrm{KCP}}_W$$r^2=x^2+y^2$$\{\begin{array}{c}{x}_{\mathrm{distortion}}=x\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+2p_1\mathrm{xy}+p_2\left(r^2+2x^2\right)\\ y_{\mathrm{distortion}}=y\left(1+k_1{r}^2+k_2{r}^4+k_3{r}^6\right)+2p_2\mathrm{xy}+p_1\left(r^2+2y^2\right)\end{array},$

(19) where: f.sub.x, f.sub.y, u.sub.0 and v.sub.0 are internal parameter coefficients; R is an external parameter rotation matrix; T is an external parameter translation vector; K is an internal parameter matrix; Z.sub.C is a Z-axis coordinate of a calibration point in a camera coordinate system; C is a spatial transformation matrix from a world coordinate system to the camera coordinate system; X.sub.W, Y.sub.W and Z.sub.W respectively represent an X-axis coordinate, a Y-axis coordinate and a Z-axis coordinate of the calibration point in the world coordinate system; u and v respectively represent an abscissa and an ordinate of the calibration point in a pixel coordinate system; x and y are an X-axis coordinate and a Y-axis coordinate of the calibration point on a normalized plane of a camera, respectively; r is a polar coordinate form of the calibration point in the normalized plane; k.sub.1, k.sub.2, k.sub.3, p.sub.1, and p.sub.2 are distortion coefficients; and x.sub.distortion, y.sub.distortion are distortion values of the camera in a direction of x, y.

(20) Step 3, respectively identifying the same feature point in the visible light image and the thermal infrared image, measuring the coordinate values of the point in the visible light pixel coordinate system and the thermal infrared pixel coordinate system, and according to the solved transformation matrix between the visible light P.sub.pixel_V and the thermal infrared pixel coordinate system P.sub.pixel_I and the world coordinate system, and according to the formula:

(21) $C_I^{-1}\left(\begin{array}{ccc}{f}_{\mathrm{x\_I}}^{-1}& 0& u_1{f}_{\mathrm{x\_I}}^{-1}\\ 0& f_{\mathrm{y\_I}}^{-1}& v_I{f}_{\mathrm{y\_I}}^{-1}\\ 0& 0& 1\end{array}\right)P_{\mathrm{pixel\_I}}=C_V^{-1}\left(\begin{array}{ccc}{f}_{\mathrm{x\_V}}^{-1}& 0& u_V{f}_{\mathrm{x\_V}}^{-1}\\ 0& f_{\mathrm{y\_V}}^{-1}& v_V{f}_{\mathrm{y\_V}}^{-1}\\ 0& 0& 1\end{array}\right)P_{\mathrm{pixel\_V}},$

(22) where: C.sub.I.sup.1 and C.sub.V.sup.1 are an inverse matrix of the internal parameter matrix of the thermal infrared camera and an inverse matrix of the internal parameter matrix of the visible light camera respectively, and P.sub.pixel_I and P.sub.pixel_V are an abscissa and an ordinate of the calibration point in the pixel coordinate system; f.sub.x_I.sup.1 and f.sub.y_I.sup.1 are reciprocals of the internal parameter coefficients of the thermal infrared camera; f.sub.x_V.sup.1 and f.sub.y_V.sup.1 are reciprocals of the internal parameter coefficients of the visible light camera; u.sub.I and v.sub.I are coordinates of the thermal infrared camera in the pixel coordinate system; and u.sub.V and v.sub.V are coordinates of visible light in the pixel coordinate system.

(23) The rotation matrix and translation matrix of pixel coordinates of visible light images and thermal infrared images are capable of being obtained by linear transformation, and the spatial geometric registration is completed.

(24) The difference values between abscissas and ordinates of pixels of two adjacent calibration points in the visible light images and the thermal infrared images are calculated respectively, where the image scale factor is a ratio of a distance between the two adjacent calibration points in visible light images and a distance between the two adjacent calibration points in thermal infrared images;

(25) ${}_{\mathrm{Scalefactor}}=\frac{X_{\mathrm{infrared}}^n-X_{\mathrm{infrared}}^{n-1}}{X_{\mathrm{visible}}^n-X_{\mathrm{visible}}^{n-1}},$

(26) where n is the number of checkerboard calibration points; the offset vector is the column vector of the same calibration point in the visible light pixel coordinate system and the column vector in the thermal infrared pixel coordinate system of a same calibration point:

(27) 0$\left[\begin{array}{c}{x}_{\mathrm{shift}}\\ y_{\mathrm{shift}}\end{array}\right]=\left[\begin{array}{c}{x}_{v\mathrm{isible}}^n\\ y_{\mathrm{visible}}^n\end{array}\right]-\left[\begin{array}{c}{x}_{\mathrm{infrared}}^n\\ y_{\mathrm{infrared}}^n\end{array}\right]$

(28) Spatial information registration of thermal infrared images and visible light images is realized by combining offset vector.

(29) Step 4, down-sampling the visible light images, unifying imaging specifications and imaging ranges of the thermal infrared images and the visible light images: unifying resolution of the images, unifying the imaging ranges to be thermal infrared image imaging ranges and unifying spatial scale to be scale parameters of the thermal infrared images, taking the imaging ranges of the thermal infrared images as the interest areas, multiplying the visible light image V(x, y) by the image scale factor to scale to the spatial scale L(u, v) of the thermal infrared image, and combining with the offset vector to realize the spatial information registration of the thermal infrared images and the visible light images:

(30) $L\left(u,v\right)=V\left(x,y\right)*{}_{\mathrm{Scalefactor}};$

(31) according to the image scale factor and offset vector, the visible light images are corrected, matched and the interest areas are reserved by using the image processing function.

(32) Step 5, using edge detection operators such as Canny operator and Sabel operator to detect the edge of the down-sampled visible light images, and the edge skeleton maps of the visible light images are extracted and binarized.

(33) Step 6, adding the thermal infrared pseudo-color image matrix and the edge feature binary image matrix by adopting a pixel-level fusion method to obtain thermal infrared images with enhanced texture details, thereby improving the number of feature points of the thermal infrared images and the recognition capability.

Embodiment

(34) The present disclosure will be further described in detail with an embodiment.

(35) (1) The unmanned aerial vehicle is equipped with a ten-lens joint imaging system to image the target, and a group of thermal infrared images and visible light images are obtained, where the thermal infrared image resolution is 640512, the focal length is 25 mm, the visible light image resolution is 40003000, and the focal length is 8 mm.

(36) (2) By extracting the coordinate values of two same calibration points in the thermal infrared image and the visible light image in the respective pixel coordinate systems, the image scale factor .sub.scalefactor=0.34 and the offset vector

(37) $\left[\begin{array}{c}{x}_{\mathrm{shift}}\\ y_{\mathrm{shift}}\end{array}\right]=\left[\begin{array}{c}372\\ 261\end{array}\right]$
are calculated.

(38) (3) Taking the imaging ranges of the thermal infrared images as the interest areas, the visible light images are down-sampled to the scale of the thermal infrared images to obtain the down-sampled visible light images with the image resolution of 640512, and the thermal infrared images are moved into the down-sampled visible light images according to the offset vector to realize spatial registration.

(39) (4) Edge detection is carried out by gradient descent method, and the detailed feature information of the down-sampled visible light image is extracted and binarized to obtain a binary image of texture feature information.

(40) (5) Using the image fusion method based on pixel level, the binary image of texture feature information is fused with the thermal infrared images, and finally the thermal infrared images with texture details are obtained.

(41) The disclosure provides a joint imaging system based on an unmanned aerial vehicle platform and an image enhancement fusion method. There are many ways and means to realize the technical scheme. The above is only the preferred embodiment of the disclosure. It should be pointed out that for ordinary technicians in the technical field, several improvements and retouching can be made without departing from the principle of the disclosure, and these improvements and retouching should also be regarded as the protection scope of the disclosure. All components that are not clear in this embodiment can be realized by existing technology.