PANORAMIC STITCHING METHOD FOR INFRARED IMAGES OF WIND TURBINE BLADE, DEVICE, STORAGE MEDIUM, AND PRODUCT

Abstract

Provided are a panoramic stitching method for infrared images of a wind turbine blade, a device, a storage medium, and a product. The stitching method includes: performing coarse registration on each frame of visible image and infrared image; separately performing background subtraction on a coarsely-registered visible image and a coarsely-registered infrared image to obtain a foreground mask visible image, a foreground mask infrared image, a background-subtracted blade visible image, and a background-subtracted blade infrared image; performing fine registration on the foreground mask visible image and the foreground mask infrared image to obtain a relative displacement; stitching a plurality of frames of blade visible images to obtain a pixel increment; calculating a pixel increment when two adjacent frames of blade infrared images are stitched; and stitching two corresponding adjacent frames of background-subtracted blade infrared images, to obtain an infrared panoramic image of the wind turbine blade.

Claims

1. A panoramic stitching method for infrared images of a wind turbine blade, wherein the stitching method comprises: obtaining a plurality of frames of visible images and infrared images of the wind turbine blade, wherein each frame of visible image corresponds to each frame of infrared image, and two adjacent frames of images overlap each other; performing coarse registration on each frame of visible image and infrared image to obtain an infrared image that is coarsely registered with the visible image; separately performing background subtraction on a coarsely-registered visible image and a coarsely-registered infrared image to obtain a foreground mask visible image, a foreground mask infrared image, a background-subtracted blade visible image, and a background-subtracted blade infrared image; performing fine registration on the foreground mask visible image and the foreground mask infrared image to obtain a relative displacement during the fine registration between the foreground mask visible image and the foreground mask infrared image; stitching a plurality of frames of background-subtracted blade visible images to obtain a pixel increment when two adjacent frames of blade visible images are stitched; calculating, based on the relative displacement and the pixel increment when two adjacent frames of blade visible images are stitched, a pixel increment when two corresponding adjacent frames of blade infrared images are stitched; and stitching, based on the pixel increment when two adjacent frames of blade infrared images are stitched, two corresponding adjacent frames of background-subtracted blade infrared images, to obtain an infrared panoramic image of the wind turbine blade.

2. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein before the obtaining a plurality of frames of visible images and infrared images of the wind turbine blade, the method comprises: adopting a Zhang's calibration method to separately perform monocular calibration on a visible light camera configured to acquire a visible image and an infrared thermal imaging camera configured to acquire an infrared image, to obtain an intrinsic matrix of the visible light camera, an intrinsic matrix of the infrared thermal imaging camera, a distortion correction parameter of the visible image, and a distortion correction parameter of the infrared image.

3. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein the obtaining a plurality of frames of visible images and infrared images of the wind turbine blade is specifically implemented as follows: locking a to-be-inspected wind turbine blade; carrying a visible light camera and an infrared thermal imaging camera on an unmanned aerial vehicle; and controlling the unmanned aerial vehicle to perform rectilinear flight and acquire an image along a single blade and in parallel to a blade surface.

4. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein the performing coarse registration on each frame of visible image and infrared image comprises: performing an un-distortion operation on the visible image based on an intrinsic matrix of a calibrated visible light camera and a distortion correction parameter of the visible image; performing an un-distortion operation on the infrared image based on an intrinsic matrix of a calibrated infrared thermal imaging camera and a distortion correction parameter of the infrared image; calculating a registration transformation matrix and an offset based on an undistorted visible image and an undistorted infrared image, wherein a calculation formula for the registration transformation matrix is as follows: $R_C=\left[\begin{array}{cc}{}_s& 0\\ 0,& {}_s\end{array}\right],{}_s=\frac{B_i^V-B_{i-1}^V}{B_i^l-B_{i-1}^l},$ wherein R.sub.C represents the registration transformation matrix, .sub.s represents a scaling factor, $B_i^V\mathrm{and}{B}_{i-1}^V$ respectively represent coordinates, in a horizontal direction or a vertical direction of the visible image, of centers of two adjacent black squares i and i1 in a chessboard calibration board, and $B_i^l\mathrm{and}{B}_{i-1}^l$ respectively represent coordinates, in a horizontal direction or a vertical direction of the infrared image, of the centers of the two adjacent black squares i and i1 in the chessboard calibration board; and the chessboard calibration board is located in front of the visible light camera and the infrared thermal imaging camera; a calculation formula for the offset is as follows: $x_d=x_{\mathrm{vis}}-x_{\inf },y_d=y_{\mathrm{vis}}-y_{\inf },$ wherein x.sub.d and y.sub.d represent offsets of the infrared image in the horizontal direction and the vertical direction, and (x.sub.vis, y.sub.vis) and (x.sub.inf, y.sub.inf) respectively represent pixel coordinates, in the visible image and the infrared image, of a center of any square in the chessboard calibration board; and calculating, based on the registration transformation matrix and the offset, the infrared image that is coarsely registered with the visible image, and a specific formula is as follows: $\left[\begin{array}{c}{\mathrm{Inf}}_{\mathrm{ix}}^{}\\ {\mathrm{Inf}}_{\mathrm{iy}}^{}\end{array}\right]=\left[\begin{array}{cc}{}_s& 0\\ 0& {}_s\end{array}\right]\left[\begin{array}{c}{\mathrm{Inf}}_{ix}\\ {\mathrm{Inf}}_{\mathrm{iy}}\end{array}\right]+\left[\begin{array}{c}{x}_d\\ y_d\end{array}\right],$ wherein (Inf.sub.ix, Inf.sub.iy) represents pixel coordinates of the undistorted infared image, and $\left({\mathrm{Inf}}_{\mathrm{ix}}^{},{\mathrm{Inf}}^{}\right)$ represents pixel coordinates of the infrared image that is coarsely registered with the visible image.

5. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein the separately performing background subtraction on a coarsely-registered visible image and a coarsely-registered infrared image comprises: separately performing background segmentation on the coarsely-registered visible image and the coarsely-registered infrared image to obtain the foreground mask visible image and the foreground mask infrared image; separately performing an edge contour smoothing operation on the foreground mask visible image and the foreground mask infrared image; and separately setting foreground RGB channel pixels of a smoothed foreground mask visible image to 1, and then multiplying the smoothed foreground mask visible image with the foreground RGB channel pixel set to 1 with the visible image to obtain the background-subtracted blade visible image; and separately setting foreground RGB channel pixels of a smoothed foreground mask infrared image to 1, and then multiplying the smoothed foreground mask infrared image with the foreground RGB channel pixel set to 1 with the coarsely-registered infrared image to obtain the background-subtracted blade infrared image.

6. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein the performing fine registration on the foreground mask visible image and the foreground mask infrared image comprises: separately performing edge detection on the foreground mask visible image and the foreground mask infrared image to obtain a blade edge visible image and a blade edge infrared image; separately performing boundary extraction on the blade edge visible image and the blade edge infrared image to obtain a first boundary coordinate list and a second boundary coordinate list, wherein the first boundary coordinate list is a boundary coordinate list of the blade edge visible image, and the second boundary coordinate list is a boundary coordinate list of the blade edge infrared image; calculating a first width information list based on the first boundary coordinate list, and calculating a second width information list based on the second boundary coordinate list; and calculating, based on the first width information list and the second width information list, the relative displacement during the fine registration between the foreground mask visible image and the foreground mask infrared image, wherein a specific calculation formula is as follows: $h_k=y_i^{},y_i^{}\min .Math.w_i^{}-w_0^{}.Math.,$ wherein h.sub.k represents a relative displacement during fine registration between a k.sup.th frame of foreground mask visible image and foreground mask infrared image, $w_0^{}$ represents a width of an uppermost part in the first width information list, $w_i^{}$ represents a width of an i.sup.th row in the second width information list, and $y_i^{}$ represents a vertical coordinate of an i.sup.th row in the second width information list.

7. The panoramic stitching method for infrared images of a wind turbine blade according to claim 1, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a k1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

8. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 1.

9. A non-transitory computer-readable storage medium on which a computer program/instruction is stored, wherein when the computer program/instruction is executed by a processor, the panoramic stitching method for infrared images of a wind turbine blade according to claim 1 is implemented.

10. The panoramic stitching method for infrared images of a wind turbine blade according to claim 2, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a k1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

11. The panoramic stitching method for infrared images of a wind turbine blade according to claim 3, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a K1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

12. The panoramic stitching method for infrared images of a wind turbine blade according to claim 4, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a k1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

13. The panoramic stitching method for infrared images of a wind turbine blade according to claim 5, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a K1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a K1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

14. The panoramic stitching method for infrared images of a wind turbine blade according to claim 6, wherein a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows: $P_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),$ wherein P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a k1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; h.sub.k represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of visible light foreground mask image and foreground mask infrared image.

15. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 2.

16. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 3.

17. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 4.

18. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 5.

19. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 6.

20. An electronic device, comprising a memory, a processor, and a computer program/instruction stored on the memory, wherein the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade according to claim 7.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] To describe the technical solutions in the present disclosure more clearly, the following briefly describes the accompanying drawings required for describing the embodiments. Apparently, the accompanying drawings in the following description show merely some embodiments of the present disclosure, and those of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.

[0046] FIG. 1 is a flowchart of a panoramic stitching method for infrared images of a wind turbine blade according to an embodiment of the present disclosure;

[0047] FIG. 2 is a schematic diagram of acquisition of a visible image and an infrared image according to an embodiment of the present disclosure, where UAV represents unmanned aerial vehicle;

[0048] FIG. 3 is a schematic diagram of a coarse registration process according to an embodiment of the present disclosure;

[0049] FIGS. 4A-4C are schematic diagrams of an effect of performing background subtraction on a visible image according to an embodiment of the present disclosure;

[0050] FIGS. 5A-5C are schematic diagrams of an effect of performing background subtraction on an infrared image according to an embodiment of the present disclosure;

[0051] FIG. 6 is a schematic diagram of fine registration according to an embodiment of the present disclosure;

[0052] FIG. 7 is a schematic diagram of an effect of stitching visible images according to an embodiment of the present disclosure;

[0053] FIG. 8 is a schematic diagram of an effect of stitching infrared images according to an embodiment of the present disclosure; and

[0054] FIG. 9 shows an infrared panoramic image of a wind turbine blade according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] The following clearly and completely describes the technical solution of the present disclosure with reference to the accompanying drawings in the embodiments of the present disclosure. Apparently, the described embodiments are merely a part rather than all of the embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present disclosure without creative efforts shall fall within the scope of protection of the present disclosure.

[0056] The technical solution of the present disclosure will be described in detail below with reference to specific embodiments. The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be repeatedly described in some embodiments.

[0057] As shown in FIG. 1, a panoramic stitching method for infrared images of a wind turbine blade according to an embodiment of the present disclosure includes the following steps.

[0058] In step 1, as shown at block 102, a visible light camera and an infrared thermal imaging camera are calibrated.

[0059] The visible light camera is configured to acquire a visible image of the wind turbine blade, and the infrared thermal imaging camera is configured to acquire an infrared image of the wind turbine blade. Before dual-spectral images (namely, the visible image and the infrared image) of the wind turbine blade are acquired, monocular calibration is separately performed on the visible light camera and the infrared thermal imaging camera by using a Zhang's calibration method. A specific calibration process is performed on OpenCV as follows: Several images of a chessboard calibration board are respectively taken from different perspectives; then, chessboard corners are determined and found from the images by utilizing the findChessboardCorners function; then, coordinates of each corner of the chessboard calibration board are further determined by using the cornerSubPix function; and finally, an intrinsic matrix of the visible light camera, an intrinsic matrix of the infrared thermal imaging camera, a distortion correction parameter of the visible image, and a distortion correction parameter of the infrared image are separately calculated by calling the calibrateCamera function.

[0060] Due to different locations and focal lengths of lenses of the visible light camera and the infrared thermal imaging camera, an image size and a location of a spatial object shot by the two lenses are also different. Therefore, the chessboard calibration board is placed in front of the two cameras, and the two cameras are ensured to capture a complete calibration board image during imaging. Image sizes of squares on the calibration board and a relative distance between the two adjacent black squares on the calibration board are obtained through calibration to subsequently calculate a coordinate mapping relationship for coarse registration and alignment between heterogeneous images, namely, a coarse registration transformation matrix R.sub.C.

[0061] In step 2, as shown at block 104, a plurality of frames of visible images and infrared images of the wind turbine blade are obtained.

[0062] In the present disclosure, the visible light camera and the infrared thermal imaging camera are carried on an unmanned aerial vehicle to capture the visible image and the infrared image of the wind turbine blade. As shown in FIG. 2, before image acquisition, a to-be-inspected wind turbine blade is firstly locked, and then the unmanned aerial vehicle is controlled to perform rectilinear flight and acquire an image along a single blade and in parallel to a blade surface. Images may be sequentially captured from a root to a tip of the wind turbine blade or images may be sequentially captured from a tip to a root of the wind turbine blade. The visible light camera and the infrared thermal imaging camera are configured to simultaneously acquire images at a same time interval. Therefore, each frame of image includes one frame of visible image and one frame of infrared image. A same frame of visible image and infrared image corresponds to a same part of the wind turbine blade. Two adjacent frames of images overlap each other, to facilitate stitching and ensure integrity of stitched images.

[0063] To simplify a track control solution of the unmanned aerial vehicle, the to-be-inspected wind turbine blade is firstly locked in a vertically downward direction, then a pitch angle of the unmanned aerial vehicle is adjusted to keep the pitch angle of the unmanned aerial vehicle at 0, and then, the unmanned aerial vehicle is controlled to perform rectilinear flight and acquire an image in a direction perpendicular to the ground.

[0064] In step 3, as depicted at block 1-6, coarse registration is performed on each frame of visible image and infrared image to obtain an infrared image that is coarsely registered with the visible image.

[0065] The coarse registration in step 3, background subtraction in step 4 (see block 108), and a fine registration operation in step 5 (see block 110) are performed on both each frame of visible image and infrared image. In a specific implementation of the present disclosure, that coarse registration is performed on each frame of visible image and infrared image includes the following steps.

[0066] In step 3.1, an un-distortion operation is performed on the visible image based on an intrinsic matrix of the visible light camera calibrated in step 1 and a distortion correction parameter of the visible image.

[0067] In step 3.2, an un-distortion operation is performed on the infrared image based on an intrinsic matrix of the infrared thermal imaging camera calibrated in step 1 and a distortion correction parameter of the infrared image, where the un-distortion operation can be implemented by calling the undistort function in OpenCV.

[0068] In step 3.3, a registration transformation matrix and an offset are calculated based on an undistorted visible image and an undistorted infrared image, where a calculation formula for the registration transformation matrix is as follows:

[00012]$\begin{array}{cc}{R}_C=\left[\begin{array}{cc}{}_s& 0\\ 0& {}_s\end{array}\right]& \left(1\right)\end{array}$$\begin{array}{cc}{}_s=\frac{B_i^V-B_{i-1}^V}{B_i^l-B_{i-1}^l},& \left(2\right)\end{array}$

where [0069] R.sub.C represents the registration transformation matrix, .sub.s represents a scaling factor,

[00013]$B_i^V\mathrm{and}{B}_{i-1}^V$ respectively represent coordinates, in a horizontal direction or a vertical direction of the visible image, of centers of two adjacent black squares i and i1 in a chessboard calibration board, and

[00014]$B_i^l\mathrm{and}{B}_{i-1}^l$ respectively represent coordinates, in a horizontal direction or a vertical direction of the infrared image, of the centers of the two adjacent black squares i and i1 in the chessboard calibration board, as shown in FIG. 3. The registration transformation matrix R.sub.C is obtained through calculation based on image sizes of the chessboard calibration board in the visible light camera and the infrared thermal imaging camera, and a relative distance between the two adjacent black squares on the calibration board. Before the registration transformation matrix is obtained through calculation, the scaling factor .sub.s is first calculated based on a pixel difference between coordinates, in the visible image and the infrared image, of centers of any two adjacent black squares in the chessboard calibration board.

[0070] There is a specific relative offset between the visible image and the infrared image. Therefore, an image offset further needs to be calculated based on pixel coordinate locations, in the visible image and the infrared image, of a center of any square in the chessboard calibration board. A specific calculation formula for the offset is as follows:

[00015]$\begin{array}{cc}{x}_d=x_{\mathrm{vis}}-x_{\inf }& \left(3\right)\end{array}$$\begin{array}{cc}{y}_d=y_{\mathrm{vis}}-y_{\inf },& \left(4\right)\end{array}$

where [0071] x.sub.d and y.sub.d represent offsets of the infrared image in the horizontal direction and the vertical direction, and (x.sub.vis, y.sub.vis) and (x.sub.inf, y.sub.inf) respectively represent pixel coordinates, in the visible image and the infrared image, of a center of any square in the chessboard calibration board.

[0072] In step 3.4, the infrared image that is coarsely registered with the visible image is calculated based on the registration transformation matrix and the offset, to implement coarse registration between the visible image and the infrared image.

[0073] In this embodiment, a specific formula for calculating the infrared image that is coarsely registered with the visible image as follows:

[00016]$\begin{array}{cc}\left[\begin{array}{c}{\mathrm{Inf}}_{\mathrm{ix}}^{}\\ {\mathrm{Inf}}_{\mathrm{iy}}^{}\end{array}\right]=\left[\begin{array}{cc}{}_s& 0\\ 0& {}_s\end{array}\right]\left[\begin{array}{c}{\mathrm{Inf}}_{\mathrm{ix}}\\ {\mathrm{Inf}}_{\mathrm{iy}}\end{array}\right]+\left[\begin{array}{c}{x}_d\\ y_d\end{array}\right],& \left(5\right)\end{array}$

where [0074] (Inf.sub.ix, Inf.sub.iy) represents pixel coordinates of the undistorted infrared image, and

[00017]$\left({\mathrm{Inf}}_{\mathrm{ix}}^{},{\mathrm{Inf}}^{}\right)$ represents pixel coordinates of the infrared image that is coarsely registered with the visible image.

[0075] In step 4, background subtraction is separately performed on a coarsely-registered visible image and a coarsely-registered infrared image to obtain a foreground mask visible image, a foreground mask infrared image, a background-subtracted blade visible image, and a background-subtracted blade infrared image.

[0076] As shown in FIGS. 4A-4C and FIGS. 5A-5C, images 140A, 140B, 140C, 150A, 150B, and 150C of the wind turbine blade shot by the visible light camera and the infrared thermal imaging camera includes a great deal of background information. Complex background information affects a subsequent stitching result, and therefore, a background subtraction operation needs to be separately performed on the visible image and the infrared image. In a specific implementation of the present disclosure, that background subtraction is separately performed on the coarsely-registered visible image and the coarsely-registered infrared image includes the following steps.

[0077] In step 4.1, background segmentation is separately performed on the coarsely-registered visible image and the coarsely-registered infrared image to obtain a foreground mask visible image A and a foreground mask infrared image B.

[0078] In step 4.2, an edge contour smoothing operation is separately performed on the foreground mask visible image A and the foreground mask infrared image B.

[0079] In step 4.3, foreground RGB channel pixels of a smoothed foreground mask visible image are separately set to 1, and then the smoothed foreground mask visible image with the foreground RGB channel pixel set to 1 is multiplied with the visible image, making a pixel value of a blade part in the visible image be kept to the original value, to obtain a background-subtracted blade visible image, as shown in FIGS. 4A-4C.

[0080] In step 4.4, foreground RGB channel pixels of a smoothed foreground mask infrared image are separately set to 1, and then the smoothed foreground mask infrared image with the foreground RGB channel pixel set to 1 is multiplied with the coarsely-registered infrared image, making a pixel value of a blade part in the infrared image be kept to the original value, to obtain a background-subtracted blade infrared image, as shown in FIGS. 5A-5C.

[0081] In this embodiment, background segmentation is separately performed on the coarsely-registered visible image and the coarsely-registered infrared image by using a U-net network. The U-net network is an effective semantic segmentation framework. Before the background segmentation is performed on the coarsely-registered visible image and the coarsely-registered infrared image by using the U-net network, a training sample dataset needs to be first constructed to train the U-net network. The training sample dataset includes a plurality of samples, and each sample includes the coarsely-registered infrared image and a label thereof or the coarsely-registered visible image and a label thereof. Pixel-level labels may be obtained by annotating both the coarsely-registered visible image and the coarsely-registered infrared image by using Labelme software.

[0082] To ensure smooth and natural edge contours of the foreground mask visible image A and the foreground mask infrared image B that are obtained through background segmentation, small gaps between the foreground mask visible image A and an image border further need to be eliminated. A 77 full-one matrix serves as a structuring element for a closing operation to perform a morphological closing operation in which dilation is followed by erosion on the foreground mask visible image A to fill and connect gaps on an edge of the image. In this way, narrow interruptions and elongated gullies between the foreground mask visible image A and the image border are bridged, to make transition along boundaries of the wind turbine blade be more natural. Similarly, the morphological closing operation in which dilation is followed by erosion is also performed on the foreground mask infrared image B.

[0083] In step 5, as shown at block 110 in FIG. 1, fine registration is performed on the foreground mask visible image and the foreground mask infrared image to obtain a relative displacement during the fine registration between the foreground mask visible image and the foreground mask infrared image.

[0084] According to the present disclosure, fine registration for dual-spectral images is implemented by searching for and matching blade width information in the foreground mask visible image and the foreground mask infrared image. In a specific implementation of the present disclosure, that fine registration is performed on the foreground mask visible image and the foreground mask infrared image includes the following steps.

[0085] In step 5.1, edge detection is separately performed on the foreground mask visible image and the foreground mask infrared image to obtain a blade edge visible image and a blade edge infrared image.

[0086] In this embodiment, the edge detection is separately performed on the foreground mask visible image and the foreground mask infrared image by using a Canny edge detector, and specifically includes: calculating a gradient magnitude and a direction of an image pixel after smoothing an image using Gaussian filtering. Specifically, convolution with an input image (namely, the foreground mask visible image or the foreground mask infrared image) is separately performed by using a Sobel horizontal operator S.sub.x and a vertical operator S.sub.y, to calculate a gradient magnitude E.sub.x and a direction E.sub.y of each pixel point in the horizontal direction and the vertical direction, so as to further calculate a gradient magnitude E and a direction of each pixel point of the image according to the following specific formulas:

[00018]$\begin{array}{cc}{S}_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right]& \left(6\right)\end{array}$$\begin{array}{cc}{S}_y=\left[\begin{array}{ccc}-1& -2& -1\\ 0& 0& 0\\ 1& 2& 1\end{array}\right]& \left(7\right)\end{array}$$\begin{array}{cc}{E}_x=f\left(x,y\right)*S_x& \left(8\right)\end{array}$$\begin{array}{cc}{E}_y=f\left(x,y\right)*S_y& \left(9\right)\end{array}$$\begin{array}{cc}E=\sqrt{E_x^2+E_y^2}& \left(10\right)\end{array}$$\begin{array}{cc}=\arctan \left(E_y/E_x\right),& \left(11\right)\end{array}$

where [0087] f(x, y) represents an image area covered by a convolution kernel centered at a point (x, y) during calculation of a gradient of each pixel point through convolution.

[0088] Non-edge pixels are filtered out through non-maximum suppression from an image obtained through the edge detection, then, spurious edges are eliminated through double-threshold detection, and finally, edges are connected to obtain a complete blade edge image, that is, a blade edge visible image A and a blade edge infrared image B are separately obtained.

[0089] In step 5.2, boundary extraction is separately performed on the blade edge visible image and the blade edge infrared image to obtain a first boundary coordinate list and a second boundary coordinate list, where the first boundary coordinate list is a boundary coordinate list of the blade edge visible image, and the second boundary coordinate list is a boundary coordinate list of the blade edge infrared image.

[0090] For the blade edge visible image A, with the top-left corner of the image as the origin, the horizontal direction as the X-axis, and the vertical direction as the Y-axis, the image is traversed to obtain points where a pixel value is not zero, and coordinates and corresponding pixel values of the points are stored in a list. The coordinates of the points in the list are judged. If a distance between any two points on each row in the list is less than a pixel threshold (for example, 3 pixels), the two points are considered to be from a same boundary. Otherwise, the two points belong to different boundaries, and the two points are respectively left and right edge points on the corresponding row. In this way, all left and right edge points on each row of the blade are obtained. On each row, coordinates of a point with a maximum pixel value in all points on left and right edges of the blade are respectively obtained, to obtain the first boundary coordinate list, recorded as follows:

[00019]$\begin{array}{cc}{L}_1=\left[\left[y_0,x_{0l},x_{0r}\right],\left[y_1,x_{1l},x_{1r}\right],.Math.,\left[y_i,x_{\mathrm{il}},x_{ir}\right],.Math.,\left[y_N,x_{\mathrm{Nl}},x_{Nr}\right]\right],& \left(12\right)\end{array}$

where [0091] L.sub.1 represents the first boundary coordinate list, x.sub.il represents a horizontal coordinate of a point with a maximum pixel value on the left edge of an i.sup.th row of the blade, x.sub.if represents a horizontal coordinate of a point with a maximum pixel value on the right edge of the i.sup.th row of the blade, y.sub.i represents vertical coordinates (namely, a vertical coordinate of the point with the maximum pixel value on the left edge of the i.sup.th row and the point with maximum pixel value on the right edge of the i.sup.th row) corresponding to x.sub.il and x.sub.ir, and N represents a sequence number of a last row, N+1 rows in total. Points with same vertical coordinates in the image form one row, that is, vertical coordinates of pixel points in each row are the same and are not zero.

[0092] Similarly, for the blade edge infrared image B, the second boundary coordinate list L.sub.2 may be obtained.

[0093] In step 5.3, a first width information list is calculated based on the first boundary coordinate list, and a second width information list is calculated based on the second boundary coordinate list.

[0094] A blade width on each row is calculated based on the first boundary coordinate list L.sub.1 or the second boundary coordinate list L.sub.2 by using a width calculation formula W.sub.i=X.sub.irX.sub.il. In this way, a first width information list

[00020]$L_{\mathrm{vis}}=\left[\left[y_0^{},w_0^{}\right],\left[y_1^{},w_1^{}\right].Math.\left[y_i^{},w_i^{}\right].Math.\left[y_N^{},w_N^{}\right]\right]$

and a second width information list

[00021]$L_{\inf }=\left[\left[y_0^{},w_0^{}\right],\left[y_1^{},w_1^{}\right].Math.\left[y_i^{},w_i^{}\right].Math.\left[y_N^{},w_N^{}\right]\right]$

are obtained, where

[00022]$y_i^{}\mathrm{and}{y}_i^{}$

respectively represent vertical coordinates on the i.sup.th row in the blade edge visible image A and the blade edge infrared image B, and

[00023]$w_i^{}\mathrm{and}{w}_i^{}$

respectively represent widths on the i.sup.th row in the blade edge visible image A and the blade edge infrared image B.

[0095] In step 5.4, a relative displacement during fine registration between the foreground mask visible image A and the foreground mask infrared image B is calculated based on the first width information list and the second width information list.

[0096] As shown in the configuration 160 of FIG. 6, the second width information list L.sub.inf is traversed based on a width (namely, a width on the first row, where a vertical coordinate on the first row is zero)

[00024]$w_0^{}$

on the topmost part in the first width information list L.sub.vis, to obtain a vertical coordinate on the i.sup.th row of the corresponding blade edge infrared image B when a width on the row is closest to

[00025]$w_0^{}.$

In this case, the vertical coordinate on the row is the relative displacement of the blade edge infrared image B relative to the blade edge visible image A during fine registration, and a specific calculation formula is as follows:

[00026]$\begin{array}{cc}{h}_k=y_i^{},y_i^{}\min .Math.w_i^{}-w_0^{}.Math.,& \left(13\right)\end{array}$

where [0097] h.sub.k represents a relative displacement during fine registration between a k.sup.th frame of foreground mask visible image and foreground mask infrared image;

[00027]$w_0^{}$ represents a width of an uppermost part (namely, the first row) in the first width information list;

[00028]$w_i^{}$ represents a width of an i.sup.th row in the second width information list, where the width

[00029]$w_i^{}$ of the row is closest to

[00030]$w_0^{};y_i^{}$ represents vertical coordinates of an i.sup.th row in the second width information list, namely, the vertical coordinate of the row corresponding to the width

[00031]$w_i^{}$ in the second width information list.

[0098] For each frame of foreground mask infrared image (or each frame of infrared image), there is one relative displacement h.sub.k.

[0099] In step 6, as shown at block 112 in FIG. 1, a plurality of frames of background-subtracted blade visible images are stitched to obtain a pixel increment when two adjacent frames of blade visible images are stitched.

[0100] In a specific implementation of the present disclosure, the plurality of frames of background-subtracted blade visible images are stitched by using a normalized cross-correlation (NCC) algorithm for grayscale images. The normalized cross-correlation (NCC) algorithm for grayscale images is the existing technology, and a principle thereof is as follows: Two adjacent frames of images are matched to obtain an optimal matching location, and the two adjacent frames of images are stitched and synthesized based on the optimal matching location. The NCC algorithm is used to obtain a similarity between a template image and a to-be-searched image at different locations by using an evaluation function, and a location with a greatest similarity is the optimal matching location. A calculation formula for the similarity is as follows:

[00032]$\begin{array}{cc}\left(i,j\right)=\frac{{.Math.}_{m=1}^m{.Math.}_{n=1}^n{S}^{i,j}\left(m,n\right)T\left(m,n\right)-\mathrm{mn}\stackrel{\_}{T}\stackrel{\_}{S^{i,j}}}{\sqrt{{.Math.}_{m=1}^m{.Math.}_{n=1}^n{\left(S^{i,j}\left(m,n\right)\right)}^2-\mathrm{mn}{\stackrel{\_}{S^{i,j}}}^2}\sqrt{{.Math.}_{m=1}^m{.Math.}_{n=1}^n{\left(T\left(m,n\right)\right)}^2-\mathrm{mn}{\stackrel{\_}{T}}^2}},& \left(14\right)\end{array}$

where [0101] T(m,n) represents the template image, S.sup.i,j represents a sub-image, covered by the to-be-searched image, of the template image, m and n respectively represent a width and a height of the template image, and T and {square root over (S.sup.i,j)} respectively represent grey average values of the template image and the sub-image.

[0102] The NCC algorithm for grayscale images is time-consuming and poses a high requirement on image contrast, and therefore, an improved NCC algorithm for boundary search is adopted. As shown in the configuration 170 of FIG. 7, the following provides a specific implementation process of the improved NCC algorithm for boundary search.

[0103] The background-subtracted blade visible image is preprocessed with histogram equalization (for details, refer to Sand-Dust Degraded Image Enhancement Algorithm Based on Histogram Equalization and MSRCR [J], Wang Chunzhi, Niu Hongxia, Computer Engineering, 2022, 48(09): 223-229. DOI: 10.19678/j.issn.1000-3428.0062764), to improve the image contrast.

[0104] When two adjacent frames of images are stitched, the subsequent frame is used as the to-be-searched image, and a part is extracted from an overlapping part of the previous frame of image to serve as the template image (as shown in the white box in the to-be-stitched image 1 in FIG. 7). A two-level image pyramid is constructed for both the to-be-searched image and the template image using downsampling.

[0105] Matching is first performed on the top layer of the pyramid image based on the first boundary coordinate list. During matching, a matching region is formed by leftward and rightward expanding by 10 pixels with a blade right boundary of the top layer of a to-be-searched image as a center. The top layer of the template image is traversed in the matching region based on the blade right boundary, to find a correct matching location. A size of the top layer of the pyramid image is of that of the original image, so that computation during matching can be reduced.

[0106] After the initial optimal matching location is determined through top-layer image matching, in the to-be-searched image, a 66-pixel rectangular region centered at the initial optimal matching location is defined corresponding to a next layer of the pyramid layer. Similarly, traversing and searching are performed by using the blade right boundary as a center, to obtain the optimal matching location through fine matching. During traversal search-based matching, a matching speed is increased by performing NCC calculation through searching and matching along a blade boundary.

[0107] Finally, a pixel increment Q.sub.k,k-1 (namely, Q) between two adjacent frames of images is obtained based on the optimal matching location. After a previous frame (namely, (k1).sup.th frame) of image is put at a pixel increment Q location of a next frame (namely, k.sup.th frame) of image, two adjacent frames of blade visible images are stitched.

[0108] In step 7, as shown at block 114 in FIG. 1, a pixel increment when two corresponding adjacent frames of blade infrared images are stitched is calculated based on the relative displacement and the pixel increment when two adjacent frames of blade visible images are stitched.

[0109] In this embodiment, a specific calculation formula for the pixel increment when two adjacent frames of blade infrared images are stitched is as follows:

[00033]$\begin{array}{cc}{P}_{k,k-1}=Q_{k,k-1}+\left(h_k-h_{k-1}\right),& \left(15\right)\end{array}$

where [0110] P.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade infrared image and a k1.sup.th frame of background-subtracted blade infrared image are stitched; Q.sub.k,k-1 represents a pixel increment when a k.sup.th frame of background-subtracted blade visible image and a k1.sup.th frame of background-subtracted blade visible image are stitched; represents the relative displacement during fine registration between the k.sup.th frame of foreground mask visible image and foreground mask infrared image; and h.sub.k-1 represents a relative displacement between a k1.sup.th frame of foreground mask visible image and foreground mask infrared image.

[0111] In step 8, as depicted at block 116 in FIG. 1, two corresponding adjacent frames of background-subtracted blade infrared images are stitched based on the pixel increment when two adjacent frames of blade infrared images are stitched, to obtain an infrared panoramic image of the wind turbine blade.

[0112] As shown in FIG. 8, the k1.sup.th frame of background-subtracted blade infrared image is put at a pixel increment P.sub.k,k-1 location of the k.sup.th frame of background-subtracted blade infrared image based on the pixel increment P.sub.k,k-1 when two adjacent frames of blade infrared images are stitched, to stitch the k1.sup.th frame and the k.sup.th frame of blade infrared images. The stitching result is stitched with a next frame (namely, k+1.sup.th frame) of blade infrared image by using a same method, that is, an k+1.sup.th frame of background-subtracted blade visible image is used as the to-be-searched image, and a part of an overlapping part (namely, a stitching result of the k1.sup.th frame and the k.sup.th frame of blade visible images) of a corresponding visible image stitching result is used as a template image for next stitching, to calculate a corresponding pixel increment P through matching and mapping. The blade infrared images are stitched based on the pixel increment P, and steps 6 to 8 are performed repeatedly to obtain an infrared panoramic image of the wind turbine blade as shown in FIG. 9.

[0113] In another specific implementation of the present disclosure, alternatively, a pixel increment for each stitching may be first obtained during blade visible image stitching, that is, a pixel increment during stitching of a first frame and a second frame of blade visible images, a pixel increment during stitching between a stitching result of the first frame and the second frame of blade visible images and a third frame of blade visible image, a pixel increment during stitching between a previous stitching result (namely, a result obtained by stitching the stitching result between the first frame and the second frame of blade visible images and the third frame of blade visible image) and a fourth frame of blade visible image . . . ; then, a corresponding pixel increment during stitching between blade infrared images is calculated according to the step 7; and finally, two corresponding frames of blade infrared images are stitched according to the step 8.

Embodiment 2

[0114] An embodiment of the present disclosure further provides an electronic device, where the electronic device includes a memory, a processor, and a computer program/instruction stored on the memory, where the processor executes the computer program/instruction to implement the panoramic stitching method for infrared images of a wind turbine blade in this embodiment of the present disclosure.

[0115] Although not shown, the electronic device includes the processor that is capable of performing various suitable actions and processing according to a program stored in a read-only memory (ROM) or a program loaded from a storage part to a random access memory (RAM). The processor may be a multi-core processor, or there may be a plurality of processors. In some embodiments, the processor may include a general-purpose main processor and one or more special coprocessors, for example, a central processing unit, a graphics processing unit (GPU), a neural processing unit (NPU), and a digital signal processor (DSP). Various programs and data required for operations of a device are further stored in the RAM. The processing unit, the ROM, and the RAM are connected to each other through a bus. An input/output (I/O) interface is also connected to the bus.

[0116] The processor and the memory are configured to execute a program/instruction stored in the memory. When the program/instruction is executed by a computer, the methods, steps, or functions described in the embodiments may be implemented.

[0117] Although not shown, an embodiment of the present disclosure further provides a computer-readable storage medium storing a computer program/instruction. When the computer program/instruction is executed by a processor, the panoramic stitching method for infrared images of a wind turbine blade in the embodiments of the present disclosure is implemented.

[0118] The readable medium includes both persistent, non-persistent, removable, and non-removable media, and storage of information may be implemented by any method or technology. The information may be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of a computer storage medium include, but are not limited to, a phase-change random access memory (PRAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), other types of RAMs, a ROM, an electrically erasable programmable read-only memory (EEPROM), a flash memory or another memory technology, a compact disc read-only memory (CD-ROM), a digital versatile disk (DVD) or another optical storage device, a magnetic cassette tape, and a magnetic tape disk storage device or another magnetic storage device or any other non-transmission medium, which can be configured to store information that can be accessed by a computing device. The computer-readable medium, as defined herein, excludes non-transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0119] Although not shown, an embodiment of the present disclosure further provides a computer medium product, including a computer program/instruction. When the computer program/instruction is executed by a processor, the panoramic stitching method for infrared images of a wind turbine blade in the embodiments of the present disclosure is implemented.

[0120] The above are merely specific implementations of the present disclosure, and the protection scope of the present disclosure is not limited thereto. Any modification or replacement easily conceived by those skilled in the art within the technical scope of the present disclosure should fall within the protection scope of the present disclosure.