Global and local binary pattern image crack segmentation method based on robot vision

Abstract

A global and local binary pattern image crack segmentation method based on robot vision comprises the following steps: enhancing a contrast of an acquired original image to obtain an enhanced map; using an improved local binary pattern detection algorithm to process the enhanced map and construct a saliency map; using the enhanced map and the saliency map to segment cracks and obtaining a global and local binary pattern automatic crack segmentation method; and evaluating performance of the obtained global and local binary pattern automatic crack segmentation method. The present application uses logarithmic transformation to enhance the contrast of a crack image, so that information of dark parts of the cracks is richer. Texture features of a rotation invariant local binary pattern are improved. Global information of four directions is integrated, and the law of universal gravitation and gray and roundness features are introduced to correct crack segmentation results, thereby improving segmentation accuracy. Crack regions can be segmented in the background of uneven illumination and complex textures. The method has good robustness and meets requirements of online detection.

Claims

1. A global and local binary pattern image crack segmentation method based on robot vision, comprising: enhancing a contrast of an acquired original image to obtain an enhanced map; using a modified local binary pattern detection algorithm to process the enhanced map and construct a saliency map, wherein the using the modified local binary pattern algorithm further includes: calculating a rotation invariant local binary pattern code of each pixel by comparing with adjacent pixels in four directions, obtaining four rotation invariant local binary pattern detection patterns, and averaging the four rotation invariant local binary pattern detection patterns to obtain the saliency map, wherein the rotation invariant local binary pattern code of each pixel is calculated based on a gray value of a target pixel, gray values of the adjacent pixels, a count of the adjacent pixels, and a relative threshold determined by fitting a Gaussian-like distribution curve to a gray histogram of the enhanced map; using the enhanced map and the saliency map to segment cracks and obtaining a global and local binary pattern automatic crack segmentation method; and evaluating performance of the obtained global and local binary pattern automatic crack segmentation method.

2. The global and local binary pattern image crack segmentation method based on robot vision according to claim 1, wherein the enhancing a contrast of an acquired original image to obtain the enhanced map comprises: using logarithmic transformation to expand a gray range of relatively dark crack defects to obtain the enhanced map.

3. The global and local binary pattern image crack segmentation method based on robot vision according to claim 1, wherein the using the modified local binary pattern detection algorithm to process the enhanced map and construct a saliency map comprises: according to changes of all pixels in the original image in horizontal, vertical and two diagonal directions, calculating a rotation invariant local binary pattern code of each pixel, and obtaining the saliency map according to the rotation invariant local binary pattern code of each pixel.

4. The global and local binary pattern image crack segmentation method based on robot vision according to claim 3, wherein prior to obtaining the saliency map according to the rotation invariant local binary pattern code of each pixel, the method comprises: processing the enhanced map to obtain a gray histogram of the enhanced map.

5. The global and local binary pattern image crack segmentation method based on robot vision according to claim 1, wherein the evaluating performance of the obtained global and local binary pattern automatic crack segmentation method comprises: testing the global and local binary pattern automatic crack segmentation method on an open crack data set, and comparing test results to obtain performance evaluation results.

6. The global and local binary pattern image crack segmentation method based on robot vision of claim 1, wherein the rotation invariant local binary pattern code of each pixel is calculated as follows: $\begin{array}{cc}{C}_d{.Math.}_n=\underset{i=0}{\overset{n-1}{.Math.}}{2}^{i}F\left(g_i-g_t\right)& F\left(x\right)=\{\begin{array}{c}1,x\ge T_R\\ 0,x<T_R\end{array},\end{array}$ wherein C is a local binary pattern code, g.sub.t is the gray value of the target pixel p.sub.t, g.sub.i is a gray value of the i-th adjacent pixel p.sub.i, n is the count of adjacent pixels, d denotes the four directions, d=1 is the upper left diagonal direction, d=2 is the horizontal direction, d=3 is the vertical direction, and d=4 is the lower left diagonal direction, and T.sub.R is the relative threshold determined by fitting a Gaussian-like distribution curve to a gray histogram of the enhanced map.

7. The global and local binary pattern image crack segmentation method based on robot vision of claim 6, wherein the Gaussian-like distribution curve is determined as follows: $y=\left(1/\sqrt{2\pi {\sigma }_H}\right)e^{{\left(x-{\mu }_H\right)}^2/2{\sigma }_H^2}$ where: μ.sub.H is a gray value corresponding to a maximum value of the Gaussian-like distribution curve, σ.sub.His a standard deviation of the Gaussian-like distribution curve; and, the relative threshold T.sup.R=ασ.sub.H, where α is a threshold coefficient.

8. A global and local binary pattern image crack segmentation apparatus based on robot vision, comprising: an image enhancement unit, configured to enhance a contrast of an acquired original image to obtain an enhanced map; a saliency map construction unit, configured to use a modified local binary pattern detection algorithm to process the enhanced map and construct a saliency map, wherein the saliency map construction unit is further configured to: calculate a rotation invariant local binary pattern code of each pixel by comparing with adjacent pixels in four directions, obtain four rotation invariant local binary pattern detection patterns, and average the four rotation invariant local binary pattern detection patterns to obtain the saliency map, wherein the rotation invariant local binary pattern code of each pixel is calculated based on a gray value of a target pixel, gray values of the adjacent pixels, a count of the adjacent pixels, and a relative threshold determined by fitting a Gaussian-like distribution curve to a gray histogram of the enhanced map; a segmentation algorithm acquisition unit, configured to use the enhanced map and the saliency map to segment cracks and obtain a global and local binary pattern automatic crack segmentation method; and a performance evaluation unit, configured to evaluate performance of the obtained global and local binary pattern automatic crack segmentation method.

9. The global and local binary pattern image crack segmentation apparatus based on robot vision of claim 8, wherein the rotation invariant local binary pattern code of each pixel is calculated as follows: $\begin{array}{cc}{C}_d{.Math.}_n=\underset{i=0}{\overset{n-1}{.Math.}}{2}^{i}F\left(g_i-g_t\right)& F\left(x\right)=\{\begin{array}{c}1,x\ge T_R\\ 0,x<T_R\end{array},\end{array}$ wherein C is a local binary pattern code, g.sub.t is the gray value of the target pixel p.sub.t, g.sub.i is a gray value of the i-th adjacent pixel p.sub.i, n is the count of adjacent pixels, d denotes the four directions, d=1 is the upper left diagonal direction, d=2 is the horizontal direction, d=3 is the vertical direction, and d=4 is the lower left diagonal direction, and T.sub.R is the relative threshold determined by fitting a Gaussian-like distribution curve to a gray histogram of the enhanced map.

10. The global and local binary pattern image crack segmentation apparatus based on robot vision of claim 9, wherein the Gaussian-like distribution curve is determined as follows: $y=\left(1/\sqrt{2\pi {\sigma }_H}\right)e^{{\left(x-{\mu }_H\right)}^2/2{\sigma }_H^2}$ where: μ.sub.H is a gray value corresponding to a maximum value of the Gaussian-like distribution curve, σ.sub.His a standard deviation of the Gaussian-like distribution curve; and, the relative threshold T.sub.R=ασ.sub.H, where a is a threshold coefficient.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) In order to more clearly explain the technical scheme of embodiments of the present application, accompanying drawings required for use in description of the embodiments will be briefly described below. It is apparent that the drawings in the following description are some of the embodiments of the present application from which other drawings may be obtained without creative effort by those of ordinary skill in the art.

(2) FIG. 1 is a flow chart of a global and local binary pattern image crack segmentation method based on robot vision provided by an embodiment of the present application;

(3) FIG. 2 is a frame diagram of a global and local binary pattern image crack segmentation method based on robot vision provided by an embodiment of the present application;

(4) FIG. 3 is a schematic diagram of an improved local binary pattern method provided by an embodiment of the present application;

(5) FIG. 4 is a gray histogram of a preprocessed image provided by an embodiment of the present application;

(6) FIG. 5 is a schematic diagram of detection results of a disclosed crack data set by different methods provided by an embodiment of the present application;

(7) FIGS. 6a-6d are parameter adaptive configuration diagrams of GLBPCS provided by an embodiment of the present application; and

(8) FIG. 7 is a histogram of quantitative analysis of detection results of different methods provided by an embodiment of the present application.

DETAILED DESCRIPTION OF EMBODIMENTS

(9) Specific implementations of the present invention will be further explained below with reference to the accompanying drawings. It should be noted here that the description of these implementations is intended to help understand the present invention, but is not intended to limit the present invention. In addition, technical features involved in each of the implementations of the present invention described below can be combined with each other as long as they do not conflict with each other.

(10) As shown in FIG. 1, a flow chart of a global and local binary pattern image crack segmentation method based on robot vision provided by an embodiment includes:

(11) S101, a contrast of an acquired original image is enhanced to obtain an enhanced map.

(12) As shown in FIG. 2a, the gray value of a crack region is lower than that of the background, and a main region of concern is a part with low gray value. Therefore, logarithmic transformation is used to expand the gray range of dark crack defects, so that the image shows more low gray details. According to minimum and maximum gray values of the transformed image, the gray range of the image is mapped to 0-255, so as to realize contrast enhancement between the crack and the background. As shown in FIG. 2b, the enhanced map is denoted as G, and the contrast between the crack and the background is enhanced by preprocessing the original image.

(13) S102, an improved local binary pattern detection algorithm is used to process the enhanced map and construct a saliency map.

(14) Local Binary Patterns (LBP) are mainly used to describe local texture features. In order to better explain a global and local binary pattern saliency map algorithm proposed in the present application, a rotation invariant local binary pattern algorithm is first explained.

(15) In the algorithm, any pixel on the image will be compared with its adjacent pixel set in turn to calculate a local binary pattern code:

(16) $\begin{array}{cc}\begin{array}{cc}C{|}_n=\underset{i=0}{\overset{n-1}{.Math.}}{2}^{i}F\left(g_i-g_t\right)& F\left(x\right)=\{\begin{array}{c}1,x\ge 0\\ 0,x<0\end{array},\end{array}& \left(1\right)\end{array}$

(17) Where: C is a LBP code, g.sub.t is a gray value of a target pixel p.sub.t, g.sub.i is a gray value of the i-th adjacent pixel p.sub.i, and n is the total number of adjacent pixels.

(18) Thus, the rotation invariant local binary pattern code obtained is:
IRC|.sub.n,r=minB2D{Rot[D2B(C),j]}|.sub.j∈[0,n−1]₍₂₎

(19) Where: IRC is a rotation invariant local binary pattern code, B2D denotes conversion of a binary system to a decimal system, D2B means conversion of the decimal system to the binary system, Rot (x, j) means performing j times of cyclic bitwise rotation on the binary value x, and min means taking a minimum value.

(20) Although the rotation invariant LBP depiction of local features is applied to texture detection, it is difficult to use such feature result for defect detection, because detection results are affected by the distance from the target pixel to the adjacent pixel

(21) $r=\sqrt{{\left(p_i-p_t\right)}^2}.$
When r is less than one-half of the shortest distance of the defect or when r is much greater than the shortest distance of the defect, the IRC of the pixel located inside the defect is almost the same as that of the pixel located in the background, which will lead to more false negatives and false positives in the detection results. In most cases, the algorithm can only detect rough defect edges.

(22) As shown in FIG. 3, a hollow circle in the middle denotes a target pixel, a square frame denotes adjacent pixels, and four straight lines composed of square frames denote four different directions: a horizontal direction, a vertical direction and diagonal directions.

(23) In each direction, the target pixel will be compared with n points in its direction, and the comparison rule is consistent with the original local binary pattern. In this paper, taking n=8, the gray range of the image is controlled at [0, 255]. The gray values of these n points can be replaced by a 7×7 rectangular domain gray mean, thereby reducing the influence of noise. The threshold constant value 0 in formula (1) is set as a relative threshold T.sub.R, namely:

(24) $\begin{array}{cc}\begin{array}{cc}{C}_d{.Math.}_n=\underset{i=0}{\overset{n-1}{.Math.}}{2}^{i}F\left(g_i-g_t\right)& F\left(x\right)=\{\begin{array}{c}1,x\ge T_R\\ 0,x<T_R\end{array},\end{array}& \left(3\right)\end{array}$

(25) Where: d denotes the above four directions, d=1 is the upper left diagonal direction, d=2 is the horizontal direction, d=3 is the vertical direction, and d=4 is the lower left diagonal direction.

(26) Thus, each target pixel has four IRC.sub.d values i.e. four rotation invariant local binary pattern detection patterns in different directions as shown in FIG. 2c to FIG. 2f. Its average is taken as a saliency map, as shown in FIG. 2g, which is denoted as S. It can be seen from the figure that the complex background is filtered out, and cracks are obvious, but there is still some discrete noise, so third crack segmentation and correction are needed.

(27) It is worth noting that the relative threshold T.sub.R is determined by a gray histogram of the contrast-enhanced map. As shown in FIG. 4, a largest hump is locked first, which is a Gaussian-like distribution.sup.[15]

(28) $\begin{array}{cc}y=\left(1/\sqrt{2\pi {\sigma }_H}\right)e^{{\left(x-{\mu }_H\right)}^2/2{\sigma }_H^2}& \left(4\right)\end{array}$

(29) Where: μ.sub.H is a gray value corresponding to the maximum value of Gaussian-like curve, and σ.sub.H is the standard deviation of Gaussian-like curve.

(30) According to Gaussian distribution, the probability of background appearance is in the middle of Gaussian distribution, while defects always appear at both ends, so it is necessary to roughly calculate the background gray range. As shown in the diagram, the place below the waist may be affected by noise. Therefore, by deriving the Gaussian-like curve, the position where the first derivative of the left part, namely μ.sub.H−σ.sub.H, of the gray value μ.sub.H corresponding to the maximum value of function y is 0 is obtained. Because the crack is a dark defect and the Gaussian-like distribution is considered to be symmetrical, it is only necessary to calculate one side, and then make the relative threshold T.sub.R=ασ.sub.H

(31) The improved local binary pattern algorithm is designed for crack detection, which not only inherits local characteristics of the original local binary pattern, but also takes into account global information.

(32) S103, cracks are segmented by using the enhanced map and the saliency map, and a global and local binary pattern automatic crack segmentation method is obtained.

(33) In crack segmentation, roundness Cir is used as a feature of crack based on a permeability model.sup.[9]

(34) $\begin{array}{cc}Cir=\frac{A_{Reg}}{\pi D_{Reg}^2}& \left(5\right)\end{array}$

(35) Where: A.sub.Reg denotes the number of pixels of a region Reg, and D.sub.Reg denotes the maximum distance from the center to the edge of the region.

(36) If the region is circular, Cir=1; if the region is hollow or very long, Cir<1. At the same time, based on the law of universal gravitation, it is considered that the larger the area of crack region, the larger the influence range of its endpoint on nearby points, that is

(37) $\begin{array}{cc}{R}_{Reg}=\sqrt{k{A}_{Reg}}& \left(6\right)\end{array}$

(38) Where: R.sub.Reg denotes the radius of influence of the region, and k is a coefficient of change.

(39) Based on the above characteristics, the following crack segmentation program is designed:

(40) Input: Enhanced map G, saliency map S, image width and height W and H, and parameter k.

(41) Output: Output Image O.

(42) 1) Initialization: making Image I=S, target area A.sub.T=H×W, and flag Flag=0.

(43) While (True):

(44) 2) A 0 Matrix M of H×W is generated.

(45) 3) If Flag=0, threshold segmentation is performed on I, a potential crack region is extracted, the mean value μ.sub.I and standard deviation σ.sub.I of I are calculated; when the pixel gray value in I is greater than or equal to μ.sub.I+λσ.sub.I, the corresponding position in O is set to 1; otherwise, it is set to 0; if Flag=1, O=I.

(46) 4) The area of the crack region is calculated: A.sub.crack=Σ.sub.O=1.

(47) 5) If the condition A.sub.crack=A.sub.T or k=0 is met, the result O is output; otherwise, A.sub.T=A.sub.crack and step 6) is carried out.

(48) 6) Connectivity analysis is carried out on the region with a median of O of 1: ∪Reg.sub.i=Σ.sub.O==1. If Flag=1, the region with the area greater than or equal to 3 and roundness less than or equal to 0.4 will be preserved, that is:

(49) $Re{g}_i=Re{g}_i{|}_{A_{\mathrm{Regi}}\ge 3\&\mathrm{Ci}{r}_{\mathrm{Regi}}\le 0.4}.$

(50) 7) Skeleton extraction is performed on Reg, and all its endpoints EP.sub.i=(x.sub.1, y.sub.1), . . . , (x.sub.j, y.sub.j), . . . , (x.sub.m, y.sub.m) are obtained, where m is the sum of the number of each skeleton endpoint.

(51) 8) The gray average value Gm.sub.Regi in G for each region and the influence radius for each region are calculated according to formula (6)

(52) $R_i=\sqrt{k{A}_{{\mathrm{Reg}}_i}}.$

(53) 9) For the neighborhood pixel set of each endpoint (x.sub.j, y.sub.j) in Reg.sub.i pixels satisfying the gray scale range of [Gm.sub.Regi−δ, Gm.sub.Regi+δ] in G are merged with Reg.sub.i.

(54) 10) If the corresponding region of the expanded Reg.sub.i satisfies Cir.sub.Regi≤0.1, the relevant region is set to 1.

(55) 11) I=M and k-- are set, and step 2) is returned to.

(56) End While

(57) S104, performance of the obtained global and local binary pattern automatic crack segmentation method is evaluated.

(58) In order to evaluate the performance of the proposed global and local binary pattern automatic crack segmentation method (GLBPCS), tests are carried out on a public crack data set (https://github.com/cuilimeng/CrackForest), detection results are compared with the controllable evidence filtering method (SEF method), the width transformation method (WT method), the permeating algorithm (P method), and the improved U-shaped convolution neural network method (Unet++ method). Performance indexes are respectively expressed by coincidence rate D.sub.c, (that is, coincidence rate between a real value and a predicted value), accuracy rate P.sub.r (that is, a ratio of a correct number of defect pixels to a total predicted defect value), recall rate R.sub.e (that is, ratio of a correct number of defect pixels to a total real defect values) and running time.

(59) $\begin{array}{cc}{D}_c=\frac{2TP}{2TP+FN+FP}& \left(7\right)\end{array}$

(60) 0$\begin{array}{cc}{P}_r=\frac{TP}{TP+FP}& \left(8\right)\end{array}$$\begin{array}{cc}{R}_e=\frac{TP}{TP+FN}& \left(9\right)\end{array}$

(61) Where: TP denotes the number of pixels in which both the true value and the predicted value are defects, FP denotes the number of pixels in which the true value is background but the predicted value is defect, and FN denotes the number of pixels in which the true value is defect but the predicted value is background.

(62) The disclosed crack data set contains 118 480×320 three-channel images, as shown in FIG. 5, where the detection results of original image (FIG. 5a), true value (FIG. 5b), SEF (FIG. 5c), WT (FIG. 5d), P (FIG. 5e), Unet++ (FIG. 5f) and GLBPCS (FIG. 5g) are respectively from top to bottom. Different types of crack shapes are selected for analysis in the original image. For example, FIGS. 4a1, 5a4 and 5a6 all show single-branch cracks, FIGS. 5a3 and 5a5 show multi-branch cracks, and FIG. 5a2 shows reticular cracks. At the same time, the image also includes the influence of noise, such as existence of paint in FIG. 5a3 and existence of oil spots in FIG. 5a6. Therefore, the data set is equally divided into a training set and a test set, and GLBPCS adopts an adaptive method to configure parameters specifically in a way of selecting 5 samples arbitrarily from the training set to calculate D.sub.c to adjust each parameter to obtain optimal results. The program automatically generates four curve graphs, namely, the relationship between threshold coefficient α, standard deviation coefficient λ, change coefficient k, allowable gray scale change range δ and coincidence rate D.sub.c. As shown in FIG. 6, the program automatically selects the values corresponding to the average maximum D.sub.c, namely α=3, λ=1, k=7, and δ=11. In view the balance between algorithm accuracy and running time, k is fine-tuned to 5.

(63) In an embodiment, after the results are obtained, experimental results need to be evaluated.

(64) The present application evaluates the experimental results from two aspects: visual comparison and quantitative analysis.

(65) For the SEF method, as shown in FIG. 5c, the method can detect various types of cracks, but its detection effect is limited by the size of a filter core. When the core is large, the method can detect wider cracks, while when the core is small, it can detect narrower cracks. For the WT method, as shown in FIG. 5d, the detection effect of this method is poor, and it is almost impossible to identify the characteristics of cracks because the method relies on the edge extraction effect. For the position where the transition between cracks and background is smooth, the edge extraction may fail, resulting in the failure or error of edge point pair search. For the P method, as shown in FIG. 5e, the method has a general effect on defect extraction in complex background, and can achieve better results only when the background is relatively pure and there is no too much noise interference. For the Unet++ method, as shown in FIG. 5f, the method can basically completely extract the crack region, mainly due to millions of sample parameters. Nevertheless, it also introduces more false positive regions. For the GLBPCS method proposed in this paper, as shown in FIG. 5g, the method can detect the crack region well without introducing too many false positives, mainly because the improved LBP algorithm combines global and local information, and the modified algorithm introduces the theory of universal gravitation and roundness features, so that the crack features can be extracted well.

(66) FIG. 7 shows the quantitative analysis results of different crack detection methods. Because the true value is artificially marked and there are subjective factors, the detection effect of the algorithm is considered to be better when the three indexes D.sub.c, P.sub.r and R.sub.e are greater than or equal to 0.5. As shown in the diagram, the three indexes of the proposed method are all over 0.6, and the detection effect is better than other methods. Although its detection speed is slower than those of the SEF method and WT method, it can still meet the requirements of online detection within 1 s.

(67) The present application provides an automatic segmentation method for cracks in a concrete structure. Logarithmic transform is used to enhance the contrast of crack image, which makes information of dark parts of the cracks richer. Texture features of a rotation invariant local binary pattern are improved. Global information of four directions is integrated to generate the crack saliency map. Finally, the law of universal gravitation and gray and roundness features are introduced to correct crack segmentation results, thereby improving segmentation accuracy. With the method, crack regions can be segmented in the background of uneven illumination and complex textures. The method has good robustness and meets requirements of online detection.

(68) In an embodiment, the present application further provides a global and local binary pattern image crack segmentation apparatus based on robot vision, including:

(69) an image enhancement unit, configured to enhance a contrast of an acquired original image to obtain an enhanced map;

(70) a saliency map construction unit, configured to use an improved local binary pattern detection algorithm to process the enhanced map and construct a saliency map;

(71) a segmentation algorithm acquisition unit, configured to use the enhanced map and the saliency map to segment cracks and obtain a global and local binary pattern automatic crack segmentation method; and

(72) a performance evaluation unit, configured to evaluate performance of the obtained global and local binary pattern automatic crack segmentation method.

(73) In an embodiment, the present application further provides a computer device which includes a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the computer program, the global and local binary pattern image crack segmentation method based on robot vision according any one of the above embodiments is realized.

(74) In an embodiment, the present application further provides the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is enabled to execute the global and local binary pattern image crack segmentation method based on robot vision according to any one of the above embodiments.

(75) Implementations of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described implementations. Various changes, modifications, substitutions and modifications made by those skilled in the art to these implementations without departing from the principles and spirit of the present invention still fall into the scope of protection of the present invention.