A NON-CONTACT VISUAL MONITORING SYSTEM AND METHOD FOR A FLEXIBLE PROTECTIVE STRUCTURE AGAINST ROCKFALL DISASTER

Abstract

The invention relates to the technical field of intelligent construction and maintenance, and relates to a non-contact visual monitoring system and method for a flexible protective structure against rockfall disasters; it comprises a hardware system and a data analysis system, and the hardware system uses a multipoint distributed high-resolution high-speed camera to capture dynamic image sequences of a protective structure under the rockfall impact in a non-contact mode; the data analysis system comprises an impact deformation state full-field tracking module and a multipoint impact large deformation extraction module, and the impact deformation state full-field tracking module captures spatio-temporal changes of impact deformation of the protective system by adopting a full-field optical flow method, constructs a two-dimensional velocity amplitude distribution diagram and performs full-field spatio-temporal tracking of large deformation of the system.

Claims

1. A non-contact visual monitoring system and method for a flexible protective structure against rockfall disasters, characterized in that it comprises a hardware system and a data analysis system, and the hardware system uses a multipoint distributed high-resolution high-speed camera to collect a dynamic image sequence of a protective structure under the rockfall impact in a non-contact mode; the data analysis system comprises an impact deformation state full-field tracking module and a multipoint impact large deformation extraction module for the flexible protective structure, and the full-field deformation state tracking module for the flexible protective structure captures the full-field deformation state of the protective structure under the rockfall impact by adopting a full-field optical flow method; the multipoint impact large deformation extraction module uses the image feature point detection and matching algorithm to simultaneously extract the multipoint dynamic impact large deformation of the protective structure under the rockfall impact and extract its maximum elongation.

2. A non-contact visual monitoring method for a flexible protective structure against rockfall disasters, characterized in that it comprises the following steps: 1) Dynamic image capture of protective structures under rockfall impact disasters; 2) Full-field deformation state tracking of the flexible protective structure; In 2), the specific method for full-field deformation state tracking of the flexible protective structure is as follows: The grayscale of the image captured by the high-speed camera at t and t+dt is equal, namely: $I\left(x,y,t\right)=I\left(x+u\left(x,y\right),y+v\left(x,y\right),t+\mathrm{dt}\right)$ Where, I(x+u(x,y),y+v(x,y),t+dt) is the image grayscale information of the pixel (x,y) at t+dt I(x,y,t) is the grayscale information of the pixel (x,y) at t u(x, y) and v(x, y) are the horizontal and vertical displacement field components of the pixel (x,y) respectively: dt is the time interval; The grayscale information of the pixel (x, y) at t+dt is approximated by the first-order Taylor expansion formula into: $I\left(x+u\left(x,y\right),y+v\left(x,y\right),t+dt\right)=I\left(x,y,t\right)+I_{x}u\left(x,y\right)+I_{y}v\left(x,y\right)+I_{t}dt$ Where, I.sub.x and I.sub.y are the gray gradients of the image in the horizontal and vertical spatial domains, respectively, denoted as I.sub.x=?I(x, y, t)/?x and I.sub.y=?I(x,y,t)/?y: I, is the gray gradient of the image in the temporal domain, denoted as I.sub.t=?I(x,y,t)/?t; Since the brightness between two adjacent frames of images remains unchanged: $I_{x}u\left(x,y\right)+I_{y}v\left(x,y\right)+I_{t}dt=0$ According to the assumption of spatial consistency, the displacement field vectors in both directions can be obtained by using 9 pixels in the 3?3 window, namely: $\left\{\begin{array}{c}u\left(x,y\right)\\ v\left(x,y\right)\end{array}\right\}={\left[\begin{array}{cc}\underset{i=1}{\overset{9}{.Math.}}{I}_{xi}^2& \underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{yi}\\ \underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{\mathrm{yi}}& \underset{i=1}{\overset{9}{.Math.}}{I}_{yi}^2\end{array}\right]}^{-1}\left\{\begin{array}{c}-\underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{ti}\\ -\underset{i=1}{\overset{9}{.Math.}}{I}_{\mathrm{yi}}{I}_{ti}\end{array}\right\}$ Accordingly, the root mean square of the velocity amplitude of the flexible protective system at this point can be calculated, namely: $V_{mag}\left(x,y\right)=\sqrt{{\left(\frac{u\left(x,y\right)}{dt}\right)}^2+{\left(\frac{v\left(x,y\right)}{dt}\right)}^2}$ The root mean square of the velocity amplitude of all pixels in the shooting field of view is formed into a two-dimensional matrix, and a two-dimensional velocity amplitude distribution diagram can be obtained; by comparing the two-dimensional velocity amplitude distribution diagrams in different frames, the whole deformation process of the flexible protective system under the rockfall impact can be tracked; 3) Multipoint impact large deformation extraction of flexible protective structure; In 3), the extraction method of the multipoint impact large deformation of the flexible protective system is as follows: According to the deformation characteristics of the protective system under the rockfall impact, the impact large deformation extraction area is determined, and the image centroids of each area are calculated separately, namely: $C=\left(\frac{m_{10}}{m_{00}},\frac{m_{01}}{m_{00}}\right)$ Where, m.sub.00 is the 0-order moment of the image, expressed as $m_{00}=\underset{x,y}{.Math.}I\left(x,y\right);$ m.sub.10 and m.sub.01 are the 1-order moments of the image, respectively, expressed as $m_{10}=\underset{x,y}{.Math.}\mathrm{xI}\left(x,y\right)$ $\mathrm{and}$ $m_{01}=\underset{x,y}{.Math.}\mathrm{yI}\left(x,y\right);$ Thus, a direction angle between the detection feature point and the image centroid is calculated as: $?=a\tan 2\left(m_{01},m_{10}\right)$ A rotation matrix is formed by the direction angle, and the detected point pairs are rotated to obtain the feature points of the protective structure with rotation invariant characteristics; by comparing the detected feature points in different frames with the detected feature points in the reference frame, the dynamic deformation d.sub.i of the protective structure under the rockfall impact can be obtained, and the correlation coefficients of dynamic deformation curves of different feature points are defined to eliminate mismatching, namely: $R_{\mathrm{ij}}=\frac{\underset{n=1}{\overset{N}{.Math.}}\left(d_{\mathrm{ni}}-{\stackrel{?}{d}}_{?}\right)\left(d_{nj}-{\stackrel{?}{d}}_j\right)}{\sqrt{\underset{n=1}{\overset{N}{.Math.}}{\left(d_{\mathrm{ni}}-{\stackrel{?}{d}}_i\right)}^2}\sqrt{\underset{n=1}{\overset{N}{.Math.}}{\left(d_{\mathrm{nj}}-\stackrel{?}{d_j}\right)}^2}}$ Where, d.sub.ni and d.sub.nj are respectively the values of the dynamic deformation of the feature point i and the feature point j at the moment n; d.sub.i and d.sub.j are respectively the average values of the dynamic deformation of the feature point i and the feature point j: R.sub.ij is the correlation coefficient of the feature point i and the feature point j; The conversion coefficient is used to convert the pixel displacement of the protective structure under the rockfall impact into physical displacement; according to the pinhole camera model and the principle of similar triangles, the following relationship can be obtained: $s=\frac{f}{p?Z}$ Where, f is the focal length of the camera; Z is the actual distance between the position of the camera and the measured object; p is the unit length of the camera sensor; By multiplying this conversion factor by the dynamic pixel displacement at multiple measuring points of the protective structure under the rockfall impact, the real displacement of the protective structure can be obtained, and then its maximum elongation can be extracted and its performance status can be evaluated.

3. A non-contact visual monitoring method for a flexible protective structure against rockfall disasters according to claim 2, characterized in that: in 1), the image capture method is as follows: according to the size of the flexible protective net system, determining multipoint distributed image data collection and arrangement plan, and capturing the dynamic image sequence of the protective net structure under the rockfall impact.

4-5. (canceled)

Description

DESCRIPTION OF DRAWINGS

[0031] FIG. 1 is a flowchart of a non-contact visual monitoring method for the flexible protective structure against rockfall disasters in Embodiment 1;

[0032] FIG. 2 is a visual measurement flow chart of a three-span flexible protective structure in Embodiment 2;

[0033] FIG. 3 is a schematic diagram of the method for detecting full-field optical flow motion in Embodiment 2;

[0034] FIG. 4 is the full-field deformation state identification of the protective net system in Embodiment 2;

[0035] FIG. 5 is the full-field deformation state tracking of the protective net at different time points in Embodiment 2;

[0036] FIG. 6 is a schematic diagram of the multipoint impact deformation extraction of the protective structure in Embodiment 2;

[0037] FIG. 7 is a contrast diagram of the multipoint dynamic deformation curve and theoretical value in Embodiment 2;

EMBODIMENTS

[0038] To further understand the content of the present invention, the present invention will be described in detail regarding the accompanying drawings and embodiments. t should be understood that the embodiments are only for explaining the present invention and not for limiting.

Embodiment 1

[0039] This embodiment provides a non-contact visual monitoring system and method for the flexible protection structure against rockfall disasters, it comprises the hardware system and the data analysis system, and the hardware system uses the high-resolution high-speed camera to capture the dynamic image sequence of the protection system under rockfall impact in a non-contact mode; the data analysis system comprises the deformation state full-field tracking module and the multipoint impact large deformation extraction module for the flexible protection structure, and the deformation state full-field tracking module captures the full-field deformation of the protective net structure under the rockfall impact by adopting a full-field optical flow method; the multipoint impact large deformation extraction module uses the image feature point detection and matching algorithm to simultaneously extract the multipoint dynamic impact large deformation of the protective structure under the rockfall impact and extract its maximum elongation.

[0040] As shown in FIG. 1, this embodiment provides the non-contact visual monitoring method for the flexible protective structure against rockfall disasters, and it comprises the following steps: [0041] 1. Full-field tracking of the impact deformation state of the flexible protective structure: process the image sequence of the protective structure under the rockfall impact captured by the high-speed camera to obtain the spatio-temporal changes of the impact deformation of the protective structure, and use the full-field optical flow method to obtain the full-field velocity vector distribution of the protective net structure, and build the two-dimensional velocity amplitude distribution diagram and track all the frame images captured by the high-speed camera, to capture the full-field and whole-process tracking of the impact deformation state of the protective structure;

Specifically Including the Following Steps:

[0042] 1) Image data collection: the high-speed camera arrangement plan is determined according to the size and installation position of the protective net system. For the protective net structure with a small length, only a single set of visual measurement systems can be arranged. Otherwise, multiple sets of visual measurement systems should be arranged at intervals along the length distribution direction of the protective net; [0043] 2) Full-field deformation state tracking of the protective structure:

[0044] The grayscale of the image captured by the high-speed camera at t and t+dt is equal, namely:

[00012]$I\left(x,y,t\right)=I\left(x+u\left(x,y\right),y+v\left(x,y\right),t+dt\right)$

[0045] Where, I(x+u(x,y),y+v(x,y),t+dt) is the image grayscale information of the pixel (x, y) at t+dt; I(x,y,t) is the grayscale information of the pixel (x, y) at t; u(x,y) and v(x,y) are the horizontal and vertical displacement field components of the pixel (x, y) respectively; dt is the time interval;

[0046] The grayscale information of the pixel (x,y) at t+dt is approximated by the first-order Taylor expansion formula into:

[00013]$I\left(x+u\left(x,y\right),y+v\left(x,y\right),t+dt\right)=I\left(x,y,t\right)+I_{x}u\left(x,y\right)+I_{y}v\left(x,y\right)+I_{t}dt$

[0047] Where, I.sub.x and I.sub.y are the gray gradients of the image in the horizontal and vertical spatial domains, respectively, denoted as I.sub.x=?I(x,y,t)/?x and I.sub.y=?I(x,y,t)/?y; I, is the gray gradient of the image in the temporal domain, denoted as I.sub.t=?I(x,y,t)/?t;

[0048] Since the brightness between two adjacent frames of images remains unchanged:

[00014]$I_{x}u\left(x,y\right)+I_{y}v\left(x,y\right)+I_{t}dt=0$

[0049] According to the assumption of spatial consistency, the displacement field vectors in both directions can be obtained by using 9 pixels in the 3?3 window, namely:

[00015]$\left\{\begin{array}{c}u\left(x,y\right)\\ v\left(x,y\right)\end{array}\right\}={\left[\begin{array}{cc}\underset{i=1}{\overset{9}{.Math.}}{I}_{xi}^2& \underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{yi}\\ \underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{yi}& \underset{i=1}{\overset{9}{.Math.}}{I}_{yi}^2\end{array}\right]}^{-1}\left\{\begin{array}{c}-\underset{i=1}{\overset{9}{.Math.}}{I}_{xi}{I}_{ti}\\ -\underset{i=1}{\overset{9}{.Math.}}{I}_{yi}{I}_{ti}\end{array}\right\}$

[0050] Accordingly, the root mean square of the velocity amplitude of the flexible protection system at this point can be calculated, namely:

[00016]$V_{\mathrm{mag}}\left(x,y\right)=\sqrt{{\left(\frac{u\left(x,y\right)}{\mathrm{dt}}\right)}^2+{\left(\frac{v\left(x,y\right)}{\mathrm{dt}}\right)}^2}$

[0051] The root mean square of the velocity amplitude of all pixels in the shooting field of view is formed into a two-dimensional matrix, and the two-dimensional velocity amplitude distribution diagram can be obtained; by comparing the two-dimensional velocity amplitude distribution diagrams in different frames, the whole deformation process of the flexible protection system under the rockfall impact can be tracked;

[0052] 2. Multipoint impact large deformation extraction of flexible protective structure: According to the size of the protective net structure, determine the displacement extraction points at multiple regions of interests of the protective structure under the rockfall impact, and use the feature point detection algorithm to detect and match the feature points in each area. Due to the characteristics of large deformation of the protective structure, it is necessary to further screen the extracted deformation of each area according to the relevant criteria, to realize the accurate extraction of dynamic deformation curves in each area, calculate the maximum elongation of the protective structure under the rockfall impact and provide key data support for the performance verification of the protective structure under the rockfall impact;

The Specific Method is as Follows:

[0053] According to the deformation characteristics of the protection structure under the rockfall impact, the impact large deformation extraction area is determined, and the image centroids of each area are calculated separately, namely:

[00017]$C=\left(\frac{m_{10}}{m_{00}},\frac{m_{01}}{m_{00}}\right)$

[0054] Where, m.sub.00 is the 0-order moment of the image, expressed as

[00018]$m_{00}=\underset{x,y}{.Math.}I\left(x,y\right);$

m.sub.10 and m.sub.01 are the 1-order moments of the image, respectively, expressed as

[00019]$m_{10}=\underset{x,y}{.Math.}\mathrm{xI}\left(x,y\right)$$\mathrm{and}$$m_{01}=\underset{x,y}{.Math.}\mathrm{yI}\left(x,y\right);$

[0055] Thus, the direction angle between the detection feature point and the image centroid is calculated as:

[00020]$?=a\tan 2\left(m_{01},m_{10}\right)$

[0056] The rotation matrix is formed by the direction angle, and the detected point pairs are rotated to obtain the feature points of the protection structure with rotation invariant characteristics; by comparing the detected feature points in different frames with the detected feature points in the reference frame, the dynamic deformation d.sub.i of the protection structure under the rockfall impact can be obtained, and the correlation coefficients of dynamic deformation curves of different feature points are defined to eliminate mismatching, namely:

[00021]$R_{\mathrm{ij}}=\frac{\underset{n=1}{\overset{N}{.Math.}}\left(d_{\mathrm{ni}}-{\stackrel{?}{d}}_{?}\right)\left(d_{\mathrm{nj}}-{\stackrel{?}{d}}_j\right)}{\sqrt{\underset{n=1}{\overset{N}{.Math.}}{\left(d_{\mathrm{ni}}-{\stackrel{?}{d}}_i\right)}^2}\sqrt{\underset{n=1}{\overset{N}{.Math.}}{\left(d_{\mathrm{nj}}-\stackrel{?}{d_j}\right)}^2}}$

[0057] Where, d.sub.ni and d.sub.nj are respectively the values of the dynamic deformation of the feature point i and the feature point j at the moment n; d.sub.i and d.sub.j are respectively the average values of the dynamic deformation of the feature point i and the feature point j; R.sub.ij is the correlation coefficient of the feature point i and the feature point j;

[0058] The conversion coefficient is used to convert the pixel displacement of the protection structure under rockfall impact into physical displacement; according to the pinhole camera model and the principle of similar triangles, the following relationship can be obtained:

[00022]$s=\frac{f}{p?Z}$

[0059] Where, f is the focal length of the camera; Z is the actual distance between the position of the camera and the measured object; p is the unit length (mm/pixel) of the camera sensor;

[0060] By multiplying this conversion factor by the dynamic pixel displacement at multiple measuring points of the protection structure under rockfall impact, the real displacement of the protection structure can be obtained, and then its maximum elongation can be extracted and its performance status can be evaluated.

Embodiment 2

[0061] According to FIGS. 2?7, a three-span flexible protective structure case illustrates the specific implementation process of the non-contact visual monitoring method for flexible protective structures against rockfall disasters in this embodiment. The specific steps are as follows: [0062] 1) Image data collection of protective structure: the aperture of the test model ring net is 0.3 m, with a total of three links, the width of each link is 10 meters, the cantilever length of the supporting steel column is 4 meters, the size of the single network piece is 4 m?10 m, and the total area is 120 m.sup.2. The vibration image sequence (as shown in FIG. 2) of the middle span of the flexible protective structure under the rockfall is captured by a high-speed camera. To ensure the vibration information covers the entire field of the large protective structure, the method of multipoint distributed simultaneous data acquisition can be adopted. [0063] 2) Full-field deformation state tracking of the protective structure: the spatio-temporal motion detection algorithm (FIG. 3) of the method of the present invention is used to analyze the captured image and obtain its velocity vector distribution under rockfall impact, and then the velocity information of all pixels is used to obtain the two-dimensional velocity amplitude distribution diagram (FIG. 4) of the protective structure, and its full-field deformation state is consistent with the numerical solution. B further extracting the whole process of deformation of the protective structure under the rockfall impact (FIG. 5), it can be seen that when the falling rocks just contact the protective net system (t=0.04 s), the protective net begins to deform, and then the deformation of the protective net increases and gradually becomes narrow (t=0.18 s, 0.35 s); To verify the correctness of the non-contact monitoring of the deformation state of the protective structure, the deformation state of the protective structure under the impact of the falling hammer is simulated by using LS-DYNA (right side of FIG. 5). It can be seen that the deformation state of the protective structure simulated by the finite element software is consistent with the result of the non-contact visual monitoring, and the correctness and effectiveness of the method of the invention are verified. [0064] 3) Multipoint impact large deformation extraction of protective structure: the method of the present invention can extract the multipoint large dynamic impact deformation of protective structure under the rockfall impact from the video shot by the high-speed camera. In this embodiment, the multipoint deformation extraction diagram is shown in FIG. 6. The vertical deformation curve of five measuring points of the flexible protective structure extracted by the feature point detection algorithm is compared with the theoretical value, as shown in FIG. 7. It can be seen that the dynamic deformation curve of each measuring point of the protective structure extracted by the feature point detection algorithm of the present invention is consistent with the theoretical value, but the maximum deformation value generated by each measuring point is different, and the maximum deformation value from the measuring point P1 to the measuring point P5 are ?2.755 m, ?3.244 m, ?6.201 m, ?3.723 m and ?2.473 m respectively. It can be seen that the maximum deformation position of the protective structure under the rockfall impact is at the measuring point P3, and its deformation value is the largest. The measured value is ?6.201 m, and the relative error between it and the theoretical value is 6.41%. The correctness and robustness of the method for extracting deformation and maximum elongation at multiple measuring points of the protective structure are verified.

[0065] The present invention and its embodiments are schematically described above, which is not restrictive, what is shown in the drawings is only one of the embodiments of the present invention, and the actual structure is not limited thereto. Therefore, if as inspired by the present invention, without departing from the inventive concept of the present invention, the structure and embodiments similar to this technical solution designed without creativity shall by those of ordinary skill in the art shall all belong to the protection scope of the present invention.