SEWER PIPE INSPECTION AND DIAGNOSTIC SYSTEM AND METHOD

Abstract

A method is disclosed for interrogating enclosed spaces such as sewers and the like by commanding a camera to travel through the enclosed space while transmitting the video feed from the camera to a remote location for viewing and processing. The processing involves image manipulation before analyzing frames of the video using a neural network developed for this task to identify defects from a library of known defects. Once a new defect is identified, it is inserted into the model to augment the library and improve the accuracy of the program. The operator can pause the process to annotate the images or override the model's determination of the defect for further enhancement of the methodology.

Claims

1. A method for interrogating an integrity of an inner surface of a wall of an enclosed space, comprising the steps of: commanding a video camera to move along the enclosed space; communicating a video feed from the camera to a remote location; extracting frames of the video feed for detecting a presence of defects; processing the extracted frames using an image processing method; using a neural network model to analyze frames against known defects; alerting an operator when the neural network model identifies a defect; and incorporating the newly detected defect into the neural network model to improve future model performance.

2. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 1, wherein the processing includes removing a central portion of the extracted frame and analyzing a remaining portion of non-extracted frame for defects.

3. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 2, wherein the processing further comprises applying a color correction and a resizing of the image.

4. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 3, wherein the operator may introduce feedback of an identified defect, said feedback including a confirmation or negation of the identified defect.

5. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 2, wherein the enclosed space is a sewer pipe.

6. The method for interrogating an integrity of an inner surface of a wall of an enclosed space claim 1, wherein the commanding step is preceded by creation of a model using a convolutional neural network using previously extracted and processed images of enclosed spaces.

7. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 1, wherein the neural network model further classifies the detected defect as a particular type.

8. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 3, wherein the processing further comprises edge enhancement of the detected defect prior to resizing.

9. The method for interrogating an integrity of an inner surface of a wall of an enclosed space of claim 1, wherein a computer processing is enhanced by removing a portion of the image prior to applying the model to the frame, and where the monitor displays the image without the removed portion of the image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0013] FIG. 1 is a photograph depicting a sewer pipe with no discernable defects;

[0014] FIG. 2 is a photograph depicting a sewer pipe with a defect;

[0015] FIG. 3 is a processed image that eliminates the non-essential data;

[0016] FIG. 4 is a flow chart of the training phase of the methodology; and

[0017] FIG. 5 is a flow chart of the autopipe phase of the methodology.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The present invention uses both hardware and software to inspect, diagnose, and catalog defects in subterranean pipes such as sewer systems and the like. The use of automated motorized cameras using closed circuit television that are controlled above ground in video surveillance vans or other remote stations are well known in the art. This invention improves upon such systems by making the task of reviewing the live feed of camera more effective and by iteratively improving the recognition of the presence and type of defects through a learning mode of the software.

[0019] The system is divided into two components: a training component and a runtime component. Training is executed in a Cloud based computing environment, whereas the runtime element of the invention occurs while the operator analyzes the video feed for defects as the camera moves along the pipe.

[0020] In the training step, the software analyzes images of defects in sewage pipes in order to learn how to differentiate between image frames containing visible defects and frames where no defects are visible. This is accomplished by annotating visible defects in a database of videos and having the software recognize those annotated defects as a catalog of all possible defects, and anything not annotated is interpreted by the software as not being a defect. This “training” aspect of the invention is ongoing and allows the process to continuously improve and become more efficient as the program learns what imagery is a defect and what is not. As a defect appears in the video, it is labeled when it first appears in the center of the frame far from the camera. This ensures the potential early detection of the defect, which is important to the invention. If a defect is not detected early, the camera may in many cases need to be stopped, backed up into position, and restarted again. This process needs to be avoided if the task is to carried out in an efficient and expedient manner. Once the defects are identified by the operator and the type annotated, the images are extracted using a computer vision program and store the image on a storage disk.

[0021] To extract the images, a three step process is followed. First, the image is cropped so that the center of the pipe is not displayed (e.g., the horizon inside the pipe), focusing on the near field image adjacent the camera. Since the center view of the image is typically dark and does not yield usable information, the excision of this portion of the image serves two purposes: a) it focuses the operator's attention on the portion of the image where defects can actually be detected and evaluated; b) and it reduces the computer processing on the image by eliminating a large portion the image, allowing the processing power to be concentrated on the remaining portion of the image. After the image has been cropped, a color correction is applied to the image to emphasize the discolorization or contrast that results from a defect as opposed to other markings and debris on the wall of the pipe that could appear to be a defect. Once the colorization processing has occurred, the edge detection algorithm focuses on the edges of the defect and creates an outline of the defect along the edge. This colorized outline is resized and stored in a defect database used to train the system for optimization.

[0022] The above-identified database is used to train a convolutional neural network (CNN), where the model is trained to detect whether a defect exists in a camera feed image. If the CNN model determines that a defect does exist, a second model can be used to classify the type of defect from among a set of classifications of defects previously established by the model. Since neural network training is very computationally taxing and therefore expensive, this step is best to a powerful computing unit or cloud computing facility. This is because the performance of the training step depends on the amount of processed images—the more images that are cataloged and the more types of defects that are recognized by the system, the more accurate the model will be at detecting and evaluating defects in real time.

[0023] Once the training phase of the invention is at least reached a level where the model is operational, the runtime phase of the invention can be initiated. In the runtime step, the results of the training phase, namely the trained neural network model, is employed in real time to evaluate a camera feed of a sewer system. The system runs on a computing device typically in an inspection vehicle under the supervision of an operator. A monitor displays a camera feed of a sewer pipe, such as that shown in FIG. 1, as it moves along the pipe. The camera is mounted on a remote controlled cart that illuminates the pipe downfield while capturing high resolution images of the pipe's interior as it moves from one end of the pipe to the other. Software processes the displayed image in real time, and the operator controls both the camera and the cart moving along the pipe. Each image captured by the camera is processed by the software and compared by the model to the library of defects to determine if a defect is present in the field of view.

[0024] As the images are received, the processing detailed above is applied to the images. As shown in FIG. 2, at some point a defect will be identified. The software crops the image to exclude the enter portion of the image, that is the portion shown in FIG. 3 is excluded from the image to concentrate on the remaining portion of the image. The cropped image is subjected to color correction and edge enhancement, and then the image is resized. The software processes the image by passing it through the model and the model returns a determination whether a defect is detected. If a defect is detected, the defect is characterized by type according to the software, and this defect is stored and added to the database for future determination of defects. If no defects are detected, the program provides no input as the camera continues to provide images to the monitor for the operator. Every time the camera moves, the software continues to analyze the frames it receives according to the model for known defects. The operator can also pause the program, causing the process to continue without processing any new images and without flagging any defects in the video stream.

[0025] Operators can override or add input to the determinations made by the model to correct or revise decisions made by the software. That is, if the program incorrectly identifies a defect that the operator concludes is an artifact, debris, marking, or other discoloration on the pipe wall, the operator will characterize the image as a non-defect to further improve the model. The CNN receives this data and incorporates it in the revised model for future predictions moving forward.

[0026] FIG. 4 is a flow chart illustrating the steps of the training phase of the present invention. In step 200, a set of videos are collected with known defects for analysis by the software of the present invention. The images that contain the defects are extracted from the videos in step 205, and the extracted images are processed in step 210. The processing involves grayscale conversion, edge processing such as sobel detection, and resizing the image such as downsampling the image to 256×256 pixels. In step 215, the processed image of the defect is fed to the Convolutional Neural Network training algorithm for developing a learning model of the known defects in step 220, which is then used in step 225 to identify and classify defects in new videos.

[0027] FIG. 5 is a flow chart of the runtime phase of the present invention, where the model developed in the preceding paragraph is used to detect and catalog new defects from new video. In step 250, the operator instructs the camera and the software to initiate the investigation of a new sewer as the software captures images from the camera feed in real time and the video is sent to the vehicle where it is viewed by the operator. In step 255, the frames of video are extracted from the feed and processed in step 260 in the same manner as in step 210 in the training phase of the invention. In step 230, the model created in step 220 is used with new images from video collected in real time from a camera feed of sewer investigations. If a defect is detected by the model from the images in the camera feed, the operator is sent a notification on the monitor in step 235 alerting the operator of the presence of a detected defect. The operator may stop the camera and annotate the data to include feedback relating to the defect in step 240, including overriding the model if the operator determines that the model has incorrectly identified a defect or mischaracterized a defect in any way. The process continues as the camera moves along the pipe until the camera reaches the end of the pipe and the length of pipe has been analyzed for defects.