System and method for detecting damage using two-dimensional imagery and three-dimensional model

Abstract

A method for inspection of a mechanical system includes the steps of: obtaining a two-dimensional image sequence of the mechanical system; generating a three-dimensional structure model from the two-dimensional image sequence; and refining the three-dimensional structure model with an existing three-dimensional model of the mechanical system to produce a refined model having intensity and/or color information from the two-dimensional image sequence and structural accuracy of the existing three-dimensional model.

Claims

1. A method for inspection of a gas turbine engine mechanical system, comprising the steps of: obtaining a two-dimensional image sequence of internal surfaces of the gas turbine engine mechanical system; generating a three-dimensional structure model from the two-dimensional image sequence using a technique selected from the group consisting of Structure from Motion (SFM), Simultaneous Localization and Mapping (SLAM), and combinations thereof; refining the three-dimensional structure model with an existing three-dimensional model of the gas turbine engine mechanical system to produce a refined model having intensity and/or color information from the two-dimensional image sequence and structural accuracy of the existing three-dimensional model.

2. The method of claim 1, wherein the two-dimensional image sequence is obtained with a borescope.

3. The method of claim 1, wherein the refining step comprises matching the three-dimensional structure model with the existing three-dimensional model.

4. The method of claim 1, wherein the refining step comprises regression between the three-dimensional structure model and the existing three-dimensional model.

5. The method of claim 1, wherein the existing three-dimensional model comprises at least one of an as-designed CAD model, an as-built model, and a previous condition model.

6. The method of claim 1, further comprising, before the refining step, mapping the two-dimensional image sequence to the existing three-dimensional model to obtain an augmented existing three-dimensional model, and wherein the refining step comprises refining the three-dimensional structure model with the augmented existing three-dimensional model.

7. A system for inspection of a gas turbine engine mechanical system, comprising: a camera positionable through the gas turbine engine mechanical system to obtain a two-dimensional image sequence of internal surfaces of the gas turbine engine mechanical system; a processor system in communication with the camera to receive the two-dimensional image sequence and configured to generate a three-dimensional structure model from the two-dimensional image sequence by running a technique selected from the group consisting of Structure from Motion (SFM), Simultaneous Localization and Mapping (SLAM), and combinations thereof, and configured to refine the three-dimensional structure model with an existing three-dimensional model of the gas turbine engine mechanical system to produce a refined model having intensity and/or color information from the two-dimensional image sequence and structural accuracy of the existing three-dimensional model.

8. The system of claim 7, wherein the processor system is in communication with a storage containing the two-dimensional image sequence and the existing three-dimensional model.

9. The system of claim 7, wherein the camera is a borescope.

10. The system of claim 7, wherein the processor system is configured to refine by matching the three-dimensional structure model with the existing three-dimensional model.

11. The system of claim 7, wherein the processor system is configured to refine by regression between the three-dimensional structure model and the existing three-dimensional model.

12. The system of claim 7, wherein the existing three-dimensional model comprises at least one of an as-designed CAD model, an as-built model, and a previous condition model.

13. The system of claim 7, wherein the processor system is configured to map the two-dimensional image sequence to the existing three-dimensional model to obtain an augmented existing three-dimensional model, and to refine the three-dimensional structure model with the augmented existing three-dimensional model.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 illustrates a typical manual inspection of a mechanical system using a borescope;

(2) FIG. 2 schematically illustrates one exemplary embodiment of a system and method according to the disclosure;

(3) FIG. 3 schematically illustrates another exemplary embodiment of a system and method according to the disclosure;

(4) FIG. 4 illustrates an exemplary embodiment wherein the images obtained are thermal images; and

(5) FIG. 5 illustrates mapping of a thermal image such as that obtained in FIG. 4 onto an existing three-dimensional model of a blade.

(6) Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

(7) The invention relates to a system and method whereby improved inspection of a mechanical system is provided by combining a three-dimensional structure model generated from the two-dimensional sequence of images with an existing three-dimensional model to produce a refined three-dimensional image wherein the issues of parallax effect and drifting can be minimized.

(8) FIG. 1 shows a typical manual inspection of a mechanical system, in this case a gas turbine engine, using a borescope which is visually monitored by the user to interpret results. Camera systems such as a borescope can store images obtained. These stored images can be processed, for example using Structure from Motion (SFM) and/or Simultaneous Localization and Mapping (SLAM) techniques, to generate a three-dimensional structure model.

(9) In many instances, there will be available an existing three-dimensional model, for example an as-designed CAD model, an as-built model, and/or a previous condition model (as might result from the system and method disclosed herein). These existing models do not have the same issues of parallax effect or drifting and are therefore structurally more accurate than the SFM or SLAM generated three-dimensional structure model. By refining the generated three-dimensional structure model with the existing three-dimensional model, a refined model can be produced, for example having the intensity, color and/or detail of the two-dimensional sequence of images and having the structural accuracy of the existing three-dimensional model.

(10) According to the disclosed method, an existing three-dimensional model (an as-designed CAD model, an as-built model, a previous condition model, etc.) of a component interior (high pressure turbine chamber, combustor chamber, etc.) can be used to assist the mosaicking of multiple images. As a result, the existing three-dimensional model is augmented with image details such as pixel intensity/color on its vertices and surfaces. Potential damage, with accurate metrology, can then be viewed and detected using the augmented three-dimensional model. A fly-through of the 3D model is also possible to get a global view of the entire structure interior. Inspection of a local area and detection of potential damage can then be performed with a clear understanding of location with respect to the global 3D model of the entire structure interior, accurate metrology, and, optionally, with a clear understanding of damage progression.

(11) FIGS. 2 and 3 show two different non-limiting examples of methods to leverage existing three-dimensional models in damage detection.

(12) In FIG. 2, a sequence of images or videos is obtained, such as labeled borescope images/videos 100 (FIG. 2). These images can be obtained using a borescope 102, and the images can be stored in a storage 104, and/or transmitted to a processing system 110 which can be configured to perform the SFM and/or SLAM techniques 106 in order to generate a three-dimensional structure model 108.

(13) Next, the accuracy of the three-dimensional structure model 108 is improved using an existing three-dimensional model such as a three-dimensional CAD model 112. Accuracy of the three-dimensional structure model 108 is improved through a model refinement step 114. Refinement step 114 can be configured to run on processing system 110, and generates a refined model 115 having, for example, intensity and/or color information from the two-dimensional image sequence and structural accuracy from the existing three-dimensional model.

(14) The refining step can be an interpolation or averaging of features from the three-dimensional structure model 108 and the existing three-dimensional model 112.

(15) Another example of this model refinement method carried out in the refining step can be regression between the two models to estimate a transformation from the first model to the second using vertex position information alone. The refined model obtains corresponding intensity and/or color information from the SFM-/SLAM-generated 3D structure model. The regression may be linear or non-linear (e.g., optical flow).

(16) In one embodiment, the transformation can be formulated as a 44 matrix containing 15 independent parameters when vertices of the two models are represented in homogeneous coordinates. These parameters define relative translation, rotation, stretching, squeezing, and shearing between the two models. A minimum of 5 pairs of 3D vertices are identified to be corresponding to each other from the two models. The identification can be either manual, semi-automatic, or fully automatic. The coordinates of these pairs of vertices can be used first to solve a linear regression problem to get an initial estimate of the 15 parameters. This initial estimate can then be used as starting point to solve a non-linear regression problem using an algorithm such as Gauss-Newton or Levenberg-Marquardt. As a result, the refined values of the 15 parameters can be used to transform one model to match closely with the other.

(17) FIG. 3 shows another non-limiting embodiment of the system and method. In this embodiment, the input images/video 100 can be registered with the existing three-dimensional model 112 to produce an augmented three-dimensional model 116 which is augmented with intensity and/or color from the images/videos 100. This registration can include finding corresponding fiducial marks, computing a homography, and the like. In addition, the images/videos can be processed with the processing system 110 to generate a three-dimensional structure model 118 as shown in FIG. 3. Model refinement 120 can then be carried out on the three-dimensional structure model 118 and the augmented three-dimensional model 116.

(18) When the two three-dimensional models are combined and refined, a refined three-dimensional model results, shown at 122, and this model combines the color and intensity of the image sequence obtained with the borescope with the structural accuracy of the existing three-dimensional model.

(19) It should be noted that the images or sequence of images or video can be images of various different type. One non-limiting example of an alternative type of image is a thermal intensity (or equivalently, temperature) image obtained from a hot high-pressure turbine blade is shown in FIG. 4. In FIG. 4, a camera 122 is schematically illustrated along with a thermal image 124 obtained with camera 122, and this image 124 can be obtained by camera 122 from a mechanical system such as a blade 126 as shown in this non-limiting example.

(20) The temperature or thermal image 124 obtained in this manner can be transformed (mapped) to the surface of the existing three-dimensional model 128 as shown in FIG. 5. The mapping of the two-dimensional temperature image to the three-dimensional model may be performed, in one embodiment, by deriving a transformation that transforms the three-dimensional model vertices to two-dimensional pixel coordinates. The transformation can be formulated as a 34 matrix containing 11 independent parameters when two-dimensional image pixels and three-dimensional model vertices are represented in homogeneous coordinates. These parameters define relative translation, rotation, stretching, squeezing, shearing, and a three-dimensional-to-two-dimensional projection. A minimum of 6 pairs of three-dimensional vertices and two-dimensional pixel coordinates are identified to be corresponding to each other from the image and the three-dimensional model. The identification can be either manual, semi-automatic, or fully automatic. The coordinates of these pixels and vertices can be used first to solve a linear regression problem to get an initial estimate of the 11 parameters. This initial estimate can then be used as starting point to solve a non-linear regression problem using an algorithm such as Gauss-Newton or Levenberg-Marquardt. As a result, the refined values of the 11 parameters can be used to transform the three-dimensional model to match closely with two-dimensional pixels. The three-dimensional model vertices can obtain temperature values from those of their projected coordinates. This mapping uses forward projection, by projecting three-dimensional model vertices to two-dimensional temperature image coordinates.

(21) The mapping of the two-dimensional temperature image to the three-dimensional model may be performed, in another embodiment, using backward projection. First the three-dimensional vertices are triangulated to form three-dimensional planar triangular surfaces. Then camera center coordinates are calculated from the transformation matrix. Next every image pixel is back-projected though finding an intersection between the line connecting the pixel and the camera center with a three-dimensional planar triangular surface patch. In this way, not only the three-dimensional model vertices obtain temperature values, but also the triangular surface patches, increasing the resolution of the three-dimensional model in terms of temperature mapping.

(22) The mapping of the two-dimensional temperature image to the three-dimensional model may be performed, in yet another embodiment, by combining the above two methods.

(23) In yet another non-limiting embodiment, if the image/video sensor location and pose are known, for example when exact location and pose of the camera or borescope are known, the images or video frames may be mapped directly to the existing three-dimensional model with local interpolation.

(24) Once a refined three-dimensional model is obtained, either through the process of FIG. 2 or FIG. 3 or alternatives, the refined three-dimensional model can be used for either manual or automated detection of needed maintenance or repair, either immediately in the event of an issue needing immediate attention, or further maintenance can be scheduled at some time in the future if more appropriate.

(25) One or more embodiments of the present invention have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. For example, different sources of images and/or types of images can be utilized. Accordingly, other embodiments are within the scope of the following claims.