SYSTEM AND METHOD FOR ASSESSING A COATED SURFACE WITH RESPECT TO SURFACE DEFECTS

Abstract

The invention relates to a method for providing a system for assessing a coated surface with respect to a type set containing at least one type of surface defect that can occur on the surface, to such a system for assessing a coated surface, to a measuring device for acquiring an image of a coated surface including such a system, and to a method for assessing a coated surface using such a system. The system can use a convolutional neural network (CNN), in particular the so-called U-Net architecture, to recognize the at least one surface defect in an image provided to the system. Moreover, the system can use a support vector machine algorithm to provide, based on the recognition results, quantitative and/or qualitative information about the depiction of the at least one surface defect in the image provided to the system.

Claims

1. A method for providing a system for assessing a coated surface with respect to a type set containing at least one type of coating surface defect that can occur on the coating surface, wherein the type set contains at least one of the types pinhole, blister, crater, and seed, the system containing a database and at least one machine learning algorithm; the method including: generating a plurality of images of coated surfaces, wherein at least one of the plurality of images is a three-dimensional image including topography information of a coated surface of the coated surfaces; generating a plurality of datasets each containing at least one image of the plurality of images of coated surfaces, wherein each dataset containing an image depicts the at least one type of coating surface defect is labeled with the at least one type of coating surface defect and/or with quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the one image, and wherein the three-dimensional image including topography information provides details about the shape of the at least one type of coating surface defect, storing the plurality of datasets in the database, using the plurality of datasets in the database to train the at least one machine learning algorithm to recognize the at least one type of coating surface defect in an image provided to the system and/or to provide quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image.

2. The method according to claim 1, wherein generating a single image of the plurality of images comprises one or more of image acquisition, surface sampling, and image processing.

3. The method according to claim 1, wherein the system contains a first machine learning algorithm, which uses a convolutional neural network to recognize the at least one type of coating surface defect in the image provided to the system.

4. The method according to claim 3, wherein the first machine learning algorithm uses U-Net architecture.

5. The method according to claim 3, wherein the system further contains a second machine learning algorithm, which uses a support vector machine algorithm to provide, based on the recognition result of the first machine learning algorithm, the quantitative and/or qualitative information about the depiction of the at least one coating surface defect in the image provided to the system.

6. The method according to of claim 5, wherein the quantitative and/or qualitative information is a human impression metric.

7. The method according to claim 1, wherein the system contains a machine learning algorithm, which uses a convolutional neural network with several layers to provide the quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image provided to the system, without having separately recognized the at least one type of coating surface defect in the image before.

8. A system for assessing a coated surface with respect to a type set containing at least one type of coating surface defect that can occur on the coating surface, wherein the type set includes one or more of a pinhole, a blister, a crater, and a seed, the system comprising: a database including a plurality of datasets each containing at least one image of a coated surface, wherein each dataset containing an image depicting the at least one type of coating surface defect is labeled with the at least one type of coating surface defect and/or with quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image, and wherein the database includes a three-dimensional image including topography information of a coated surface providing details about the shape of the at least one type of coating surface defect, at least one machine learning algorithm trained with the plurality of datasets to recognize the at least one type of coating surface defect in an image provided to the system and/or to provide quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image, and a scanner for capturing an image of a coated surface to provide to the system.

9. A measuring device for acquiring an image of a coated surface, it includes the measuring device comprising the system according to claim 8 and a display.

10. A method for assessing a coated surface with respect to a type set containing at least one type of coating surface defect that can occur on the coated surface, wherein the type set includes one or more of a pinhole, a blister, a crater, and a seed, the method comprising: providing a system for assessing a coated surface with respect to a type set containing at least one type of coating surface defect that can occur on the coating surface, the system comprising: a database including a plurality of datasets each containing at least one image of a coated surface, wherein each dataset containing an image depicting the at least one type of coating surface defect is labeled with the at least one type of coating surface defect and/or with quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image, and wherein the database includes a three-dimensional image including topography information of a coated surface providing details about the shape of the at least one type of coating surface defect, at least one machine learning algorithm trained with the plurality of datasets to recognize the at least one type of coating surface defect in an image provided to the system and/or to provide quantitative and/or qualitative information about the depiction of the at least one type of coating surface defect in the image, and a scanner for capturing an image of a coated surface to provide to the system, generating at least one image of a coated surface and providing the at least one image to the system, wherein an image of at least one image is a three-dimensional image including topography information of the coated surface, using the at least one machine learning algorithm of the system to assess the coated surface depicted in the at least one image by outputting a statement that the at least one image of the coated surface depicts the at least one coating surface defect, and/or outputting quantitative and/or qualitative information about the depiction of the at least one coating surface defect in the at least one image.

11. The method for assessing a coated surface according to claim 10, wherein generating the at least one image of a coated surface comprises one or more of image acquisition, surface sampling, and image processing.

Description

[0070] Further advantageous embodiments of the invention are described in the following description in connection with the attached drawings.

[0071] FIG. 1 shows an exemplary representation of a method for assessing a coated surface according to the prior art;

[0072] FIG. 2 shows an exemplary representation of the method for assessing a coated surface according to the fourth aspect of the invention;

[0073] FIG. 3 shows a block diagram of an implementation of the method for providing a system for assessing a coated surface according to the first aspect of the invention and of the method for assessing a coated surface according to the fourth aspect of the invention;

[0074] FIG. 4 shows another exemplary representation of the method for assessing a coated surface according to the fourth aspect of the invention.

[0075] FIG. 1 shows an exemplary representation of a method for assessing a coated surface according to the prior art. In this method, the coated surface to be assessed is scanned by a 3D portal scanner (not shown), whereby not only color and brightness information, but also height information for each pixel scanned is obtained. From the data obtained by the 3D scanner, a three-dimensional image of the coated surface is obtained (first picture). For a certain partial area of the coated surface, a height profile of a cross-section is calculated from the image (second picture).

[0076] This height profile is then analyzed by statistical methods like the determination of the maximum peak value and the average value (third picture). From this analysis, statistical parameters can then be derived which are representative of certain surface defects (not shown).

[0077] FIG. 2 shows an exemplary representation of a method for assessing a coated surface according to the fourth aspect of the invention. In this method, the coated surface to be assessed is scanned by a specialized surface measuring device (not shown).

[0078] The system for assessing a coated surface according to the second aspect of the invention can be included in the surface measuring device, in which case the surface measuring device is a measuring device according to the third aspect of the invention, or it can be installed on a separate computer, in a computer network, or even in the so-called “cloud”. The decision whether to include the system for assessing a coated surface in the surface measuring device itself depends on parameters of the surface measuring device like its form factor, its computing power, its battery capacity, etc.

[0079] A further alternative for the arrangement of the system for assessing a coated surface with respect to the surface measuring device is a “distributed arrangement”. This means that the training of the one or more machine learning algorithms is performed offline, i. e. in a computer or computer network outside the coating surface measuring device, the trained system is then transferred to the surface measuring device as software and/or hardware, and the assessment of the coated surface measured is performed online, i. e. within the coating surface measuring device itself, in particular in an autonomous way, i. e. without any connection to a computer outside the surface measuring device.

[0080] This process of offline training and online use is sometimes described in the way that the surface measuring device is “pinned” for a certain application domain or customer.

[0081] From the data obtained by the measuring device, a three-dimensional image of the coated surface is obtained as above (first picture). Furthermore, a two-dimensional image including topographical information represented by different colors and/or gray scale values is obtained (second picture, above). Also, for a certain partial area of the coated surface, a height profile of a cross-section is calculated from the image (third picture, right).

[0082] This height profile is then analyzed for detecting certain surface defects like pinholes, craters, or bubbles (blisters) using a machine learning algorithm which has been trained before with images of similar surface defects (fourth image). A final result of the assessment is a list of the different types of surface defects to be detected together with the number of occurrences of each of these types of surface defects on the analyzed coated surface (Defect Statistics).

[0083] FIG. 3 shows a block diagram of an implementation of the method for providing a system for assessing a coated surface according to the first aspect of the invention and of the method for assessing a coated surface according to the fourth aspect of the invention.

[0084] Both methods use the same image generation steps S1, S1a, and S2, optionally leading to Imaging Model R1. The method for providing the system (left branch, “Training”) additionally includes steps S3 and S4, leading to Defect Model R2, an S5, leading to Quality Model R3. The method for assessing a coated surface (right branch, “Evaluation Process”) additionally includes steps S6, S7, and S8. All these steps and models are explained in detail in the following.

[0085] The invention enables a semantic classification and quantification of different defect types observed when the quality of a surface varnish or coating is characterized. In the specific embodiment described here, two different types of defects are distinguished: pinholes and blisters. However, the method described is not limited to these defect types. Given suitable training samples, this can be extended to cover further defect types such as craters or seeds. These defects are in the lateral and vertical size range of μm to mm. Therefore, high quality image acquisition is a crucial prerequisite for the use-cases described here, and the exact configuration depends on the type of image acquisition used.

[0086] Image Acquisition (Step S1)

[0087] As an image acquisition device, the “spectro2profiler” device developed by BYK-Gardner® is used. Using photometric stereo, the height profile of a given area can be measured with a spatial resolution of 30 μm/pixel, an imaging area of 225 mm.sup.2, and a vertical resolution of approximately 1-2 μm.

[0088] Further techniques available to create high-resolution images of surface topologies that can be used in Step S1 include, but are not limited to, shade-by-shading, interferometric measurements, confocal microscopy, or structured illumination.

[0089] Surface Sampling (Step S1a)

[0090] The inspected coated surface areas are much larger than the high-resolution imaged area (approx. 225 mm.sup.2). Therefore, several samples are taken to quantify the defects of a surface (see below) and attribute the individual samples with the human assessment metric.

[0091] Image Preprocessing (Step S2)

[0092] To ensure high and constant quality images, a median image subtraction method is performed to limit image acquisition device specific characteristics. For coating surfaces with low frequency spatial noise (e. g. structured wall surface, as determined in step S1), furthermore difference of Gaussian (DoG) is applied as a high-pass filter. Finally, images are normalized to have zero mean and unit variance.

[0093] When deep learning approaches are taken, preprocessing based on image statistics can usually be left out. However, in this case, it can optionally be used to ensure rather constant image quality and to notify (and understand) if image statistics indicate major changes in the nature of the imaged surface, leading to Imaging Model R1.

[0094] Semantic Defect Classification Algorithm (“Training” Branch in FIG. 3)

[0095] Creating a Training Data Set (Step S3)

[0096] In the initial embodiment, a training data set of 56 lab-prepared samples is created across four different application domains (architectural, wood, automotive, industrial coatings), using Steps S1, S1a, and S2. Domain experts classify the quality of the samples (metrics from 1 to 5, with 1=good and 5=bad, are applied). For each of the samples, several non-overlapping measurements are obtained, resulting in a total of 672 measurements. Pixel-wise labeling of the two different defect types considered here (pinholes and blisters) are performed by the domain experts as well.

[0097] Training the Classification Algorithm (Step S4)

[0098] In modern digital image processing, convolutional neural networks (CNN) are used in many tasks, including semantic segmentation. More specifically, the U-Net architecture is used in this embodiment, originally proposed by Ronneberger et al. (see above) for biomedical image segmentation.

[0099] Advantages of this architecture are that it works with relatively few training samples and has strong image augmentation. Due to its fully convolutional nature, it can work with arbitrarily large input images.

[0100] The network structure employed in this embodiment deviates from the originally proposed U-Net structure by scaling the images with a factor of 0.5, reducing the feature dimensions for the initial embodiment by 50%, which results in a feature space size of 512 at the bottom layer. No further data augmentation is applied.

[0101] In Step S4, the network is trained with standard stochastic gradient descent and a learning rate of 0.01. The loss function is the dice coefficient. For training and testing, an 80/20 split of the data sample level is used.

[0102] Defect Model (R2)

[0103] This results in a trained algorithm that is able to detect segmentation masks for the different types of defects (Defect Model R2). Based on this defect model, quantitative features are derived such as the number of detected defects, the percentage of the surface area affected by defects, the size distributions of the defects, the uniformity of spatial distributions across samples and measurements, etc.

[0104] Automatic Representation of Human Impression Metric (Step S5)

[0105] Based on the segmentation masks derived with the classification approach, a support vector machine algorithm is used to learn the human impression metric (Step S5).

[0106] Quality Model (R3)

[0107] Features are the metrics obtained by the semantic classification. Typical data processing is performed such as feature normalization. This results in a trained algorithm (R3) to predict human impression (surface quality) based on segmentation masks (R2).

[0108] Instead of the pair of a CNN for the segmentation and a support vector machine algorithm for learning the human impression metric, also a single classical CNN can be employed, where classical refers to a standard architecture of a couple of convolutional layers to extract features, followed by two fully connected layers with the last layer only containing one single neuron for the regression output.

[0109] This approach is an alternative to Steps S4 and S5, directly predicting human impression without segmentation masks and features, i. e. omitting Defect Model R2. This directly results in the predicted metric for a given image and shows similar results as in the two-step approach.

[0110] The advantage of separating the semantic pixel-based classification and the metric estimation (Steps S4 and S5) is interpretability, and it is very transparent to also included features for a sample across different measurements.

[0111] Testing Novel Images (“Evaluation Process” Branch in FIG. 3)

[0112] When a novel image has been acquired in Steps S1, S1a, and S2 (with the same image acquisition method and/or device type as in the “Training” branch), it is first tested whether the image meets imaging model criteria to detect whether the expected quality of subsequent results is sufficient (Step S6). Subsequently, defects are detected and quantified (Step S7) based on the Defect Model R2.

[0113] Optionally, the quality can be assessed (Step S8) by comparing the result against the Quality Model R3.

[0114] Moreover, the user can evaluate the quantitative and qualitative results, optionally feeding adjustments back to the database described above.

[0115] FIG. 4 shows another exemplary representation of the method for assessing a coated surface according to the fourth aspect of the invention, i. e. an example of segmentation results, in more detail.

[0116] Two samples (“Sample 1”, “Sample 2”) from an example training database created in step S4 are shown (left part of FIG. 4), the left image of each sample showing the original image and the right image showing the corresponding positions of the expert labeled defects. In this case, simple defect types (“Type A”, “Type B”) are used (here: holes and bubbles).

[0117] Based on these training samples, a Trained Defect Model (R2) is generated.

[0118] When a novel image is acquired, it is compared against the trained model (step S7), and likely defective areas and their likely types (as existing in the training set) are detected (middle part of FIG. 4).

[0119] The identified defects can subsequently be quantified with a variety of blob based metrics (step S8), e. g. counts, areas, or intensities (right part of FIG. 4).