METHOD AND ASSISTANCE SYSTEM FOR CHECKING SAMPLES FOR DEFECTS

Abstract

A method for checking samples for defects is provided, in which image data of the samples are recorded and classified into predeterminable defect categories by a defect detection algorithm, and the samples classified into a defect category are represented in a multi-dimensional confusion matrix as a classification result of the defect detection algorithm, characterized in that—miniature images which reproduce the image data are assigned according to the classified defect categories of the image data to segments of the confusion matrix which represent the defect categories, and these miniature images are displayed visually, —the miniature image is assigned by an interaction with a user or a software robot to a different segment from the assigned segment of the confusion matrix, and is either provided as training image data for the defect detection algorithm or is output as training image data for the defect detection algorithm.

Claims

1. A method for checking samples for defectiveness, the method comprising: recording image data of the samples to be checked; classifying the image data associated with the samples into predefinable defect categories by a defect recognition algorithm; presenting a number of the samples classified in the predefinable defect categories in a multi-dimensional confusion matrix as a classification result of the defect recognition algorithm; assigning miniature images which reproduce the image data associated with the samples according to the predefinable defect categories of the image data, to segments of the multi-dimensional confusion matrix which represent the predefinable defect categories, wherein the miniature images are presented visually within the segments: assigning, by way of an interaction with a user or a software robot, a miniature image into a different segment than the assigned segment of the multi-dimensional confusion matrix, which is either provided as training image data for the defect recognition algorithm or is provided and output as training image data for the defect recognition algorithm.

2. The method as claimed in claim 1, wherein the size of the miniature images is adapted such that the miniature images assigned to a segment optically fit into the segment.

3. The method as claimed in claim 1, wherein a size of the miniature image is individually adapted after selection of the miniature image.

4. The method as claimed in claim 1, wherein a number of miniature images within a segment is optically identified.

5. The method as claimed in claim 1, wherein the miniature images are positioned within a segment of the multi-dimensional confusion matrix in a manner sorted according to at least one predefinable criterion, the at least one predefinable criterion being a confidence value, an entropy over all defect categories, a dimension reduction, a similarity, or a distance metric.

6. The method as claimed in claim 1, wherein an assignment of one or more miniature images from an assigned segment into a different segment of the multi-dimensional confusion matrix is carried out by way of comparison of selected or selectable miniature image regions on a basis of the at least one criterion or on a basis of at least one further criterion including a poor confidence value, a similar image brightness, a visually similar sample shape and/or a recognizable defect.

7. An assistance system for checking samples for defectiveness by a defect recognition device, which records image data of the samples to be checked and classifies the image data associated with the samples into predefinable defect categories by a defect recognition algorithm, wherein a number of the samples classified in the defect categories are presentable in a multi-dimensional confusion matrix as a classification result of the defect recognition algorithm, the assistance system comprising: at least one processing unit having at least one storage unit, wherein the processing unit is configured to assign miniature images which reproduce the image data associated with the samples, according to the classified defect categories of the image data, to segments of the multi-dimensional confusion matrix which represent the predefinable defect categories, and to present the miniature images visually within the segments, to assign a miniature image, by way of an interaction with a user or a software robot, into a different segment than the assigned segment of the multi-dimensional confusion matrix, either to provide the miniature image as training image data for the defect recognition algorithm in the at least one storage unit or to provide the miniature image as training image data for the defect recognition algorithm in the at least one storage unit and to output the miniature image at an output unit.

8. The apparatus as claimed in claim 7, wherein the miniature images are positionable within a segment in a manner sorted according to at least one predefinable criterion.

9. The method as claimed in claim 7, wherein the number of miniature images within a segment is optically identifiable.

10. A computer program product, comprising a computer readable hardware storage device having computer readable program code stored therein, said program code executable by a processor of a computer system to implement a method as claimed in claim 1, comprising program code parts designed to carry out the method.

Description

BRIEF DESCRIPTION

[0043] Some of the embodiments will be described in detail, with reference to the following figures, wherein like designations denote like members, wherein:

[0044] FIG. 1 shows one exemplary embodiment of the method according to the invention as a flow diagram;

[0045] FIG. 2 shows an embodiment of a confusion matrix, in the segments of which miniature image are presented virtually;

[0046] FIG. 3 shows an embodiment of a confusion matrix, in the segments of which miniature image are presented virtually;

[0047] FIG. 4 shows an embodiment of a confusion matrix, in the segments of which miniature image are presented virtually;

[0048] FIG. 5 shows an embodiment of a confusion matrix, in the segments of which miniature images are presented visually;

[0049] FIG. 6 shows a confusion matrix with time series data originating from a sensor; and

[0050] FIG. 7 shows a manufacturing environment by way of example.

DETAILED DESCRIPTION

[0051] FIG. 7 shows a manufacturing environment. An assistance system A comprising a processing unit, e.g. a processor, and at least one storage unit can be used in a manufacturing environment for example in order to assist or carry out the following steps. In step 1, a defect recognition device (not illustrated), which can also be embodied as an edge device, can set up a connection to the assistance system A or can be integrated into the assistance system in some other way. In step 2, a camera or some other image recording device is connected to the defect recognition device. The classification result of a defect recognition algorithm can be displayed to a user B on an output apparatus e.g. in the form of a display D arranged close to a manufacturing machine F.

[0052] FIG. 1 shows a flow diagram concerning the procedure according to embodiments of the invention, the steps being designated by 1 to 7.

[0053] In step 3, a training mode can be started, in which the defect recognition algorithm, which can be configured in the form of an intelligent learning algorithm, e.g. a neural network, conducts a data classification. In the example, the user or a bot supplies marked image data recorded from a sample, which are classified or allocated into a predefinable defect category. According to embodiments of the invention, a multi-dimensional confusion matrix is used, defect categories—optionally sorted in a rank order—being plotted on each axis or dimension. By way of example, in accordance with FIG. 1, the defect categories designated there by K11 to K14 may read as follows, for example: “increased area”, “defects/holes in the material”, “segment missing” and “normal”. The vertical y-axis would then be assignable to the user. The defect categories K11 to K15 with “increased area”, “defects/holes in the material”, “segment missing” and “normal” are arranged on the horizontal x-axis, with defect category K14 additionally representing “others” or “not classified” or “unknown”. The vertical y-axis would be an initial starting solution for the defect recognition algorithm or would already be a defect classification result if the defect algorithm has previously been executed once already. The image data, in the form of miniature images M of the respective samples, are assigned to the segments of the confusion matrix and presented visually. In FIGS. 2, 3 and 5, one of the many miniature images is designated by M. In FIG. 4, the miniature images M1, M2, M3, M4 are indicated as being presented with a box around them and correspondingly magnified and arranged below the confusion matrix, in which case these miniature images can be shown in a separate window or a separate display. A correspondence between the result of the defect algorithm and that of the user or optionally bot can then be found on the diagonal of the confusion matrix. The miniature images which, by way of an interaction with the user B or bot, are assigned into a different segment than the assigned segment of the confusion matrix are provided as training image data for the defect recognition algorithm in the at least one storage unit and/or are output at an output unit, e.g. display D.

[0054] If present, an already pretrained model can be used for the defect algorithm in step 4. Said pretrained model could be represented with defect categories possibly by way of a further z-axis in the depth.

[0055] In step 5, a training of the defect recognition algorithm with the aid of the content of the confusion matrix or training image data is carried out, and in step 6 the trained model is introduced into the defect recognition unit. In step 7, finally, samples can be checked for defectiveness by means of the defect recognition device while a manufacturing machine F is running (online) or after the manufacturing process (post-processing).

[0056] FIGS. 2 to 6 show various embodiments of a confusion matrix. In the example, said confusion matrix is of two-dimensional design. The image data associated with a sample form a miniature image (referred to as: thumbnails) and are assigned, according to their classified defect category K11 to K26 of a segment S representing the defect category, to the confusion matrix and are presented visually.

[0057] In FIGS. 2 to 6, the size of the miniature images is adapted such that the miniature images assigned to a segment optically fit into the segment. In this case, the miniature images can lie in a segment without being sorted—as shown in FIG. 2—or they are arranged in the segment in a manner sorted according to a specific criterion—as is evident in FIGS. 3 to 5. One possible criterion is the probability (see e.g. FIG. 3: 0.94) of the association with this defect category, also referred to as a confidence factor. Other criteria are conceivable, such as e.g. entropy over all defect categories, dimension reduction, similarity or distance metric, random, etc.

[0058] FIG. 4 indicates that a miniature image M, by way of an interaction (e.g. “drag and drop” using the mouse or selection with the aid of lasso tools, etc.) with a user or bot, can be assigned into a different segment than the assigned segment S of the confusion matrix. The assignment of one or more miniature images from an assigned or classified segment into a different segment of the confusion matrix can be carried out by way of comparison of selected or selectable miniature image regions on the basis of at least one criterion. Such a criterion can be a poor confidence value, similar brightness of the image, visually similar shape of the samples, recognizable defects such as welding spatter, scratches, etc.

[0059] Particularly miniature images M which are initially assigned to the defect category “not classified” or could not be assigned by the defect recognition algorithm and are therefore in the defect category “unknown” are assignable to one of the other defect categories by the user. The defect categories on an axis of the confusion matrix can be arranged according to a predefinable rank order.

[0060] Various background hatchings of the segments of the confusion matrix are discernible in FIGS. 2 to 6. The number of miniature images within a segment can thus be optically identified. These hatchings, e.g. “close spacing of the lines” (=dark) and “wide spacing of the lines” (=light) can represent t different lightness levels of a color. By way of example, the color “green”—for example in FIG. 5 in the segments (K11;21), (K12;K22), (K13; K23), (K14; K24) —would indicate that the miniature images shown in this segment are correctly assigned or classified. “Dark green” could indicate that a high number (within a predefinable interval) of miniature images are assigned correctly. The number of correctly assigned miniature images within a segment decreases with gradation to “light green”. The color “blue”—e.g. in FIG. 5 in segments (K13; K21), (K14; K21), (K14; K22), (K11; K23), (K11; K24), (K12; K25) —with its lightness gradations analogous to “green” could indicate that the miniature images within this segment are not correctly assigned. A color “purple” could represent a kind of “intermediate level” between correctly assigned miniature images and non-correctly (incorrectly) assigned miniature images. Other color combinations are also conceivable. Likewise, instead of a background hatching or color, the border of a segment could also be correspondingly optically identified.

[0061] FIG. 4 shows with the miniature images M1 to M4 by way of example that a selection of miniature images can be magnified. Magnifying techniques such as fish eye, magnifying glass, zoom, new window or presentation on a different display are possible for this purpose.

[0062] FIG. 6 shows a confusion matrix in which time series data from sensors are classified. In this regard, instead of an image recording device, sensors can also supply data which are presented visually in the confusion matrix. A time series is interpreted as a sample in this case. Embodiments mentioned above can be applied to this confusion matrix as well.

[0063] Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention.

[0064] For the sake of clarity, it is to be understood that the use of “a” or “an” throughout this application does not exclude a plurality, and “comprising” does not exclude other steps or elements.