COMPUTER-IMPLEMENTED METHOD AND TOOL FOR OPTICAL QUALITY CONTROL OF INTERMEDIATE OR END PRODUCTS OF PRODUCTION INSTALLATIONS, AND PRODUCTION INSTALLATION CONTROLLER

Abstract

For optical quality control, a product image of a production installation captured by an image capture device for given intrinsic and extrinsic parameters is used and digital twin data of a digital twin of the production installation are used, to render a synthetic simulation image based on the digital twin data, wherein the rendered synthetic simulation image is based on the same intrinsic and extrinsic parameters as during product image capture, to transfer the product image from a real domain into an artificial domain by a trained domain adaptation and in the process to generate a synthetic product image from the product image with domain transfer parameters obtained by the training, to compare the synthetic product image with the synthetic simulation image by a comparison operator, and to output a comparison result which qualitatively assesses the product.

Claims

1. A computer-implemented method for optical quality control of intermediate or end products of production installations, wherein for the quality control a product image of a production installation captured by an image capture device for given intrinsic and extrinsic parameters is used, digital twin data of a digital twin of the production installation are used, wherein the digital twin is synchronized with the production installation at the time of operation thereof, the method comprising: wherein a) rendering a synthetic simulation image based on the digital twin data, wherein the rendered synthetic simulation image is based on the same intrinsic and extrinsic parameters as during product image capture; b) transferring the product image from a real domain into an artificial domain by a trained domain adaptation which contains domain transfer parameters obtained by the training and with which a synthetic product image is generated from the product image in accordance with the domain transfer parameters, whereby an image pair formed from the synthetic simulation image and the synthetic product image arises in an artificial image space for comparison purposes; c) comparing the synthetic product image with the synthetic simulation image by a comparison operator; and d) outputting a comparison result which qualitatively assesses the product, by way of an output unit of the production installation or a production installation controller of the production installation.

2. The computer-implemented method as claimed in claim 1, wherein the domain adaptation is implemented as a machine learning model according to the principle of a generative adversarial network (GAN), wherein data are generated by the use of two competing artificial neural networks referred to as generator and discriminator, of which the generator generates artificial data which the discriminator checks on the basis of authentic data, captured with the aid of images, and wherein the two networks are logically and mathematically combined with one another in such a way that the artificial data generated by the generator seem more and more genuine and at the end the discriminator is no longer able to differentiate the genuine data from the authentic data.

3. The computer-implemented method as claimed in claim 1, wherein the trained domain adaptation with the domain transfer parameters is implemented in a two-stage training with the following steps S1 and S2 S1: generating a data set on the basis of a multiplicity n of image pairs which are formed from captured product images and associated synthetic simulation images for uniformly given intrinsic and extrinsic parameters; S2: training the transfer of product-image-related data to simulation-image-related data with the aid of the generated data set by way of learning methods.

4. The computer-implemented method as claimed in claim 1, wherein the comparison operator is configured in such a way that the comparison is implemented pixel by pixel.

5. The computer-implemented method as claimed in claim 1, wherein the production installation is a robot system or automation system with a universally usable automatic movement machine for executing handling, service and/or manufacturing tasks.

6. A computer-implemented tool, configured as an APP, for optical quality control of intermediate or end products of production installations, wherein for the quality control a product image of a production installation captured by an image capture device for given intrinsic and extrinsic parameters is used, digital twin data of a digital twin of the production installation are used, wherein the digital twin is synchronized with the production installation at the time of operation thereof, wherein a non-volatile, readable memory, wherein processor-readable control program instructions of a program module for optical quality control are stored, and a processor connected to the memory, the processor executing the control program instructions of the program module for optical quality control of the intermediate or end products of production installations, wherein the program module is constituted in such a way, and the processor that executes the control program instructions of the program module for optical quality control is configured in such a way, that a) a synthetic simulation image based on the digital twin data is rendered, wherein the rendered synthetic simulation image is based on the same intrinsic and extrinsic parameters as during product image capture, b) the product image is transferred from a real domain into an artificial domain by a trained domain adaptation which contains domain transfer parameters obtained by the training and with which a synthetic product image is generated from the product image in accordance with the domain transfer parameters, whereby an image pair formed from the synthetic simulation image and the synthetic product image arises in an artificial image space for comparison purposes, c) the synthetic product image is compared with the synthetic simulation image by a comparison operator; and d) a comparison result which qualitatively assesses the product is output, by way of an output unit of the production installation or a production installation controller of the production installation.

7. The computer-implemented tool as claimed in claim 6, wherein the processor and the program module for optical quality control are configured in such a way that the domain adaptation is implemented as a machine learning model according to the principle of a generative adversarial network (GAN), wherein data are generated by the use of two competing artificial neural networks referred to as generator and discriminator, of which the generator generates artificial data which the discriminator checks on the basis of authentic data, captured with the aid of images, and wherein the two networks are logically and mathematically combined with one another in such a way that the artificial data generated by the generator seem more and more genuine and at the end the discriminator is no longer able to differentiate the genuine data from the authentic data.

8. The computer-implemented tool as claimed in claim 6, wherein the processor and the program module for optical quality control are configured in such a way that the trained domain adaptation with the domain transfer parameters is implemented in a two-stage training with the following steps S1 and S2 S1: generating a data set on the basis of a multiplicity n of image pairs which are formed from captured product images and associated synthetic simulation images for uniformly given intrinsic and extrinsic parameters; S2: training the transfer of product-image-related data to simulation-image-related data with the aid of the generated data set by way of learning methods.

9. The computer-implemented tool as claimed in claim 6, wherein the processor and the program module for optical quality control and also the comparison operator are configured in such a way that the comparison is implemented pixel by pixel.

10. The computer-implemented tool as claimed in claim 6, wherein the production installation is a robot system or automation system with a universally usable automatic movement machine for executing handling, service and/or manufacturing tasks.

11. A production installation controller for optical quality control of intermediate or end products of a production installation, comprising: an image capture device which captures a product image of the production installation for given intrinsic and extrinsic parameters either is part of the production installation and as such is connected to the production installation controller or is assigned to the production installation and as such is connected to the production installation controller, a database which stores digital twin data of a digital twin of the production installation is assigned to the production installation and as such is connected to the production installation controller, wherein the digital twin is synchronized with the production installation at the time of operation thereof wherein a computer-implemented tool as claimed in claim 6, which is loadable into the production installation controller in order to implement a method for optical quality control of intermediate or end products of production installations.

12. The production installation controller as claimed in claim 11, wherein a control unit for a robot system or automation system with a universally usable automatic movement machine for executing handling, service and/or manufacturing tasks.

Description

BRIEF DESCRIPTION

[0037] Some of the embodiments will be described in detail, with reference to the following figures, wherein like designations denote like members, wherein:

[0038] FIG. A shows a production installation PA with optical quality control for intermediate or end products of the production installation, which is usable in various technical fields (in technical domains) for product production by adequate methods and techniques, e.g., in factories, production lines and large apparatuses; and

[0039] FIG. B shows a production installation PA with optical quality control for intermediate or end products of the production installation, which is usable in various technical fields (in technical domains) for product production by adequate methods and techniques, e.g., in factories, production lines and large apparatuses.

DETAILED DESCRIPTION

[0040] In a first embodiment variant of the invention for optical quality control of intermediate or end products in accordance with an option A, the production installation PA contains a production installation controller PAS, a database DB, an image capture device BEE configured e.g., in the form and shape of a camera, and an output unit AEH. While the database DB and the image capture device BEE are connected to the production installation controller PAS for accesses thereof, the output unit AEH either in a first embodiment in accordance with option I, like the database DB and the image capture device BEE, is connected to the production installation controller PAS for accesses thereof or in a second embodiment in accordance with option II is contained in the production installation controller PAS.

[0041] As an alternative to the first embodiment variant of the invention in accordance with option A, in a second embodiment variant of the invention for optical quality control of intermediate or end products in accordance with an option B, the production installation PA, the production installation controller PAS, the database DB and the image capture device BEE are not combined under one roof, the roof of the production installation PA, rather they all function as separate units, wherein the production installation controller PAS is connected to the production installation PA, the database DB and the image capture device BEE for accesses, while the output unit AEH either in the first embodiment in accordance with option I is contained in the production installation PA for accesses of the production installation controller PAS or in the second embodiment in accordance with option II is contained in the production installation controller PAS.

[0042] Between these two extreme embodiment variants of the invention, other variants are also conceivable (not explicitly illustrated in FIGS. A and B where either only the database DB, the image capture device BEE or the production installation controller PAS is contained in the production installation PA or the production installation PA contains in each case two of the units mentioned.

[0043] In both illustrated embodiment variants in accordance with options A and B, the production installation controller PAS is a control unit STE for a robot system or automation system with a universally usable automatic movement machine for executing handling, service and/or manufacturing tasks.

[0044] With regard to the illustrated embodiment variant in accordance with option B, it is alternatively also possible for the production installation controller PAS to be a customary personal computer or controller.

[0045] The explanations below concerning the description of FIGS. A and B apply to both illustrated embodiment variants of the invention for optical quality control of intermediate or end products in accordance with options A and B.

[0046] For this purpose [0047] the image capture device BEE captures both a product image PB of the production installation PA, which can concern an intermediate or end product, for given intrinsic and extrinsic parameters of the image capture device BEE and also product images PB1, . . . , PBn for uniformly given intrinsic and extrinsic parameters of the image capture device BEE for the purpose of generating a data set on the basis of a multiplicity n of image pairs. [0048] the database DB stores digital twin data DZD of a digital twin DZ of the production installation PA, wherein the digital twin DZ is synchronized with the production installation PA at the time of operation thereof. [0049] a computer-implemented tool CIW is used, which is a computer program product (non-transitory computer readable storage medium having instructions, which when executed by a processor, perform actions) CPP configured as an APP and is loadable into the production installation controller PAS for optical quality control for intermediate or end products of the production installation PA.

[0050] The computer-implemented tool CIW contains a non-volatile, readable memory SP, wherein processor-readable control program instructions of a program module PGM for optical quality control are stored, and a processor PZ connected to the memory SP, the processor executing the control program instructions of the program module PGM for optical quality control of the intermediate or end products of production installation PA.

[0051] For this purpose, the computer-implemented tool CIW uses [0052] the product image PB of the production installation PA captured by the image capture device BEE for given intrinsic and extrinsic parameters, and the product images PB1, . . . , PBn captured by the image capture device BEE for given intrinsic and extrinsic parameters for the purpose of generating the data set on the basis of a multiplicity n of image pairs, [0053] the digital twin data DZD of the digital twin DZ of the production installation PA that are stored in the database DB, wherein the digital twin DZ is synchronized with the production installation PA at the time of operation thereof.

[0054] During the loading of the computer-implemented tool CIW into the production installation controller PAS, these data are requested from the processor PZ as input data by access and then either collected or supplied.

[0055] The program module PGM of the computer-implemented tool CIW is constituted in such a way, and the processor PZ of the computer-implemented tool CIW that executes the control program instructions of the program module PGM for optical quality control is configured in such a way, that the following steps for optical quality control are carried out: [0056] rendering rdn a synthetic simulation image SB.sub.syn based on the digital twin data DZD. The rendered synthetic simulation image SB.sub.syn here is based on the same intrinsic and extrinsic parameters as during product image capture. [0057] transferring trf the product image PB from a real domain into an artificial domain by a trained trn domain adaptation DA.

[0058] The domain adaptation is e.g., implemented as a machine learning model according to the principle of a generative adversarial network <GAN>, wherein data are generated by the use of two competing artificial neural networks referred to as generator and discriminator, of which the generator generates artificial data which the discriminator checks on the basis of authentic data, e.g., captured with the aid of images, and wherein the two networks are logically and mathematically combined with one another in such a way that the artificial data generated by the generator seem more and more genuine and at the end the discriminator is no longer able to differentiate the genuine data from the authentic data.

[0059] The domain adaptation DA contains domain transfer parameters DTP obtained by the training trn. The trained trn domain adaptation DA with the domain transfer parameters DTP is implemented in a two-stage training trn with the following steps S1 and S2 [0060] S1: generating a data set on the basis of a multiplicity n of image pairs which are formed from the captured product images PB1, . . . , PBn and associated synthetic simulation image SB1syn, . . . , SBn.sub.syn for uniformly given intrinsic and extrinsic parameters; [0061] S2: training the transfer of product-image-related data to simulation-image-related data with the aid of the generated data set by way of learning methods such as e.g., generative adversarial network <GAN>.

[0062] After the training, a synthetic product image PB.sub.syn is generated from the product image PB with the trained domain adaptation DA in accordance with the domain transfer parameters DTP. As a result, an image pair formed from the synthetic simulation image SB.sub.syn and the synthetic product image PB.sub.syn arises in an artificial image space for comparison purposes. [0063] comparing vgl the synthetic product image PB.sub.syn with the synthetic simulation image SB.sub.syn by a comparison operator VO. The comparison operator VO is e.g., configured in such a way that the comparison is implemented pixel by pixel. [0064] outputting asg a comparison result VGE which qualitatively assesses the product. This comparison result VGE is e.g., output asg by way of the output unit AEH of the production installation PA or the production installation controller PAS of the production installation PA.

[0065] Although the present invention has been disclosed in the form of embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention.

[0066] For the sake of clarity, it is to be understood that the use of a or an throughout this application does not exclude a plurality, and comprising does not exclude other steps or elements.