CLASSIFICATION MODEL FOR CONTROLLING A MANUFACTURING PROCESS

Abstract

Controlling a manufacturing process by a computer-generated classification model is provided. This is combined with a reward system based on a distributed ledger and smart contracts. The classification model is trained by: Providing data entities being indicative of a property of a manufacturing of a product. Acquiring labels for each of the data entities from an agent. Determining labeling metrics based on the acquiring of the agent. Training the classification model, wherein the training set includes the data entities and their labels. Validating the trained classification model yielding a classifier score. Training a labeling score model based on the data entities, the respective labels, the labeling metrics and the classifier score. Determining a labeling score for the agent based on the labeling score model, the labels and the set of labeling metrics.

Claims

1. A computer-implemented method for training a classification model for controlling a manufacturing process, wherein products are manufactured according to at least one process parameter and wherein at least one property is indicative of the manufacturing of the products, the method comprising, for generating a training set and for training the classification model: providing a set of data entities, each of the data entities being indicative of at least one property of a manufacturing of a respective product; acquiring one or more labels for each of the data entities from an agent; determining a set of labeling metrics based on the acquiring from the agent; training the classification model, wherein the training set comprises the data entities and the respective one or more labels; validating the trained classification model based on predefined criteria and yielding a classifier score; training a labeling score model based on the data entities, the respective one or more labels, the sets of labeling metrics and the classifier score; and determining a labeling score for the agent based on the labeling score model and the respective one or more labels and set of labeling metrics.

2. The computer-implemented method of claim 1, wherein: the manufacturing process is a process for additive manufacturing; each of the data entities comprises an image of a powder layer of a powder bed for manufacturing a layer of the respective product; and the at least one property is a homogeneity of powder of the respective powder layer.

3. The computer-implemented method of claim 1, wherein each of the data entities comprises one of a group comprising: an image, a video, a material density, a sound recording, and a concentration of a chemical substance.

4. The computer-implemented method of claim 1, wherein the set of labeling metrics comprises one of the following groups: a time span during which the one or more labels for the respective data entity have been acquired from the agent; a time span between acquiring one label and a further label from the agent; an amount of required energy; an effort for labeling a data entity; an importance score for a data entity; a count of labels for a data entity of the data entities; a count of labels for the set of data entities; a count of labels acquired from a group of agents; a count of labels acquired from the agent; a measure of the similarity between labels across the group; a measure of similarity of labels across multiple/different groups of agents; assigning a new label to a particular data entity; and an agent classification score.

5. The computer-implemented method of claim 1, being iteratively performed with a first and one or more further iterations, wherein in the further iterations the labeling score is determined based on the labeling score model of a respective previous iteration.

6. The computer-implemented method of claim 5, wherein in the further iterations, the data entities of the first set or a further set are pre-filtered by the classification model, which has been trained in the respective previous iteration.

7. The computer-implemented method of claim 1, further comprising a labeling method for generating the training set and a validation method for training the classification model; wherein the labeling method at least comprises the acquiring of the labels, the determining of the labeling metrics, and the determining of the labeling score and further comprises: storing, for each of the data entities, the one or more labels and the set of labeling metrics in a distributed database; and retrieving the labeling score model from the distributed database; and wherein the validation method at least comprises the training of the classification model, the validation of the classification model, and the training of the labeling score model and further comprises: retrieving the one or more labels for each of the data entities from the distributed database; and storing the trained classification model and the labeling score model in the distributed database.

8. The computer-implemented method of claim 7, wherein the agent is a first agent of a first group of agents; wherein the labeling method further comprises: acquiring one or more labels for each of the data entities from a second agent of a second group of agents; determining a set of labeling metrics based on the acquiring from the second agent; storing, for each of the data entities and for each of the agents, the one or more labels and the set of labeling metrics in a distributed database; determining a labeling score for each of the agents based on the labeling score model and the respective one or more labels and set of labeling metrics; and wherein the validation method further comprises: retrieving the one or more labels for each of the data entities and for each of the agents from the distributed database.

9. The computer-implemented method of claim 8, wherein the labeling score for an agent of one of the groups of agents, the respective set of metrics and the one or more labels of one of the data entities acquired from this agent are provided to this agent or a further agent of the respective group.

10. The computer-implemented method of claim 8, wherein the labeling score for an agent of one of the groups of agents, the respective set of metrics and the one or more labels of one of the data entities acquired from this agent are provided to an agent of another group of the groups of agents depending on whether, during a current iteration of the labeling method, labels for the data entities may still be acquired from the other group.

11. The computer-implemented method of claim 8, wherein the labeling method further comprises: acquiring at least from the first agent, after determining the labeling score at least for the first and the second agent, an agent classification score with respect to the labels for one or more of the data entities acquired from the second agent and an agent labeling score with respect to the labeling score and the set of labeling metrics of the second agent; and wherein: the training set for training the classification model further comprises the agent classification score; and the training of the labeling score model is further based on the agent classification score and the agent labeling score.

12. The computer-implemented method of claim 1, wherein, for each of the data entities, the one or more labels and the set of labeling metrics of the first agent are encrypted by a public key and are stored in encrypted form in a data storage; and wherein, after acquiring labels during a current iteration of the labeling method has been finished, the encrypted form is retrieved from the data storage and is decrypted.

13. A method for controlling a manufacturing process, wherein products are manufactured according to at least one process parameter and wherein at least one property is indicative of the manufacturing of the products, comprising: acquiring a data entity, the data entity being indicative of at least one property of a manufacturing of a respective product; classifying the manufacturing of the product based on the data entity and a classification model; and adapting the at least one process parameter based on the classifying; wherein the classification model is training by a computer-implemented method comprising: providing a set of data entities, each of the data entities being indicative of at least one property of a manufacturing of a respective product; acquiring one or more labels for each of the data entities from an agent; determining a set of labeling metrics based on the acquiring from the agent; training the classification model, wherein the training set comprises the data entities and the respective one or more labels; validating the trained classification model based on predefined criteria and yielding a classifier score; training a labeling score model based on the data entities, the respective one or more labels, the sets of labeling metrics and the classifier score; and determining a labeling score for the agent based on the labeling score model and the respective one or more labels and set of labeling metrics.

14. A controlling apparatus for controlling a manufacturing process, wherein products are manufactured by a manufacturing system according to at least one process parameter and wherein at least one property is indicative of the manufacturing of the products, the controlling apparatus comprising: a sensor assembly adapted to acquire a data entity, the data entity being indicative of at least one property of a manufacturing of a respective product; a data processing apparatus adapted to classify the manufacturing of the product based on the data entity and a classification model; and a control interface adapted to output a control signal such that the at least one process parameter is changed based on the classifying; wherein the data processing apparatus is further adapted to receive the classification model from a data storage of the controlling apparatus, onto which the classification model is stored, or from a distributed database, the classification model being generated by a computer-implemented method comprising: providing a set of data entities, each of the data entities being indicative of at least one property of a manufacturing of a respective product; acquiring one or more labels for each of the data entities from an agent; determining a set of labeling metrics based on the acquiring from the agent; training the classification model, wherein the training set comprises the data entities and the respective one or more labels; validating the trained classification model based on predefined criteria and yielding a classifier score; training a labeling score model based on the data entities, the respective one or more labels, the sets of labeling metrics and the classifier score; and determining a labeling score for the agent based on the labeling score model and the respective one or more labels and set of labeling metrics.

15. A computer-implemented method for generating a smart contract for determining a labeling score, the method comprising: training a labeling score model based on data entities, at least one label for each data entity and at least one set of labeling metrics, wherein the at least one labeling metrics is indicative of an acquiring of the label for the respective data entity; storing, after training, the labeling score model in a distributed database; generating a smart contract that comprises a method for determining a labeling score, wherein the method at least comprises applying a set of labeling metrics, which are indicative of an acquiring of labels from an agent, to the trained labeling score model, whereby the labeling score model is performed and yields, depending on the labeling metrics of the agent, a labeling score; and storing the smart contract in the distributed database.

Description

BRIEF DESCRIPTION

[0077] Some of the embodiments will be described in detail, with reference to the following figures, wherein like designations denote like members, wherein:

[0078] FIG. 1 schematically illustrates a manufacturing system for additive manufacturing comprising an additive manufacturing apparatus as well as a controlling apparatus according to an embodiment;

[0079] FIG. 2 represents a flowchart of a method for training a classification model for controlling a manufacturing process according to an embodiment;

[0080] FIG. 3 represents a flowchart of a method for controlling a manufacturing process according to an embodiment;

[0081] FIG. 4 represents a flowchart of a labeling method for generating a training set according to an embodiment;

[0082] FIG. 5 represents a flowchart of a validation method for training a classification model according to an embodiment;

[0083] FIG. 6 represents a flowchart of a method for generating artificial intelligence-based labeling agents as a further modification of a validation method according to an embodiment;

[0084] FIG. 7 represents a flowchart of a method for generating a smart contract as a further modification of a validation method according to an embodiment; and

[0085] FIG. 8 represents a flowchart of a method implementing a smart contract for determining a labeling score according to an embodiment.

DETAILED DESCRIPTION

[0086] In the following, various embodiments of the invention will be described in detail with reference to the appended figures. It is to be understood that the following description of embodiments is not to be taken in a limiting sense. The scope of embodiments of the invention are not intended to be limited by the embodiments described hereinafter or by figures, which are taken to be illustrative only.

[0087] The figures are to be regarded as being schematic representations and elements illustrated in the figures, which are not necessarily shown to scale. Rather, the various elements are represented such that their function and their general purpose become apparent to a person skilled in the art. Any connection or coupling between functional blocks, devices, components or other physical or functional units shown in the figures or described herein may also be implemented as an indirect connection or coupling. A coupling between components may be established over a wireless connection. Functional blocks may be implemented in hardware, firmware, software or a combination thereof.

[0088] FIG. 1 schematically illustrates a manufacturing system 10 for additive manufacturing comprising an additive manufacturing apparatus 60, a controlling apparatus 100, a labeling system 120, a validation system 150, and a distributed database 140; the controlling apparatus 100, the labeling system 120 and the validation system 150, their combination with the distributed database 140 and/or some or all of the further illustrated parts of the manufacturing system 10 each according to an embodiment of the invention.

[0089] In an exemplary embodiment, the distributed database 140 is implemented as a joint distributed infrastructure network including multiple infrastructure nodes (e.g., blockchain full nodes). According to an modification, the distributed database may be implemented as a blockchain supporting smart contracts like Ethereum. Furthermore, according to an modification, the multiple infrastructure nodes are private nodes in a private network, wherein access to the private nodes is restricted to the private network and/or to external organizations that have access to the private network.

[0090] In an exemplary embodiment, the additive manufacturing apparatus 60 comprises a powder bed 62 for metal powder, a storage container 64 for metal powder, and a recoater blade with an actuator 66, and a laser device 68. With the additive manufacturing apparatus 60 products may be manufactured by moving, by the recoater blade 66, metal powder from the storage container 64 into the powder bed 62, thereby creating a powder layer within the powder bed, and melting the metal powder of the powder layer by the laser device 68, such that such areas of the powder layer are connected that are part of the corresponding product or products to be manufactured. For manufacturing the whole product or products, this manufacturing process may be repeated for several consecutive layers. For some modifications of this manufacturing process, a property indicative of the manufacturing of the products is the homogeneity of the powder layer. For example, for a high quality of products a high homogeneity of the powder layer is required, wherein deviations in the distribution of the powder layer or one of the consecutive powder layers would result in deformations, instabilities, and/or missing parts of the manufactured products and/or maybe this result of a failure during melting or distributing powder for a previous layer. Also, according to some modifications of this manufacturing process and the respective additive manufacturing apparatus 60, a process parameter is the repetition of distributing metal powder for one of the powder layers, whereby in homogeneities of the respective powder layer may be reduced or removed. Moreover, the process parameter or a further process parameter according to some modifications, is a selection of one or more products manufactured during the manufacturing process, wherein one or more powder layers had an inhomogeneity within an area for the respective product or products, whereby these products, which might have missing or defective parts, can be removed from the further production chain.

[0091] In an exemplary embodiment, the controlling apparatus 100 comprises a sensor assembly 102, a data processing apparatus 108, and a control interface 106. Furthermore, in some modifications, the controlling apparatus 100 comprises a data storage 104 for storing a classification model. Alternatively, or additionally, the data processing apparatus 108 is adapted to retrieve the classification model from the distributed database 140.

[0092] The sensor assembly 102 is adapted to acquire a data entity, wherein the data entity is indicative of the homogeneity of a current powder layer within the powder bed 62. In some modifications, the sensor assembly 102 is an image capturing device directed to the powder bed 62, such that images of consecutive layers of powder can be captured.

[0093] The data processing apparatus 108 is adapted to classify the manufacturing of the product based on the data entity and the classification model. So, in some modifications, the data processing apparatus 108 is adapted to capture an image of the current powder layer by the image capturing device such as a camera 102 and classifies the currently created powder layer with regard to the homogeneity of the powder layer by the classification model. Moreover, the data processing apparatus 108 is adapted to output by the control interface 106 a control signal such that the homogeneity of the current powder layer is changed on the classifying. In some modifications, the data processing apparatus 108 is adapted to trigger the actuator of the recoater blade 66 and—in some further modifications—also the storage container 64, such that new metal powder is distributed within the powder bed 62 as the current powder layer to ensure sufficient homogeneity of the powder layer.

[0094] In an exemplary embodiment, the labeling system 120 comprises or is connected to a user interface 122 and is adapted to perform a labeling method, wherein labels are acquired via the user interface 122 from a first agent being a human user. Moreover, the labeling system 120 is adapted to store the acquired labels in the distributed database 140.

[0095] In an exemplary embodiment, the validation system comprises or is connected to a user interface 152 and is adapted to perform a validation method, wherein the classification model, which is used for classifying the data entity in the manufacturing process carried out by the manufacturing system 10, is trained based on the labels stored in the distributed database 140 and respective data entities for training and wherein the trained classification model is validated and adapted, in particular, if validation fails, by a supervisor, being a human user, via the user interface 152. Moreover, the validation system, in some modifications, is adapted to store the trained classification model in the distributed database 140.

[0096] While the exemplary embodiments are mainly elucidated by an additive manufacturing process using metal powder, some embodiments of the invention can also be carried out with other powders such as plastics or for other manufacturing processes, or more general, for other processes, wherein generating a training set for training a classification model is required.

[0097] FIG. 2 shows a flowchart of a method 200 for training a classification model for controlling a manufacturing process, the method 200 according to an embodiment of the invention. Moreover, in some modifications, the manufacturing process and a respective manufacturing system adapted to perform the manufacturing process is according to an embodiment of the invention and, in some further implementations, is according to FIG. 1.

[0098] In an exemplary embodiment, the method 200 comprises the method conditions 210, 211 and 218, and further comprises the method steps 222, 223, 224, 225, 226, 227, 228, 229, 232, 233, 234, 235, 242, 244, 252, 254, 256 and 258. The method starts at the start of the method 202 and ends at the end of the method 204. Furthermore, one or more of the method steps, in particular a sequence of method steps and/or the whole method 200 may repeatedly be performed.

[0099] At method step 222, a first set of data entities is provided, wherein each of the data entities is indicative of at least one property of a manufacturing of a respective product. In some modifications, each of the data entities is an image of a powder layer and is indicative of the homogeneity of the powder within the powder layer.

[0100] At method step 223, the first set of data entities is prefiltered by a predetermined, in particular, pre-trained classification model. Alternatively, the first set of data entities is prefiltered by predetermined statistical criteria. Alternatively, the first set of data entities is manually prefiltered.

[0101] At method step 224, one or more labels are acquired for a data entity of the first set of data entities from a first agent of a first group of agents. In some modifications, the labels are acquired via a user interface from a first human user being the first agent.

[0102] At method step 226, a set of labeling metrics is determined based on the acquiring from the first agent.

[0103] At method condition 210, it is determined, whether labels have been acquired for all data entities of the first set of entities or if other criteria such as the elapse of a predetermined time span are fulfilled, and in this case—which is symbolized by <y>—the method is continued at method step 242. Otherwise—symbolized by <n>—the method steps 224 and 226 are repeated for one or more further data entities of the first set of data entities.

[0104] In parallel or in sequence to method steps 222, 226 and evaluating method condition 210, the method steps 225 and 227 are performed and the method condition 211 is evaluated, wherein the method step 225 is like the method step 224, but performed for a second agent of a second group of agents, also being a human user or being an agent based on artificial intelligence. Correspondingly, at method step 227, a set of labeling metrics for the second agent is determined; and at the method condition 211 it is evaluated whether labels for all data entities of the first set of data entities have been acquired from the second agent or if other criteria such as the elapse of the predetermined time is fulfilled.

[0105] At method step 242, the labels and the set of labeling metrics are stored in a distributed database for each of the data entities, for which labels have been acquired, and for each of the agents. In some modifications, the method steps 222-227 as well as the method conditions 210 and 211 are comprised by a labeling method 220 and may be carried out by a labeling system. Moreover, also the method step 242 may be comprised by the labeling method 220 for generating a training set and may be carried out by a labeling system.

[0106] At method step 252, the labels and the data entities are retrieved from the distributed database. In some modifications, the method step 252 is comprised by a validation method 250 for training a classification model and may be carried out by a validation system.

[0107] At method step 254, which may also be comprised by the validation methods 250, the classification model is trained based on a training set, wherein the training set comprises the data entities and the respective labels.

[0108] At method step 256, which may also be comprised by the validation method 250, the trained classification model is validated based on predefined criteria, whereby a classifier score is yielded.

[0109] At method step 258, which may also be comprised by the validation method 250, a labeling score model is trained based on the data entities, the respective labels, the sets of labeling metrics and the classifier score.

[0110] At method step 244, which may also be comprised by the validation method 250, the trained classification model and the labeling score model are stored in the distributed database.

[0111] As method step 248, which may—in some modifications—be comprised by the labeling method 220 and, accordingly, be carried out by the labeling system, the labeling score model is retrieved from the distributed database.

[0112] At method step 228, which may also be comprised by the labeling method 220, a labeling score is determined for the first agent based on the labeling score model, the respective labels and set of labeling metrics. In some implementations, the labeling score is provided to the first agent, which allows to give continuous automated feedback on the performance/quality/accuracy of the labeling of the first agent.

[0113] At method step 232, which may also be comprised by the labeling method 220, the labeling score for the first agent, the respective set of metrics and labels are provided to the second agent, in particular, after acquiring labels from the first and/or second agent has been finished.

[0114] At method step 234, which may also be comprised by the labeling method 220, an agent classification score with respect to one or more of the data entities acquired from the first agent and an agent labeling score with respect to the labeling score and the set of labeling metrics of the first agent are acquired, after determining the labeling score for the first and for the second agent, from the second agent.

[0115] Correspondingly, the method steps 229, 233 and 235 are performed, wherein the method step 229 corresponds to method step 228, the method step 223 corresponds to method step 232 and the method step 235 corresponds to method step 234, wherein the roles of the first and the second agent have been exchanged.

[0116] At method condition 218, it is determined, whether a further iteration of the method 200—i.e., starting from method step 222 or 223 or method steps 224 and 225 is to be performed, which is symbolized by <y>. Otherwise, symbolized by <n>, the method ends at the method end 204.

[0117] When one or more further iterations are to be performed—i.e., method condition 218 yields <y>—, these iterations may be based on the first set of data entities or on one or more further sets of data entities, so that, consequently, a further set of data entities is provided at method step 222. Moreover, at method step 223, in some modifications, the respective set of data entities is filtered with the classification model of the previous iteration. Moreover, in some modifications, the training set for training the classification model further comprises the agent classification scores acquired in a previous iteration. Moreover, in some modifications, the training of the labeling score model is further based on the agent classification scores and/or the agent labeling scores of one of the previous iterations.

[0118] In some modifications, the labeling method 220 and the validation method 250 are performed in parallel, wherein a labeling score is directly determined and provided to the respective agent for a particular data item and respective labels after the respective labels and the respective set of labeling metrics have been acquired based on a predetermined labeling score model or on a labeling score model of one of the previous iterations. This may beneficially allowed to reduce a time span between entering labels and receiving feedback, so that the quality of the labeling may be enhanced.

[0119] FIG. 3 shows a flowchart of a method 300 for controlling a manufacturing process according to an embodiment of the invention. In some implementations, the method 300 is performed by the controlling apparatus 100 of FIG. 1, which is adapted accordingly.

[0120] In an exemplary embodiment, the method 300 comprises the method steps 322, 324, 326 and 328. The method 300 starts at the start of the method 302 and ends at the end of the method 304. Furthermore, one or more of the method steps, in particular, a sequence of method steps and/or the whole method 300 may be performed repeatedly. Thereby, in some implementations, multiple production runs may be performed and controlled.

[0121] At method step 322, a data entity being indicative of at least one property of a manufacturing of a respective product such as the homogeneity of a powder layer is acquired.

[0122] At method step 324, a classification model is retrieved. In some modifications, the classification model is retrieved from a data storage 104 of the controlling apparatus 100. Alternatively, or additionally, the classification model is retrieved from a distributed database such as the distributed database 140.

[0123] At method step 328, the manufacturing of the product is classified based on the data entity and the classification model.

[0124] At method step 326, the at least one process parameter is adapted based on the classifying.

[0125] In some modifications, the classification model is trained by the method 200.

[0126] FIG. 4 represents a flowchart of a labeling method 220 for generating a training set, the method 220 according to an embodiment of the invention. Moreover, FIG. 4 shows details of an implementation of the method 220 based on encryption and smart contracts.

[0127] In an implementation, the labeling method 220 starts at the start of the method 402 and ends at the end of method 404 and comprises the following method steps:

[0128] At method step 420, an event is received that indicates that one or more additional data entities are available. Such data entities are, in some modifications, continuously acquired from a manufacturing system, in particular, from a sensor assembly of the manufacturing system.

[0129] At method step 422, the data entities are received and arranged into a set of data entities. In some modifications, the set of data entities may also comprise data entities received from previous events and being stored in a data storage such as a distributed database.

[0130] At method step 423, the data entities of the set of data entities are prefiltered. In some modifications, this prefiltering is implemented as described with respect to method step 223.

[0131] At method step 424, which may relate to method step 224, a user starts to review the data entities and to enter one or more labels for some or all of the data entities of the set of data entities.

[0132] While the user enters labels, labeling metrics are acquired at method step 426.

[0133] At method step 425, a label or several labels with respect to one of the data entities are created based on the input entered by the user.

[0134] At method step 427, the created labels and the respective data entities are encrypted.

[0135] At method step 428, a transaction is started based on a smart contract stored in the distributed database, wherein the following steps of the smart contract are performed: At method step 482, the smart contract receives the encrypted data. At method step 484, the data is decrypted. At method step 486, the decrypted data is evaluated based on the data entities, the labels and the set of labeling metrics and, at method step 487, a labeling score for the user based on the data of the transaction is calculated. At method step 488, a further transaction is started, wherein, at method step 489, the calculated labeling score is stored in the distributed database and accounted to the user or to a group to which the user is assigned to.

[0136] FIG. 5 shows a flowchart of a validation method 250 with respect to training a classification model, the method 250 according to an embodiment of the invention. Moreover, FIG. 5 shows further details of some implementations of the method 250.

[0137] In an exemplary embodiment, the method 250 starts at the start of the method 502 and ends at the end of the method 504, and comprises the method condition 510 as well as the method steps 520, 544, 552, 554, 556 and 557.

[0138] At method step 520, it is determined, whether acquiring labels for a set of data entities is finished. This may, in some modifications, be the case if a certain time span elapsed or if some or all of the data entities of the set of data entities have been acquired or if a labeling method such as the labeling method 220 or a respective iteration of it is finished.

[0139] After entering of labels has been finished according to step 520, all data such as the data entities of a particular set of data entities, respective labels and labeling metrics are collected from a distributed database, at method step 552.

[0140] At method step 554, which may correspond to method step 254, a classification model is trained based on a training set comprising the data collected from the distributed database.

[0141] At method step 556, which may correspond to and may be implemented according to method step 256, the trained classification model is validated based on predefined criteria, wherein a classifier score is yielded.

[0142] At method condition 510, it is determined whether the trained classification model fits with predetermined conditions such as a predetermined range for the classifier score or a classification of a predetermined set of data entities as a validation set of data entities.

[0143] If this is not the case—symbolized by <n>—, an input from a supervisor is acquired at method step 557, whereby one or more parameters of the classification model, i.e., the trained classification model, are fine-tuned.

[0144] After step 557, or when method condition 510 was fulfilled—symbolized by <y>—, method step 544, which may relate to method step 244, is performed, wherein the trained classification model is sent to a labeling system and/or is stored in the distributed database.

[0145] FIG. 6 represents a flowchart of a further modification of the validation method 250—or a part thereof—according to an embodiment of the invention. This further modification relates to an implementation, wherein one or more or all of the agents are agents based on artificial intelligence. Moreover, this implementation is combined with one of the previous implementations for a system of human users as agents as well as artificial intelligence-based agents.

[0146] In an exemplary embodiment of this further modification of method 250, method 250 further comprises the method steps 553, 560, and 564.

[0147] Method step 553 is performed after collecting all data from the distributed database—i.e., method step 552—and at method step 553, their labels and sets of labeling metrics as well as respective data entities acquired from agents that are agents based on artificial intelligence are evaluated and compared against corresponding data, such as corresponding labels, sets of labeling metrics and data entities acquired from human users or previously stored as a predetermined evaluation data.

[0148] At method step 560, one or more agents based on artificial intelligence are generated for a further iteration based on the collected data and/or the trained classification model. In some implementations, the artificial intelligence agents are generated by modifying the trained classification model by random variations of parameters of the classification model. In some implementations, the artificial intelligence agents are generated by a genetic algorithm. For this purpose, in some further implementations, the artificial intelligence agents are selected based of their performance of previous iterations and/or further agents are generated based on the most successful agents and/or wherein a predetermined number of worse performing agents—i.e., having a low labeling score—are removed from a pool of agents.

[0149] Likewise, human user may be excluded from a group of (human) agents, if the labeling score of this user is below a predetermined threshold and/or if this user is among a set of (human) agents of a predetermined size which perform worst—i.e., in particular, if this user is among the 5 worst performing (human) agents.

[0150] At method step 564, the agents based on artificial intelligence, generated at method step 560, are stored in a distributed database or an AI labeling hub. Accordingly, the artificial intelligence agents are retrieved, in some modifications of a labeling method such as the labeling method 220, from the distributed database and, at method step 224 or 225, labels are acquired from these agents based on artificial intelligence. In some implementations the AI labeling hub is a computer system such as an arrangement of one or more servers, wherein the agents based on artificial intelligence are stored on the server(s) and wherein the server(s) provide an environment for running the artificial intelligence-based agents.

[0151] FIG. 7 represents a flowchart of a method for generating a smart contract as part of a further modification of a validation method according to an embodiment of the invention, wherein the smart contract is adapted to determine a labeling score.

[0152] In an exemplary embodiment of this further modification of method 250, method 250 further comprises the method conditions 512 and 514 and the method steps 558, 559, 576, 577, 580, 582, 584, and 586.

[0153] After collecting all data from the distributed database—i.e., method step 552—, at method condition 512, it is determined whether one or more agent classification scores and/or one or more agent labeling scores have been acquired. If this is the case—symbolized by <y>—the labeling score model is trained based on the data entities, the labels, the sets of labeling metrics and the classifier score as well as the agent classification scores or, respectively, the agent labeling scores, at method step 559. Furthermore, prior to training the labeling score model, the agent labeling scores and/or the agent classification scores are, in some further modifications, transformed to one or more numerical parameters based on a statistical method or based on artificial intelligence. This may allow to gather agent labeling scores or agent classification scores—at least partially—based on a free-form text input, which is then transformed into numerical parameters.

[0154] Otherwise—symbolized by <n>—, at method step 558, which may relate—in some implementations—to method step 258, the labeling score model is trained based on the data entities, the labels, the sets of labeling metrics and the classifier score.

[0155] At method step 576, after training the labeling score model in one of the steps 558 or 559, the trained labeling score model is validated based on predefined criteria such as predetermined minimum or maximum labeling scores, test data for the labeling score model and/or a statistic analysis.

[0156] At method condition 514, it is determined whether the trained labeling score model fits with predetermined conditions such as the validation of step 576, and if this is not the case—symbolized by <n>, an input from a supervisor is acquired at method step 577, whereby one or more parameters of the labeling score model are fine-tuned, whereby the labeling score model is adapted such that, after fine-tuning, it passes the validation and such that it yields, at least for a training set of validation data, labeling scores in a predetermined range for respective sets of labeling metrics of the training set of validation data.

[0157] According to some embodiments of the invention, the labeling score model, especially the trained labeling score model, is adapted such that based on a respective set of labeling metrics, it yields a higher labeling score for an agent:

[0158] who entered more labels than another agent,

[0159] who entered a predetermined amount of labels in a shorter time span,

[0160] who entered labels for entities that have a high influence on training the classification model—i.e., in the sense of active learning;

[0161] who entered labels that are similar to labels of a group of this agent or of other groups of agents or

[0162] who received a high agent classification score or agent labeling score from one or more other agents.

[0163] Likewise, another agent, for whom the previous criteria are not met or are only met to a lower extent, will be yielded a lower labeling score by the labeling score model, wherein, in some further modifications, the labeling score is restricted to a predetermined range, wherein the predetermined range starts at a predetermined positive number, in particular the bigger than five, which may allow to mitigate the effects caused by singularities with regard to the convergence of the labeling score model, when training that labeling score model, and/or instabilities when determining the respective labeling score based on the labeling score model.

[0164] Moreover, in some embodiments, the determining of the labeling score is controlled by fixed boundaries such as relaxations or weights between different labeling metrics. In some modifications, a higher quality of labeling has a higher weight than the time span required for labeling. According to some further modifications, a wrong label may be equated with a time malus of 2 Minutes, wherein, in particular, a wrong label may be determined based on the labeling of other agents.

[0165] After method step 577, or when method condition 514 was fulfilled—symbolized by <y>—, method step 580 is performed, wherein a smart contract for determining labeling scores is generated based on the trained—and potentially fine-tuned—labeling score model.

[0166] At method step 582, a transaction on the distributed database is started, wherein at method step 584 the generated smart contract is deployed to the distributed database—i.e., in particular stored in the distributed database—, and wherein at method step 586 smart contracts of previous iterations—if any—are invalidated.

[0167] FIG. 8 shows a smart contract 800 for determining a labeling score and its implementation as a method 840 according to an embodiment of the invention.

[0168] According to some modifications, such a smart contract 800 is generated by the validation method 250. Moreover, according to some modifications, such a smart contract is retrieved from a distributed database and is executed by/within a labeling method 220 in order to determine the respective labeling score of an agent.

[0169] In an exemplary embodiment, the method 840 comprises the method steps 842, 844, 846, 847, 848, and 849, starts at method stand 802 and ends at method end 804.

[0170] At method step 842, for a respective agent, the one or more labels and the set of labeling metrics and the labeling score model are provided. In some implementations, these may be retrieved from a distributed database. Alternatively, this may be retrieved from a labeling system. Alternatively, or additionally, the smart contract may be performed on a labeling system such that the labels, the labeling metrics and the labeling score model are readily stored and available at the labeling system.

[0171] At method step 844, if the data is stored in encrypted form and e.g., retrieved from the distributed database, the data is decrypted.

[0172] At method step 846, the set of labeling metrics and the labels are applied to the labeling score model and, accordingly at method step 847, by the labeling score model, a respective labeling score is yielded for the respective agent, i.e., the first agent or another agent.

[0173] At method step 848, a further transaction is started—in some implementations on the distributed database—, wherein, at method step 849, the yielded labeling score is stored in a data storage, in particular in the distributed database, and accounted to the respective agent or to a group to which the agent is assigned to.

[0174] In some modifications, the smart contract 800 is used and, accordingly the method 840 is performed for determining the labeling score in method step 228.

[0175] Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention.

[0176] For the sake of clarity, it is to be understood that the use of “a” or “an” throughout this application does not exclude a plurality, and “comprising” does not exclude other steps or elements.