SYSTEMS METHODS AND COMPUTATIONAL DEVICES FOR AUTOMATED CONTROL OF INDUSTRIAL PRODUCTION PROCESSES

Abstract

A system and method for optimized industrial production using machine learning. The method includes creating a model defining dependencies among a plurality of parameters for an industrial production process, the plurality of parameters including a plurality of controlled parameters and a plurality of monitored parameters; training an agent via reinforcement learning based on iterative application of the model, wherein the agent is trained to determine new values for the plurality of controlled parameters based on current values of the plurality of monitored parameters in order to optimize the industrial production process with respect to at least one predetermined objective; and iteratively modifying, by the trained agent, current values of the plurality of controlled parameters in real-time during operation of the industrial production process.

Claims

1-31. (canceled)

32. A method for optimized industrial production using machine learning, comprising: creating a model defining dependencies among a plurality of parameters for an industrial production process, the plurality of parameters including a plurality of controlled parameters and a plurality of monitored parameters; training an agent via reinforcement learning based on iterative application of the model, wherein the agent is trained to determine new values for the plurality of controlled parameters based on current values of the plurality of monitored parameters in order to optimize the industrial production process with respect to at least one predetermined objective; and iteratively modifying, by the trained agent, current values of the plurality of controlled parameters in real-time during operation of the industrial production process.

33. The method of claim 32, wherein training the agent further comprises: simulating a portion of the plurality of monitored parameters using the model in order to generate artificial data, wherein the agent is trained at least partially using the artificial data.

34. The method of claim 33, wherein the artificial data includes a plurality of artificial parameters, further comprising: validating the model by comparing the plurality of artificial parameters to a plurality of test parameters measured during a production run of the industrial production process.

35. The method of claim 34, wherein validating the model further comprises: determining a difference between the plurality of artificial parameters and the plurality of test parameters, wherein the model is validated when the difference is below a threshold.

36. The method of claim 34, wherein validating the model further comprises: selecting a plurality of input values; and processing the plurality of input values using the model in order to determine the plurality of artificial parameters, wherein the plurality of test parameters includes historical monitored parameters for the industrial production process.

37. The method of claim 32, wherein training the agent further comprises: iteratively determining at least one reward, wherein each reward is a score function defined with respect to one of the at least one predetermined objective; and updating the agent based on the at least one reward determined at each iteration.

38. The method of claim 37, wherein the agent has at least one weight value defining the dependency between the plurality of controlled parameters and the plurality of monitored parameters, wherein updating the agent further comprises: determining at least one new value for the at least one weight value; and changing at least a portion of the at least one weight value based on the determined at least one new value.

39. The method of claim 32, further comprising: dividing the industrial production process into a plurality of phases; and determining an initial set of controlled parameters for each of the plurality of phases, wherein the plurality of controlled parameters is initialized to the respective initial set of controlled parameters at the beginning of each phase.

40. The method of claim 32, wherein the at least one predetermined objective includes at least one of: high product yield, short fermentation duration, low impurity value, product quality, and process efficiency.

41. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising: creating a model defining dependencies among a plurality of parameters for an industrial production process, the plurality of parameters including a plurality of controlled parameters and a plurality of monitored parameters; training an agent via reinforcement learning based on iterative application of the model, wherein the agent is trained to determine new values for the plurality of controlled parameters based on current values of the plurality of monitored parameters in order to optimize the industrial production process with respect to at least one predetermined objective; and iteratively modifying, by the trained agent, current values of the plurality of controlled parameters in real-time during operation of the industrial production process.

42. A system for optimized industrial production using machine learning, comprising: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: create a model defining dependencies among a plurality of parameters for an industrial production process, the plurality of parameters including a plurality of controlled parameters and a plurality of monitored parameters; train an agent via reinforcement learning based on iterative application of the model, wherein the agent is trained to determine new values for the plurality of controlled parameters based on current values of the plurality of monitored parameters in order to optimize the industrial production process with respect to at least one predetermined objective; and iteratively modify, by the trained agent, current values of the plurality of controlled parameters in real-time during operation of the industrial production process.

43. The system of claim 42, wherein the system is further configured to: simulate a portion of the plurality of monitored parameters using the model in order to generate artificial data, wherein the agent is trained at least partially using the artificial data.

44. The system of claim 43, wherein the artificial data includes a plurality of artificial parameters, wherein the system is further configured to: validate the model by comparing the plurality of artificial parameters to a plurality of test parameters measured during a production run of the industrial production process.

45. The system of claim 44, wherein the system is further configured to: determine a difference between the plurality of artificial parameters and the plurality of test parameters, wherein the model is validated when the difference is below a threshold.

46. The system of claim 44, wherein the system is further configured to: select a plurality of input values; and process the plurality of input values using the model in order to determine the plurality of artificial parameters, wherein the plurality of test parameters includes historical monitored parameters for the industrial production process.

47. The system of claim 42, wherein the system is further configured to: iteratively determine at least one reward, wherein each reward is a score function defined with respect to one of the at least one predetermined objective; and update the agent based on the at least one reward determined at each iteration.

48. The system of claim 47, wherein the agent has at least one weight value defining the dependency between the plurality of controlled parameters and the plurality of monitored parameters, wherein the system is further configured to: determine at least one new value for the at least one weight value; and change at least a portion of the at least one weight value based on the determined at least one new value.

49. The system of claim 42, wherein the system is further configured to: divide the industrial production process into a plurality of phases; and determine an initial set of controlled parameters for each of the plurality of phases, wherein the plurality of controlled parameters is initialized to the respective initial set of controlled parameters at the beginning of each phase.

50. The system of claim 42, wherein the at least one predetermined objective includes at least one of: high product yield, short fermentation duration, low impurity value, product quality, and process efficiency.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The subject matter disclosed herein is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the disclosed embodiments will be apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0074] FIG. 1 schematically shows CO.sub.2 concentration (curve A), biomass concentration (curve B), and carbon source concentration (curve C) as functions of time;

[0075] FIG. 2 is a graph showing CO.sub.2 concentration as a function of time for an actual production run that shows the effect of lack of nitrogen source on the CO.sub.2 concentration during the production phase of an example fermentation process emphasizing how a non-carbon source can affect and be described by the CO.sub.2 dynamic;

[0076] FIG. 3 is a graph showing how a method for controlling the process can save time in an example fermentation process;

[0077] FIG. 4 schematically shows a closed loop system for optimized feeding of the carbon source and the nitrogen source feeding in an example fermentation process;

[0078] FIG. 5 shows a comparison of yield of fed-batch produced product, for production runs carried out by following the protocol previously used with yield obtained by using the system;

[0079] FIG. 6 shows graphs of CO.sub.2 concentration and carbon source feeding as functions of time during a production run for a secondary derivative in which the carbon source was fed according to the standard protocol followed for production of the product;

[0080] FIG. 7 shows graphs of CO.sub.2 concentration and carbon source feeding as functions of time during a production run for a secondary derivative in which the carbon source was fed according to the method; and

[0081] FIG. 8 schematically shows the reinforcement learning iterative training phase of an embodiment of the method;

[0082] FIG. 9 schematically shows control of the fermenter by the trained agent during live production runs;

[0083] FIG. 10 schematically shows an embodiment of a closed loop system configured for carrying out an embodiment of the method for optimizing values of the nutrient feeding and physical parameters in an example fermentation process;

[0084] FIG. 11 is a schematic diagram of an example computing environment according to some embodiments;

[0085] FIG. 12 shows graphs of the predicted and measured biomass concentration (top left graph), dissolved oxygen (bottom left graph), carbon source concentration (top right graph), and desired product concentration (bottom right graph) as measured during the validation phase of the model according to some embodiments;

[0086] FIG. 13 shows a graph displaying the learning process of a reinforcement learning algorithm according to some embodiments;

[0087] FIG. 14 is an example code of a reinforced learning machine according to some embodiments.

DETAILED DESCRIPTION

[0088] It is important to note that the embodiments disclosed herein are only examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise indicated, singular elements may be in plural and vice versa with no loss of generality. In the drawings, like numerals refer to like parts through several views.

[0089] Some disclosed embodiments relate to controlling particularly an example industrial production process by a reactor controller that regulates controlled parameters of the reactor continually during the process. In one or more embodiments, the industrial production process inter alia includes research and development processes, processes of pilot facilities, processes of demo facilities, fermentation processes, bio-reactor processes, and chemical processes. In one or more embodiments, the reactor includes various vessel processes including, but not limited to a bio-reactor, a chemical reactor and a fermenter. In one or more embodiments, the controlled parameters, inter alia include the amounts of nutrient sources to feed the process, the timing of the feedings, and/or physical parameters such as agitation, aeration rates and temperature control.

[0090] In an embodiment, the method generally comprises several phases for obtaining an agent trained using a mathematical model that simulates the actual production process of a reactor.

[0091] A controller as herein disclosed includes a trained agent or the controller may be trained to obtain a trained agent that can maximize objectives of the production process.

[0092] Thus, some embodiments include methods for regulating a production process of a reactor.

[0093] Some embodiments include a controller with a trained agent or an agent that can be trained based on a model that mimics the production process of a reactor.

[0094] Some embodiments include systems with a controller, a local agent and a reactor.

[0095] In one or more embodiments, the method includes a phase in which a mathematical model is constructed; a learning/optimization phase, and a production, i.e. “real time”, phase. These phases of the method are unique for each industrial production or fermentation process and the exact steps required to carry them out must be determined specifically for each particular process. In one or more embodiments, a model constructed for a particular process is optimized during the learning/optimization phase and the optimized model is then used to obtain a trained agent. In one or more embodiments an agent is trained during the learning/optimization phase and the optimized agent or trained agent is used to automatically control and optimize production runs of an example fermentation process.

[0096] The mathematical model is optionally a set of equations, optionally differential equations collectively comprising parameters that describe different aspects of the specific example production process being controlled and optimized. The equations may be based on the academic literature, past data collected on the process, and the results of specifically designed experiments.

[0097] The use of differential equations as a basis for a model representing growth and activity of microorganisms is known and used in research and several industries for a better understanding of the interactions of different elements in the process, and in some cases as a basis for improving current protocols using knowledge gained from the model. Other control mechanisms such as pH control or dO.sub.2 control, used in the example fermentation processes, calculate input values using a strict set of rules, wherein each variable measured could usually influence changes in one input value controlled. For example, pH can be titrated to adjust a specific set point and dissolved oxygen (dO.sub.2 control) can be used as a set point to control fermentation parameters such as temperature and pressure (agitation/airflow adjustment).

[0098] The disclosed method, using a model of the specific process, integrates all live measured data, along with data from past measurements of the process for a full image of the current conditions of the fermenter. The controller integrates machine learning and optimization methods to find the best possible input or controlled parameters to the fermentation vessel at each time during the process, e.g. quantity of C and N source, temperature, agitation or aeration rate.

[0099] Optionally, machine learning and optimization methods as herein disclosed make use of past and/or real time data collected from a reactor to find the best possible controlled parameters to the fermentation vessel.

[0100] A model can be created for different products produced by the example fermentation process. In a specific example, the production of biomass, i.e. microbial cells or biomass is sometimes the intended product of an example fermentation process. Non limiting examples of such processes include production of single cell protein, baker's yeast, lactobacillus, E. coli, and other, extracellular primary metabolites and secondary metabolites. Some examples of primary metabolites are ethanol, citric acid, glutamic acid, lysine, vitamins and polysaccharides. Some examples of secondary metabolites are penicillin, cyclosporin A, gibberellin, and lovastatin. These compounds are of obvious value to humans wishing to prevent the growth of bacteria, either as fed-batch produced products or as antiseptics (such as gramicidin S) or fungicides, such as griseofulvin, which are also produced as secondary metabolites. Typically, secondary metabolites are not produced in the presence of glucose or other carbon sources which would encourage growth and like primary metabolites are released into the surrounding medium without rupture of the cell membrane. Of primary interest among the intracellular components are microbial enzymes: catalase, amylase, protease, pectinase, glucose isomerase, cellulase, hemicellulase, lipase, lactase, streptokinase and many others. Examples of recombinant proteins that are produced in fermentation processes include insulin, hepatitis B vaccine, interferon, granulocyte colony-stimulating factor, and streptokinase.

[0101] A specific example of creation of a model for secondary metabolites in a fed-batch fermentation process follows:

[0102] In this model, t represents the time and the model is updated with a time differential of dt.

[0103] The biomass trend is given by equation (1):

X(t+1)=X(t)+dt(X(t)(μ(t−K.sub.d))  (1)

[0104] wherein: X(t) is the biomass concentration in the fermenter at time t, K.sub.d is the death factor constant of the cells, and μ(t) is the growth rate of the cells at time t, X(t+1) is the value of X(t) one minute after t, and μ(t) is given by equation (2):

[00001]$\begin{array}{cc}\mu \left(t\right)={\mu }_x\left(\frac{S\left(t\right)}{K_x+S\left(t\right)}\right)\left(\frac{CL\left(t\right)}{CL\left(t\right)+K_{ox}}\right)\left(\frac{A\left(t\right)}{A\left(t\right)+K_{xa}}\right)& \left(2\right)\end{array}$

[0105] wherein: μ.sub.x is the maximal growth rate constant of the cells, K.sub.x is the carbon source limitation constant for growth, K.sub.ox is the oxygen limitation constant for growth, S(t) is the carbon source concentration in the fermenter at time t, and CL(t) is the dissolved oxygen concentration at time t, A(t) is the nitrogen source concentration in the fermenter at time t, and K.sub.xa is the nitrogen source limitation constant for growth.

[0106] The production trend is given by equation (3):

P(t+1)=P(t)+dt(μ.sub.pp(t)X(t)−KP(t))  (3)

[0107] wherein: P(t) is the product concentration in the fermenter, K is the product hydrolysis rate constant, and μ.sub.pp(t) is the production rate at time t, μ.sub.pp(t) is given by equation (4):

[00002]$\begin{array}{cc}{\mu }_{pp}\left(t\right)={\mu }_p\left(\frac{A\left(t\right)}{K_p+A\left(t\right)}\right)\left(\frac{CL\left(t\right)}{CL\left(t\right)+K_{op}}\right)\left(\frac{S\left(t\right)}{K_I+S\left(t\right)}\right)& \left(4\right)\end{array}$

[0108] wherein, μ.sub.p is the maximal production rate constant, K.sub.p is the production inhibition constant for ammonia, K.sub.op is the production inhibition constant for dissolved oxygen, and K.sub.I is the inhibition constant for dextrose.

[0109] The carbon source is used for cell growth, production and maintenance of the fermentation process. The amount of carbon source in the fermentation vessel decreases with time and can be increased by feeding during the process. The carbon source trend is given by equation (5):

[00003]$\begin{array}{cc}S\left(t+1\right)=S\left(t\right)+dt\left[X\left(t\right)\left(-\frac{\mu \left(t\right)}{Y_{x/s}}-m_x-\frac{{\mu }_{pp}\left(t\right)}{Y_{p/s}}\right)+\frac{S_{in}\left(t\right)}{\mathrm{Vessel}\mathrm{volume}}\right]& \left(5\right)\end{array}$

[0110] wherein, S(t) is the carbon source concentration in the fermenter at time t, Y.sub.x/s is the growth yield constant for the carbon source, Y.sub.p/s is the production yield constant for the carbon source, m.sub.x is the maintenance constant of the carbon source, and S.sub.in is the carbon source feeding value.

[0111] Nitrogen is needed for production. The amount of nitrogen can be increased when needed by feeding. The nitrogen source trend is given by equation (6):

[00004]$\begin{array}{cc}A\left(t+1\right)=A\left(t\right)+dt*(X\left(t\right)\left(-\frac{{\mu }_{pp\left(t\right)}}{Y_{p/a}}-\frac{\mu \left(t\right)}{Y_{p/s}}+\frac{A_{in}\left(t\right)}{\mathrm{Vessel}\mathrm{Volume}}\right)& \left(6\right)\end{array}$

[0112] wherein: A(t) is the nitrogen source concentration in the fermenter at time t, Y.sub.p/a-Growth Yield constant for the nitrogen source, μ.sub.pp is the specific fed-batch produced product production rate, and A.sub.in is the Nitrogen source feeding value.

[0113] The dissolved oxygen trend, showing the uptake of oxygen by the cells, is given by equation (7):

[00005]$\begin{array}{cc}\mathrm{CL}\left(t+1\right)=\mathrm{CL}\left(t\right)+dt\left[X\left(t\right)\left(\frac{-\mu \left(t\right)}{Y_{x/o}}-m_o-\frac{{\mu }_{pp}\left(t\right)}{Y_{p/o}}\right)+K_{la}\left(C{L}^{*}-CL\left(t\right)\right)\right]& \left(7\right)\end{array}$

[0114] wherein: CL(t) is the dissolved oxygen level in the fermenter at time t, CL* is the maximal dissolved oxygen concentration, Y.sub.x/o is the growth yield constant for dissolved oxygen, Y.sub.p/o is the production yield constant for dissolved oxygen, m.sub.o is the maintenance constant of dissolved oxygen, and K.sub.la is the oxygen insertion constant.

[0115] Each process has its specific properties and different fermentation processes will have different values of these properties in the above equations as well as a different set of equations. Properties could be added or removed for example in cases of inducers, a second carbon/nitrogen source, or a second product. The equations could change as well due to different kinetics and relations between variables. Different processes could be a result of different fed-batch produced products, different organism (bacterium or fungi), or different fermentation procedures.

[0116] In view of the above, one or more of the herein disclosed systems and methods include one or more of the following stages:

[0117] A Model Creation Phase

[0118] Stage 1—The mathematical model is built, i.e., theoretical equations that describe various aspects of the process are chosen. The equations that are selected collectively comprise various, optionally all parameters that describe different aspects of the specific fermentation process being investigated.

[0119] Stage 2—Data relating to the values of the parameters in the equations is gathered from production runs and observation trials. In this stage, data is collected from as many real production runs and from variations to the real runs that are performed during the observation trials.

[0120] Stage 3—The data of controlled parameters collected in stage 2 is inserted into the equations, which are simultaneously solved to obtain predictive output of predictive monitored parameters.

[0121] Stage 4—a comparison is then performed between real time output of monitored parameters and the predictive values, and a base model that best fits the production process is chosen.

[0122] An Agent Creation Phase

[0123] Stage 5—Machine learning techniques and the model are used to create a trained agent that is used for future production runs.

[0124] In one or more embodiments, the above stages 1 to 5 are conducted offline, i.e., when not connected to an actual real time production process but rather performed artificially using the model that simulates the actual production process of the reactor.

[0125] The following is an example explanation of Carbon and Nitrogen source feeding based on the disclosed model.

[0126] One of the factors leading to less than optimal yields and profitability of fermentation processes as carried out today in industries such as the pharmaceutical industry is that material, such as the carbon source that is necessary to promote cell growth and production is added to the fermentation vessel in predetermined quantities at fixed times that have been determined by trial and error during an initial running-in period of the process before commercial production of a new product begins. The nitrogen source is added in real time by titrating the pH during the fermentation process.

[0127] Based on their conviction that yields and profitability can be significantly increased by feeding the carbon and nitrogen sources only in the amount and at the time that is needed, the inventors have developed a method and a controller for using the model derived as described herein to dynamically provide optimal values of selected controlled parameters to a reactor. The controller as herein disclosed receives real-time data of monitored parameters measured by sensors attached to the fermenter and instructs a local agent and/or equipment controller devices of the reactor (e.g., a pump, an agitation device, a nutrient feeding device, etc.,) to adjust the controlled parameters of the reactor in order to optimize the process with respect to quantity and purity of the final product and the overall cost. For example, the value of the pH will be controlled by addition of nitrogen source, e.g. ammonia; dO.sub.2 will be controlled by adjusting pressure or temperature; CO.sub.2 concentration will be controlled by addition of carbon source, e.g. glucose, by agitation, or by adjusting the values of other parameters that will affect the biomass trend. Specifically, since overfeeding can cause toxicity and underfeeding will cause increased CO.sub.2 levels with increased biomass growth and no production. Both situations are described in the model, which is optimized to provide the fermentation controller with controlled parameters values that will prevent either of them from occurring.

[0128] Nitrogen source along with carbon source are two substrates necessary for an example fermentation process. During the growth phase, the carbon source is used in a “Krebs cycle” (glycolysis cycle) and CO.sub.2 is released. During the production phase, cell growth is reduced and equilibrium between carbon and nitrogen source is required for high yield production. Lack of carbon source concentration will cause reduced production, cell maintenance and cell growth (biomass), and therefore will cause a decrease in CO.sub.2 levels. On the other hand, lack of nitrogen source concentration needed for product creation will shift the culture back to the growth phase, meaning that the carbon source will be used for glycolysis, and the CO.sub.2 concentration will increase. These traits are modeled, as described herein above, by adjusting the equations, finding the relevant values of the properties, and optimized for an efficient and productive process.

[0129] During the rapid growth rate of cells, a minimal medium containing, e.g. glucose, is required as the sole source of carbon. During the growth phase metabolism of glucose to smaller molecules (e.g., CO2, ethanol, or acetic acid) can generate the ATP necessary for energy-requiring activities of the cells. The sole nitrogen source in a minimal medium can be ammonium (NH4+), from which the cells can synthesize all the necessary amino acids and other nitrogen-containing metabolites.

[0130] FIG. 1 schematically shows CO.sub.2 concentration (curve A), biomass concentration (curve B), and carbon source concentration (curve C) as functions of time for a typical fermentation process. High correlation between CO.sub.2 and biomass concentrations, especially during the growth stage, is visible. This is with negative correlation between the CO.sub.2 concentration and the carbon source concentration, meaning that the carbon source is being used for the biomass growth. During the production phase, biomass growth rate decreases and the resources are used for the second metabolite formation as well.

[0131] FIG. 2 is a graph showing CO.sub.2 concentration as a function of time for an actual production run that shows the effect of lack of nitrogen source on the CO.sub.2 concentration during the production phase of an example fermentation process. In the figure the dashed line is the set point for the CO.sub.2 concentration with the values of the set point written above the line. The rate at which the carbon source (sugar) is fed at various stages into the fermenter is written next to the curve. Carbon source is fed starting after about 5.75 hours in equal doses every minute, e.g. during the growth phase 2.2 kg of sugar are added each minute. Feeding with ammonia begins at about 6.75 hours (indicated by a downward pointing arrow). The amount and timing of ammonia feeding was controlled to keep the pH within predetermined upper and lower limits. Between 7.25 and 7.5 hours, during the period marked by the ellipse with the vertically pointing arrow at its bottom, the ammonia supply was exhausted, and no ammonia was fed until a new supply was prepared. During this time it can be seen how the process shifted from the production to the growth phase accompanied by a rapid rise in CO.sub.2 concentration.

[0132] As discussed above and shown schematically in FIG. 1, during the growth phase the biomass state is closely coordinated with the state of the CO.sub.2 concentration; and, as shown in FIG. 2, during the production stage, the CO.sub.2 concentration is influenced by both the nitrogen and the carbon source concentrations. The CO.sub.2 concentration at any time during the process depends on the metabolism of cells in the fermenter, which in turn depends directly on the feeding rate of the carbon and nitrogen sources. These facts indicate that CO.sub.2 concentration can be useful in feeding control of both C and N; and, in view of them, the inventors have developed a closed loop system that uses mathematical model of the process derived by the method described herein above to control the level of the CO.sub.2 concentration as a function of time, thereby providing a means of controlling the feeding of the carbon and nitrogen sources in an example fermentation process in accordance with the requirements of the process.

[0133] Some disclosed embodiments include a method of controlling an example fermentation process by a fermenter controller that regulates controlled parameters of the fermenter. Steps of such a method may include construction of a digital model that mimics the behavior of the fermentation process, processing input controlled parameter values of actual real time production runs by the model and obtaining predictive values of the monitored parameters, and comparing the values of these parameters to monitored values obtained received in real time during production runs from sensors at the fermenter. Comparison of the output from the model to the real time output data of real production runs is then used to obtain a model that mostly fits or mimics the actual behavior of the process. The input values calculated by the model may include controlled parameters obtained by real production processes. A trained agent based on the model obtained utilizing machine learning technique is then used to instruct the fermenter controller to adjust controlled parameters relating to the operation of the fermenter.

[0134] The controller that provides the input to the fermenter controller devices is based on biological mimicry model. The model is utilized for an example fermentation process for production of a specific product. The model contains various, optionally all parameters of the fermenter's operation and its contents that are related to the fermentation process.

[0135] In an optional embodiment, data is gathered from actual and experimental production runs. The data is inserted into the model and various algorithms are employed to determine a set of values for all parameters that best fits the data. Machine learning using input from subsequent production runs is used to optimize and continually update the model.

[0136] The model is useful for production in fermentation processes. Creating the model for a specific process comprises two phases: in the first phase experimental data on the fermentation process is gathered from which a digital model of the fermentation process is generated; second, by implementing optimization and machine learning methods, productivity increase is achieved.

[0137] In the first phase of creating the model, a base model is generated, which simulates the different interactions of the conditions inside the real fermenter for a specific fermentation process. Specifically, a mathematical model is created such that monitored parameters are linked to controlled parameters in a manner where changes in controlled parameters result in changes in the controlled parameters. The model may be based on a set of partial differential equations, representing the condition of the culture inside the fermenter at any time, while relations between variables (i.e., monitored and controlled parameters) are integrated in the equations.

[0138] The base model receives initial conditions, as well as input data from measurements of properties which effect the culture's state, e.g., carbon source/ammonia feeding values, agitation and air flow, along the simulated fermentation, and calculates the variable's values, e.g. carbon dioxide concentration, biomass concentration, carbon source/ammonia concentration, product concentration, and dissolved oxygen concentration—all as functions of time along the duration of the fermentation process.

[0139] After understanding the mathematical equations representing the process, the next step is approximation of the mathematical model to the physical process by finding accurate values for the properties in these equations. This approximation/validation is done using data collected from actual production batches, and from R&D experimental batches specially designed for understanding of certain aspects of the model. These experiments may optionally include specific properties which may be strictly controlled creating a different environment than the usual production state.

[0140] These measurements contain both the input data, such as feeding quantities and physical measurements (temperature, weight, airflow, agitation frequency and more) and the various variable values of the properties at all times. Feeding and physical measurements data are loaded into the model, which calculates the values of the properties. Then, the accuracy of the model is measured by comparing measurements of the real batch's properties to the output of the model. In this way several models are derived. An optimized model that represents the actual process with the highest precision is chosen where in such model the difference between real batch's measurements and output or predictive measurements of the model is minimal or does not exceed a specific or predefined threshold. Optionally, a performance score through a specially designed goal/objective function, where the most accurate model has the lowest goal/objective function score. Non-limited examples of goals/objectives include product yield, short fermentation duration, product quality, process efficiency, low impurity value and a combination thereof. Finally, various optimization methods, fitted for this purpose are activated, adjusting the values of the properties for an optimized model, with the lowest possible goal function score, that represents the actual process with the highest precision.

[0141] The model is optimized for a specific fermentation process for production of a specific product, for example, production of secondary derivative, enzyme or a specific fed-batch produced product, by a specific strain, and the optimization is done using data from real fermentation processes where all the values of the properties are measured and saved. This data is used for obtaining the values in the differential equations of the model that match the relevant process so that the digital fermenter created will behave in the same way as the physical fermenter. For that reason, the more data that is collected, with more diversity, a better, more accurate model can be created. In fermentation processes, particularly in secondary metabolites production, values of the properties of the process are closely related to medium composition and feed composition. When constructing a model for a specific process, it is essential to assess the adequacy of experiments for their validity and appropriateness of the kinetic and operation properties to use them with different medium and strain conditions. The model's fitting process is done using optimization methods that use the input data received for the construction of the simulated model to minimize the differences between the simulated values and the values measured in the actual fermentation process conducted.

[0142] The following is an example explanation of process enhancement using the disclosed model.

[0143] The model obtained in the first phase serves as a digital simulation of the real fermentation process. Therefore, after creation and validation of the model, it may be updated by machine learning techniques to obtain an optimized digital clone that is incorporated into a controller and to a local agent that can instruct dedicated instrumentation of the reactor to apply selected controlled parameters to a reactor based on parameters monitored by one or more sensors of the reactor. In one or more embodiments, machine learning and optimization methods take one or more of the following three final objectives into consideration: (1) high product yield, (2) short fermentation duration, (3) low impurity value (for processes with impurity). Achieving these objectives increases profitability by: creating more product; saving usage time of the fermenter, which can be used for more batches of the same process or of other processes; and saving resources used for purifying the product.

[0144] Different approaches may be used for calculation of the best possible controlled parameters:

[0145] (1) Creating an optimized digital fermentation process using optimization methods based on the created model. In addition, interactions between monitored parameters and controlled parameters are deduced from the model. The optimized digital process that is created is used as a template for model, which will aim for the preferred conditions at any time along the process through interactions knowledge obtained. An example of how this process works is to use a controller that uses a proportional-integral-derivative (PID) mechanism for each of the monitored parameters. A set point and bias are calculated for each of the parameters. Then close loop feeding control is achieved by the PID calculation for the specific bias and set point values that were calculated by using the model and the output rate is given by the PID controller.

[0146] (2) Dividing the process into phases (such as growth phase, production phase with abundant/lack of carbon source concentration in solution, stationary phase due to lack of necessary substrate etc.) that will be identified using supervised machine learning methods with measured data as features and past data as training. Each phase will have different preferred conditions that the processor will aim to at any time along the process through interactions knowledge obtained from the model.

[0147] (3) Activation of the model with various controlled parameters values every specified time period (usually according to measurements frequency), using data from current and past measurements, where initial conditions are set to be the current state of the fermenter. The results of the model will be treated by optimization methods in order to find the input values which leads to the best conditions in the future. This approach can be implemented after deciding the current process phase, using machine learning methods in a manner similar to the 2.sup.nd approach.

[0148] All of these approaches reflect a model that uses all measured data as a base for the input values controlled through utilization of sophisticated algorithms; as a result, the method described herein is capable of achieving better profitability improvements compared to control mechanisms currently used in the art.

[0149] In outline the two phases of the method of generating the model described herein can be described as comprising the following six stages:

[0150] FIG. 3 is a graph showing how the method for controlling the process can save time in an example fermentation process during recombinant protein production. The figure shows the CO.sub.2 concentration as a function of time and five measurements of optical density (OD) made during a process for production of a recombinant protein. In the method presently used by the operators of the system the measurements of the OD are used to determine when to add inducer to the process and start the recombinant protein production. At this stage, when inducer was added to the culture media, CO.sub.2 concentration dramatically decreased, thus emphasizing the cells' state by terminating their replication stage/“birthing”/CO.sub.2 release, and initiating use of their energy for the recombinant protein production. According to this method when the increase in OD is observed at 18 hours and after increased OD that show cell growth initiates again after the production stage has been accomplished, the process is stopped. According to the an embodiment, the CO.sub.2 concentration is continuously monitored and, according to the understanding that the rapid rise of CO.sub.2 at 10 hours is the result of rapid cell growth caused by end of production stage, the process would be stopped at 10 hours saving approximately nine hours.

[0151] FIG. 4 schematically shows a closed loop system for optimized values of the nutrient feeding and physical parameters in an example fermentation process.

[0152] A fermenter processor, which may constitute part of the fermenter controller, receives, from sensors in a fermenter in which an example fermentation process is being carried out, instantaneous values of a set of monitored parameters as a function of time during the entire time of the process. The monitored parameters include inter alia: CO.sub.2 concentration, nitrogen source and carbon source concentration, dO.sub.2, pH, temperature, air flow, and agitation. The fermenter processor may optionally receive from the model predicted values of the monitored parameters. This is particularly relevant in cases where one or more of the monitored parameters are cannot be detected and/or assessed by the sensors of the fermenter. Software in the fermenter processor comprises a trained agent integrating the model updated using algorithms of machine learning and optimization methods to thereby generate controlled parameters. The values of the controlled parameters are sent in real time to the fermenter controller equipment in order to control operation of the fermenter. The controlled parameters may be the feeding of the nutrient sources, and physical parameters like agitation and aeration. For example, the instructions could be to change the agitation rate or to add a specified amount of carbon or nitrogen source.

[0153] In an optional embodiment, software in the fermenter processor comprises algorithms that use machine learning and optimization methods to generate controlled parameters that are based, inter alia, on various options of predicted values of the monitored parameters received from the model processer, after the model processer was activated with various options of controller parameters values. The values of the controlled parameters are sent in real time to the fermenter controller in order to control operation of the fermenter. The controlled parameters are the feeding of the nutrient sources, and physical parameters like agitation and aeration. For example, the instructions could be to change the agitation rate or to add a specified amount of carbon or nitrogen source. Data which might include the values as a function of time of the monitored or controlled parameters and the difference between the predicted and measured monitored parameters are sent in real time from the fermenter processor to the model processor, which uses the data to update the current model and optimize it generating a new model and to predict updated values of the monitored parameters, which in turn are sent back to the fermenter processer in real time.

[0154] It is noted that FIG. 4 depicts fermenter processor and the fermenter controller as separate physical entities, embodiments may comprise only a single controller with a processor containing software configured to carry out the functions described above.

[0155] In one or more embodiments, the criteria in the algorithms in software in the fermenter controller that are used to determine time and quantity of carbon source and nitrogen source feeding are based on the values and trends of the following parameters:

[0156] μ.sub.pp(t) presented in equations numbers 4 and 6, is the parameter that describes the production and makes the connection between the N source and the model, this parameters basically shows that the production rate is effected from substrate utilization and ammonia uptake by the cells;

[0157] μ(t) presented in equation number 2, describes the specific growth rate which is directly connected to the CO.sub.2 and is influenced, in the growth stage, by both increases and decreases in the levels of carbon source and, during the production stage, by increases and decreases of both C and N.

[0158] Equation 2 describes the growth with dependence on both carbon S(t), oxygen concentration CL(t), and Ammonia A(t)

[0159] Although any commercially available CO.sub.2 and pH sensor can be used in the system, for a CO.sub.2 sensor, the inventors prefer the VAYU Meter, which is a very accurate non-invasive meter that provides very sensitive measurements of CO.sub.2 concentration in the exhaust of a fermentation vessel. U.S. Pat. No. 9,441,260, assigned to the parent company of the applicant of the present application, describes the method used by a processor to determine the CO.sub.2 concentration in the fermentation vessel from the measured CO.sub.2 concentration in the exhaust pipe. The Vayu Meter is manufactured by the applicant of the present application. Embodiments of the VAYU Meter are described in detail in co-pending international patent application number PCT/IL2019/050750 to the applicant of the present application. The VAYU Meter is coupled to a controller that comprises a processor, a data storage device, and a graphic user interface. The VAYU Meter provides a real time output control via analog/digital connection. The VAYU Meter comprises an infrared laser, detector and optical components configured to provide identical optical paths through the gases that exit the fermenter, thereby enabling continuous metabolic gas detection for highly sensitive monitoring of the process in any size fermenter with the same optical path. The VAYU Meter records and analyzes CO.sub.2 metabolic gas concentrations produced during the respiration and growth of living cells. Continuous, automatic measurements via the IR optical system allow in-situ detection of metabolic gases without interrupting the fermentation process for invasive sampling.

[0160] FIG. 5 is a graph comparing yield of fed-batch produced product for production runs carried out by following the protocol previously used (lower curve) and by using the disclosed system, method and controller for feeding the carbon source only. The graph shows as increase in yield above 20% and a potential savings of time of approximately 24 hours.

[0161] FIG. 6 shows graphs of CO.sub.2 concentration (gray curve) and carbon source feeding (black curve) as functions of amount of carbon source feed vs. time during a production run for a secondary derivative in which the carbon source was fed regardless the amount of biomass in the culture according to the standard protocol followed for production of the product. According to the protocol, during the production stage of the process, starting at about 24 hours until the process is terminated, the carbon source is fed in fixed predetermined constant amounts according to a fixed predetermined schedule at a constant rate.

[0162] FIG. 7 shows graphs of CO.sub.2 concentration (gray curve) and carbon source feeding (black curve) as functions of time during a production run for the same secondary derivative as in FIG. 6. In FIG. 7 the carbon source was fed according to a PID controller that was using set point and bias values that were calculated for feeding according to CO.sub.2 only according to part of the method described herein above. In the production run shown in this figure, the carbon source was fed with opposite correlation to the culture state according to the CO.sub.2 concentration using a close loop feedback control—when CO.sub.2 went up less carbon source was added and when CO.sub.2 went down more carbon source was added with the amounts of carbon source depending on the deviation of the instantaneous value of the CO.sub.2 concentration from the time varying value of the set point derived from the PID controller.

[0163] Comparison of FIG. 6 with FIG. 7 illustrates some of the advantages of the present method over the traditional protocol. In particular during most of the production stage of the process the CO.sub.2 concentration in FIG. 7 is constant indicating equilibrium between cell growth and death and ideal conditions for product formation. In contrast, in FIG. 6 the CO.sub.2 concentration during the production phase is very uneven indicating conditions that are not conducive to optimal production of product. Also referring to FIG. 7 the sever drop in CO.sub.2 level is followed immediately by a large feed of carbon source after which there is an immediately rise in the CO.sub.2 level followed by a rapid drop in CO.sub.2. Another injection of carbon source again briefly raises the CO.sub.2 concentration, which again falls rapidly after the carbon source feeding ceases and continues to drop even when carbon source is added between about 110 and 115 hours. This behavior of the CO.sub.2 concentration indicates that the process should be terminated at about 115 hours. This is in stark contrast to FIG. 6, where the protocol dictates termination of the process at 150 hours.

[0164] In one or more embodiments, methods as herein disclosed include a first off-line stage of building a mathematical model. The model is a mathematical description which comprises both controlled and monitored parameters. The main guidelines for generating the model are the academic literature and good fitness of the model to data measured in experimental runs of the process.

[0165] Following the building of the model a machine learning based training phase is conducted off line with the goal of creating a trained agent capable of making state dependent decisions (actions), which will eventually optimize the process according to predetermined goals that are determined by the customer, for example: achieving one or more of high yield, low impurities, and time reduction.

[0166] FIG. 8 schematically shows the reinforcement learning iterative training phase, wherein each cycle represents one time step or an episode of several time steps of the training. If, for example, the time step is one minute and the episode is 10,000 minutes long, then the cycle of FIG. 8 is exemplarily carried out 10,000 times during the learning based training phase. Since the learning based training phase is conducted off line, the actual 10,000 minutes long cycle is executable in silico at fraction of this time, which allows the carry out during the learning based training of a reasonable duration, such as several hours or days, a 10,000 minutes long cycle 10,000 times.

[0167] In FIG. 8, S.sub.t is the state at time t representing in the present case monitored parameters (e.g. DO concentration, carbon source concentration and nitrogen concentration) generated only by using the model; a.sub.t is the action calculated by the agent at time t representing in the present case the controlled parameters (e.g. carbon source feeding, nitrogen source feeding and agitation); r.sub.t is the reward at time t, representing the quality of the action at time t−1. In some cases, r.sub.t may also the quality of several previous actions, e.g. t−1, t−2, etc. For example, if one of the criteria of performance is yield, then high yield rate reflects an advantageous value of a.sub.t-1, therefore the learning algorithm will increase the probability for the action that caused it in the next episode and low yield rate, on the other hand, will cause a decrease in the probability of this action.

[0168] In order to achieve high performance, the learning process requires a large data set to learn from. In a specific embodiment of the method, the machine learning technique used during the training phase to generate the agent is reinforcement learning (RL). While most machine learning algorithms use prefabricated data sets, reinforcement learning as herein disclosed uses a mathematical model describing the process to generate an unlimited amount of artificial data. In this case, the RL algorithm does not use monitored parameters measured in fermenter, but the RL algorithm uses the model to generate monitored parameters, represented by S.sub.t, to be used in a following episode based on the reward it determines for the process run based on the parameters that it had generated in the previous episode. In each cycle, the controlled parameters are calculated according to the current agent, represented by a.sub.t. For the first few episodes, arbitrary values of the parameters are entered into the algorithm in order to initiate the iterative learning process. During the training phase, the training consists of a large number of consecutive episodes. In one specific example, about 20,000 episodes were required; but in general, for different processes, more numerous or fewer episodes might be required to achieve the desired performance.

[0169] An episode is a simulated way to predict a whole real fermentation process with controlled parameters of each episode determined using the agent achieved from all previous episodes. All of the episodes are governed by the same model, but each episode differs from the others by its unique protocol, i.e. action, for each time step. The updates of the agent, namely the changes in weight values in the decision policy, i.e. improving the probability of an action leading to a higher reward (or vice versa) leads to an iterative improvement of the reward value, meaning better goal values, e.g. higher yield, lower impurity, shorter fermentation time, etc. During the training stage based on the r.sub.t feedback, the agent is being iteratively improved. This upgrade stops when the agent reaches sufficient, optionally maximal performance, which occurs when the agent achieves repetitive high reward values for simulated runs. In one or more embodiments, the model does not change during the training stage of the agent; however, the model has been developed for a specific fermentation process. For a different process the algorithm that is responsible for training the agent is unchanged; however, the model will change as well as the action and the system state. These differences will force a completely new training process.

[0170] FIG. 9 schematically shows control of the fermenter by the trained agent during live production runs. After the training phase, monitored parameters may no longer be calculated by the model but are measured by sensors located in the fermenter. Nevertheless, the algorithms of the agent may have been trained using a model that comprises parameters for which live measurements aren't available during production runs, e.g. parameters that have to be measured off-line such as carbon or nitrogen source concentration determined by titration. To deal with this situation parameters that don't have live measurements are simulated using the model and are sent to the agent during live batches. This option enables sending as detailed data as possible to the agent at any time. In FIG. 9, S.sub.t is the state at time t, representing the monitored parameters measured by sensors in the fermenter (and simulated by the model if necessary). The state is sent to the agent that uses, for example, a deep neural network (DNN), which has been trained to optimize the process by, for example, increasing yield, decreasing impurity and short fermentation duration. a.sub.t is the action calculated by the agent at time t, i.e. the values of controlled parameters that are sent to the fermenter.

[0171] FIG. 10 schematically shows an embodiment of a closed loop system 30 configured for carrying out an embodiment of the method for optimizing controlled parameters in an example fermentation process. The system 30 is comprised of three main units: fermenter 16 containing sensors 14; services (controller) 34, which comprises an agent 38 that comprises an algorithm trained to find the optimal action to take at a particular time based on the system state at that time; and local agent 32, which is a mediator configured to transfer data to and from both the fermenter and services. Optionally, services 34 comprise a digital model 36 that represents the fermentation process being carried out in fermenter 16. As for the first embodiment, monitored parameters 18 are parameters that are measured by sensors 14 in fermenter 16, e.g. CO.sub.2 concentration, nitrogen concentration, dO.sub.2, pH, temperature, air flow, and agitation and controlled parameters are parameters that are allowed to be changed, e.g. feeding of a carbon source, feeding of a nitrogen source, agitation, temperature, and aeration.

[0172] Live connection to agent 38 (with or without local agent 32 if a wired communication link between fermenter 16 and services 34 is used) is utilized for troubleshooting, software updating and data withdrawal. It is possible to provide services 34 incorporated in a computer located in the facility housing the fermenter with remote access; however, cloud-based architecture is preferred to provide higher security since the algorithms are not physically located in the costumer's facility, data access, connection speed, and reliability. In the cloud based architecture the local agent 32 encrypts data received from the sensors 14 before sending the data to services 34 and decrypts encrypted data received from services 34 before sending it to fermenter 16.

[0173] With reference to FIG. 11, an example system for implementing aspects described herein includes a computing device, such as computing device 400. In its most basic configuration, computing device 400 typically includes at least one processing unit 402 and memory 404. Depending on the exact configuration and type of computing device, memory 404 may be volatile (such as random-access memory (RAM)), non-volatile (such as read-only memory (ROM), flash memory, etc.), or some combination of the two. This most basic configuration is illustrated in FIG. 11 by dashed line 406.

[0174] Computing device 400 may have additional features/functionality. For example, computing device 400 may include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated in FIG. 11 by removable storage 408 and non-removable storage 410.

[0175] Computing device 400 typically includes a variety of computer readable media. Computer readable media can be any available media that can be accessed by computing device 400 and include both volatile and non-volatile media, and removable and non-removable media. Computer storage media include volatile and non-volatile, and removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

[0176] Memory 404, removable storage 408, and non-removable storage 410 are all examples of computer storage media. Computer storage media include, but are not limited to, RAM, ROM, electrically erasable program read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 400. Any such computer storage media may be part of computing device 400.

[0177] Computing device 400 may contain communications connection(s) 412 that allow the device to communicate with other devices. Computing device 400 may also have input device(s) 414 such as a keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s) 416 such as a display, speakers, printer, etc. may also be included. All these devices are well known in the art and need not be discussed at length here.

[0178] It should be understood that the various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the processes and apparatus of the presently disclosed subject matter, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium where, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the presently disclosed subject matter.

[0179] FIG. 12 shows graphs obtained during the model validation step. The graphs illustrate a comparison between the predicted monitored parameters (herein “model”) and real measurements of monitored parameters (herein “data”) of: biomass oxygen, dissolved oxygen, carbon source, and desired product (e.g., an antibiotic). In one or more embodiments, the step of validating the model is conducted by: i) selecting initial input values of monitored parameters and controlled parameters, ii) processing the initial input values by the model to thereby obtain calculated predictive values for the monitored parameters, iii) determining a difference between the calculated predictive values of the monitored parameters and respective values of the monitored parameters as obtained in previous data of the reactor, and iv) further determining that the difference does not exceed a predefined threshold.

[0180] In one or more embodiments, the predefined threshold includes one or more values (absolute and/or relative values, e.g., a percentage) for allowing to determine the compatibility of the model for a process of a reactor. The model chosen should mimic the actual dynamic behavior of the reactor such that difference between the calculated predictive values and the respective values of the monitored parameters in the historical data (previous data of a reactor) that does not exceed the predetermined threshold may indicate compatibility of the model.

[0181] FIG. 13 illustrates the learning process of a reinforcement learning algorithm. X axis denotes the number of episodes executed, Y axis denotes the reward value. The black lines represent specific value of each reward. The central bold line denotes averaging the last 50 episodes, showing the learning trend. This graph displays 4500 episodes of the learning phase where the average reward value continuously improves due to policy update following each episode.

[0182] FIG. 14 shows an example computer program code configured for evaluation and/or update of the decision policy or policy function. This function gets as an input the state of the process (“observation1”) at some time point and it returns the action which supposed to be optimal (with high confidence) for that state. The function which appears in lines 16-22 extracts from the file named “agentData.mat” the final policy. This policy helps us to decide on the optimal action for each state (line 21).

[0183] Although example implementations may refer to utilizing aspects of the presently disclosed subject matter in the context of one or more stand-alone computer systems, the subject matter is not so limited, but rather may be implemented in connection with any computing environment, such as a network or distributed computing environment. Still further, aspects of the presently disclosed subject matter may be implemented in or across a plurality of processing chips or devices, and storage may similarly be effected across a plurality of devices. Such devices might include PCs, network servers, and handheld devices, for example.

[0184] The following are working empirical examples. This following example was performed for the optimization of an antibiotic production process, based on the embodiments described hereinabove. This activity was conducted, with the objective of increasing the yield for the selected fermentation process.

[0185] The instant fermentation process concerns a species of Streptomyces bacteria which produces an antibiotic compound.

[0186] The fermentation process begun with a small number of bacteria inserted to the fermenter that contained a grow medium. The process was divided into two main phases: (1) growth phase, where the bacteria replicated itself, thus increasing the biomass inside the fermenter; and (2) production phase where the vast majority of production was conducted, and the biomass has not changed dramatically. Each of these phases was composed of several sub-phases which shows different behaviors.

[0187] The physical conditions of dissolved oxygen concentration, carbon source concentration, nitrogen source concentration, and pH, were measured using sensors in the fermenter and were selected as monitored parameters. Controlled parameters of carbon source feeding, nitrogen source feeding, agitation and airflow were selected.

[0188] The development protocol of the intelligent controller was composed of a combination of constant values to some of the monitored parameters. This protocol was developed using an understanding of the biological properties of the of the process, as well as try and error R&D experiments.

[0189] The main objective was to increase the desired antibiotic production, with secondary objectives of decreasing the impurity (relative amount other compounds produced, which making the purification process less efficient). A significant improvement has been achieved by creating an intelligent controller based, as described hereinabove. The controller was activated every predetermined period of time, where the input was a set of monitored parameters and the output was a set of controlled parameters.

[0190] A model which describes the dynamics of a single fermentation process was formed. This model contained the dependency of the controlled/monitored parameters given a simulative prediction of the yield obtained in various simulated experiments, differentiated in initial conditions and controlled parameters values.

[0191] The mathematical model that was formed included a set of differential equations which collectively comprised parameters describing different aspects of the subject fermentation process. The equations were based inter alia on academic literature, past data collected on the process, and the results of specifically designed experiments. The model contained several parameters which were calibrated based on collected data.

[0192] Following the building of the model, a machine learning-based training phase was conducted offline. The training stage included a large amount of simulative processes (episodes). It started with an arbitrarily agent and based on the simulative yield it improved, iteratively, the agent's performances. All of the experiments were governed by the same model, but each episode differed from the others by its unique protocol, i.e. action, for each time step (state). The updates of the agent, i.e. improving the probability of an action leading to a higher reward (or vice versa) led to an iterative improvement of the reward value, with better goal values, in instant case, higher yield and lower impurities. This training stage ended up with a trained (optimal) agent capable of making state dependent decisions (actions).

[0193] The model was realistic only for well-defined range of monitored parameters, i.e. the model succeeded to predict the dynamic of monitored parameters as long as these values were within the realistic range. Additional restrictions regarding the controlled and monitored parameters were raised from FDA restrictions and customer request, such as maximum amount of dextrose feeding. As a consequence, restrictions which cancel actions that may lead to such undesired scenarios were introduced.

[0194] This agent was embedded in the process (during several experiments) as a controller of the controlled parameters. It showed sufficient results, i.e., it improved the final yield of the process.

[0195] Results—the performance of the trained agent was examined during 3 experiments. Each of the experiments was composed from 2 fermentation processes which were executed simultaneously. While one of the fermentation processes was governed by the standard protocol the other was governed by the controller. The average improvement in terms of production yield was around 13%. The minimal improvement was 9%. Thus, at least some embodiments provide an improvement in one or more objectives of a production process by at least about 5%, at least about 7%, or at least 9%.

[0196] In this model, t represented the time and the model was updated with a time differential of dt Presented herein are the differential equations of the model.

[0197] The biomass trend was given by equation (1):

X(t+1)=X(t)+dt(X(t)(μ(t)−K.sub.d))  (1)

[0198] wherein: X(t) is the biomass concentration in the fermenter at time t, K.sub.d is the death factor constant of the cells, and μ(t) is the growth rate of the cells at time t, X(t+1) is the value of X(t) one minute after t, and μ(t) is given by equation (2):

[00006]$\begin{array}{cc}\mu \left(t\right)={\mu }_x\left(\frac{S\left(t\right)}{K_x+S\left(t\right)}\right)\left(\frac{CL\left(t\right)}{CL\left(t\right)+K_{ox}}\right)\left(\frac{A\left(t\right)}{A\left(t\right)+K_{xa}}\right)& \left(2\right)\end{array}$

[0199] wherein: μ.sub.x is the maximal growth rate constant of the cells, K.sub.x is the carbon source limitation constant for growth, K.sub.ox is the oxygen limitation constant for growth, S(t) is the carbon source concentration in the fermenter at time t, CL(t) is the dissolved oxygen concentration at time t, A(t) is the nitrogen source concentration in the fermenter at time t, and K.sub.xa is the nitrogen source limitation constant for growth.

[0200] The production trend was given by equation (3):

P(t+1)=P(t)+dt(μ.sub.pp(t)X(t)−KP(t))  (3)

[0201] wherein: P(t) is the product concentration in the fermenter, K is the product hydrolysis rate constant, and μ.sub.pp(t) is the production rate at time t, μ.sub.pp(t) is given by equation (4):

[00007]$\begin{array}{cc}{\mu }_{pp}\left(t\right)={\mu }_p\left(\frac{A\left(t\right)}{K_p+A\left(t\right)}\right)\left(\frac{CL\left(t\right)}{CL\left(t\right)+K_{op}}\right)\left(\frac{S\left(t\right)}{K_I+S\left(t\right)+\frac{{s\left(t\right)}^2}{K_{ps2}}}\right)& \left(4\right)\end{array}$

[0202] wherein, μ.sub.p is the maximal production rate constant, K.sub.p is the production inhibition constant for nitrogen source, K.sub.op is the production inhibition constant for dissolved oxygen, K.sub.I is the first inhibition constant for carbon source and K.sub.ps2 is the second inhibition constant for carbon source.

[0203] The carbon source was used for cell growth, production, and maintenance of the fermentation process. The amount of the carbon source in the vessel decreased with time and was increased by feeding during the process. The carbon source trend was given by equation (5):

[00008]$\begin{array}{cc}S\left(t+1\right)=S\left(t\right)+dt\left[X\left(t\right)\left(-\frac{\mu \left(t\right)}{Y_{x/s}}-m_x-\frac{{\mu }_{pp}\left(t\right)}{Y_{p/s}}\right)+\frac{S_{in}\left(t\right)}{\mathrm{Vessel}\mathrm{Volume}}\right]& \left(5\right)\end{array}$

[0204] wherein, S(t) is the carbon source concentration in the fermenter at time t, Y.sub.x/s is the growth yield constant for the carbon source, Y.sub.p/s is the production yield constant for the carbon source, m.sub.x is the maintenance constant of the carbon source, and S.sub.in is the carbon source feeding value.

[0205] The various embodiments disclosed herein can be implemented as hardware, firmware, software, or any combination thereof. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit or computer readable medium consisting of parts, or of certain devices and/or a combination of devices. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPUs”), a memory, and input/output interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit. Furthermore, a non-transitory computer readable medium is any computer readable medium except for a transitory propagating signal.

[0206] All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the principles of the disclosed embodiment and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosed embodiments, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0207] It should be understood that any reference to an element herein using a designation such as “first,” “second,” and so forth does not generally limit the quantity or order of those elements. Rather, these designations are generally used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to first and second elements does not mean that only two elements may be employed there or that the first element must precede the second element in some manner. Also, unless stated otherwise, a set of elements comprises one or more elements.

[0208] As used herein, the phrase “at least one of” followed by a listing of items means that any of the listed items can be utilized individually, or any combination of two or more of the listed items can be utilized. For example, if a system is described as including “at least one of A, B, and C,” the system can include A alone; B alone; C alone; 2A; 2B; 2C; 3A; A and B in combination; B and C in combination; A and C in combination; A, B, and C in combination; 2A and C in combination; A, 3B, and 2C in combination; and the like.