Constraint adaptor for reinforcement learning control

Abstract

A system for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine is provided. The apparatus includes an input interface to accept data indicative of a state of the machine, a memory configured to store an optimization problem for computing the safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set (CIS) to a control input satisfying the control input constraints, and a processor configured to iteratively perform a reinforcement learning (RL) algorithm to jointly control the machine and update the control policy.

Claims

1. A system for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine, comprising: an input interface to accept data indicative of a state of the machine; a memory configured to store an optimization problem for computing a safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set (CIS) to a control input satisfying the control input constraints, wherein a control of the machine having the state within the CIS according to the control policy maintains the state of the machine within the CIS, wherein the memory includes a supervisor algorithm that obtains the state of the machine and computes a desired safety margin, wherein the supervisor algorithm generates a safe command when a reinforcement learning (RL) algorithm generates a command that is deemed unsafe, wherein the safe command is a modification of the unsafe command according to optimization (SO):
c(t)=min α Σ.sub.k=1.sup.N∥u(k|t)∥.sub.1, where c(t) is a cost function, u (k|t) is a command vector predicted value of u(t+k), α is a scaling factor, k, N are integers, t is a current time of the system; and a processor configured to iteratively perform the RL algorithm to jointly control the machine and update the control policy, wherein, for performing the joint control and update, the processor is configured to control the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determine a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and update the control policy that improves a cost function of operation of the machine according to the determined reward.

2. The system of claim 1, wherein the RL algorithm is a deep-deterministic policy gradient (DDPG) algorithm.

3. The system of claim 2, wherein the DDPG algorithm learns both a critic network to estimate long-term values for a given policy and an actor network to sample optimal actions according to the estimated long-term values.

4. The system of claim 1, wherein the reward function is modified to an updated reward by subtracting the cost function from the reward function, wherein the updated reward is expressed by
{circumflex over (r)}(t)=r(t)−c(t), where {circumflex over (r)}(t) is the updated reward, r(t) is the reward function, c(t) is the cost function, and t is a current time of the system.

5. The system of claim 1, wherein the scaling factor α is chosen such that the cost c(t) is between +c.sub.b, wherein Cb are bounds on the cost function.

6. The system of claim 1, wherein a maximum penalty G for performing the RL algorithm is about twice a value of c.sub.b:G≈2c.sub.b, wherein c.sub.b are bounds on the cost function.

7. The system of claim 1, wherein the machine is a suspension system of a vehicle.

8. A computer-implemented method for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine, wherein the computer-implemented method is stored in a memory, comprising steps of: accepting data indicative of a state of the machine; computing a safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set (CIS) to a control input satisfying the control input constraints, wherein a control of the machine having the state within the CIS according to the control policy maintains the state of the machine within the CIS, wherein the memory includes a supervisor algorithm that obtains the state of the machine and computes a desired safety margin, wherein the supervisor generates a safe command when a reinforcement learning (RL) algorithm generates a command that is deemed unsafe, wherein the safe command is a modification of the unsafe command according to optimization (SO):
c(t)=min α Σ.sub.k=1.sup.N∥u(k|t)∥.sub.1, where c(t) is a cost function, u(k|t) is a command vector predicted value of u(t+k), α is a scaling factor, k, N are integers, t is a current time of the system; and iteratively performing the RL algorithm to jointly control the machine and update the control policy, wherein, for performing the joint control and update, wherein the iteratively performing step comprises: controlling the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determining a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and updating the control policy that improves a cost function of operation of the machine according to the determined reward.

9. The method of claim 8, wherein the RL algorithm is a deep-deterministic policy gradient (DDPG) algorithm.

10. The method of claim 9, wherein the DDPG algorithm learns both a critic network to estimate long-term values for a given policy and an actor network to sample optimal actions according to the estimated long-term values.

11. The method of claim 8, wherein the reward function is modified to an updated reward by subtracting the cost function from the reward function, wherein the updated reward is expressed by
{circumflex over (r)}(t)=r(t)−c(t), where {circumflex over (r)}(t) is the updated reward, r(t) is the reward function, c(t) is the cost function, and t is a current time of the system.

12. The method of claim 8, wherein the scaling factor α is chosen such that the cost c(t) is between +c.sub.b, wherein c.sub.b are bounds on the cost function.

13. The method of claim 8, wherein a maximum penalty Gfor performing the RL algorithm is about twice a value of c.sub.b:G≈2c.sub.b, wherein c.sub.b are bounds on the cost function.

14. The method of claim 8, wherein the machine is a suspension system of a vehicle.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The presently disclosed embodiments will be further explained with reference to the attached drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments.

(2) FIG. 1A presents a conventional scheme for controlling a system by a conventional reinforcement learning controller;

(3) FIG. 1B is a schematic diagram presenting a constraint set;

(4) FIG. 1C is a flowchart of a conventional reinforcement learning algorithm;

(5) FIG. 2A is a diagram indicating a safe supervisor scheme for controlling a system by a reinforcement learning controller, according to embodiments of the present invention;

(6) FIG. 2B is a flowchart indicating a safety supervisor algorithm, according to embodiments of the present invention;

(7) FIG. 3 is a schematic of a quarter-car model based on a safety supervisor algorithm, according to embodiments of the present invention;

(8) FIG. 4 shows the addition of noise to the output of a deep-deterministic policy gradient algorithm, according to embodiments of the present invention;

(9) FIG. 5 shows a diagram indicating the entire operation of the scheme for safe reinforcement learning, according to embodiments of the present invention; and

(10) FIG. 6 is a diagram illustrating an example of a control apparatus for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine, according to embodiments of the present invention.

DETAILED DESCRIPTION

(11) Various embodiments of the present invention are described hereafter with reference to the figures. It would be noted that the figures are not drawn to scale elements of similar structures or functions are represented by like reference numerals throughout the figures. It should be also noted that the figures are only intended to facilitate the description of specific embodiments of the invention. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an aspect described in conjunction with a particular embodiment of the invention is not necessarily limited to that embodiment and can be practiced in any other embodiments of the invention.

(12) It is an object of some embodiments to provide a system and a method for controlling an operation of a machine using a data-driven state feedback optimal controller. It is another object of some embodiments to provide such a controller that is suitable for controlling a machine subject to safety constraints. An example of such a data-driven optimal controller uses reinforcement learning (RL) to determine control policies based on data obtained during the operation of the controlled machine and a supervisor to provide feedback to the RL-based controller on safe operation of the control.

(13) FIG. 1A shows an example of an ordinary RL controller 100 in feedback with a system 109. The controller passes a command 106 to the system and the system performs actions according to the command. When the system is linear, it can be modeled according to a difference equation:
x(t+1)=Ax(t)+Bu(t)+B.sub.ww(t)   (1)
where x is a vector containing the system states, u is a vector of commands, and w is a vector of disturbances. When the system is nonlinear, it can in the majority of practical purposes be modeled as a linear system. The RL controller receives a feedback signal 112 from the system which is generally a function of both the system state and command vectors and not a function of the disturbance input vector because that is generally unknown. The controller modifies the command according to the feedback. In general, the feedback q (t) is a function of all the vectors above:
q(t)=f(x(t), u(t), w(t))   (2)

(14) The system 109 is output-constrained, meaning that the output 115 is subject to constraints. The output can be described mathematically as a linear combination of the system state vector, command vector, and disturbance input vector:
y(t)=Cx(t)+Du(t)+D.sub.ww(t)   (3)

(15) The output is a vector and the constraints that it is subject to are modeled as a set. FIG. 1B shows a two-dimensional output vector 120 and a constraint set 123 plotted against a set of axes. The axes together form a basis for the vector space which describe the output. Let ê.sub.1 be a vector parallel to axis 121 and ê.sub.2 be a vector parallel to axis 122. Then the output y(t) can be expressed as
Y(t)=y.sub.1(t)ê.sub.1+y.sub.2(t)ê.sub.2
where y.sub.1(t) and y.sub.2(t) are appropriately valued scalars. For safe operation, the output must remain in the constraint set 123. Mathematically, the constraints are represented as linear-inequality requirements:
Sy(t)≤s   (4)
which represent the polytope or polygon that geometrically represents the constraints.

(16) Algorithms used in RL generally do not protect against constraint violation. Conventional RL algorithms work through a trial-and-error process that aims to maximize an accumulation of discounted rewards:
Σ.sub.t=0.sup.∞γ.sup.tr(t),   (5)
where r(t) is the reward function and γ<1 is a positive discount factor. FIG. 1A shows the reward 113 as an output of a function that obtains the feedback 112 and computes the reward. The trial-and-error process works to determine a policy π.sub.θ which is a function that maps the feedback 112 to the command 106 and is parametrized by some parameter vector θ. The policy is the function that maps the feedback to the command:
u(t)=π.sub.θ(q(t))   (6)

(17) FIG. 1C shows the general working of the RL controller according to the present invention. During each sample time t of the system, the reward 132 is sent to the RL algorithm 134, which determines the parameter vector θ to parameterize the policy 136, π.sub.θ. The feedback signal 130 is transformed by the policy 136 to the control command 138.

(18) The main idea behind this invention is to modify the reward function r(t) to be the reward function minus a cost function c(t) that measures the danger of constraint violation. The updated reward is therefore:
{circumflex over (r)}(t)=r(t)−c(t)   (7)

(19) The cost function c(t) is determined by an add-on element called a supervisor.

(20) FIG. 2A is a diagram indicating a safe supervisor scheme (method) used in a control system 20 for controlling a system by a reinforcement learning controller, according to embodiments of the present invention.

(21) FIG. 2A shows a control system 20 that includes the supervisor 203 placed between the RL controller 200 and the constrained system 209. The supervisor 203 obtains the state measurement or estimate 220 from the system 209 and computes a desired safety margin that gets converted to a penalty 217 and sent to the reward computation 218 for obtaining the modified reward 213. At the same time, the supervisor transmits a safe command 216 in case the command 206 was deemed unsafe. The safe command is a modification of the unsafe command determined according to the optimization (SO):
c(t)=min α Σ.sub.k=1.sup.N∥u(k|t)∥.sub.1   (8)
subject to the constraint:
Sy(k|t)+y.sub.w.sup.β(k, S)≤s   (9)
for k=0, . . . , N−1 and subject to the constraint:
Hy(k|t)+y.sub.w.sup.β(k, H)≤h   (10)

(22) The term y(k|t) is the predicted value of y(t+k) at time t according to the dynamics:
x(k+1|t)=Ax(k|t)+Bu(k|t)   (11)
y(k|t)=Cx(k|t)+Du(k|t)   (12)
With the initial conditions x(0|t)=x(t), which is obtained from the system, and u(0|t)=u(t), which is obtained from the RL controller. The term y.sub.w.sup.β(k, S) is the support of the disturbance set with probability β. This set is the set Ξ.sub.k satisfying:
Pr(y.sub.w(t+k)−y(k|t)∈ Ξ.sub.k)=β  (13)
so that y.sub.w.sup.β(k, S) is the solution to:
min S.sup.Ty.sub.w   (14)
subject to the constraint:
y.sub.w ∈ Ξ.sub.k   (15)

(23) The multiplicative factor α in the (SO) problem is a scaling factor that modifies the size of the cost c(t). In the above, k, N are integers, and t is a current time of the system.

(24) The solution to the (SO) problem is the minimum effort required to keep the system within constraints according to the system model. The system model is not perfect and hence the need for RL to obtain a more optimal control. Furthermore, RL cannot handle constraints and hence the need for a supervisor (or supervisor algorithm) to inform RL of constraint violation. In this way, the functions of the RL algorithm and the supervisor are complimentary to each other. The supervisor is model-based and can determine the optimal value according to the linear model through a relatively simple computation. For example, in the case of the supervisor, we can obtain a strong bound on the value of N, which is the maximum number of steps needed to return to the zero-effort set. To obtain N, we compute the control invariant set (CIS), which is the set of all system states x(t) for which there exists a command u(t) that would return the state into the CIS according to the system dynamics and satisfy the set-membership constraint Sy(t)≤s. Therefore, if a state is not in the CIS, the system is guaranteed to eventually violate constraints.

(25) One way to compute the CIS is to compute the set of all combinations of initial states and commands that guarantee constraint enforcement and project this onto the x-axis. Once the projection is no longer growing the resulting set, we have found the limit N. Specifically, we compute the CIS by defining the set:
C.sub.0={(x, u.sub.0):S(Cx+Du.sub.0)≤s}  (16)
and then recursively computing the sets:
C.sub.k={(x, u.sub.0, . . . , u.sub.k):Ax+Bu.sub.i ∈ C.sub.k−1, i=1, . . . , k,(x, u.sub.0) ∈ C.sub.0}  (17)
The CIS is the projection of lim.sub.k.fwdarw.∞ C.sub.k onto the x-axis. When the projection at step k is the same size as the projection at step k−1 we set N=k−1. For practical purposes, we can stop the algorithm a little earlier when the difference in projections is deemed negligible.

(26) The existence of the CIS set implies that there sometimes does not exist a solution to the (SO) problem, as the state may not be inside the CIS. Furthermore, if the state is outside of the CIS then, according to the model, the system will inevitably violate constraints as there does not exist a solution to the (SO) problem. If this occurs, we set the penalty to c(t)=−G, where G is a very large number that is larger than any other possible penalty and performs a procedure to determine a modified command.

(27) The zero-effort set itself is the set of states for which the solution to the (SO) problem is nil. This set can be characterized a set of linear inequalities
Hy(k|t)+y.sub.w.sup.β(k, H)≤h   (18)
for k=0, . . . , N*. So far, it is unknown how to compute N*, but it is know that the value is finite and that it is related to the rate of decay of the linear system. Therefore we choose an N* which is much larger than the settling time of the linear system.

(28) FIG. 2B is a flowchart indicating a safety supervisor algorithm, according to embodiments of the present invention. FIG. 2B describes the procedure that determines the function of the safety supervisor algorithm of a control apparatus. The supervisor obtains the state 240 and attempts to solve the (SO) problem 241. If a solution exists, the supervisor sets the penalty c(t) to the solution of the problem 244 and passes the command received from the RL controller to the system 245. If a solution does not exist, it means that constraints will very likely be violated; therefore the supervisor sets the penalty c(t) to the maximum penalty 242 and modifies the command received from the RL controller and passes the modified command to the system 243.

(29) After some experimentation, we realized that the safe command should be chosen at random. When a command is deemed unsafe, it means that applying it will lead to constraint violation. If we apply a slightly modified command, it does not greatly diminish the risk of violating constraints. Furthermore, staying within the neighborhood of an unsafe region leads the RL controller to not explore all possible regions. Therefore, we instead take drastic action and randomly sample a command satisfying constraints. We do this using a hit-and-run technique. We generate a sequence of commands
{u(0|t), u(1|t), . . . , u(N−1|t)}={u.sub.0(0|t), u.sub.0(1|t), . . . , u.sub.0(N−1|t)}   (19)
that satisfy the following constraints:
Sy(k|t)+y.sub.w.sup.β(k, S)≤s   (20)
Hy(N+k′|t)+y.sub.w.sup.β(N+k′, H)≤h   (21)
we then pick a random sequence {p.sub.0, p.sub.1, . . . , p.sub.N−1} and set
{u(0|t), u(1|t), . . . , u(N−1|t)}={u.sub.1(0|t), u.sub.1(1|t), . . . , u.sub.1(N−1|t)}   (22)
where
u.sub.1(k|t)=u.sub.0(k|t)+λp.sub.k   (23)
for all k=0, . . . , N−1, k′=0, . . . , N* and some scalar λ. We then find the smallest λ that satisfies the above constraints. We repeat the above to find sequences of u.sub.2, u.sub.3, . . . Since we are guaranteed that as k.fwdarw.∞ for u.sub.k, then the sequence of u.sub.k will be truly random and we will have sampled the constraints uniformly randomly.

(30) According to some embodiments of this invention, a control system or a control apparatus for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine is realized. To that end, the system or the apparatus may include an input interface to accept data indicative of a state of the machine; a memory configured to store an optimization problem for computing the safety margin of a state and action pair satisfying the state constraints and a control policy mapping a state of the machine within the control invariant set (CIS) to a control input satisfying the control input constraints, wherein a control of the system having the state within the CIS according to the control policy maintains the state of the system within the CIS; and a processor configured to iteratively perform reinforcement learning (RL) to jointly control the machine and update the control policy, wherein, for performing the joint control and update. In this case, the processor is configured to control the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs; determine a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS; and update the control policy that improves a cost function of operation of the machine according to the determined reward.

(31) The control method (safe supervisor algorithm) used in the control system or the apparatus according to the present invention can be applied to the machines used in a factory automation system, actuators and suspensions used in a robotic system or vehicles, or a plant system.

(32) FIG. 3 is a schematic of a quarter-car model based on a safety supervisor algorithm, according to embodiments of the present invention. Suspension systems can be modeled according to the quarter-car model which is a simplified model of the complex suspension system dynamics. For example, the simplified model assumes that the suspension springs are linear, which is not truly the case, especially when the spring is greatly stretched; in this case, the spring force is better modeled by, for example, a cubic relationship to the spring displacement.

(33) Although a vehicle suspension system is described below as an example, the safe supervisor (safe supervisor control method) according to the present invention is not limited to the vehicle suspension, the safe supervisor can be applied to control actuators and suspensions used in a robotic system or a factory automation system.

(34) FIG. 3 provides a schematic of the quarter-car, with the sprung mass 304 modeling a quarter of the vehicle-body mass resting on the suspension, the unsprung mass 305 modeling the vehicle wheel. The displacement of the sprung mass height from equilibrium 301 is denoted by z.sub.s and the displacement of the unsprung mass height from equilibrium 302 is denoted by z.sub.us. The sprung and unsprung mass are connected by an active suspension system 306 which is modeled as a spring-damper assembly with an actuation force F from an actuator. The actuator is commanded by the controller, which runs on an on-board computer 311. The algorithm on this computer determines and actuator command 310, which is the force F. The contact between the road 308 and the wheel is modeled as a spring-damper system. The height of the road 303 is denoted by z.sub.r. Since the system is modeled as an interconnection of masses, springs, and dampers, its dynamics are given by a two degree-of-freedom set of dynamical equations:
M.sub.s({umlaut over (z)}.sub.s−{umlaut over (z)}.sub.us)+c.sub.s(ż.sub.s−ż.sub.us)+k.sub.s(z.sub.s−z.sub.us)=F   (24)
M.sub.us{umlaut over (z)}.sub.us+c.sub.tż.sub.us+k.sub.t(z.sub.us−z.sub.r)−c.sub.s(ż.sub.s−ż.sub.us)−k.sub.s(z.sub.s−z.sub.us)=F+M.sub.us{umlaut over (z)}.sub.r+c.sub.tż.sub.r   (25)
Because they are linear, these equations can be transformed into the required format by letting:
x(t)=z.sub.s(t)−z.sub.us(t), ż.sub.s(t), z.sub.us(t)−z.sub.r(t), ż.sub.us(t))   (26)
v(t)=F(t)   (27)
w(t)=M.sub.us{umlaut over (z)}.sub.r(t)+c.sub.tż.sub.r(t)   (28)
We then let:
v(t)=−Kx(t)+u(t)   (29)
where K is a stabilizing feedback, i.e., the matrix A−BK is a stable matrix and hence the dynamics of x(t) are stable. The matrix K represents a stabilizing feedback controller which has been designed to control the active suspension. Our intention is to use the RL algorithm to improve the controller in the presence of constraints. In this explanation, the feedback state x(t) is assumed to be directly measured using some instrumentation. This is because in our experimentation, we were able to measure all the states. However, it is likely that in real-world application, one would implement a state estimator 314 to obtain the state estimates 315 by measuring the vertical displacement of the sprung mass 301 using displacement sensors, such as linear variable displacement transducers. The nominal feedback controller has not been designed with constraints in mind and therefore the RL algorithm should improve the controller performance especially in the presence of constraints.

(35) The constraints we consider are that 1) z.sub.s−z.sub.us≥.sub.s,− to protect the spring from breaking due to compression; 2) z.sub.s−z.sub.us≤.sub.s+ to protect the spring from breaking due to stretching; 3) |ż.sub.s|≤f.sub.s to ensure ride comfort for the passengers of the vehicle; and 4) z.sub.s−z.sub.us≥.sub.us,− to protect the wheel from being damaged due to compression of the tire; the terms .sub.s,−, .sub.s,+, f.sub.s and .sub.us,− are positive scalar limits on the functions of the variables above. Since these constraints are linear, they can be modeled in the required form.

(36) The RL algorithm we apply is the deep-deterministic policy gradient (DDPG) algorithm due to its ability to deal with continuous control systems. DDPG learns both a critic network to estimate the long-term value for a given policy and an actor network to sample the optimal action. In application to the suspension system, the critic network is composed of two hidden fully-connected layers, with 160 and 120 neurons respectively, and the actor network is composed of two hidden fully-connected layers, with 160 and 120 neurons respectively, and a softmax layer to sample the optimal action. For the rest of the design of the DDPG algorithm, default hyperparameters have been used. Importantly, since it makes for better learning, DDPG does not apply the optimal control that it has learned:
u(t)=π.sub.θ(q(t))   (30)

(37) Instead, it applies the optimal control modified with some colored noise signal:
u(t)=π.sub.θ(q(t))+∈.sub.OU   (31)
where ∈.sub.OU is the output of a colored noise (also called Ohrstein-Uhlenbeck) process.

(38) FIG. 4 shows a schematic of this procedure. Colored noise 403 is the output of a color filter 402 with white noise 401 as an input. The output of the policy 404 is the optimal command 405. This command is summed with the noise 406 and the output is a test command 407. The test command 407 is not the actual actuator command. Before a command can be passed to the actuator, it needs to be checked by the supervisor 203 and modified to adhere to safety constraints if it is determined to violate safety constraints.

(39) FIG. 5 shows the steps of the full safe RL method used in the control system or the control apparatus. The algorithm modifies the ordinary DDPG algorithm 501 to improve safety of the system. The optimal command 505 is generated by the policy 503. The algorithm adds colored noise 507 to the optimal command to determine the test command and checks safety 509 by solving the (SO) problem. As a result of solving the (SO) problem, a safety margin 511 is obtained, which is set to the maximum penalty if a solution does not exist. If a solution does exist, it means that the test command is safe and it is passed as the actuator command 517; if a solution does not exist, it means that the test command is unsafe, so the algorithm generates a random, safe actuator command. The safe actuator command is passed to the system 519 which returns a feedback signal 521 via measurement devices. The feedback signal is sent to the reward computation 523 which generates the reward 513. The safety margin is added to the reward 515 and the modified reward is passed to the DDPG algorithm.

(40) It remains to explain the implementation details specific to suspension systems. The reward function that we wish to maximize is rider comfort:
r(t)=−|ż.sub.s(t)|  (32)

(41) In other words, we wish that the motion of the sprung mass comfort be minimized. As stated above, the rider comfort is constrained between ±f.sub.s. This means that the reward is also constrained between these bounds. Through computation of the CIS, and experimentation of application of the safe RL algorithm, we have found that a good choice of the scaling factor α in the (SO) problem is one which ensures that the cost c(t) is always between ±c.sub.b where c.sub.b are bounds on the cost function which, through experimentation, we set as:
c.sub.b=100f.sub.s   (33)

(42) This is because the effort needed to return to safety is very large at the boundaries of the CIS, and so the scaling factor needs to be large to ensure that the cost is high enough nearer to the zero-effort set. Through experimentation, we also found that the maximum penalty G should always be about twice the value of c.sub.b, that is:
G≈2c.sub.b   (34)

(43) If the maximum penalty is too large, it induces a dominant effect on the learning process of the RL algorithm and the algorithm generates commands that overly avoid constraints. Therefore choosing G to be on the order of magnitude of the largest possible solution the (SO) problem is appropriate.

(44) FIG. 6 shows a block diagram of a control system (apparatus) 600 controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine according to some embodiments of the invention. The system 600 may include a human machine interface (HMI) 610 connectable with a keyboard 611 and a pointing device/medium 612, a processor 620, a storage device 630, a memory 640, a network interface controller 650 (NIC) connectable with a data communication network 690 including local area networks and internet network, a control interface 670 to transmit a control input 675 to the machine. The control system 600 can receive data indicative of state of a machine 695 via the network 690 connected to the NIC 650. The storage device 630 includes a reinforcement learning (RL) algorithm (program) 631, a supervisor algorithm 633, a reward function, cost function, and maximum penalty parameters for the RL and supervisor algorithms 634, inequalities describing the constraints 632 on the system 600, and inequalities describing the zero-effort set 635. The pointing device/medium 612 may include modules that read programs stored on a computer readable recording medium.

(45) Accordingly, some embodiments of the present invention can provide a computer-implemented method for controlling an operation of a machine subject to state constraints in continuous state space of the machine and subject to control input constraints in continuous control input space of the machine. In this case, the method may include steps of accepting data indicative of a state of the machine, computing a safety margin of a state and action pair satisfying the state constraints and a control policy mapping the state of the machine within a control invariant set (CIS) to a control input satisfying the control input constraints, wherein a control of the machine having the state within the CIS according to the control policy maintains the state of the machine within the CIS, and iteratively performing a reinforcement learning (RL) algorithm to jointly control the machine and update the control policy, wherein, for performing the joint control and update, wherein the iteratively performing step comprises controlling the machine using the control policy to collect data including a sequence of control inputs generated using the control policy and a sequence of states of the machine corresponding to the sequence of control inputs determining a reward for a quality of the control policy on the state of the machine using a reward function of the sequence of control inputs and the sequence of states of the machine augmented with an adaptation term determined as the minimum amount of effort needed for the machine having the state to remain within the CIS, and updating the control policy that improves a cost function of operation of the machine according to the determined reward. In some cases, the computer-implemented method can be used to control a suspension system (systems) of a vehicle.

(46) Although a vehicle suspension system is described above as an example, the safe supervisor (safe supervisor control method) according to the present invention is not limited to the vehicle suspension, the safe supervisor can be applied to control actuators and suspensions used in a robotic system or a factory automation system.

(47) The above-described embodiments of the present invention can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component. Though, a processor may be implemented using circuitry in any suitable format.

(48) Also, the embodiments of the invention may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

(49) Use of ordinal terms such as “first,” “second,” in the claims to modify a claim element does not by itself connote any priority, precedence, or order of one claim element over another or the temporal order in which acts of a method are performed, but are used merely as labels to distinguish one claim element having a certain name from another element having a same name (but for use of the ordinal term) to distinguish the claim elements.

(50) Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention.

(51) Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.