DEVICE AND METHOD FOR CONTROLLING A ROBOT DEVICE

Abstract

A method for controlling a robot device. The method includes providing a selection model and executing multiple instances of a task, including, in each execution, when a function of the robot device needs to be selected to perform the task instance, checking whether the selection model provides a selection of a function and, if yes, controlling the robot device to perform the function selected by the selection model and if no, receiving user input indicating a selection of a function, selecting a function according to the selection indicated by the user input, controlling the robot device to perform the function selected according to the selection indicated by the user input and training the selection model according to the selection indicated by the user input.

Claims

1. A method for controlling a robot device, comprising the following steps: providing, for each function of a plurality of functions, a control model for controlling the robot device to perform the function; providing a selection model for selecting among the plurality of functions; executing multiple instances of a task by the robot device, including, in each execution, when a function of the plurality of functions needs to be selected to perform the task instance, checking whether the selection model provides a selection of a function and, based on the selection model providing a selection of a function, controlling the robot device to perform the function selected by the selection model, using the control model for the selected function; and based on the selection model not providing a selection of a function, receiving user input indicating a selection of a function, selecting a function according to the selection indicated by the user input, controlling the robot device to perform the function selected according to the selection indicated by the user input using the control model for the selected function. and training the selection model according to the selection indicated by the user input.

2. The method of claim 1, wherein the selection model outputs indications of confidences for selections of functions of the plurality of functions and wherein training the selection model according to the selection indicated by the user input includes adjusting the selection model to increase a confidence output by the selection model to select the function selected according to the selection indicated by the user input.

3. The method of claim 1, wherein the selection model outputs indications of confidences for selections of functions of the plurality of functions, and the checking of whether the selection model provides a selection of a function includes checking whether the selection model outputs an indication of a confidence for a selection of the function which is above a predetermined lower confidence bound.

4. The method of claim 1, wherein the functions include skill and branches of skills and the selection function is trained to provide, for sets of alternative skills, a selection of a skill, and, for sets of alternative branches of skills, a selection of a branch.

5. The method of claim 1, wherein the providing of the control model for each function includes performing demonstrations of the function and training the control model using the demonstrations.

6. The method of claim 1, wherein the selection model is a logistic regression model.

7. The method of claim 1, further comprising: based on the selection model not providing a selection of a function, pausing operation of the robot device until user input indicating a selection of a function has been received.

8. A robot controller configured to control a robot device, the robot controller configured to: provide, for each function of a plurality of functions, a control model for controlling the robot device to perform the function; provide a selection model for selecting among the plurality of functions; execute multiple instances of a task by the robot device, including, in each execution, when a function of the plurality of functions needs to be selected to perform the task instance, checking whether the selection model provides a selection of a function and, if the selection model provides a selection of a function, control the robot device to perform the function selected by the selection model, using the control model for the selected function; and if the selection model does not provide a selection of a function, receive user input indicating a selection of a function, select a function according to the selection indicated by the user input, control the robot device to perform the function selected according to the selection indicated by the user input using the control model for the selected function. and train the selection model according to the selection indicated by the user input.

9. A non-transitory computer-readable medium on which are stored instructions controlling a robot device, the instructions, when executed by a computer, causing the computer to perform the following steps: providing, for each function of a plurality of functions, a control model for controlling the robot device to perform the function; providing a selection model for selecting among the plurality of functions; executing multiple instances of a task by the robot device, including, in each execution, when a function of the plurality of functions needs to be selected to perform the task instance, checking whether the selection model provides a selection of a function; if the selection model provides a selection of a function, controlling the robot device to perform the function selected by the selection model, using the control model for the selected function; and if the selection model does not provide a selection of a function, receiving user input indicating a selection of a function, selecting a function according to the selection indicated by the user input, controlling the robot device to perform the function selected according to the selection indicated by the user input using the control model for the selected function. and training the selection model according to the selection indicated by the user input.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG. 1 shows a robot according to an example embodiment of the present invention.

[0027] FIG. 2 illustrates robot control for a task goal according to an example embodiment of the present invention.

[0028] FIG. 3 illustrates the determination of feature vectors for an edge selector and a branch selector, according to an example embodiment of the present invention.

[0029] FIG. 4 shows a flow diagram illustrating a method for controlling a robot device, according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0030] The following detailed description refers to the figures that show, by way of illustration, specific details and aspects of this disclosure in which the present invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure can be combined with one or more other aspects of this disclosure to form new aspects.

[0031] In the following, various examples will be described in more detail.

[0032] FIG. 1 shows a robot 100.

[0033] The robot 100 includes a robot arm 101, for example an industrial robot arm for handling or assembling a work piece (or one or more other objects). The robot arm 101 includes manipulators 102, 103, 104 and a base (or support) 105 by which the manipulators 102, 103, 104 are supported. The term “manipulator” refers to the movable members of the robot arm 101, the actuation of which enables physical interaction with the environment, e.g. to carry out a task. For control, the robot 100 includes a (robot) controller 106 configured to implement the interaction with the environment according to a control program. The last member 104 (furthest from the support 105) of the manipulators 102, 103, 104 is also referred to as the end-effector 104 and may include one or more tools such as a welding torch, gripping instrument, painting equipment, or the like.

[0034] The other manipulators 102, 103 (closer to the support 105) may form a positioning device such that, together with the end-effector 104, the robot arm 101 with the end-effector 104 at its end is provided. The robot arm 101 is a mechanical arm that can provide similar functions as a human arm (possibly with a tool at its end).

[0035] The robot arm 101 may include joint elements 107, 108, 109 interconnecting the manipulators 102, 103, 104 with each other and with the support 105. A joint element 107, 108, 109 may have one or more joints, each of which may provide rotatable motion (i.e. rotational motion) and/or translatory motion (i.e. displacement) to associated manipulators relative to each other. The movement of the manipulators 102, 103, 104 may be initiated by means of actuators controlled by the controller 106.

[0036] The term “actuator” may be understood as a component adapted to affect a mechanism or process in response to be driven. The actuator can implement instructions issued by the controller 106 (the so-called activation) into mechanical movements. The actuator, e.g. an electromechanical converter, may be configured to convert electrical energy into mechanical energy in response to driving.

[0037] The term “controller” may be understood as any type of logic implementing entity, which may include, for example, a circuit and/or a processor capable of executing software stored in a storage medium, firmware, or a combination thereof, and which can issue instructions, e.g. to an actuator in the present example. The controller may be configured, for example, by program code (e.g., software) to control the operation of a system, a robot in the present example.

[0038] In the present example, the controller 106 includes one or more processors 110 and a memory 111 storing code and data based on which the processor 110 controls the robot arm 101. According to various embodiments, the controller 106 controls the robot arm 101 on the basis of a machine learning model 112 stored in the memory 111. The machine learning model 112 includes control models for skills and skill branches as well as a selection model to select among skills and skill branches.

[0039] A robot 100 can take advantage of learning-from-demonstration (LfD) approaches to learn to execute a skill or collaborate with a human partner. Human demonstrations can be encoded by a probabilistic model (also referred to as control model). The controller 106 can subsequently use the control model, which is also referred to as robot trajectory model, to generate the desired robot movements, possibly as a function of the state (configuration) of both the human partner, the robot and the robot's environment.

[0040] The basic idea of LfD is to fit a prescribed skill model such as GMMs to a handful of demonstrations. Let there be M demonstrations, each of which contains T.sub.m data points for a dataset of N=Σ.sub.mT.sub.m total observations ξ={ξ.sub.t}.sub.t=1.sup.N, where ξ.sub.t∈.sup.d. Also, it is assumed that the same demonstrations are recorded from the perspective of P different coordinate systems (given by the task parameters such as local coordinate systems or frames of objects of interest). One common way to obtain such data is to transform the demonstrations from a static global frame to frame p by ξ.sub.t.sup.(p)=A.sup.(p).sup.−1(ξ.sub.t−b.sup.(p)). Here, {(bb.sup.(p), A.sup.(p))}.sub.p=1.sup.P is the translation and rotation of (local) frame p w.r.t. the world (i.e. global) frame. Then, a TP-GMM is described by the model parameters {π.sub.k, {μ.sub.kb.sup.(p), Σ.sub.kb.sup.(p)}.sub.p=1.sup.P}.sub.k=1.sup.K where K represents the number of Gaussian components in the mixture model, π.sub.k is the prior probability of each component, and {μ.sub.kb.sup.(p), Σ.sub.kb.sup.(p))}.sub.p=1.sup.P are the parameters (mean and covariance) of the k-th Gaussian component within frame p.

[0041] Differently from standard GMM, the mixture model above cannot be learned independently for each frame. Indeed, the mixing coefficients π.sub.k are shared by all frames and the k-th component in frame p must map to the corresponding k-th component in the global frame. Expectation-Maximization (EM) is a well-established method to learn such models.

[0042] Once learned, the TP-GMM can be used during execution to reproduce a trajectory for the learned skill.

[0043] Hidden semi-Markov Models (HSMMs) extend standard hidden Markov Models (HMMs) by embedding temporal information of the underlying stochastic process. That is, while in HMM the underlying hidden process is assumed to be Markov, i.e., the probability of transitioning to the next state depends only on the current state, in HSMM the state process is assumed semi-Markov. This means that a transition to the next state depends on the current state as well as on the elapsed time since the state was entered. They can be applied, in combination with TP-GMMs, for robot skill encoding to learn spatio-temporal features of the demonstrations, resulting in a task-parameterized HSMM (TP-HSMM) model. A task-parameterized HSMM (TP-HSMM) model is defined as:

Θ={{a.sub.hk}.sub.h=1.sup.K,(μ.sub.k.sup.D,σ.sub.k.sup.D),π.sub.k,{(μ.sub.kb.sup.(p),Σ.sub.kb.sup.(p))}.sub.p=1.sup.P}.sub.k=1.sup.K,  (1)

[0044] where a.sub.hk is the transition probability from state h to k; (μ.sub.k.sup.D, σ.sub.k.sup.D) describe the Gaussian distributions for the duration of state k, i.e., the probability of staying in state k for a certain number of consecutive steps; {π.sub.k, {μ.sub.kb.sup.(p), Σ.sub.kb.sup.(p)}.sub.p=1.sup.P}.sub.k=1.sup.K equal the TP-GMM introduced earlier, representing the observation probability corresponding to state k. Note that herein the number of states corresponds to the number of Gaussian components in the “attached” TP-GMM.

[0045] Consider now a multi-DoF (degree of freedom) robotic arm 101 within a static and known workspace, of which the end-effector 104 has state r such as its 6-D pose and gripper state. Also, there are multiple objects of interest 113 denoted by O={o.sub.1, . . . o.sub.J} such as its 6-D pose.

[0046] It is assumed that there is a set of primitive skills that enable the robot to manipulate these objects, denoted by A={a.sub.1, a.sub.2, . . . , a.sub.H}. For each skill, a human user performs several kinaesthetic demonstrations on the robot. Particularly, for skill a∈A the set of objects involved is given by O.sub.a.Math.O and the set of demonstrations is given by

[0047] D.sub.a={D.sub.1, . . . , D.sub.M.sub.o}, where each demonstration D.sub.m is a timed sequence of states consists of the end-effector state r and object states {P.sub.o, o∈O.sub.a}, i.e.

D.sub.m=[s.sub.t].sub.t=1.sup.T.sup.m=[(r.sub.t,{p.sub.t,o,o∈O.sub.a})].sub.t=1.sup.T.sup.m.

[0048] Thus, for each (primitive) skill a∈A, a TP-HSMM Θ.sub.a (i.e. a control model) is learned as in equation (1).

[0049] Via a combination of these skills, the objects 113 can be manipulated by the robot arm 101 to reach different states. It is desirable that the robot 100 (specifically controller 106) is trained for a generic manipulation task, i.e. should be able to perform may different instances of a generic task. Each task instance is specified by an initial state so and a set of (at least one) desired goal states s.sub.G. A task (instance) is solved when the system state (also referred to as configuration) is changed from so to s.sub.G.

[0050] So, given a new task (s.sub.0; s.sub.G), the controller 106 should determine (i) the discrete sequence of (primitive) skills and (ii) the continuous robot trajectory to execute each skill. Here, a task may be an instance of a complex manipulation task where the sequence of desired skills and the associated trajectories depend significantly on the scenario of the task instance.

[0051] FIG. 2 illustrates robot control for a task goal 201 according to an embodiment.

[0052] According to various embodiments, the controller implements (extended) primitive skill models 202 and a GTN (geometric task network) which are trained interactively during online execution by human inputs.

Primitive Skill Learning

[0053] As illustrated for the skills in the skill models 202, there are often multiple ways of executing the same skill under different scenarios (called branches). For instance, there are five different ways of picking objects from a bin, i.e., approaching with different angles depending on the distances to each boundary. To handle the branching, the controller implements, for each (primitive) skill, a branch selector 207 as an extension to the TP-HSMM model Θ.sub.a for the skill.

[0054] The controller 106 trains the branch selector 207 from demonstrations 205 and online instructions 204 to choose, for each skill 202 to be performed for achieving the task goal 201, a branch 206. A branch 206 is, in other words, a variant of the skill. For example, the skill may be to pick an object from a bin and branches may be to pick the object from the left from the right, depending on where the object is located in the bin. For example, if it is positioned near a bin wall with it right side, the branch selector 207 selects the branch to pick the object from the left.

[0055] p∈{1, . . . , P} Consider a skill primitive a with M demonstrations (from demonstrations 205 provided for all skills) and B different branches. Each execution trajectory or demonstration of the skill is denoted by J.sub.m=[s.sub.t].sub.t=1.sup.T.sup.m which is associated with exactly one branch

[0056] p∈{1, . . . , P}b.sub.m∈B.sub.a={1, . . . , B}. Let J.sub.a denote the set of such trajectories, initialized to be the set of demonstrations D.sub.a (and supplemented during operation by online instructions 204). The frames associated with J.sub.m, are computed from the initial state s0, by abstracting the coordinates of the robot arm 101 and the objects 113, denoted by (F.sub.0, F.sub.1, . . . , F.sub.P), where F.sub.p=(b.sup.p, A.sup.p) is the coordinate of frame of frame; their order can be freely chosen but fixed afterwards. Then, the controller 106 derives a feature vector

v.sub.m=(F.sub.01,F.sub.12, . . . ,F.sub.(P-1)P)  (2)

[0057] where F.sub.ij=(b.sub.ij, α.sub.ij)∈.sup.6 is the relative transformation from frame F.sub.i to frame F.sub.j, b.sub.ij∈.sup.3 is the relative pose and α.sub.ij∈.sup.3 is the relative orientation. Thus, given J.sub.a, the controller 106 can construct the training data for the branch selector 207:

τ.sub.a.sup.B={(v.sub.m,b.sub.m),∀J.sub.m∈J.sub.a},

[0058] where b.sub.m is the branch label of trajectory J.sub.m; v.sub.m is the associated feature vector. The controller 106 can then train the branch selector 207, denoted by .sub.a.sup.B, via any multi-nominal classification algorithm. For example, logistic regression under the “one-vs-rest” strategy yields an effective selector from few training samples. Given a new scenario with state s.sub.t, the controller 106 chooses branch b with the probability:

ρ.sub.b=.sub.a.sup.B(s.sub.t,b),∀b∈B.sub.a,

[0059] where ρ.sub.b∈[0, 1]. Since most skills contain two or three frames, the feature vector v.sub.m normally has dimension 6 or 12.

Task Network Construction

[0060] As mentioned above, complex manipulation tasks often contain various sequences of skills 202 to account for different scenarios. For example, if a bar-code to be scanned is at the top of an object, the robot 100 needs to turn the object (i.e. execute a turn skill) before picking up the object and showing it to a camera (i.e. executing a show skill). This may not be required if the bar-code is already at the bottom. A high-level abstraction of such relations between skills is referred as task network. A valid plan evolves by transition from one skill 202 to another until the task is solved. The conditions on these transitions are particularly difficult and tedious to specify manually. Therefore, according to various embodiments, the controller 106 uses a coordination structure referred to as geometric task network (GTNs) 203, where the conditions are learned from task executions.

[0061] The a GTN 203 has a structure defined by a triple =(V, E, f). The set of nodes V is a subset of the primitive skills A; the set of edges E.Math.V×V contains the allowed transitions from one skill to another; the function ƒ:v.fwdarw. maps each node to a edge selector w.r.t. all of its outgoing edges. Intuitively, (V, E) specifies how skills can be executed consecutively for the given task, while function ƒ(v) models the different geometric constraints among the objects and the robot, for the outgoing edges of node v. It should be noted that ƒ(⋅) is explicitly conditioned on both the current system state and the goal state.

[0062] A complete plan of a task is given by the following sequence:

ξ=as.sub.0a.sub.0s.sub.1a.sub.1s.sub.2 . . . s.sub.Gā,

[0063] where a and ā are virtual “start” and “stop” skills, respectively. For different initial and goal states instances of the task the resulting plans can be different. Let Ξ={ξ} denote the set of complete plans for a set of given task instances. Then, for each “action-state-action” triple (a.sub.n, s.sub.n+1, a.sub.n+1) within ξ, first, the pair (a.sub.n, a.sub.n+1) is added to an edge set Ê if not already present; second, for each unique skill transition (a.sub.n, a.sub.n+1), a set of augmented states is collected, denoted by ŝ.sub.a.sub.n.sub.a.sub.n+1={ŝ}, where ŝ=(s.sub.n+1, s.sub.G) The controller 106 trains an edge selector 208 to select edges and thus, skills 202 to be performed. For this, for each augmented state =(s.sub.t,s.sub.G)∈ŝ.sub.a.sub.n.sub.a.sub.n+1, the controller 106 derives the following feature vector:

h.sub.l=(h.sub.tG,v.sub.G),  (3)

[0064] where h.sub.tG=(H.sub.r, H.sub.o.sub.1, . . . , H.sub.o.sub.H), where H.sub.o=(b.sub.o, α.sub.o)∈.sup.6 is the relative translation and rotation of robot r and objects o.sub.1, o.sub.2, . . . , o.sub.H∈O.sub.a.sub.n, from the current system state s.sub.t to the goal state s.sub.G; V.sub.G is the feature vector defined in equation (2) associated with the goal state s.sub.G. It should be noted that encapsulates features from both the relative transformation to the goal, and the goal state itself. Its dimension is linear to the number of objects relevant to skill a.sub.n, as shown in FIG. 3.

[0065] FIG. 3 illustrates the determination of feature vectors v.sub.m for the edge selector 208 and for the branch selector 207, given skill frames F.sub.p.

[0066] Once the controller 106 has processed all plans within Ξ, it can construct the GTN 203 as follows. First, its nodes and edges are directly derived from Ê. Then, for each node a, the set of its outgoing edges in Ê is given by Ê.sub.a={(a,)∈Ê}. Thus the function ƒ(a) returns the edge selector 208 .sub.a.sup.E over Ê.sub.a. To compute this selector, we first construct the following training data:

τ.sub.a.sup.E={(.sub.l,e),∀∈ŝ.sub.e,∀e∈Ê.sub.a},

where e is the label for an edge e=(a, )∈Ê.sub.a; is the feature vector derived from equation (3). Then the edge selector .sub.a.sup.E can be learned via a multi-nominal classification algorithms. Similar to the branch selector 207, given a new scenario with state s.sub.t and the specified goal state s.sub.G, the controller 106 then chooses an edge e with a probability of

ρ.sub.e=.sub.a.sup.E((s.sub.t,s.sub.G),e),∀e∈Ê.sub.a,

[0067] where ρ.sub.e∈[0, 1]. It should be noted that ρ.sub.e is trivial for skills with only one outgoing edge (i.e. with only one possibility for the subsequent skill).

[0068] In the previous two sections, the approaches to learn an extended skill model (involving a branch selector 207) and the task network 203 (involving edge selectors 208) were described. The required training data are execution trajectories of the skills and complete plans of the task. According to various embodiments, the controller 106 generates training data for the branch selectors 207 and the edge selectors 208 from human instructions 204 provided during run time. This allows improving both the skill model and the task network on-the-fly.

[0069] The GTN is initialized as empty (i.e. is initially untrained). Consider a problem instance of the task, namely (s.sub.O, s.sub.G). The system starts from state s.sub.n whereas the GTN 203 starts from the virtual start node a.sub.n=a for n=0. Then the associated edge selector .sub.a.sub.n.sup.E is used to compute the probability ρ.sub.e of each outgoing edge e∈Ê.sub.a.sub.n. Then, the next skill to execute is chosen as:

[00001]$a_{n+1}^{\star }=\underset{e\in {\hat{E}}_{a_n}}{\mathrm{argmax}}\left\{{\rho }_e\left(s_n,s_G\right)\mathrm{where}{\rho }_e>{\underset{¯}{\rho }}^E\right\},$

[0070] where ρ.sup.E>0 is a design parameter as the lower confidence bound. It should be noted that if the current set of outgoing edges is empty, i.e., Ê.sub.a.sub.n=Ø, or the maximal probability of all edges is less than ρ.sup.E, the controller 106 asks the human operator to input the preferred next skill a.sub.n+1* (as online instruction 204), e.g. by pausing execution, i.e. the robot 100 waits until the human operator inputs the online instruction, e.g. guides the robot arm 101 or inputs a branch number. Consequently, the controller 106 adds an additional data point to the training data τ.sub.a.sub.n.sup.E, i.e.,

τ.sub.a.sub.n.sup.E←(h(s.sub.n,s.sub.G),(a.sub.n,a.sub.n+1*)),  (4)

where the feature vector h is computed according to (3). Thus, a new edge (a.sub.n, a.sub.n+1*) is added to the graph topology (V,E) if not present, and the embedded function ƒ(⋅) is updated by re-learning the edge selectors .sub.n.sup.E given this new data point.

[0071] With regarding execution and update the branch selectors 207, let a.sub.n+1 be chosen as the next skill (according to the edge selector 208). Then the controller 106 uses the branch selector .sub.a.sub.n+1.sup.B to predict the probability of each branch ρ.sub.b, ∀b∈B.sub.a.sub.n+1. Then it chooses the most likely branch for a.sub.n+1 by

[00002]$b_{n+1}^{\star }=\underset{b\in B_{a_{n+1}}}{\mathrm{argmax}}\left\{{\rho }_b\left(s_n\right),\mathrm{where}{\rho }_b>{\underset{¯}{\rho }}^B\right\},$

[0072] where ρ.sup.B>0 is another parameter as the lower confidence bound for the branch selection. Again, as for the edge selection, if the controller 106 cannot find a branch in this manner, it asks the human operator to input the preferred branch b.sub.n+1* for skill a.sub.n+1, e.g. by guiding the robot arm 101 or inputting an edge number. In that case, the controller 106 adds an additional data point to the training data τ.sub.a.sub.n+1.sup.B, i.e.,

τ.sub.a.sub.n+1.sup.B←(v(s.sub.n),b.sub.n+1*),  (5)

[0073] where the feature vector v is computed according to equation (2).

[0074] Once the controller 106 has selected a branch b* for the desired next skill a.sub.n+1*, the controller 106 can retrieve its trajectory using the skill model θ.sub.a.sub.n+1. The retrieval process consists of two steps: First, the most-likely sequence of GMM components within the desired branch (denoted by K.sub.T*) can be computed via a modified Viterbi algorithm. Then, a reference trajectory 209 generated by an optimal controller 210 (e.g. LQG controller) to track this sequence of Gaussians in the task space. Thus this reference trajectory 209 is then sent to the low-level impedance controller to compute the control signal u*.

[0075] Afterwards, the system state is changed to S.sub.n+2 with different poses of the robot arm 113 and the objects 113, i.e., obtained from the state estimation and perception modules (such as a camera) providing observations 211. Given this new state, the controller 106 repeats the process to choose the next skill and its optimal branch, until the goal state 201 is reached.

[0076] In the following, an exemplary overall algorithm is given in pseudo code (using the usual English keywords like “while”, “do”, etc.).

TABLE-US-00001 Input: {D.sub.a, ∀a ∈ A}, human inputs {a.sub.n*, b.sub.n*}. Output: , {   .sub.a.sup.B}, u*. /* offline learning */ 1 Learn θ.sub.a and {   .sub.a.sup.B}, ∀a ∈ A. 2 Initialize or load existing   . /* online execution and learning */ 3 while new task (s.sub.0, s.sub.G) given do 4  | Set a.sub.n ← a and s.sub.n ← s.sub.0. 5  | while s.sub.n ≠ s.sub.G do 6  |  |  , a.sub.n+1 = ExUpGtn(   , a.sub.n, (s.sub.n, s.sub.G), a.sub.n+1*). 7  |  |  .sub.a.sub.n+1.sup.B, b.sub.n+1 = ExUpBrs(   .sub.a.sub.n+1.sup.B, s.sub.n, b.sub.n+1*). 8  |  | Compute u* for branch b.sub.n+1 of skill a.sub.n+1. 9  |  |.sub.— Obtain new state s.sub.n+1. Set n ← n + 1.  |.sub.—

[0077] During the online execution for solving new tasks, the algorithm executes and updates the GTN 203 as described above. This is done by the function ExUpGtn(⋅) in Line 6, with possible human input a.sub.n* if required. Once the next skill a.sub.n+1 is chosen, the algorithm executes and updates the branch selector 208, This is done by the function ExUpBrs(⋅) in Line 7, with possible human input b.sub.n+1* if required. Consequently the GTN 203 and branch selectors 208 are updated and improved according to equation (4) and equation (5) on-the-fly. Compared with the manual specification of the transition and branching conditions, the human inputs are more intuitive and easier to specify.

[0078] In summary, according to various embodiments, a method is provided as illustrated in FIG. 4.

[0079] FIG. 4 shows a flow diagram 400 illustrating a method for controlling a robot device.

[0080] In 401, for each function of a plurality of functions, a control model (skill model or skill branch model) for controlling the robot device to perform the function is provided.

[0081] In 402, a selection model for selecting among the plurality of functions is provided.

[0082] In 403, multiple instances of a task are executed by the robot device (i.e. the robot device is controlled to perform multiple instances of a task), comprising, in each execution, when a function of the plurality of functions needs to be selected to perform the task instance, checking whether the selection model provides a selection of a function and, if the selection model provides a selection of a function, controlling the robot device to perform the function selected by the selection model using the control model for the selected function in 404 and, if the selection model does not provide a selection of a function, receiving user input indicating a selection of a function, selecting a function according to the selection indicated by the user input, controlling the robot device to perform the function selected according to the selection indicated by the user input using the control model for the selected function and training the selection model according to the selection indicated by the user input in 405.

[0083] The approach of FIG. 4 can be used to compute a control signal for controlling a physical system generally referred to as “robot device”, like e.g. a computer-controlled machine, like a robot, a vehicle, a domestic appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. According to various embodiments, a policy for controlling the physical system may be learnt and then the physical system may be operated accordingly.

[0084] Various embodiments may receive and use image data (i.e. digital images) from various visual sensors (cameras) such as video, radar, LiDAR, ultrasonic, thermal imaging, motion, sonar etc., for example as a basis for the descriptor images.

[0085] According to one embodiment, the method is computer-implemented.

[0086] Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate and/or equivalent implementations may be substituted for the specific embodiments shown and described without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein.