Provided is a robot system including: a robot including a hand; a unit job storage section configured to store a unit job; a linking job generation section configured to generate a linking job being a command to move the hand from an end position at which a first unit job has ended to a start position at which a second unit job to be executed subsequently after the first unit job is started; an action command generation section configured to generate an action command for the robot by connecting the unit jobs and the linking job in series, based on arrangement of a plurality of processing symbols; and a required time calculation section configured to calculate a required time of the action command by adding required times of the unit jobs and a required time of the linking job.

Implementations described herein relate to training and refining robotic control policies using imitation learning techniques. A robotic control policy can be initially trained based on human demonstrations of various robotic tasks. Further, the robotic control policy can be refined based on human interventions while a robot is performing a robotic task. In some implementations, the robotic control policy may determine whether the robot will fail in performance of the robotic task, and prompt a human to intervene in performance of the robotic task. In additional or alternative implementations, a representation of the sequence of actions can be visually rendered for presentation to the human can proactively intervene in performance of the robotic task.

This disclosure discloses a robot system including one or more work facilities and a teaching information database. The work facilities comprise a robot and robot controller. The robot controller controls the movement of the robot based on teaching information stored in a storage part. The teaching information database stores a plurality of types of the teaching information associated with work information. Each work facility includes an interface device configured to receive an input of search condition information, to search teaching information highly relevant to the search condition information among the plurality of types of teaching information, and to receive a selection of desired teaching information among one or more sets of the teaching information hit in the search. The robot system further comprises a first transferring part configured to transfer the teaching information from the teaching information database to the storage part.

A computer-implemented method of determining actions for controlling a robot, in particular an assembly robot, includes (i) receiving a first and second input, wherein the first input is a sentence describing an action which should be carried out by the robot, wherein the second input is an image of a current state of an environment of the robot, (ii) feeding the first input into a first machine learning model and feeding the second input into a second machine learning model, wherein the first and second machine learning models are configured to determine tokens for their respective inputs, and (iv) feeding the tokens into a third machine learning model, wherein the third machine learning model outputs two outputs, wherein the first output is a switch for incorporating specialized skill networks and the second output are actions.

A method for controlling a robotic manipulator according to a task comprises accepting a feedback signal including a sequence of multi-modal observations of a state of execution of the task. The multi-modal observations are processed with a neural network having a self-attention module with a hierarchically conditioned output to produce a skill of the robotic manipulator and an action conditioned on the skill. The neural network is trained in a supervised manner with demonstration data to produce a sequence of skills and a corresponding sequence of actions for the actuators of the robotic manipulator to perform the task. The method further comprises determining one or more control commands for the one or more actuators based on the produced action and submitting the one or more control commands to the one or more actuators causing a change of the state of execution of the task.