Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a robotic control plan. One of the methods includes obtaining, from a user device, image data depicting an instruction manual for assembling a plurality of assembly components; processing the image data using a machine learning model to generate instruction data representing a sequence of instructions for assembling the plurality of assembly components, wherein the machine learning model has been configured through training to process images depicting instruction manuals and to generate instruction data characterizing sequences of instructions identified in the instruction manuals; processing the instruction data to generate a robotic control plan to be executed by one or more robotic components for assembling the plurality of assembly components; and providing the robotic control plan to a robotic control system for executing the robotic control plan using the one or more robotic components.

A device and a method for controlling a robotic device, including a control model. The control model includes a robot trajectory model, which for the pickup includes a hidden semi-Markov model with one or multiple initial states, a precondition model, which for each initial state of the robot trajectory model includes a probability distribution of robot configurations before the pickup is carried out, and an object pickup model, which for a depth image outputs a plurality of pickup robot configurations having a respective associated probability of success.

An operation range setting device 1X includes a first recognition means 15Xa, a second recognition means 15Xb, and an operation range setting means 17X. The first recognition means 15Xa is configured to recognize positions of plural reference objects. The second recognition means 15Xb is configured to recognize combinations of reference objects, the combinations each being selected to be a pair of the reference objects from the plural reference objects. The operation range setting means 17X is configured to set an operation range of a robot based on line segments, the line segments each connecting a pair of the reference object for each of the combinations.

A training system for a robot includes: a model having a transformer architecture and configured to determine how to actuate at least one of arms and an end effector of the robot; a training dataset including sets of demonstrations for the robot to perform training tasks, respectively; and a training module configured to: meta-train a policy of the model using first ones of the sets of demonstrations for first ones of the training tasks, respectively; and optimize the policy of the model using second ones of the sets of demonstrations for second ones of the training tasks, respectively, where the sets of demonstrations for the training tasks each include more than one demonstration and less than a first predetermined number of demonstrations.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for robotic control using demonstrations to learn category-level manipulation task. One of the methods includes obtaining a collection of object models for a plurality of different types of objects belonging to a same object category and training a category-level representation in a category-level space from the collection of object models. A category-level trajectory is generated the demonstration data of a demonstration object. For a new object in the object category, a trajectory projection is generated in the category-level space, which is used to cause a robot to perform the robotic manipulation task on the new object.

A robot device according to various embodiments comprises a camera, a robot arm, and a control device electrically connected to the camera and the robot arm, wherein the control device can be configured to collect a robot arm control record about a random operation, acquire, from the camera, a camera image in which a working space of the robot arm is photographed, implement an augmented reality model by rendering a virtual object corresponding to an object related to objective work in the camera image, and update a control policy for the objective work by performing image-based policy learning on the basis of the augmented reality model and the control record.

Systems and a method for predicting motion-outcome data of a robot moving between a given pair of robotic locations. Data on a given pair of robotic locations are received as input data. A function trained by a machine learning algorithm is applied to the input data, wherein a related robotic motion-outcome data is generated as output data. The robotic motion-outcome data is provided as output data.

Example implementations described herein are directed to a simulation environment for a real world system involving one or more robots and one or more sensors. Scenarios are loaded into a simulation environment having one or more virtual robots corresponding to the one or more robots, and one or more virtual sensors corresponding to the one or more virtual system to train a control strategy model from reinforcement learning, which is subsequently deployed to the real world environment. In cases of failure of the real world environment, the failures are provided to the simulation environment to generate an updated control strategy model for the real world environment.

Robotic devices may be trained by a user guiding the robot along a target trajectory using a correction signal. A robotic device may comprise an adaptive controller configured to generate control commands based on one or more of the trainer input, sensory input, and/or performance measure. Training may comprise a plurality of trials. During an initial portion of a trial, the trainer may observe robot's operation and refrain from providing the training input to the robot. Upon observing a discrepancy between the target behavior and the actual behavior during the initial trial portion, the trainer may provide a teaching input (e.g., a correction signal) configured to affect robot's trajectory during subsequent trials. Upon completing a sufficient number of trials, the robot may be capable of navigating the trajectory in absence of the training input.

Methods and systems for controlling a robot. In one aspect, the method (1300) comprises obtaining (s1302) first input information associated with a first simulation environment to which a first level of realism is assigned and obtaining (s1304) second input information associated with a second simulation environment to which a second level of realism is assigned. The first level of realism is different from the second level of realism. The method further comprises associating (s1306) the first input information with a first realism value representing the first level of realism; and associating (s1308) the second input information with a second realism value representing the second level of realism. The method further comprises modifying (s1310), based on the associated first input information and the associated second input information, one or more parameters of a machine learning (ML) process used for controlling the robot.