Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food preparation recipe. In one embodiment, a robotic control platform, comprises one or more sensors; a mechanical robotic structure including one or more end effectors, and one or more robotic arms; an electronic library database of minimanipulations; a robotic planning module configured for real-time planning and adjustment based at least in part on the sensor data received from the one or more sensors in an electronic multi-stage process file, the electronic multi-stage process recipe file including a sequence of minimanipulations and associated timing data; a robotic interpreter module configured for reading the minimanipulation steps from the minimanipulation library and converting to a machine code; and a robotic execution module configured for executing the minimanipulation steps by the robotic platform to accomplish a functional result.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for integrating sensor streams for robotic demonstration learning. One of the methods includes selecting, by a learning system for a robot, a base update rate for combining multiple sensor streams into a task state representation. The learning system repeatedly generates the task state representation at the base update rate, including combining, during each time period defined by the update rate, the task state representation from most recently updated sensor data processed by the plurality of neural networks. The learning system repeatedly uses the task state representations to generate commands for the robot at the base update rate.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for providing user feedback for robotic demonstration learning. One of the methods includes initiating a local demonstration learning process to collect respective local demonstration data for each of one or more demonstration subtasks defined by a skill template to be executed by a robot. Local demonstration data is repeatedly collected for each of the one or more demonstration subtasks of the skill template while a user manipulates a robot to perform each of the one or more demonstration subtasks defined by the skill template. A respective progress value for each of the one or more demonstration subtasks defined by the skill template is maintained. A user interface presentation is generated that presents a suggested demonstration to be performed by the user based on a respective progress value for each demonstration subtask.

Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food preparation recipe. In one embodiment, a robotic control platform, comprises one or more sensors; a mechanical robotic structure including one or more end effectors, and one or more robotic arms; an electronic library database of minimanipulations; a robotic planning module configured for real-time planning and adjustment based at least in part on the sensor data received from the one or more sensors in an electronic multi-stage process file, the electronic multi-stage process recipe file including a sequence of minimanipulations and associated timing data; a robotic interpreter module configured for reading the minimanipulation steps from the minimanipulation library and converting to a machine code; and a robotic execution module configured for executing the minimanipulation steps by the robotic platform to accomplish a functional result.

A method of deriving autonomous control information involves receiving one or more sets of associated environment sensor information and device control instructions. Each set of associated environment sensor information and device control instructions includes environment sensor information representing an environment associated with an operator controllable device and associated device control instructions configured to cause the operator controllable device to simulate at least one action taken by at least one operator experiencing a representation of the environment generated from the environment sensor information. The method also involves deriving autonomous control information from the one or more sets of associated environment sensor information and device control instructions, the autonomous control information configured to facilitate generating autonomous device control signals from autonomous environment sensor information representing an environment associated with an autonomous device, the autonomous device control signals configured to cause the autonomous device to take at least one autonomous action.

An automated kitchen assistant system inspects a food preparation area in the kitchen environment using a novel sensor combination. The combination of sensors includes an Infrared (IR) camera that generates IR image data and at least one secondary sensor that generates secondary image data. The IR image data and secondary image data are processed to obtain combined image data. A trained convolutional neural network is employed to automatically compute an output based on the combined image data. The output includes information about the identity and the location of the food item. The output may further be utilized to command a robotic arm, kitchen worker, or otherwise assist in food preparation. Related methods are also described.

There is disclosed a method of training an RPA robot to use a GUI. The method comprises capturing video of the GUI as an operator uses the GUI to carry out a process; capturing a sequence of events triggered as the operator uses the GUI to carry out said process; and analyzing said video and said sequence of events to thereby generate a workflow. The workflow, when executed by an RPA robot, causes the RPA robot to carry out said process using the GUI.

A method for the surface treatment of an article (2) by means of a robotic device (3) comprising a robotic arm (5) and a spraying head (4) fitted on the robotic arm (5); the method comprises a learning step, during which the operator moves the spraying head (4) by means of a handling device (9) and the movements made by the spraying head (4) are stored by a storage unit (8); and a reproduction step, which is subsequent to the learning step and during which the robotic arm (5) is operated so that the spraying head (4) repeats the movements stored by the storage unit (8).

Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food preparation recipe. In one embodiment, a robotic control platform, comprises one or more sensors; a mechanical robotic structure including one or more end effectors, and one or more robotic arms; an electronic library database of minimanipulations; a robotic planning module configured for real-time planning and adjustment based at least in part on the sensor data received from the one or more sensors in an electronic multi-stage process file, the electronic multi-stage process recipe file including a sequence of minimanipulations and associated timing data; a robotic interpreter module configured for reading the minimanipulation steps from the minimanipulation library and converting to a machine code; and a robotic execution module configured for executing the minimanipulation steps by the robotic platform to accomplish a functional result.

Described are techniques for storing and retrieving items using a robotic manipulator. Images depicting a human interacting with an item, sensor data from sensors instrumenting the human or item, data regarding physical characteristics of the item, and constraint data relating to the robotic manipulator or the item may be used to generate one or more configurations for the robotic manipulator. The configurations may include points of contact and force vectors for contacting the item using the robotic manipulator.