The present disclosure is directed to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the human chef's movements and actions. Monitoring a human chef is carried out in an instrumented application-specific setting, a standardized robotic kitchen in this instance, and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef.

A robot teaching and testing system and method that performs human-operated robot tasks according to instructions generated from generative AI models. The process starts with a user prompt and combines the user prompt with predefined prompt templates to generate well-formatted text prompts. Generative AI models take the text prompts and convert them into high-level instructions or control codes that can be deployed on a robot. The high-level instructions are then converted into human-operated robot tasks for a human data collector using a mixed reality (MR) device. The human data collector will attempt to follow the instructions to complete the human-operated robot tasks and may overwrite the suggested instructions by performing a different action, demonstrate a task without instructions, or leave feedback or comments regarding the tasks. Feedback data will be captured and saved for improving the robot system.

Implementations described herein relate to training and refining robotic control policies using imitation learning techniques. A robotic control policy can be initially trained based on human demonstrations of various robotic tasks. Further, the robotic control policy can be refined based on human interventions while a robot is performing a robotic task. In some implementations, the robotic control policy may determine whether the robot will fail in performance of the robotic task, and prompt a human to intervene in performance of the robotic task. In additional or alternative implementations, a representation of the sequence of actions can be visually rendered for presentation to the human can proactively intervene in performance of the robotic task.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of feedback-enabled user-robot collaboration includes obtaining a robot capability that models a robot's functionality for performing task actions, specializing the robot capability with an information kernel that encapsulates task-related parameters associated with the task actions, and providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the task-related parameters. The method also includes obtaining, based on the robot capability element's user interaction requirements, user interaction capability elements, via which the robot capability element receives user input and provides user feedback, controlling, based on the task-related parameters, the robot's functionality to perform the task actions in collaboration with the user input; and providing the user feedback including task-related information generated by the robot capability element in association with the task actions.

A unified collaboration environment is formed by establishing a local workspace positional frame of reference using a plurality of UWB transceivers. With a frame of reference established a communication link is established between each of the workspaces, and a collaboration module to establish a peer-to-peer network. Data is received from each of the workspaces including the local workspace frame of reference, the set of available assets and workspace behavior (tasks). The collaboration module crafts a unified collaboration environment by transforming the local workspace into a collaborative positional frame of reference. A user, through a user interface, can offer real-time input to a virtualized version of the workspace to augment actions within the workspace environment.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of enabling user-robot collaboration includes providing a composition of a robot capability that models a robot's functionality for performing a type of task action and user interaction capabilities; specializing the robot capability with an information kernel to provide a specialized robot capability, the information kernel encapsulating a set of task-related parameters associated with the type of task action; providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the set of task-related parameters; providing instances of the user interaction capabilities as interaction capability elements; executing the robot capability element to receive user input via the user interaction capability elements; and controlling, based on the user input and the set of task-related parameters, the robot's functionality to perform a task action of the type of task action in collaboration with the user input.

Embodiments of the present disclosure are directed to methods, computer program products, and computer systems of a robotic apparatus with robotic instructions replicating a food preparation recipe. In one embodiment, a robotic control platform, comprises one or more sensors; a mechanical robotic structure including one or more end effectors, and one or more robotic arms; an electronic library database of minimanipulations; a robotic planning module configured for real-time planning and adjustment based at least in part on the sensor data received from the one or more sensors in an electronic multi-stage process file, the electronic multi-stage process recipe file including a sequence of minimanipulations and associated timing data; a robotic interpreter module configured for reading the minimanipulation steps from the minimanipulation library and converting to a machine code; and a robotic execution module configured for executing the minimanipulation steps by the robotic platform to accomplish a functional result.

A machine learning device for a robot that allows a human and the robot to work cooperatively, the machine learning device including a state observation unit that observes a state variable representing a state of the robot during a period in that the human and the robot work cooperatively; a determination data obtaining unit that obtains determination data for at least one of a level of burden on the human and a working efficiency; and a learning unit that learns a training data set for setting an action of the robot, based on the state variable and the determination data.

A method for training and/or testing a robot control module. The method includes generating an instruction specified by a robot control module configured for robot training and/or testing, the instruction indicating how a human-driven robot task is to be performed when training and/or testing the robot control module; providing the instruction to a mixed reality device worn by a human data collector, the mixed device rendering the instruction in a manner that shows the human data collector how to perform the human-driven robot task; collecting performance data and environmental data in response to the human data collector attempting to perform the human-driven robot task using the data collection device; receiving feedback data in response to the human data collector attempting to perform the human-driven robot task specified by the instruction; and updating the robot control module using the feedback data and the collected performance and environmental data.

The invention relates to a method of controlling an industrial machine, the industrial machine comprising a control unit, e.g. a PLC, and at least one actuator and/or sensor which is controlled by the control unit, wherein a Language Model is provided, a Language Model Interface is provided, a Context Information Library is provided, which stores information on the industrial machine, particularly commands executable by the industrial machine, wherein the Language Model Interface provides information on the industrial machine from the Context Information Library to the Language Model, the Language Model sends commands to be executed by the industrial machine to the Language Model Interface, the Language Model Interface translates the commands received from the Language Model into machine commands for the control unit.