Utilization of user interface inputs, from remote client devices, in controlling robot(s) in an environment. Implementations relate to generating training instances based on object manipulation parameters, defined by instances of user interface input(s), and training machine learning model(s) to predict the object manipulation parameter(s). Those implementations can subsequently utilize the trained machine learning model(s) to reduce a quantity of instances that input(s) from remote client device(s) are solicited in performing a given set of robotic manipulations and/or to reduce the extent of input(s) from remote client device(s) in performing a given set of robotic operations. Implementations are additionally or alternatively related to mitigating idle time of robot(s) through the utilization of vision data that captures object(s), to be manipulated by a robot, prior to the object(s) being transported to a robot workspace within which the robot can reach and manipulate the object.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of feedback-enabled user-robot collaboration includes obtaining a robot capability that models a robot's functionality for performing task actions, specializing the robot capability with an information kernel that encapsulates task-related parameters associated with the task actions, and providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the task-related parameters. The method also includes obtaining, based on the robot capability element's user interaction requirements, user interaction capability elements, via which the robot capability element receives user input and provides user feedback, controlling, based on the task-related parameters, the robot's functionality to perform the task actions in collaboration with the user input; and providing the user feedback including task-related information generated by the robot capability element in association with the task actions.

A robot movement teaching apparatus including a movement path extraction unit configured to process time-varying images of a first workpiece and fingers or arms of a human working on the first workpiece, and thereby extract a movement path of the fingers or arms of the human; a mapping generation unit configured to generate a transform function for transformation from the first workpiece to a second workpiece worked on by a robot, based on feature points of the first workpiece and feature points of the second workpiece; and a movement path generation unit configured to generate a movement path of the robot based on the movement path of the fingers or arms of the human extracted by the movement path extraction unit and based on the transform function generated by the mapping generation unit.

A method of operating a robotic control system comprising a master apparatus in communication with an input device having a handle and a slave system having a tool having an end effector whose position and orientation is determined in response to a position and orientation of the handle. The method involves producing a desired end effector position and a desired end effector orientation of the end effector, in response to a current position and a current orientation of the handle. The method further involves causing the input device to provide haptic feedback that impedes translational movement of the handle, while permitting rotational movement of the handle and preventing movement of the end effector, when a rotational alignment difference between the handle and the end effector meets a first criterion. The method further involves re-enabling translational movement of the handle when the rotational alignment difference meets a second criterion.

A method, a non-transitory computer readable medium, and an apparatus for operating the robotic control system comprising a master apparatus (64) in communication with an input device (58, 60) having a handle (102) and a slave system (54, 74) having a tool (66, 67) having an end effector (73) whose position and orientation is determined in response to a current position and current orientation of the handle. The method involves producing a desired end effector position and orientation in response to a current position and orientation of the handle. The method involves causing the input device to provide haptic feedback that impedes translational movement of the handle, while permitting rotational movement of the handle and preventing movement of the end effector, when a rotational alignment difference between the handle and the end effector meets a disablement criterion. The method further involves re-enabling translational movement of the handle when the rotational alignment difference meets an enablement criterion.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of feedback-enabled user-robot collaboration includes obtaining a robot capability that models a robot's functionality for performing task actions, specializing the robot capability with an information kernel that encapsulates task-related parameters associated with the task actions, and providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the task-related parameters. The method also includes obtaining, based on the robot capability element's user interaction requirements, user interaction capability elements, via which the robot capability element receives user input and provides user feedback, controlling, based on the task-related parameters, the robot's functionality to perform the task actions in collaboration with the user input; and providing the user feedback including task-related information generated by the robot capability element in association with the task actions.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of feedback-enabled user-robot collaboration includes obtaining a robot capability that models a robot's functionality for performing task actions, specializing the robot capability with an information kernel that encapsulates task-related parameters associated with the task actions, and providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the task-related parameters. The method also includes obtaining, based on the robot capability element's user interaction requirements, user interaction capability elements, via which the robot capability element receives user input and provides user feedback, controlling, based on the task-related parameters, the robot's functionality to perform the task actions in collaboration with the user input; and providing the user feedback including task-related information generated by the robot capability element in association with the task actions.

Utilization of user interface inputs, from remote client devices, in controlling robot(s) in an environment. Implementations relate to generating training instances based on object manipulation parameters, defined by instances of user interface input(s), and training machine learning model(s) to predict the object manipulation parameter(s). Those implementations can subsequently utilize the trained machine learning model(s) to reduce a quantity of instances that input(s) from remote client device(s) are solicited in performing a given set of robotic manipulations and/or to reduce the extent of input(s) from remote client device(s) in performing a given set of robotic operations. Implementations are additionally or alternatively related to mitigating idle time of robot(s) through the utilization of vision data that captures object(s), to be manipulated by a robot, prior to the object(s) being transported to a robot workspace within which the robot can reach and manipulate the object.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of feedback-enabled user-robot collaboration includes obtaining a robot capability that models a robot's functionality for performing task actions, specializing the robot capability with an information kernel that encapsulates task-related parameters associated with the task actions, and providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the task-related parameters. The method also includes obtaining, based on the robot capability element's user interaction requirements, user interaction capability elements, via which the robot capability element receives user input and provides user feedback, controlling, based on the task-related parameters, the robot's functionality to perform the task actions in collaboration with the user input; and providing the user feedback including task-related information generated by the robot capability element in association with the task actions.

Methods and systems for enabling human-machine collaborations include a generalizable framework that supports dynamic adaptation and reuse of robotic capability representations and human-machine collaborative behaviors. Specifically, a method of enabling user-robot collaboration includes providing a composition of a robot capability that models a robot's functionality for performing a type of task action and user interaction capabilities; specializing the robot capability with an information kernel to provide a specialized robot capability, the information kernel encapsulating a set of task-related parameters associated with the type of task action; providing an instance of the specialized robot capability as a robot capability element that controls the robot's functionality based on the set of task-related parameters; providing instances of the user interaction capabilities as interaction capability elements; executing the robot capability element to receive user input via the user interaction capability elements; and controlling, based on the user input and the set of task-related parameters, the robot's functionality to perform a task action of the type of task action in collaboration with the user input.