Techniques described herein relate to using reduced-dimensionality embeddings generated from robot sensor data to identify predetermined semantic labels that guide robot interaction with objects. In various implementations, obtaining, from one or more sensors of a robot, sensor data that includes data indicative of an object observed in an environment in which the robot operates. The sensor data may be processed utilizing a first trained machine learning model to generate a first embedded feature vector that maps the data indicative of the object to an embedding space. Nearest neighbor(s) of the first embedded feature vector may be identified in the embedding space. Semantic label(s) may be identified based on the nearest neighbor(s). A given grasp option may be selected from enumerated grasp options previously associated with the semantic label(s). The robot may be operated to interact with the object based on the pose and using the given grasp option.

A set of one or more potentially graspable features for one or more objects present in a workspace area are determined based on visual data received from a plurality of cameras. For each of at least a subset of the one or more potentially graspable features one or more corresponding grasp strategies are determined to grasp the feature with a robotic arm and end effector. A score associated with a probability of a successful grasp of a corresponding feature is determined with respect to each of a least a subset of said grasp strategies. A first feature of the one or more potentially graspable features is selected to be grasped using a selected grasp strategy based at least in part on a corresponding score associated with the selected grasp strategy with respect to the first feature. The robotic arm and the end effector are controlled to attempt to grasp the first feature using the selected grasp strategy.

A control method of a manipulator is provided. The method includes photographing a target using a camera and detected the target using the photographed data. A holding motion for the target is set based on the detected target and a robot is operated to hold the target based on the set holding motion.

Techniques described herein relate to using reduced-dimensionality embeddings generated from robot sensor data to identify predetermined semantic labels that guide robot interaction with objects. In various implementations, sensor data obtained from one or more sensors of a robot includes data indicative of an object observed in an environment in which the robot operates. The sensor data is processed utilizing a first trained machine learning model to generate a first embedded feature vector that maps the data indicative of the object to an embedding space. Nearest neighbor(s) of the first embedded feature vector is identified in the embedding space. Semantic label(s) are identified based on the nearest neighbor(s). A given grasp option is selected from enumerated grasp options previously associated with the semantic label(s). The robot is operated to interact with the object based on the pose and using the given grasp option.

Embodiments are generally directed to generating a training dataset of labelled examples of sensor images and grasp configurations using a set of three-dimensional (3D) models of objects, one or more analytic mechanical representations of either or both of grasp forces and grasp torques, and statistical sampling to model uncertainty in either or both sensing and control. Embodiments can also include using the training dataset to train a function approximator that takes as input a sensor image and returns data that is used to select grasp configurations for a robot grasping or targeting mechanism.

Systems and methods for optimizing body and object interactions are provided. Based on obtained contact pressure maps and coefficient of friction (COF) maps at a contact interface where at least a portion of a body is in physical contact with a surface of an object, friction force maps can be determined, which can be used to optimize body and object interactions.

A system and method of operation of a robotic system including: receiving a sensor reading associated with a target object; generating a base plan for performing a task on the target object, wherein generating the base plan includes determining a grip point and one or more grip patterns associated with the grip point for gripping the target object based on a location of the grip point relative to a designated area, a task location, and another target object; implementing the base plan for performing the task by operating an actuation unit and one or more suction grippers according to a grip pattern rank, to generate an established grip on the target object, wherein the established grip is at a grip pattern location associated with the grip patterns; measuring the established grip; comparing the established grip to a force threshold; and re-gripping the target object based on the established grip falling below the force threshold.

A machine learning device that learns a waiting time for at least one of grasping of a workpiece by a spindle chuck and releasing of the workpiece by a loader chuck during transfer of the workpiece between the spindle chuckthat grasps and sends the workpiece and the loader chuck that grasps and receives the workpiece. The machine learning device includes a state observing unit that observes the waiting time and a FB current from a drive unit that moves the loader chuck as state variables, and a learning unit that learns the waiting time with which a transfer time of the workpiece is shortened, in accordance with a data set created based on the state variables.

A system and method of operation of a robotic system including: receiving a sensor reading associated with a target object; generating a base plan for performing a task on the target object, wherein generating the base plan includes determining a grip point and one or more grip patterns associated with the grip point for gripping the target object based on a location of the grip point relative to a designated area, a task location, and another target object; implementing the base plan for performing the task by operating an actuation unit and one or more suction grippers according to a grip pattern rank, to generate an established grip on the target object, wherein the established grip is at a grip pattern location associated with the grip patterns; measuring the established grip; comparing the established grip to a force threshold; and re-gripping the target object based on the established grip falling below the force threshold.

This control device is provided with a processing unit for calculating a permissible value of an external force that is permissible to be applied on a robot, a workpiece, or a hand. The permissible value of load that can be exerted on a constituent member of the robot is determined in advance. The processing unit calculates a permissible value of an external force in an application direction in which the external force is applied, on the basis of the position and attitude of the robot, the position of an application point where the external force is applied, and the permissible value of load of the constituent member of the robot.