Apparatuses, systems, and techniques to grasp objects with a robot. In at least one embodiment, a neural network is trained to determine a grasp pose of an object within a cluttered scene using a point cloud generated by a depth camera.

Mitigating the reality gap through utilization of technique(s) that enable compliant robotic control and/or compliant robotic contact to be simulated effectively by a robotic simulator. The technique(s) can include, for example: (1) utilizing a compliant end effector model in simulated episodes of the robotic simulator; (2) using, during the simulated episodes, a soft constraint for a contact constraint of a simulated contact model of the robotic simulator; and/or (3) using proportional derivative (PD) control in generating joint control forces, for simulated joints of the simulated robot, during the simulated episodes. Implementations additionally or alternatively relate to determining parameter(s), for use in one or more of the techniques that enable effective simulation of compliant robotic control and/or compliant robotic contact.

A controller includes: an acquisition unit configured to acquire process data for an actually used controlled object; and a calculation unit configured to, based on at least one of a target set value and parameters for the actually used controlled object, convert the process data acquired by the acquisition unit, and calculate a manipulated variable for the actually used controlled object by use of the converted process data and a trained model. Upon receiving input of process data for a specific controlled object, the trained model outputs a manipulated variable for approximating process data for the specific controlled object to a specific target set value. The parameters include parameters for specifying the relation between manipulated variables for the actually used controlled object and process data obtained by use of the manipulated variables.

Techniques are disclosed for training and applying machine learning models to control robotic assembly. In some embodiments, force and torque measurements are input into a machine learning model that includes a memory layer that introduces recurrency. The machine learning model is trained, via reinforcement learning in a robot-agnostic environment, to generate actions for achieving an assembly task given the force and torque measurements. During training, experiences are collected as transitions within episodes, the transitions are grouped into sequences, and the last two sequences of each episode have a variable overlap. The collected transitions are stored in a prioritized sequence replay buffer, from which a learner samples sequences to learn from based on transition and sequence priorities. Once trained, the machine learning model can be deployed to control various types of robots to perform the assembly task based on force and torque measurements acquired by sensors of those robots.

A method for controlling a robotic item handler is described. The method includes obtaining first point cloud data related to a first three-dimensional (3D) image and second point cloud data related to a second 3D image, captured by a first sensor device and a second sensor device respectively. Further, the method can include transforming the first point cloud data and the second point cloud data to combined point cloud data that is used as an input to a convolutional neural network, to construct a machine learning model. The machine learning model can output a decision classification indicative of a first probability associated with a first operating mode and a second probability associated with a second operating mode. Furthermore, the method can include operating the robotic item handler according to the first operating mode or the second operating mode based on a comparison of the first probability and the second probability.

Methods of positioning a surgical instrument can involve advancing a first medical instrument to a treatment site of a patient, the first medical instrument comprising a camera, recording a target position associated with a target anatomical feature at the treatment site, generating a first image of the treatment site using the camera of the first medical instrument, identifying the target anatomical feature in the first image using a pretrained neural network, and adjusting the target position based at least in part on a position of the identified target anatomical feature in the first image.

A data driven approach for fault detection in robotic actuation is disclosed. Here, a set of robotic tasks are received and analyzed by a Deep Learning (DL) analytics. The DL analytics includes a stateful (Long Short Term Memory) LSTM. Initially, the stateful LSTM is trained to match a set of activities associated with the robots based on a set of tasks gathered from the robots in a multi robot environment. Here, the stateful LSTM utilizes a master slave framework based load distribution technique and a probabilistic trellis approach to predict a next activity associated with the robot with minimum latency and increased accuracy. Further, the predicted next activity is compared with an actual activity of the robot to identify any faults associated robotic actuation.

The present disclosure provides action imitation method as well as a robot and a computer readable storage medium using the same. The method includes: collecting at least a two-dimensional image of a to-be-imitated object; obtaining two-dimensional coordinates of each key point of the to-be-imitated object in the two-dimensional image and a pairing relationship between the key points of the to-be-imitated object; converting the two-dimensional coordinates of the key points of the to-be-imitated object in the two-dimensional image into space three-dimensional coordinates corresponding to the key points of the to-be-imitated object through a pre-trained first neural network model, and generating an action control instruction of a robot based on the space three-dimensional coordinates corresponding to the key points of the to-be-imitated object and the pairing relationship between the key points, where the action control instruction is for controlling the robot to imitate an action of the to-be-imitated object.

The disclosure is to constitute, while reducing a cost for collecting training data used in machine learning that makes a control module acquire an ability to control a robot device, the control module operatable in an actual environment by the machine learning. A learning device according to one aspect of the present invention executes machine learning of an extractor by using a first learning data set constituted by a combination of simulation data and first environmental information and a second learning data set constituted by a combination of actual data and second environmental information. Further, a learning device according to one aspect of the present invention executes machine learning of a controller by using a third learning data set constituted by a combination of third environmental information, state information, and a control command.

A tool control system may include: a tactile sensor configured to, when a tool holds a target object and slides the target object downward across the tool, obtain tactile sensing data from the tool; one or more memories configured to store a target velocity and computer-readable instructions; and one or more processors configured execute the computer-readable instructions to: receive the tactile sensing data from the tactile sensor; estimate a velocity of the target object based on the tactile sensing data, by using one or more neural networks that are trained based on a training image of an sample object captured while the sample object is sliding down; and generate a control parameter of the tool based on the estimated velocity and the target velocity.