In an embodiment, a method includes determining a given material to manipulate to achieve a goal state. The goal state can be one or more deformable or granular materials in a particular arrangement. The method further includes, for the given material, determining, a respective outcome for each of a plurality of candidate actions to manipulate the given material. The determining can be performed with a physics-based model, in one embodiment. The method further can include determining a given action of the candidate actions, where the outcome of the given action reaching the goal state is within at least one tolerance. The method further includes, based on a selected action of the given actions, generating a first motion plan for the selected action.

A controller of the robot apparatus performs approaching control for making a second workpiece approach a first workpiece and position adjustment control for adjusting a position of the second workpiece with respect to a position of the first workpiece. The approaching control includes control for calculating a movement direction and a movement amount of a position of the robot based on an image captured by a first camera, and making the second workpiece approach the first workpiece. The position adjustment control includes control for calculating a movement direction and a movement amount of a position of the robot based on an image captured by the first camera and an image captured by the second camera, and precisely adjusting a position of the first workpiece with respect to the second workpiece.

Systems and methods for detecting pose and force against an object are provided. A method includes receiving a signal from a deformable sensor comprising data from a deformation region in a deformable membrane resulting from contact with the object utilizing an internal sensor disposed within an enclosure and having a field of view directed through a medium and toward a bottom surface of the deformable membrane. The method also determines a pose of the object based on the deformation region of the deformable membrane. The method also determines an amount of force applied between the deformable membrane and the object is determined based on the deformation region of the deformable membrane.

Systems and methods for moving or manipulating robotic arms are provided. A group of robotic arms are configured to form a virtual rail or line between the end effectors of the robotic arms. The robotic arms are responsive to outside force such as from a user. When a user moves a single one of the robotic arms, the other robotic arms will automatically move to maintain the virtual rail alignments. The virtual rail of the robotic arm end effectors may be translated in one or more of three dimensions. The virtual rail may be rotated about a point on the virtual rail line. The robotic arms can detect the nature of the contact from the user and move accordingly. Holding, shaking, tapping, pushing, pulling, and rotating different parts of the robotic arm elicits different movement responses from different parts of the robotic arm.

In an embodiment, a method during execution of a motion plan by a robotic arm includes determining a voice command from speech of a user said during the execution of the motion plan, determining a modification of the motion plan based on the voice command from the speech of the user, and executing the modification of the motion plan by the robotic arm.

Systems and methods for detecting pose and force against an object are provided. A method includes receiving a signal from a deformable sensor comprising data from a deformation region in a deformable membrane resulting from contact with the object utilizing an internal sensor disposed within an enclosure and having a field of view directed through a medium and toward a bottom surface of the deformable membrane. The method also determines a pose of the object based on the deformation region of the deformable membrane. The method also determines an amount of force applied between the deformable membrane and the object is determined based on the deformation region of the deformable membrane.

In an embodiment, a method and system use various sensors to determine a shape of a collection of materials (e.g., foodstuffs). A controller can determine a trajectory which achieves the desired end-state, possibly chosen from a set of feasible, collision-free trajectories to execute, and a robot executes that trajectory. The robot, executing that trajectory, scoops, grabs, or otherwise acquires the desired amount of material from the collection of materials at a desired location. The robot then deposits the collected material in the desired receptacle at a specific location and orientation.

Systems and methods for detecting pose and force against an object are provided. A method includes receiving a signal from a deformable sensor comprising data from a deformation region in a deformable membrane resulting from contact with the object utilizing an internal sensor disposed within an enclosure and having a field of view directed through a medium and toward a bottom surface of the deformable membrane. The method also determines a pose of the object based on the deformation region of the deformable membrane. The method also determines an amount of force applied between the deformable membrane and the object is determined based on the deformation region of the deformable membrane.

A robotic system includes control circuitry configured to cause actuation of one or more actuators of each of a first robotic arm and a second robotic arm. The control circuitry is configured to determine a position of a first end effector of the first robotic arm and a position of a second end effector of the second robotic arm, the positions of the first end effector and the second end effector forming a virtual rail, receive manual positioning input for the first robotic arm based at least in part on sensor signals from one or more sensors of the first robotic arm, and in response to the manual positioning input, generate a first movement command to move the first robotic arm in accordance with the manual positioning input and generate a second movement command to move the second robotic arm in a manner as to maintain at least one of a position or orientation of the second end effector relative to a point on the virtual rail.

A method for controlling a robot includes applying a setpoint force to a contact point; measuring a contact stiffness at the contact point; and slowing down the moving robot using its drives and/or braking the robot to apply the setpoint force to the contact point by the slowing down and/or slowed down robot depending on the measured contact stiffness, wherein the robot is slowed down before the setpoint force is reached.