Methods and systems for detecting and reconstructing environments to facilitate robotic interaction with such environments are described. An example method may involve determining a three-dimensional (3D) virtual environment representative of a physical environment of the robotic manipulator including a plurality of 3D virtual objects corresponding to respective physical objects in the physical environment. The method may then involve determining two-dimensional (2D) images of the virtual environment including 2D depth maps. The method may then involve determining portions of the 2D images that correspond to a given one or more physical objects. The method may then involve determining, based on the portions and the 2D depth maps, 3D models corresponding to the portions. The method may then involve, based on the 3D models, selecting a physical object from the given one or more physical objects. The method may then involve providing an instruction to the robotic manipulator to move that object.

Example systems and methods allow for dynamic updating of a plan to move objects using a robotic device. One example method includes determining a virtual environment by one or more processors based on sensor data received from one or more sensors, the virtual environment representing a physical environment containing a plurality of physical objects, developing a plan, based on the virtual environment, to cause a robotic manipulator to move one or more of the physical objects in the physical environment, causing the robotic manipulator to perform a first action according to the plan, receiving updated sensor data from the one or more sensors after the robotic manipulator performs the first action, modifying the virtual environment based on the updated sensor data, determining one or more modifications to the plan based on the modified virtual environment, and causing the robotic manipulator to perform a second action according to the modified plan.

A robot performs, after i-th (i is a natural number) work, i+1-th work different from the i-th work and performs, after j-th (j is a natural number satisfying ji) work, j+1-th work different from the j-th work. The robot performs the i+1-th work after the i-th work without changing information concerning correction in a joint of the robot during the i-th work, performs robot calibration after the j-th work, and performs the j+1-th work after performing the robot calibration.

A method to control operation of a robot includes generating at least one virtual image by an optical 3D measurement system and with respect to a 3D measurement coordinate system, the at least one virtual image capturing a surface region of a component. The method further includes converting a plurality of point coordinates of the virtual image into point coordinates with respect to a robot coordinate system by a transformation instruction and controlling a tool element of the robot using the point coordinates with respect to the robot coordinate system so as to implement the operation.

A controlling method of a robot system is provided with highly accurately determination of an origin offset at individual joints, even with a small number of cameras. A controlling unit 08 controls a robot 01 and a camera 04 to perform a photographing step for each of pivotal joints 021, 031 and 051 to acquire photographed data, and subsequently performs computational control. The photographing step assigns predetermined coordinate angles to multiple joints of the robot 01, respectively, to cause the joints to take predetermined positions and orientations, and subsequently causes the camera 04 to photograph a mark 03 during a process of causing the robot 01 to rotate at one of the multiple joints from the predetermined position and orientation. The computational control identifies the joint causing a rotational axis offset among the multiple joints of the robot 01, based on the photographed data acquired by trajectory acquiring control.

A controller is provided for performing dynamic grasping of a target object using visual sensory inputs. The controller includes a robotic interface connected to a robotic arm including links connected by joints having actuators and encoders, and a gripper of the end-effector of the robotic arm configured to grasp the target object in response to robot control signals, and a vision sensor configured to continuously provide visual observations for tracking poses of the target object in a workspace and compute grasp poses, wherein the vision sensor is mounted on a distal end of the robotic arm adjacent to the gripper. The controller trains the Eye-on-Hand reinforcement learner policy, tracks the poses of the target object, and generates robot control signals to follow the target object while keeping it in the field of view of the vision sensor and grasp the target object in the workspace.

A first characteristic portion of a first workpiece and a second characteristic portion of a second workpiece are previously determined. A characteristic amount detection unit detects a first characteristic amount related to the position of the first characteristic portion and a second characteristic amount related to the position of the second characteristic portion in an image captured by a camera. A calculation unit calculates, as a relative position amount, the difference between the first characteristic amount and the second characteristic amount. A command generation unit generates a movement command for operating a robot based on a relative position amount in the image captured by the camera and a relative position amount in a predetermined reference image.

Example methods and systems for determining 3D scene geometry by projecting patterns of light onto a scene are provided. In an example method, a first projector may project a first random texture pattern having a first wavelength and a second projector may project a second random texture pattern having a second wavelength. A computing device may receive sensor data that is indicative of an environment as perceived from a first viewpoint of a first optical sensor and a second viewpoint of a second optical sensor. Based on the received sensor data, the computing device may determine corresponding features between sensor data associated with the first viewpoint and sensor data associated with the second viewpoint. And based on the determined corresponding features, the computing device may determine an output including a virtual representation of the environment that includes depth measurements indicative of distances to at least one object.

A robotic system includes a manipulator assembly and a processing system. The manipulator assembly includes a first manipulator, a second manipulator, and a drivable structure. The first manipulator and the second manipulator are mechanically coupled to the drivable structure. The processing system is configured to determine a drivable structure motion for effecting a commanded motion for a first end effector of a first tool mechanically coupled to the first manipulator. Performing only the drivable structure motion would cause motion of the first end effector simultaneously with motion of a second end effector, the second end effector being of a second tool mechanically coupled to the second manipulator. The processing system is further configured to determine a movement of the second manipulator and the second tool that, when performed simultaneously with the drivable structure motion, would compensate for the motion of the second end effector.

An Ethernet processing unit (310) receives a plurality of payload packets that stores a plurality of divided images obtained by dividing a captured image. The Ethernet processing unit (310) and a packet discard unit (311) select from the plurality of payload packets, a payload packet that stores a divided image subject to image processing.