A control apparatus that is connectable to a memory that stores computer-readable instructions, comprising a processor that is configured to execute the computer-readable instructions so as to control driving of a robot and a camera, create setting details on an image processing sequence containing image processing of a captured image captured by the camera, and setting details on a calibration of correlating a coordinate system of the robot and a coordinate system of the camera, and call up the image processing sequence in creation of the setting details on the calibration.

A control apparatus, a robot and a robot system to which settings for calibration can be easily and appropriately made are to be provided.

A control apparatus that can control driving of a robot, an imaging unit and a display unit based on input of an input unit, includes a display control unit that allows the display unit to display an input window for inputting a robot as an object to be controlled, and allows the display unit to display an imaging unit input part that guides input of an attachment position of the imaging unit corresponding to the input robot, and a calibration control unit that performs a calibration of correlating a coordinate system of the robot and a coordinate system of the imaging unit based on the input attachment position of the imaging unit.

A control device, which controls a robot having a movable unit including a plurality of arms, includes a processor that performs calibration between a coordinate system of an imaging unit disposed in an arm different from an arm positioned on a most distal side of the movable unit and a coordinate system of the robot. The processor performs the calibration, based on a captured image obtained by causing the imaging unit to image a marker.

Described are machine vision systems and methods for simultaneous kinematic and hand-eye calibration. A machine vision system includes a robot or motion stage and a camera in communication with a control system. The control system is configured to move the robot or motion stage to poses, and for each pose: capture an image of calibration target features and robot joint angles or motion stage encoder counts. The control system is configured to obtain initial values for robot or motion stage calibration parameters, and determine initial values for hand-eye calibration parameters based on the initial values for the robot or motion stage calibration parameters, the image, and joint angles or encoder counts. The control system is configured to determine final values for the hand-eye calibration parameters and robot or motion stage calibration parameters by refining the hand-eye calibration parameters and robot or motion stage calibration parameters to minimize a cost function.

A method is used for calibrating a robot camera and external camera system relative to a medical robot. The robot camera is guided on an arm. The camera system has an external camera. The method includes: moving the robot camera via the robot arm during sensing and capturing; detecting a pose of a calibration pattern and/or an external tracker, each having a transformation to a pose of the external camera, and/or detecting a pose of the external camera; determining a transformation between the robot camera and external camera, and determining a field of view; moving a flange into at least three poses in the field of view and sensing the at least three poses via the external camera, and simultaneously sensing a transformation between the robot base and the flange; and performing a hand-eye calibration. The method can be used with a surgical assistance system and computer-readable storage medium.

A robot system includes a robot, a vision sensor, and a controller. The vision sensor is configured to be detachably attached to the robot. The controller is configured to measure a reference object by using the vision sensor and calibrate a relative relationship between a sensor portion of the vision sensor and an engagement portion of the vision sensor, and teach the robot by referring to the relative relationship and by using the vision sensor, after the vision sensor is attached to the robot.

A method is performed by a computing system. The method includes receiving image data from an imaging device, and determining, using the image data, a plurality of image-space tools, each image-space tool associated with a tool of a plurality of tools, each tool controlled by a manipulator of a plurality of manipulators. The method further includes determining a first correspondence between a first image-space tool of the plurality of image-space tools and a first tool of the plurality of tools based on a first disambiguation setting associated with the first tool.

A method is performed by a computing system. The method includes receiving image data from an imaging device, and determining, using the image data, a plurality of image-space tools, each image-space tool associated with a tool of a plurality of tools, each tool controlled by a manipulator of a plurality of manipulators. The method further includes determining a first correspondence between a first image-space tool of the plurality of image-space tools and a first tool of the plurality of tools based on a first disambiguation setting associated with the first tool.

A robot system includes a robot, a vision sensor, and a controller. The vision sensor is configured to be detachably attached to the robot. The controller is configured to measure a reference object by using the vision sensor and calibrate a relative relationship between a sensor portion of the vision sensor and an engagement portion of the vision sensor, and teach the robot by referring to the relative relationship and by using the vision sensor, after the vision sensor is attached to the robot.

Robot positioning is facilitated by obtaining, for each time of a first sampling schedule, a respective indication of a pose of a camera system of a robot relative to a reference coordinate frame, the respective indication of the pose of the camera system being based on a comparison of multiple three-dimensional images of a scene of an environment, the obtaining providing a plurality of indications of poses of the camera system; obtaining, for each time of a second sampling schedule, a respective indication of a pose of the robot, the obtaining providing a plurality of indications of poses of the robot; and determining, using the plurality of indications of poses of the camera system and the plurality of indications of poses of the robot, an indication of the reference coordinate frame and an indication of a reference point of the camera system relative to pose of the robot.