A remote controlled, tele-operated welder includes a multi-axis robot arm, video cameras, sensors a specialized control station that allows an operator to perform flood-fill welding operations at a remote location to avoid the heat, smoke and other environmental effects produced through typical flood-welding operations. The operator accesses the control unit (OCU) using a GUI and mouse, keyboard, joystick, or other custom controls, and observe the piece via the cameras (visual, thermal, or other) placed in the welding station via a feed displayed on the OCU display(s). Audio, video, and/or tactile feedback may be provided to indicate arm, welder, or other system status, for collision warning and arm motion singularity avoidance. Augmented reality informational graphic/textual overlays may provide guidance to an operator, and the apparatus may further include the ability to repeat series of steps needed to handle flood-weld on a given piece, repeatedly across many pieces.

A method and system for calibration of an augmented reality (AR) device's position and orientation based on a robot's positional configuration. A conventional visual calibration target is not required for AR device calibration. Instead, the robot itself, in any pose, is used as a three dimensional (3D) calibration target. The AR system is provided with a CAD model of the entire robot to use as a reference frame, and 3D models of the individual robot arms are combined into a single object model based on joint positions known from the robot controller. The 3D surface model of the entire robot in the current pose is then used for visual calibration of the AR system by analyzing images from the AR device camera in comparison to the surface model of the robot in the current pose. The technique is applicable to initial AR device calibration and to ongoing device tracking.

A real-time control system for industrial robots based on virtual reality (VR), including: a VR headset, a VR controller, a model construction module, a motion control module, a data communication system and a robot controller. Through the VR controller, a user can touch and drag an end of the gripper to move the robot in the Cyber environment. When the gripper moves into a bounding box of a control point, the data communication system is triggered, and a communication code corresponding to the control point is sent to the robot controller via a Unity engine script. After receiving the code, the robot controller sends the corresponding motion command such that the robot such that the robot moves in real time according to the control data.

A method and a control arrangement for determining a relation R.Math.MA between a robot coordinate system of a robot and an MA coordinate system of a moveable apparatus, the movable apparatus including a sensor device and a localization mechanism configured to localize a sensor coordinate system of the sensor device in the MA coordinate system, wherein a marker is arranged in a fixed relation with a reference location on the robot. The method includes positioning the marker in a plurality of different poses in relation to the robot coordinate system. For each pose of the plurality of different poses, the method includes: determining, on the basis of sensor information, a relation C.Math.M between the sensor coordinate system and a marker coordinate system; determining a relation MA.Math.C between the MA coordinate system and the sensor coordinate system; determining a relation R.Math.E between the robot coordinate system and a reference location coordinate system. The method also includes determining the relation R.Math.MA using the relation C.Math.M, the relation MA.Math.C, and the relation R.Math.E, in the plurality of different poses.

A robot system according to the present invention comprises a control apparatus for controlling a robot, and a video display apparatus connected to the control apparatus. The video display apparatus comprises a display unit which displays an image of a real space containing the robot, in real time as the image is taken by a camera, and an augmented reality image processing unit which causes a virtual image of an end effector or robot peripheral equipment of the robot to be displayed on the display unit in superimposed fashion on a real image of the robot taken by the camera. According to the robot system, even when the end effector or the robot peripheral equipment is not present, a robot teaching task can be performed by assuming that they are present.

The present invention relates to a robot teaching system based on image segmentation and surface electromyography and robot teaching method thereof, comprising a RGB-D camera, a surface electromyography sensor, a robot and a computer, wherein the RGB-D camera collects video information of robot teaching scenes and sends to the computer; the surface electromyography sensor acquires surface electromyography signals and inertial acceleration signals of the robot teacher, and sends to the computer; the computer recognizes a articulated arm and a human joint, detects a contact position between the articulated arm and the human joint, and further calculates strength and direction of forces rendered from a human contact position after the human joint contacts the articulated arm, and sends a signal controlling the contacted articulated arm to move along with such a strength and direction of forces and robot teaching is done.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a system for decentralized and validated robotic planning. One of the methods includes obtaining data representing an optimization challenge for a task to be performed by one or more robots in a robotic operating environment, wherein the optimization challenge has one or more associated goal criteria for the task; providing, by the validation platform system to a development platform system operated by a different entity than the validation platform system, information related to the optimization challenge; obtaining a candidate robotic control plan; executing the candidate robotic control plan using the digital representation of the robotic operating environment; determining that the candidate robotic control plan is valid according to the one or more goal criteria; and in response, providing the valid robotic control plan for deployment in the robotic operating environment.

A sensing system with a detecting device that is used to detect a position of a target and a controller, where, for display on a display device or projection by a projection apparatus, the controller creates an augmented-reality image that shows: at least one of a setting related to detection of the target using the detecting device, a setting of a moving apparatus, and a setting of a robot that performs work on the target, a position of the target being recognized by the controller, a result of the detection of the target, a work plan of the moving apparatus, a work plan of the robot, a determination result of the controller and a parameter related to the target.

A robot controller configured to assist an operation of a user, by effectively utilizing both techniques of augmented reality and mixed reality. The robot controller includes: a display device configured to display information generated by a computer so that the information is overlapped with an actual environment, etc.; a position and orientation obtaining section configured to obtain relative position and orientation between the display device and a robot included in the actual environment; a display controlling section configured to display a virtual model of the robot, etc., on the display device; an operation controlling section configured to operate the virtual model displayed on the display device; and a position and orientation determining section configured to determine the position and/or orientation of the robot by using the position and/or orientation of the operated virtual model and using the relative position and orientation between the robot and the display device.

A system has a virtual-world (VW) controller and a physical-world (PW) controller. The pairing of a PW element with a VW element establishes them as corresponding physical and virtual twins. The VW controller and/or the PW controller receives measurements from one or more sensors characterizing aspects of the physical world, the VW controller generates the virtual twin, and the VW controller and/or the PW controller generates commands for one or more actuators affecting aspects of the physical world. To coordinate the corresponding virtual and physical twins, (i) the VW controller controls the virtual twin based on the physical twin or (ii) the PW controller controls the physical twin based on the virtual twin. Depending on the operating mode, one of the VW and PW controllers is a master controller, and the other is a slave controller, where the virtual and physical twins are both controlled based on one of VW or PW forces.