A digital twin modeling method to assemble a robotic teleoperation environment, including: capturing images of the teleoperation environment; identifying a part being assembled; querying the assembly assembling order to obtain a list of assembled parts according to the part being assembled; generating a three-dimensional model of the current assembly from the list and calculating position pose information of the current assembly in an image acquisition device coordinate system; loading a three-dimensional model of the robot, determining a coordinate transformation relationship between a robot coordinate system and an image acquisition device coordinate system; determining position pose information of the robot in an image acquisition device coordinate system from the coordinate transformation relationship; determining a relative positional relationship between the current assembly and the robot from position pose information of the current assembly and the robot in an image acquisition device coordinate system; establishing a digital twin model of the teleoperation environment.

A method for picking objects is specified, in which at least one object is removed from a source loading aid and placed into a target loading aid using a robot. After the operation of removing the object, a first sensor system of the robot is used to check whether at least one object is held by the robot. A number and/or a type of the at least one removed object is ascertained using the second sensor system. The operation of placing the at least one object into the target loading aid is aborted or modified if no object is held by the robot or the number and/or the type of the at least one removed object does not contribute to completing the picking order, which defines a desired number and/or desired type of objects in the target loading aid. Furthermore, a device and a computer program product for performing the presented method is specified.

There are provided systems, methods, and apparatus, for optimizing a policy controller to control a robotic agent that interacts with an environment to perform a robotic task. One of the methods includes optimizing the policy controller using a neural network that generates numeric embeddings of images of the environment and a demonstration sequence of demonstration images of another agent performing a version of the robotic task.

A binding device communicable with an information processing device, the binding device including: a binding machine that binds reinforcing bars with a wire; and a transfer robot that moves the binding machine to a binding position by a relative movement between the binding machine and the reinforcing bars. The binding machine includes: an information acquisition unit configured to acquire binding related information related to an operation of binding the reinforcing bars with the wire, and an information communication unit configured to notify the information processing device of the binding related information acquired by the information acquisition unit.

A robot is provided. The robot includes a microphone, a camera, a communication interface including a circuit, a memory storing at least one instruction, and a processor, wherein the processor is configured to acquire a user voice through the microphone, identify a task corresponding to the user voice, determine whether the robot can perform the identified task, and control the communication interface to transmit information on the identified task to an external robot based on the determination result.

A method for characterizing a state of an end effector of an ultrasonic device is disclosed. The ultrasonic device including an electromechanical ultrasonic system defined by a predetermined resonant frequency. The electromechanical ultrasonic system further including an ultrasonic transducer coupled to an ultrasonic blade. The method including applying, by an energy source, a power level to the ultrasonic transducer; measuring, by a control circuit coupled to a memory, an impedance value of the ultrasonic transducer; comparing, by the control circuit, the impedance value to a reference impedance value stored in the memory; classifying, by the control circuit, the impedance value based on the comparison; characterizing, by the control circuit, the state of the electromechanical ultrasonic system based on the classification of the impedance value; and adjusting, by the control circuit, the power level applied to the ultrasonic transducer based on the characterization of the state of the end effector.

A robot system includes: a robot including an end effector connected to an arm thereof; a vision sensor mounted to the robot; and a controller configured to output an operation signal that enables the robot to operate when an input is generated through a touch screen. Each of an object and a target to which the object is placed is inputted through the touch screen. The touch screen displays a recommendation region of the target in a distinguished manner.

A method for detecting boxes includes receiving a plurality of image frame pairs for an area of interest including at least one target box. Each image frame pair includes a monocular image frame and a respective depth image frame. For each image frame pair, the method includes determining corners for a rectangle associated with the at least one target box within the respective monocular image frame. Based on the determined corners, the method includes the following: performing edge detection and determining faces within the respective monocular image frame; and extracting planes corresponding to the at least one target box from the respective depth image frame. The method includes matching the determined faces to the extracted planes and generating a box estimation based on the determined corners, the performed edge detection, and the matched faces of the at least one target box.

Systems and methods for AI assisted reconfigurable, fixtureless manufacturing is disclosed. The invention eliminates geometry-setting tools (hard points, pins and nets—traditionally known as 3-2-1 fixturing schemes) and to replace the physical geometry setting with virtual datums driven by learning AI algorithms. A first type of part and a second type of part may be located by a machine vision system and moved by material handling devices and robots to locations within an assembly area. The parts may be aligned with one another and the alignment may be checked by the machine vision system which is configured to locate datums, in the form of features, of the parts and compare such datums to stored virtual datums. The parts may be joined while being held by the material handling devices or robots to form a subassembly in a fixtureless fashion. The material handling devices are able to grasp a number of different types of parts so that a number of different types of subassemblies are capable of being assembled. The system enables one skilled in the art to develop a product design with self-locating parts that will eliminate and minimize the need for geometry setting dedicated line tools and fixtures. This leads to the development of a manufacturing process that utilizes the industry 4.0 technologies to once again eliminate or significantly reduces the need for geometry setting line tools.

Techniques of automated framing for use in the construction of building structures are described. Examples of such structures includes walls, wall panels, roofs, and the like. In one scenario, a robotic automated framing system assists with construction of a building structure. The robotic automated framing system can analyze an architectural plan and determine a project, based at least in part, on the architectural plan. The robotic automated framing system can also schedule a robot to perform the project, and cause the robot to perform at least some of the project.