A method includes determining a voxel grid representation of occupied voxels of an environment of a robotic device based on sensor data from a depth sensor on the robotic device. The method further includes assigning a plurality of occupied voxels from the voxel grid representation to a surface within the environment. The method additionally includes determining an image to project onto the surface with a projector on the robotic device. The method further includes modifying the image to fit the surface within the environment based on the plurality of occupied voxels assigned to the surface. The method also includes causing the projector coupled to the robotic device to project the modified image onto the surface in the environment.

Implementations described herein relate to training and refining robotic control policies using imitation learning techniques. A robotic control policy can be initially trained based on human demonstrations of various robotic tasks. Further, the robotic control policy can be refined based on human interventions while a robot is performing a robotic task. In some implementations, the robotic control policy may determine whether the robot will fail in performance of the robotic task, and prompt a human to intervene in performance of the robotic task. In additional or alternative implementations, a representation of the sequence of actions can be visually rendered for presentation to the human can proactively intervene in performance of the robotic task.

A system for teaching a robot includes a wearable device having a plurality of sensors for sensing signals representing at least one movement, orientation, position, force, and torque of any part of a user's body applied on a target. The system further includes a processing device configured to receive the sensed signals, the processing device further configured to store the sensed signals defining as data of teaching commands, a controller for controlling a robot receives the data of teaching commands and operates the robot according to the received data of teaching commands, an object sensor to detect position and orientation of an object as a workpiece of the robot, and a visual input device communicatively coupled to at least one of the processing device and the controller.

A robot control device 10 is provided with: a reading device 40 that reads a marker 21 attached to a robot 20 and markers 31 attached to individual objects 30; and a CPU 111 that performs image analysis concerning the individual positions of the robot 20 and the individual objects 30 in real space based on information read from the markers 21, 31, and that simulates the operation of the robot 20 while disposing three-dimensional shape models of the robot 20 and the individual objects 30 in a virtual space based on information indicating the positions of the robot 20 and the individual objects 30 in real space, information concerning the three-dimensional shape model of the robot 20, associated with the marker 21, and information concerning the three-dimensional shape models of the individual objects 30, associated with the individual markers 31.

There is provided a teaching apparatus that teaches a robot, in particular an industrial robot, a task by direct teaching. The teaching apparatus includes a wearable display that displays information in a worker's visual field, an acquisition section that acquires position and posture data capable of specifying a position and a posture of an industrial robot, and a production section that produces information to be displayed on the wearable display. When a worker operates the robot using the teaching apparatus, the apparatus produces state information that indicates at least one of a position and a posture of the robot relative to a reference object located around the robot, based on position and posture data as acquired and displays the produced information on the wearable display. There is also provided a teaching method having teaching functions equivalent to those of the apparatus.

This invention is directed to a plant management system that presents a position to be measured by a sensor that performs measurement for inspection of a plant or the like when the measurement is executed, thereby preventing a measurement error. The plant management system includes a measurement position instructor that, when performing the measurement by the sensor on a work site, instructs the position to be measured by the sensor on a measurement target, and a presenter that, when the measurement is executed, performs presentation of an image representing the position to be measured by the sensor on the work site by superimposing the image on an image of the measurement target or projecting the image onto the measurement target. Here, the presenter includes a superimposition display that superimposes and displays the image representing the position to be measured by the sensor on the image of the measurement target using an augmented reality technology or a projection device that projects the image representing the position to be measured by the sensor in association with the measurement target.

A method includes determining a movement of an industrial robot in a manufacturing environment from a first position to a second position. The method also includes displaying an image showing a trajectory of the movement of the robot on a wearable headset. The displaying of the image comprises at least one of: displaying an augmented reality (AR) graphical image or video of the trajectory superimposed on a real-time actual image of the robot, or displaying a virtual reality (VR) graphical image or video showing a graphical representation of the robot together with the trajectory.

A system and method for authoring and performing Human-Robot-Collaborative (HRC) tasks is disclosed. The system and method adopt an embodied authoring approach in Augmented Reality (AR), for spatially editing the actions and programming the robots through demonstrative role-playing. The system and method utilize an intuitive workflow that externalizes user's authoring as demonstrative and editable AR ghost, allowing for spatially situated visual referencing, realistic animated simulation, and collaborative action guidance. The system and method utilize a dynamic time warping (DTW) based collaboration model which takes the real-time captured motion as inputs, maps it to the previously authored human actions, and outputs the corresponding robot actions to achieve adaptive collaboration.

Various methods and systems are provided for puppeteering in augmented reality. Generally, an augmented or virtual reality device for each user generates a virtual 3D environment comprising a virtual representation of a physical room and a 3D asset. An author can record a 3D path for a puppeteering animation of the 3D asset using a 3D interface. At the same time, a coordinated rendering of a corresponding 3D image moving along the 3D path is updated among author devices substantially in real-time. Distinct states of the 3D asset can be assigned to different portions of the 3D path, and authors can set behavior parameters to assign template behaviors such as obstacle avoidance, particle effects, path visualizations and physical effects. The behavior parameters are distributed among presenter and audience devices, and a coordinated rendering of an animation of the 3D image corresponding to the puppeteering animation is triggered.

A simulation device includes an image sensor which captures an image of a real space including an actual robot and a peripheral device arranged at the periphery of the actual robot, an augmented reality display section which displays a virtual robot overlaid on the actual robot shown in the captured image, a workpiece management section which manages a position of a moving workpiece, and a motion control section which controls a motion of the virtual robot on the basis of the position of the workpiece.