Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing interpolated robot control parameters. One of the methods includes receiving, by a real-time bridge from a control agent for a robot, a non-real-time command for the robot, wherein the non-real-time command specifies a trajectory to be attained by a component of the robot and a target value for a control parameter, wherein the control parameter controls how a real-time controller will cause the robot to react to one or more external stimuli encountered during a control cycle of the real-time controller. The real-time bridge provides the one or more real-time commands translated from the non-real-time command and interpolated control parameter information to the real-time controller, thereby causing the robot to effectuate the trajectory of the non-real-time command according to the interpolated control parameter information.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for integrating sensor streams for robotic demonstration learning. One of the methods includes selecting, by a learning system for a robot, a base update rate for combining multiple sensor streams into a task state representation. The learning system repeatedly generates the task state representation at the base update rate, including combining, during each time period defined by the update rate, the task state representation from most recently updated sensor data processed by the plurality of neural networks. The learning system repeatedly uses the task state representations to generate commands for the robot at the base update rate.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for computing interpolated robot control parameters. One of the methods includes receiving, by a real-time bridge from a control agent for a robot, a non-real-time command for the robot, wherein the non-real-time command specifies a trajectory to be attained by a component of the robot and a target value for a control parameter, wherein the control parameter controls how a real-time controller will cause the robot to react to one or more external stimuli encountered during a control cycle of the real-time controller. The real-time bridge provides the one or more real-time commands translated from the non-real-time command and interpolated control parameter information to the real-time controller, thereby causing the robot to effectuate the trajectory of the non-real-time command according to the interpolated control parameter information.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for integrating sensor streams for robotic demonstration learning. One of the methods includes selecting, by a learning system for a robot, a base update rate for combining multiple sensor streams into a task state representation. The learning system repeatedly generates the task state representation at the base update rate, including combining, during each time period defined by the update rate, the task state representation from most recently updated sensor data processed by the plurality of neural networks. The learning system repeatedly uses the task state representations to generate commands for the robot at the base update rate.

A robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.

A robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.

A robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.

A method for controlling a robotic system includes determining a location and/or a pose of a power system component based on data received from one or more sensors, and determining a mapping of a location of a robotic system within a model of an external environment of the robotic system based on the data. The model of the external environment provides locations of objects external to the robotic system. A sequence of movements of components of the robotic system is determined to perform maintenance on the power system component based on the locations of the objects external to the robotic system and/or the location or pose of the power system component. One or more control signals are communicated to remotely control movement of the components of the robotic system based on the sequence or movements of the components to perform maintenance on the power system component.

A robotic system includes a controller configured to obtain image data from one or more optical sensors and to determine one or more of a location and/or pose of a vehicle component based on the image data. The controller also is configured to determine a model of an external environment of the robotic system based on the image data and to determine tasks to be performed by components of the robotic system to perform maintenance on the vehicle component. The controller also is configured to assign the tasks to the components of the robotic system and to communicate control signals to the components of the robotic system to autonomously control the robotic system to perform the maintenance on the vehicle component.

A robot system and method are provided that move an articulable arm relative to a target object. Perception information corresponding to a position of the arm relative to the target object is acquired. Movement of the arm is controlled based on the perception information. After movement of the arm, predicted position information representative of a predicted positioning of the arm is provided using the perception information and control signal information. The arm is subsequently controlled using the predicted position information.