A smart container yard includes systems for intelligently controlling operations of vehicles in the container yard using teleoperation and/or autonomous operations. A remote support server controls remote support sessions associated with vehicles in the container yard to provide teleoperation support for loading and unloading operations. Aerial drones may be utilized to maintain positions above a teleoperated vehicle and act as signal re-transmitters. An augmented reality view may be provided at a teleoperator workstation to enable a teleoperator to control vehicle operations in the smart container yard.

The system, devices and methods herein enable autonomous and tele-operation of tele-operated robots for maintenance of a property around known and unknown obstacles. A method may include using an unmanned aerial vehicle for obtaining additional data relating to the property and obstacles within the property and plan a path around the obstacles using data from sensors on-board the tele-operated robot and the aerial image. A method may also provide optimization of total time needed for performing the property maintenance and the labor costs in situations where manual intervention is needed for navigating the tele-operated robot around obstacles on the property or for removing obstacles on the property.

A surgical robotic system includes: a surgical console having a display and a user input device configured to generate a user input and a surgical robotic arm having a surgical instrument configured to treat tissue and being actuatable in response to the user input; and a video camera configured to capture video data that is displayed on the display. The system also includes a control tower coupled to the surgical console and the surgical robotic arm. The control tower is configured to: process the user input to control the surgical instrument and to record the user input as input data; train a machine learning system using the input data and the video data; and execute the at least one machine learning system to determine probability of failure of the surgical instrument.

A robot system includes a robot installed in a work area and controlled by a second control device, a 3D camera operated by an operator, a sensor that is disposed in a manipulation area that is a space different from the work area, and wirelessly detects position information and posture information on the 3D camera, a display, and a first control device. The first control device acquires image information on a workpiece imaged by the 3D camera, acquires, from the sensor, the position information and the posture information when the workpiece is imaged by the 3D camera, displays the acquired image information on the display, forms a three-dimensional model of the workpiece based on the image information, and the acquired position information and posture information, displays the formed three-dimensional model on the display, and outputs first data that is data of the formed three-dimensional model to the second control device.

Methods and systems to remotely operate robotic devices are provided. A number of embodiments allow users to remotely operate robotic devices using generalized consumer devices (e.g., cell phones). Additional embodiments provide for a platform to allow communication between consumer devices and the robotic devices. Further embodiments allow for training robotic devices to operate autonomously by training the robotic device with machine learning algorithms using data collected from scalable methods of controlling robotic devices.

A digital twin modeling method to assemble a robotic teleoperation environment, including: capturing images of the teleoperation environment; identifying a part being assembled; querying the assembly assembling order to obtain a list of assembled parts according to the part being assembled; generating a three-dimensional model of the current assembly from the list and calculating position pose information of the current assembly in an image acquisition device coordinate system; loading a three-dimensional model of the robot, determining a coordinate transformation relationship between a robot coordinate system and an image acquisition device coordinate system; determining position pose information of the robot in an image acquisition device coordinate system from the coordinate transformation relationship; determining a relative positional relationship between the current assembly and the robot from position pose information of the current assembly and the robot in an image acquisition device coordinate system; establishing a digital twin model of the teleoperation environment.

A computer-assisted teleoperated system includes a pre-load assembly in an instrument manipulator that is under the control of a controller. The controller can automatically cause the preload assembly to engage and disengage a preload. A surgical apparatus includes an instrument manipulator assembly and a sterile adapter assembly. The sterile adapter assembly is mounted in the distal face of the instrument manipulator assembly. When the preload assembly configures the instrument manipulator assembly to apply a preload force on the sterile adapter assembly, the sterile adapter assembly is removable from the distal face of the instrument manipulator. The sterile adapter assembly includes a mechanical sterile adapter assembly removal lockout and a mechanical instrument removal lockout.

A teleoperational medical system includes a teleoperational control system, a teleoperational manipulator, an operator controller. The control system includes a processing unit configured to: determine a position and orientation of a marker on an operator of the operator controller; determine whether a head portion of the operator is directed toward or away from a display region of a display; based on a determination that the head portion is directed toward the display region, initiate an operator following mode in which a movement of the operator controller provides a corresponding movement to the teleoperational manipulator; and based on a determination that the head portion is directed away, suspend the operator following mode. The determination that the head portion is directed away from the display region of the display device includes determining whether the head portion of the operator is directed away from the display region for a threshold time period.

Embodiments of the present disclosure provide a system, method, apparatus and computer-readable medium for teleoperation. An exemplary system includes a robot machine having a machine body, at least one sensor, at least one robot processor, and at least one user processor operable to maintain a user simulation model of the robot machine and the environment surrounding the robot machine, the at least one user processor being remote from the robot machine. The system further includes at least one user interface comprising a haptic user interface operable to receive user commands and to transmit the user commands to the user simulation model, a display operable to display a virtual representation of the user simulation model.

Systems and methods for collision detection and avoidance are provided. In one aspect, a robotic medical system including a first set of links, a second set of links, a console configured to receive input commanding motion of the first set of links and the second set of links, a processor, and at least one computer-readable memory in communication with the processor. The processor is configured to access the model of the first set of links and the second set of links, control movement of the first set of links and the second set of links based on the input received by the console, determine a distance between the first set of links and the second set of links based on the model, and prevent a collision between the first set of links and the second set of links based on the determined distance.