This technology enables a receipt of a plurality of readings from a sensor monitoring a piece of equipment in a building. The receipt enables an identification of a present fault or a projected fault in the piece of equipment. The identification enables a generation of an augmented reality content related to the present fault or the projected fault. The augmented reality content is sent to a mobile device when the mobile device is in proximity of the piece of equipment.

A system comprises a first robotic arm adapted to support and move a tool and a second robotic arm adapted to support and move a camera. The system also comprises an input device, a display, and a processor. The processor is configured to, in a first mode, command the first robotic arm to move the camera in response to a first input received from the input device to capture an image of the tool and present the image as a displayed image on the display. The processor is configured to, in a second mode, display a synthetic image of the first robotic arm in a boundary area around the captured image on the display, and in response to a second input, change a size of the boundary area relative a size of the displayed image.

An apparatus comprises a memory device and a processor coupled to a display device, an image capture device, and the memory device. The processor is configured to: cause images captured by the image capture device to be displayed in a viewing area on the display device; determine a position of a tool in a reference frame of the image capture device; determine a position to display a non-depictive symbol for the tool in a boundary area circumscribing the viewing area to indicate a direction of the determined position of the tool relative to a field of view of the image capture device, by determining a trajectory of the tool; and cause the non-depictive symbol to be displayed at the determined position in the boundary area while images that were captured by the image capture device are restricted to being displayed in the viewing area.

A robot teaching device includes a display device, a microphone, a voice recognition section configured to perform voice recognition processing on a voice signal and output character information representative of the voice, a correspondence storage configured to store information associating each of a plurality of types of operating screens with a recognition target word according to robot teaching, a recognition target word extracting section configured to extract, from the information stored in the correspondence storage, one or more recognition target words having a predetermined relevance to a word representative of the character information, and a screen list display unit configured to generate, on the basis of the information stored in the correspondence storage, an image showing a list of one or more operating screens corresponding to one or more recognition target words thus extracted, and display the image on the display device.

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

Method for controlling an object, wherein the object is arranged in the real space and is linked to at least one virtual element of a character space, wherein the object, a depiction device, an operator control device and/or a control device are functionally connected to one another, having the steps of ascertaining a position and an orientation of the operator control device and/or of the depiction device in relation to the object, generating a virtual pointer from the ascertained position and the ascertained orientation of the operator control device and/or of the depiction device, selecting the object by means of the virtual pointer, depicting the object, augmented with the at least one element of the character space, by means of the depiction device, and/or controlling the object by means of the operator control device.

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

A synthetic representation of a robot tool for display on a user interface of a robotic system. The synthetic representation may be used to show the position of a view volume of an image capture device with respect to the robot. The synthetic representation may also be used to find a tool that is outside of the field of view, to display range of motion limits for a tool, to remotely communicate information about the robot, and to detect collisions.

An endoscope captures images of a surgical site for display in a viewing area of a monitor. When a tool is outside the viewing area, a GUI indicates the position of the tool by positioning a symbol in a boundary area around the viewing area so as to indicate the tool position. The distance of the out-of-view tool from the viewing area may be indicated by the size, color, brightness, or blinking or oscillation frequency of the symbol. A distance number may also be displayed on the symbol. The orientation of the shaft or end effector of the tool may be indicated by an orientation indicator superimposed over the symbol, or by the orientation of the symbol itself. When the tool is inside the viewing area, but occluded by an object, the GUI superimposes a ghost tool at its current position and orientation over the occluding object.

A remote control manipulator system includes a manipulator controlled remotely by an operator; a camera to capture an image including the manipulator; a posture sensor to detect posture data; an action instruction inputter with which the operator inputs an action instruction instructing action to move or stop the manipulator; a control device including a structural data storage to store manipulator structural data representing a structure of the manipulator, a model image generator to generate a model image with referring to the structural data storage and the posture data, and a presentation image generator to generate a presentation image by superimposing a model image on the captured image; and a display to display the presentation image.