Robot vision super visor for hybrid homing, positioning and workspace UFO detection enabling industrial robot use for consumer applications

Abstract

A robot vision supervisor system for use with industrial robots being placed with average home consumers or at restaurants that is capable of guiding the robot to its homing position for initial homing or when a loss of reference has occurred. Further the robot vision supervisor is able to detect presence of unidentified foreign objects in the workspace and guide the robot to navigate around and prevent collisions. Image processing algorithms are used with primary, orthogonal and cross camera arrays to increase reliability and accuracy of the robot vision supervisor. Lookup tables of images are captured and stored by controller for discretized positions in the full robot workspace when robot is in known good calibration to compare with images during regular operation for detecting anomalies.

Claims

1. A robot vision supervisor comprising: a robot with an end effector moving along a plurality of axes with reference to a frame; a robot controller directing said end effector to commanded positions with reference to said frame; a plurality of cameras attached to said reference frame; at least one of said plurality of cameras viewing the said end effector and said robot controller computing current position of said end effector along one of said plurality of axes; wherein, said robot controller compares the said current position determined by said camera to said commanded position by said robot controller to detect any loss of referencing; wherein, the said cameras use image processing to extract boundaries of said robot workspace and compare with the boundaries of the said robot moving parts and said end effector to prevent collisions and guide said robot to its home position.

2. A robot vision supervisor as in claim 1, wherein the said cameras are able to detect any unidentified foreign objects other than known objects in the workspace to prevent collisions with the robot moving parts and notify the said controller of their presence.

3. A robot vision supervisor as in claim 1, wherein the said cameras are able to compare position of said end effector against reference features on said frame.

4. A robot vision supervisor as in claim 1, wherein the said cameras are able to compare position of said end effector against features on said frame.

5. A robot vision supervisor as in claim 1, wherein the said cameras are arranged in arrays and employ image stitching algorithms to improve positioning accuracy by joining the viewed images to that of adjacent said cameras.

6. A robot vision supervisor as in claim 1, wherein an image lookup table is generated against all position and orientation combinations conducted when said robot controller is known to work accurately to use for comparison when the said robot is in regular operation later.

7. A robot vision supervisor as in claim 1, further comprising: a plurality of primary cameras; a plurality of orthogonal cameras viewing in a perpendicular direction of said primary cameras.

8. A robot vision supervisor as in claim 5, further comprising: a plurality of cross cameras viewing in a mutually perpendicular direction of said primary cameras and said orthogonal cameras.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

(1) The following is a description, by way of example only, of different embodiments of the mechanism, its variations, derivations and reductions.

(2) FIG. 1 shows a robot vision supervisor watching over a 4-axis robot.

(3) FIG. 2 shows a robot end effector crashing into a known workspace object while trying to move after losing its sense of position relative to known reference frame.

(4) FIG. 3 shows a view of this crash as seen by one of the cameras of the consumer robot vision supervisor.

(5) FIG. 4 shows the end effector of a 4 axis robot in its home position.

(6) FIG. 5 shows a view of the end effector of the robot as seen by a camera of the home robot vision supervisor.

(7) FIG. 6 shows an unidentified foreign object or UFO appearing in the robot workspace which was not expected by the workspace state based robot control system.

(8) FIG. 7 shows the view of a crash as seen by one of the cameras of the robot vision supervisor when the robot was set in a motion by the controller above the known workspace object but hit the UFO.

(9) FIG. 8 shows the robot vision supervisor using the reference features on the frame to compute the absolute positions of the end effector in the plane normal to the viewing direction of camera.

(10) FIG. 9 shows a view of the end effector being closer to the camera and using this relative size comparison to compute end effector relative position along the viewing direction of the camera.

(11) FIG. 10 shows a side view of the robot vision supervisor with an orthogonal camera to compute the relative difference in the viewing direction of the first camera utilizing a corner feature on the reference frame.

(12) FIG. 11 shows an alternate side view as seen from the opposite viewing direction to that in FIG. 10.

DETAILED DESCRIPTION OF THE INVENTION

(13) Now referring to the drawings, wherein like numerals designate like components, FIG. 1 shows a robot vision supervisor watching over a 4-axis robot by way of example. A frame 1 is a stationary member and is the reference for all motion and positioning coordinates. A first degree of freedom along y-axis defined as back to front is achieved by a first carriage guide 3 moving along a first guide rail 2, a second degree of freedom along x-axis defined as going from left to right is achieved by a second carriage guide 5 moving along a second guide rail 4 fixed to first carriage guide 3, a third degree of freedom long z-axis defined as pointing up is achieved by a guide rail 7 moving across a carriage guide 6 fixed to second carriage guide 5. A fourth degree of freedom along a tilt axis is achieved by fixing a rotational joint 8 on third guide rail 7 and mounting an end effector 9 to rotational joint 8. The end effector 9 is able to translate along x, y and z axes and tilt along the axis of the rotational joint 8. A known workspace object 10 sits at the bottom of the robot workspace.

(14) A robot vision supervisor comprised of one or plural primary direction cameras 13, 14, 15 fixed to frame 1, viewing along x-axis. One or plural orthogonal direction cameras 15, 16, 17, 18 mounted on frame 1 viewing in a direction perpendicular to primary direction cameras along y-axis and one of plural third cross cameras 20, 21, 22 mounted on the roof, mounting structure not shown for clarity, also fixed with reference to frame 1 viewing in a mutually perpendicular direction to viewing directions or primary and orthogonal direction cameras. Robot vision supervisor is also assisted by one or plural fixed primary reference marks 23, 24, 25 on frame 1, one or plural secodary reference marks 26, 27, 28 on frame 1, a cutout feature 29 and a corner feature 30 on frame 1.

(15) The 4-axis robot is operated using a robot motion controller directing it to go to different positions and orientations as needed to perform its tasks. Each axis may be comprised of motion transmission elements including motors, encoders, belts, pulleys, clutches as is known in the art. When the robot is first started a homing sequence is run where each robot axis is manually guided to a home position where a sensor trips letting the controller know it has reached a designated origin or home position. As seen in FIG. 1 y-home optical sensor 11 is tripped when guide rail 4 moves into a position that blocks an optical beam from one of its ends to reach a photo sensor on the other side. A second x-home optical sensor 12 is mounted on second guide rail 4 and is tripped when guide rail 7 moves to home and trips it. The home sensors can be optical or mechanical limit switches or like. The home sensors for the z-axis and tilt-axis have not been shown for clarity. Once the robot is homed the controller asks it to go to other positions relative to this home position fixed with reference to frame 1. For open loop control the controller remembers all the requests it made and computes the current position of the end effector based on assumption that all past motion was achieved accurately and successfully. Any errors unknown to the controller will keep compounding resulting in desired positions be other than what is requested or commanded by robot. In case of closed loop controllers encoders are used which may also measure any resulting errors and keep adjusting the actual position but not able to adjust for belt slips, backlashes, deflections and others that do not impact at the encoders.

(16) Once a robot is homed it can successfully move around the workspace doing tasks. It can also go around known workspace objects or obstacles as the programmers would be aware of such obstacles and program the motion path around it. This is based on the condition that the robot is sincerely following all the motion commands and no errors or drifts are building up. Sometimes an event such as a power failure can erase the current positions in controller memory as can be seen in FIG. 2 when the robot has drifted or lost its current positions and needs to be homed but runs into a known workspace object 10 while approaching home position. In such a case a technician needs to be present to guide the robot to its home and repeat the homing procedure when a robot vision supervisor is not used. The robot vision supervisor can predict this collision as seen in FIG. 3 viewed by orthogonal camera 18 by extracting a silhouette of the known workspace object 10 and a silhouette of the end effector 9 and guide the end effector 9 such that the workspace object silhouettes and the robot silhouettes do not meet.

(17) FIG. 4 shows the robot in home position where the orthogonal camera 16 can compare its orientation by extracting boundary features of end effector 9 and comparing to reference features 23 and 26. The robot vision supervisor can guide the end effector in a position by minimizing the errors using the view as seen by camera 16 with image processing and bring it close to the homing position where a fine homing can be done using the traditional homing sensors. This hybrid vision and traditional homing eliminates the need for a technician to be present for manual observance and guidance. Based on the positioning accuracy requirements the traditional homing sensors can be ignored or eliminated from the system.

(18) FIG. 6. shows a robot that has not lost its position reference however ran into an unidentified foreign object or UFO which was left by an user but unknown to the controller. The robot vision supervisor camera 16 can predict this as seen in FIG. 7 by image processing and identifying workspace object boundaries. This unexpected object boundary can allow the controller to raise an alarm to notify about an UFO in the workspace.

(19) As can be seen in camera viewed images FIG. 8 and FIG. 9 a single primary camera can successful compute all 3 positions and orientation. As seen in FIG. 8 two orthogonal positions axes to the view direction are computed by referencing the features 23, 24 and 26. The position along the viewing direction is computed by comparing the size of end effector 9 to that of a size at a prior known position such as home as seen in FIG. 9. Alternatively an orthogonal camera viewing in an orthogonal direction as seen in FIG. 10 and FIG. 11 can be used to compute the position along the viewing direction of the primary camera. Further a cross camera 21 can be used to improve ease of position measurement and accuracy.

(20) Even though a single primary camera can successfully compute all positions a plurality of primary cameras is used to cover large robot motion range due to limited field of views of a single camera and image distortions causing lack of accuracy. Image stitching algorithms are employed to improve positioning accuracy between the primary and secondary camera arrays by joining the viewed images to that of adjacent cameras creating panoramic views. Further an image library of positions or a lookup table of images is stored when robot is in good calibration. This image library position lookup table is compared to compute positioning errors of traditional controls and detecting anomalies.

(21) Although the invention has been described herein in connection with various preferred embodiments, there is no intention to limit the invention to those embodiments. It should be understood that various changes and modifications to the preferred embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the present invention and without diminishing its attendant advantages. Therefore, the appended claims are intended to cover such changes and modifications.