NON-CONTACT METHOD AND SYSTEM FOR CONTROLLING AN INDUSTRIAL AUTOMATION MACHINE

Abstract

A method and system are provided for controlling an industrial automation machine using non-contact (virtual) position encoding. The system and method can be used to determine the position of an object under assembly on a conveyor system without mechanical coupling to a line or drive mechanism of the line.

Claims

1. A non-contact method of controlling an industrial automation machine, the method comprising: providing at least one 3D or depth sensor at a vision station located in an industrial environment, each sensor having a field of view at the vision station to obtain a stream of sensor data representative of a surface shape of an inanimate object moving along or about an axis in the vision station within its field of view; tracking the pose of the object within the vision station as a function of time based on each stream of sensor data to obtain at least one stream of poses; and generating a command signal for use by the industrial automation machine based on the at least one stream of poses to control the machine.

2. The method as claimed in claim 1, wherein the industrial automation machine is configured to perform a task on the object.

3. The method as claimed in claim 1, wherein the command signal is a trigger signal.

4. The method as claimed in claim 1, wherein the command signal is a stream of quadrature signals for each axis.

5. The method claimed in claim 2, wherein the machine is an inspection machine to inspect the object.

6. The method as claimed in claim 2, wherein the machine is configured to change or affect the object.

7. The method as claimed in claim 1, wherein motion of the object includes a linear motion in the vision station.

8. The method as claimed in claim 1 further comprising determining a multidimensional offset of the object from a reference pose and generating an offset signal for use by the industrial automation machine based on the offset.

9. The method as claimed in claim 1, wherein the step of tracking includes the steps of: processing each stream of sensor data in real time to obtain at least one stream of estimated poses; processing the at least one stream of estimated poses in real time to obtain kinematic state estimates of the object; and processing the kinematic state estimates in real time to obtain an evolution of the state of the object.

10. The method as claimed in claim 9, wherein the step of processing the kinematic state estimates utilizes a transient model.

11. The method as claimed in claim 9, wherein the step of processing the kinematic state estimate utilizes a steady state model.

12. The method claims in claim 1, wherein each of the sensors is mounted in a fixed position within the vision station.

13. A non-contact system for controlling an industrial automation machine, the system comprising: at least one 3D or depth sensor, each sensor having a field of view at a vision station to obtain a stream of sensor data representative of a surface shape of an inanimate object moving along or about an axis in the vision station within its field of view; a tracker to track the pose of the object within the vision station as a function of time based on each stream of sensor data to obtain at least one stream of poses; and a signal generator to generate a command signal for use by the industrial automation machine based on the at least one stream of poses to control the machine.

14. The system as claimed in claim 13, wherein the industrial automation machine is configured to perform a task on the object.

15. The system as claimed in claim 13, wherein the command signal is a trigger signal.

16. The system as claimed in claim 13, wherein the command signal is a stream of quadrature signals for each axis.

17. The system claimed in claim 14, wherein the machine is an inspection machine to inspect the object.

18. The system as claimed in claim 14, wherein the machine is configured to change or affect the object.

19. The system as claimed in claim 13, wherein motion of the object is a linear motion in the vision station.

20. The system as claimed in claim 13, wherein the tracker includes at least one programmed processor to: process each stream of sensor data in real time to obtain at least one stream of estimated poses; process the at least one stream of estimated poses in real time to obtain kinematic state estimates of the object; and process the kinematic state estimates in real time to obtain an evolution of the state of the object.

21. The system as claimed in claim 20, wherein the programmed processor utilizes a transient model to process the kinematic state estimates.

22. The system as claimed in claim 20, wherein the programmed processor utilizes a steady state model to process the kinematic state estimates.

23. The system claims in claim 13, wherein each of the sensors is mounted in a fixed position within the vision station.

24. The system as claimed in claim 13 further comprising means for determining a multidimensional offset of the object from a reference pose and a signal generator to generate an offset signal for use by the industrial automation machine based on the offset.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG. 1 is a block diagram, schematic view of a system constructed in accordance with at least one embodiment of the present invention;

[0029] FIG. 2 is a view similar to the view of FIG. 1 except the signal generator is a quadrature signal generator and not a position trigger signal generator; and

[0030] FIGS. 3a and 3b are schematic diagrams illustrating the generation of quadrature signals.

DESCRIPTION OF PREFERRED EMBODIMENTS

[0031] As required, detailed embodiments of the present invention are disclosed herein; however, it is to be understood that the disclosed embodiments are merely exemplary of the invention that may be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the present invention.

Definitions and Mathematical Systems

[0032] Following the Linetracking the pose of an object as it moves along a conveyor

Target Objectan object moving along a conveyor.
Effectora Tool which changes a Target Object in some way, e.g. a robot.
Inspectora Tool which measures a Target Object, e.g. a Gap and Flush sensor.

Toolan Effector or an Inspector.

[0033] The Virtual Encoder method and system of the present invention is a method and apparatus to estimate the 6DOF pose of a Target Object as a function of time, then to communicate this information in some usable fashion. The consumer of the information is a Tool, which may be an Effector or an Inspector. The Virtual Encoder is distinguished from a Mechanical Encoder in that the Virtual Encoder is a non-contact device.

[0034] The Virtual Encoder method and apparatus is a superior means for determining the position of objects along a conveyor line, compared with present methods using Mechanical Encoders. The Virtual Encoder enables assembly methods which have heretofore not been practical because of the limitations of present methods.

Symbols and Mathematical Definitions

[0035] =(x, y, z, , , ).sup.tThe pose of a rigid body requires 3 positions+3 rotations
{.sub.i, .sub.i, .sub.i, . . . }the predicted pose, velocity, acceleration, . . . of an object at time t.sub.i
{.sub.i, .sub.i, .sub.i, . . . }The measured pose, velocity, acceleration, . . . of an object at time t.sub.i
.sub.i={.sub.i, .sub.i, .sub.i, . . . }The actual dynamic state of an object
.sub.i={.sub.i, .sub.i, .sub.i, . . . }The predicted dynamic state of an object
.sub.i={.sub.i, .sub.i, .sub.i, . . . }The measured dynamic state of an object
*.sub.i={*.sub.i, *.sub.i, *.sub.i, . . . }The commanded dynamic state of an object
(t.sub.n)={.sub.0, . . . , .sub.n}A (historical) trajectory of an object's pose through time
(t.sub.n)={.sub.0, . . . , .sub.n}The (historical) trajectory of an object's state through time
(t.sub.n)={.sub.0, . . . , .sub.n, .sub.n+1, .sub.n+32, . . . }The (predicted) evolution of an object's state
(.sub.0, . . . .sub.n)=(t.sub.n)A kinematic estimator estimates the dynamic state of
M((t.sub.n))=(t.sub.n)A kinematic model M predicts an object's state's evolution.

[0036] The Virtual Encoder method and system typically estimates the pose of a Target Object between meter and 5 meters in length in motion at speeds less than 1000 mm/sec along a linear conveyor.

[0037] The Virtual Encoder method and system typically estimates the pose of Target Objects in motion along a one-dimensional axis (the axis of travel of the linear conveyor) to enable Assembly Operations on the Target Object in motion:

(t)=(x(t),y,z,,,).sup.t

[0038] In at least one embodiment, the five steady-state dimensions {y, z, , , } may be modeled assuming they are fixed in a reference pose. That is: the position and orientation of any point on the Target Object is determined by x(t) and knowledge of the original reference orientation and position of the object, .sub.0=(x.sub.0, y.sub.0, z.sub.0, .sub.0, .sub.0, .sub.0).sup.t

.sub.A1(t)=({circumflex over (x)}(t),0,0,0,0,0).sup.t+.sub.0

[0039] In at least one embodiment, the five steady-state dimensions may be modeled by assuming they have undergone a rigid transform, , relative to the reference pose. One embodiment requires a measurement of this rigid transform.

.sub.A2(t)=({circumflex over (x)}(t),0,0,0,0,0).sup.t+.Math..sub.0

[0040] The Virtual Encoder method and system is preferably designed for assembly processes requiring position errors less than 25 mm: e.g. Paint Application, Badging Inspection, Paint Inspection, Gap and Flush Inspection. This translates to the requirement that

|(t)(t)|.sub.max25 mm

[0041] Communication of the object position as a function of time should occur by mimicking a quadrature position encoder signal to take advantage of the pre-existing, low-latency, encoder inputs which are supported by current-generation robot controllers.

[0042] The Virtual Encoder system and method operate in two modes. In the first mode one assumes the Target Object is in motion along an assembly line while the Tool is immobile. In the second mode one assumes the Target Object is in motion and the Tool is tracking.

[0043] Trigger an operation at a desired position, while the Target Object moves along the assembly line. The Tool is immobile. This case can be thought of as taking a snap shot of a Target Object when it reaches a predetermined position. The position (pose) at which an assembly process happens is controlled, but not the velocity.

[0044] Track the object in motion so that the Target Object is at a relative standstill relative to the Tool. This case can be thought of as working to fool the Tool into thinking the object is stationary. The pose and the velocity at which an assembly process happens are controlled.

[0045] Design Specification: Block Diagram of FIG. 1 shows a continuous stream {.sub.0, . . . , .sub.n} of pose estimates for a Target Object. The Virtual Encoder method and system emits one or more trigger signal(s) when the Target Object reaches one or more predetermined position(s)without physically contacting the Target Object.

[0046] Design Specification: Block Diagram of FIG. 2 shows a continuous stream {.sub.0, . . . , .sub.n} of pose estimates for a Target Object. The Virtual Encoder method and system generates a quadrature signal indicative of the position and velocity of the Target Objectwithout physically contacting the Target Object.

Design Specification Details

[0047] One or more Volumetric Sensors (i.e. 3D sensors) gather 3D data (Point Clouds) from Target Objects in motion through an assembly station. These Volumetric Sensors are mounted in fixed positions at distances between 1 and 4 meters from the Target Object(s). A given assembly station may be monitored by 1 to 4 Volumetric or 3D Sensors.

[0048] Calculation of objects' positions as a function of time uses CICPEA technology (Continuously Iterated Cloud-Based Pose Estimation Algorithms). See CICPEA below.

[0049] Quadrature signal emulation occurs as described in Quadrature Signal Generation below.

[0050] Low-latency and low-jitter signaling (to trigger a Tool and/or to trigger a pose measurement) is important. For example, on a hypothetical assembly line moving at 200 mm/sec, a 50 millisecond trigger delay corresponds to a position error of 10 millimeters.

[0051] Volumetric Sensors: Several volumetric sensors are known to the art and available commercially which are capable of producing streams of point clouds indicative of the surface shape of objects in motion within their fields of view. For example: Microsoft Kinect, PrimeSense Carmine, Orbbec Astra, Intel RealSense, etc.

[0052] The sensors used to create streams of point clouds for the Virtual Encoder method and system are chosen according to the specific requirements of an application, and may or may not be modified or improved versions of commercially available sensors.

[0053] CICPEA Pose Estimation

[0054] CICPEA=Continuously Iterated Cloud-Based Pose Estimation Algorithms

[0055] Under the assumption that a Target Object is moving slowly compared to the rate of point cloud sampling it is reasonable to estimate that as time evolves, object poses change slowly. Hence, a good estimate for the pose of an object at t.sub.n+1 is the pose of the object at t.sub.n:

[00001]${\hat{}}_{n+1}{\stackrel{~}{}}_n.Math..Math.\mathrm{or}.Math..Math.{\hat{}}_{n+1}{\stackrel{~}{}}_n+{\widetilde{\stackrel{.}{}}}_n.Math.\mathrm{dt}+{\widetilde{\stackrel{\mathrm{.Math.}}{}}}_n.Math.\frac{{\mathrm{dt}}^2}{2}+\mathrm{.Math.}$

[0056] Such approximations can improve the accuracy and speed of algorithms which operate on Point Clouds of data to produce Pose Estimates. Massively parallel geometric processors (such as NVIDIA computing hardware) enable Continuously Iterated Cloud-Based Pose Estimation Algorithms=CICPEA technology. CICPEA technology is used for pose estimation by the Virtual Encoder method and system.

[0057] A variety of CICPEA algorithms are known to the art for continuous pose estimation, a prominent example being the KinFu algorithm.

Kinematics Estimator

[0058] Several methods are known to the art for producing kinematic state estimates in one dimension from streams of (potentially noisy) pose estimates. In the context of the Virtual Encoder method and system these estimators are termed Kinematics Estimators.

[0059] The Kinematics Estimator, , block of the Virtual Encoder method and system, receives a stream of pose estimates from CICPEA algorithms and produces an estimate of the kinematic state of a Target Object:

(.sub.0, . . . , .sub.n)=(t.sub.n)

[0060] The kinematics estimator in use for any particular application is configurable by the Virtual Encoder method and system depending on the needs of that installation. Successful Kinematics Estimators used by the Virtual Encoder method and system include: Kalman Filters, a variety of robust estimators for position and velocity, linear least squares fit for position and velocity, and so on. Other appropriate methods will suggest themselves to persons versed in the art.

[0061] Kinematics Model: Given a historical state estimate from a Kinematics Estimator, the task of the Kinematics Model is to predict the evolution of the state of the Target Object. Any number of Kinematics Models will suggest themselves to persons versed in the art, but a favored method for the Virtual Encoder is the following:

[0062] For each new state estimate in the sequence {.sub.0, . . . , .sub.n} extract the position and velocity estimates from the final two (most recent) state estimates {{tilde over (x)}.sub.n1, {dot over ({tilde over (x)})}.sub.n1, {tilde over (x)}.sub.n, {dot over ({tilde over (x)})}.sub.n};

[0063] Calculate the sample frequency =t.sub.nt.sub.n1;

[0064] Set the steady-state velocity to the most recent velocity estimate .sub.={dot over ({tilde over (x)})}.sub.n and set a slow velocity threshold .sub.slow to 6.sub. where .sub. is the conveyor velocity uncertainty;

[0065] If v.sub..sub.slow calculate a 1.sup.st-order (position/velocity) transient model: M.sup.T=M.sub.1.sup.T(t.sub.n+dt): x(dt)=x+{dot over (x)}dt;

[0066] else calculate a 3.sup.rd-order (position/velocity/acceleration/jerk) transient model:

[00002]$M^T=M_3^T\left(t_n+\mathrm{dt}\right).Math.\text{:}.Math..Math.x\left(\mathrm{dt}\right)=x+\stackrel{.}{x}.Math..Math.\mathrm{dt}+\stackrel{\mathrm{.Math.}}{x}.Math.\frac{{\mathrm{dt}}^2}{2}+\stackrel{\mathrm{.Math.}}{x}.Math.\frac{{\mathrm{dt}}^3}{6};$

[0067] Calculate the evolution of the state of the Target Object anew as follows:

[0068] For t<t.sub.n+ use the transient model to calculate M.sup.T(t): x(t.sub.n+t);

[0069] For tt.sub.n+ use the steady state model:

M.sup.S(t): x(t(t.sub.n+))={tilde over (x)}.sub.n+.sub.dt;

[0070] Transient Models:

[0071] If v.sub..sub.slow the 1.sup.st-order (position/velocity) transient model is calculated as follows

[00003]$M^T=M_1^T\left(t_n+\mathrm{dt}\right).Math.\text{:}.Math..Math.x\left(\mathrm{dt}\right)={\tilde{x}}_{n-1}+\frac{{\tilde{x}}_{n-1}-{\tilde{x}}_{n-1}}{}.Math.\mathrm{dt}$

[0072] Else if .sub.>.sub.slow the 3.sup.d-order (position/velocity/acceleration/jerk) transient kinematic model is calculated by solving the following equations

[00004]$x\left(\mathrm{dt}\right)=x+\stackrel{.}{x}.Math..Math.\mathrm{dt}+\stackrel{\mathrm{.Math.}}{x}.Math.\frac{{\mathrm{dt}}^2}{2}+\stackrel{\mathrm{.Math.}}{x}.Math.\frac{{\mathrm{dt}}^3}{6}$$x\left(0\right)={\tilde{x}}_{n-1};x\left(\right)={\tilde{x}}_n$$\stackrel{.}{x}\left(0\right)={\widetilde{\stackrel{.}{x}}}_{n-1};\stackrel{.}{x}\left(\right)={\widetilde{\stackrel{.}{x}}}_n$

Which yields:

[00005]$M^T=M_3^T\left(t_n+\mathrm{dt}\right).Math.\text{:}.Math..Math.x\left(\mathrm{dt}\right)=x_0+v_0.Math.\mathrm{dt}+a.Math.\frac{{\mathrm{dt}}^2}{2}+j.Math.\frac{{\mathrm{dt}}^3}{6},\mathrm{where}$$x_0={\tilde{x}}_{n-1};x_1={\tilde{x}}_n;v_0={\widetilde{\stackrel{.}{x}}}_{n-1};v_1={\widetilde{\stackrel{.}{x}}}_n$$j=\frac{\left(x_1-x_0-v_0.Math.\right)-\frac{\left(v_1-v_0\right).Math.}{2}}{-\frac{{}^3}{12}}$$a=\frac{\left(x_1-x_0-v_0.Math.\right)-j.Math..Math.{}^3/6}{{}^2/2}$

[0073] Position Trigger Generator: Given a sequence of trigger positions {q.sub.0, . . . , q.sub.k} at which an assembly operation for a Target Object should be triggered: [0074] Each time the Kinematics Model is updated, calculate the predicted trigger times {t.sub.0, . . . , t.sub.k} for these assembly operations via:

[00006]$\begin{array}{c}x\left(t_0+\right)=q_0\\ \mathrm{.Math.}\\ x\left(t_k+\right)=q_k\end{array}$

where is the signal latency for the assembly operation signal transmission. [0075] When a trigger time is reached, signal the trigger for that event and remove the trigger from the list of trigger positions for that Target Object.

[0076] Quadrature Signal Generator: The task of the Quadrature Signal Generator is to create a stream of output quadrature signals to emulate the position evolution of the Target Object. Each time a new state estimate is available the Quadrature Signal Generator must be reinitialized.

[0077] The time-resolution of the quadrature signal generating circuitry .sub.g and the time-resolution of the quadrature signal receiving circuitry .sub.r should be predetermined. The rate of production of output signals is limited by dt=4*max{.sub.g, .sub.r}. Pseudo-code for a quadrature signal is as follows:

TABLE-US-00001 Begin Quadrature Signal Generator while (t t.sub.n + ){ x=x.sub.0 x* = integer(M.sup.S(t)) t=t.sub.n if (x* x) { dt = 4*max (.sub.g, .sub.r) SendQuadraturePulse (x*x) = t.sub.n t.sub.n1 x = x* while (t < t.sub.n + ){ } x* = integer(M.sup.T(t)) t += dt if (x* x) { } SendQuadraturePulse (x*x) End QuadratureSignalGenerator x = x* } t += dt }

[0078] Quadrature Signal Generation: The signal generation of a Mechanical Rotary Encoder is driven by the rotation of a wheel. The Virtual Encoder emulates a quadrature signal in response to the stream of Target Object pose measurements as-if a mechanical encoder was measuring the position of the Target Object.

[0079] A quadrature encoder signal is a two-channel binary signal which indicates both the direction and rate of change of an object's position. The rate of change is indicated by the rate at which HIGH-LOW transitions occur. The direction is indicated by the relative phase of the A and B channels as illustrated in FIGS. 3a and 3b. Pseudo-code for a quadrative pulse is as follows:

TABLE-US-00002 Subroutine Send Quadrature Pulse else { Parameter Integer dx* digital Output (1, true) if (dx* > 0) { wait () digital Output (A, true) digital Output (0, true) wait () wait () digital Output (B, true) digital Output (1, false) wait () wait () digital Output (A, false) digital Output (0, false) wait () wait () digital Output (B, false) } wait () End Subroutine Send Quadrature Pulse }

[0080] The advantages of the above-noted method and system are numerous including but not limited to: [0081] 1. The Virtual Encoder method and system achieve much greater accuracies than the mechanical encoder method; [0082] 2. The apparatus of the virtual encoder method (3D sensors and computational hardware necessary to run algorithms for tracking the linear motion of an object) may be installed nearby the assembly station where the information is needed. Thus, there is no problem with installing multiple such stations along the same segment of an assembly line; [0083] 3. The Virtual Encoder system is not mechanically coupled to the conveyor line, so it has none of the robustness problems of the mechanical encoder; and [0084] 4. Since the virtual encoder method and system are implemented via computer algorithms, the virtual encoder is also a smart encoder, and may implement a variety of advanced signal filtering methods (such as Kalman filtering) which are well-known to improve the performance of position prediction methods in the presence of noise, but which are unavailable to purely mechanical encoders.

[0085] The pose of an object can be estimated using a sensor capable of measuring range (depth) data. Location of the object relative to the sensor can be determined from one or more range measurements. Orientation of the object can be determined if the sensor provides multiple range measurements for points on the object. Preferably a dense cloud of range measurements is provided by the sensor so that orientation of the object can be determined accurately.

[0086] Use of at least one embodiment of the present invention improves accuracy and robustness of position encoding along a conveyor system, and, consequently, enables manufacturing techniques which heretofore have not been possible or have not been not economically feasible. Examples of such practices and techniques are: [0087] 1. Accurately following contours of a known object being painted, scanned, or otherwise coated when the contours of the object are already known, hence improving efficiency of painting operations; [0088] 2. Allowing inspections to be performed at accurate locations on a vehicle while in motion, for example: measuring body panel gap and flush, making ultrasonic measurements for leaks around windows. [0089] 3. When coupled with a separate system for measuring the contours of an unknown system, allows to follow the contours of an unknown object being painted, scanned or otherwise coated, hence improving efficiency of such operations.

[0090] In one preferred embodiment, the system includes one or more volumetric or 3D sensors configured to observe an object as it traverses an assembly or inspection station. The point cloud data from these sensors is fed to a computer, which implements algorithms for tracking the 1D motion of an object. The position and velocity estimates from these tracking algorithms may be fed through linear or non-linear filtering means such as Kalman filters, model-predictive algorithms, or other filters known to the art for improving position estimations. The result is translated to a time series of quadrature signals by electronic means. The quadrature signal train is fed to a robot or other device configured to use such a signal train for tracking the linear motion of an object.

[0091] While exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms of the invention. Rather, the words used in the specification are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the invention. Additionally, the features of various implementing embodiments may be combined to form further embodiments of the invention.