HAND-EYE CALIBRATION OF CAMERA-GUIDED APPARATUSES

Abstract

The invention describes a generic framework for hand-eye calibration of camera-guided apparatuses, wherein the rigid 3D transformation between the apparatus and the camera must be determined. An example of such an apparatus is a camera-guided robot.

Claims

1. A hand-eye calibration method for determining the parameters of the hand-eye pose of camera-guided apparatuses, the method comprising the following steps: (a) controlling a plurality of apparatus poses with the apparatus; (b) capturing a camera image at each apparatus pose; (c) extracting image features in the captured camera images; (d) determining approximate values for the parameters of the hand-eye pose; and (e) determining the parameters of the hand-eye pose assuming erroneous apparatus poses and erroneous extracted image features to implicitly account for erroneous camera poses, which comprises the steps of (e1) statistically modeling the accuracies of the parameters describing the apparatus pose and the accuracies of the extracted image features, wherein the number of parameters describing the apparatus pose is at least equal to the number of degrees of freedom of the apparatus; (e2) optimizing the parameters of the hand-eye pose by simultaneously minimizing a back-projection error of the image features in the captured camera images and the errors in the parameters describing the apparatus pose, taking into account the accuracies from step (e1); (e3) computing improved accuracies of the parameters describing the apparatus pose and the extracted image features based on the results of the optimization from step (e2) using variance component estimation; and (e4) repeating steps (e1) through (e3) until the accuracies of the parameters describing the apparatus pose and the accuracies of the extracted image features converge.

2. The method according to claim 1, wherein the apparatus is a robot and apparatus poses represent robot poses.

3. The method according to claim 1, wherein capturing a camera image at each apparatus pose in step (b) comprises capturing a camera image from a calibration object, and wherein extracting image features in the captured camera images in step (c) comprises extracting calibration marks on the calibration object in the captured camera images.

4. The method according to claim 1, wherein capturing a camera image at each apparatus pose in step (b) comprises capturing a camera image of a scene suitable for extracting salient image points, and wherein extracting image features in the captured camera images in step (c) comprises extracting salient image points in the captured camera images.

5. The method according to claim 3, wherein determining approximate values for the hand-eye pose parameters in step (d) comprises the following steps: (d1) determining a camera pose for each apparatus pose using the calibration marks on the calibration object extracted in the camera image; and (d2) determining the approximate values for the hand-eye pose parameters using a hand-eye calibration approach using the camera poses and the apparatus poses.

6. The method according to claim 4, wherein determining approximate values for the hand-eye pose parameters in step (d) comprises the following steps: (d1) determining a scaled camera pose for each apparatus pose using the salient image points extracted in the camera image; (d2) determining a normalization factor using the scaled camera poses and the apparatus poses; (d3) determining the camera poses by normalizing the scaled camera poses using the normalization factor; and (d4) determining the approximate values for the parameters of the hand-eye pose using a hand-eye calibration approach using the camera poses and the apparatus poses.

7. The method according to claim 1, further comprising the step: (f) determining improved (calibrated) apparatus poses from the result of optimizing the parameters of the hand-eye pose.

8. The method according to claim 1, further comprising the step: (f) determining the accuracy of the apparatus from the result of optimizing the parameters of the hand-eye pose.

9. The method according to claim 1, wherein steps (d) and (e2) are replaced by: (d) determining approximate values for the parameters of the hand-eye pose and for the parameters of an inner orientation of the camera; and (e2) optimizing the parameters of the hand-eye pose and the parameters of the inner orientation of the camera by simultaneously minimizing a back-projection error of the image features in the captured camera images and the errors in the parameters describing the apparatus pose, taking into account the accuracies from step (e1).

Description

[0042] In the Figures

[0043] FIG. 1 shows the coordinate systems relevant to the hand-eye calibration of a robot (apparatus) with a moving camera; and

[0044] FIG. 2 shows the coordinate systems relevant to the hand-eye calibration of a robot (apparatus) in the case of a stationary camera.

DETAILED DESCRIPTION OF THE INVENTION

[0045] First, the camera model and the calibration model underlying the invention are described, i.e., the relationship between 3D world points and their projection into the camera. For ease of description of the invention, it is assumed that the camera-guided apparatus is a camera-guided robot. The descriptions can be readily applied by a person skilled in the art to other camera-guided apparatuses as described above. Further, it is understood that the cameras are mounted on the end effector of the robot. Thus, the description of the invention primarily relates to the case of a moving camera. It is known from the literature that the case of a stationary camera is equivalent to the case of a moving camera. Therefore, in the following, the case of a stationary camera will be discussed only in those places where the equivalence is not obvious. Subsequently, three alternative optimization methods for hand-eye calibration are described. Finally, methods are described which provide the necessary initial values for the unknowns in the optimization procedures.

Camera Model

[0046] In a preferred embodiment of the invention, the camera is described by the perspective camera model described in Steger et al. ([Chapter 3.9.1], 2018): When using homogeneous coordinates, a 3D point p.sub.w in the WCS is transformed into a point p.sub.c in the CCS by

[00001]$\begin{array}{cc}{p}_c{=}^c{H}_w{p}_w=\left(\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right)p_w,& \left(4\right)\end{array}$

wherein is a translation vector and R is a rotation matrix parameterized by Euler angles: R=R.sub.x(α)R.sub.y(β)R.sub.z(γ). The parameters (t.sub.x,t.sub.y,t.sub.z,α,β,γ) describe the outer orientation of the camera and thus represent a rigid transformation in 3D. In an alternative embodiment of the invention, the outer orientation is represented by another suitable parameterization. For example, the rotational part of the outer orientation may be represented by quaternions, or the entire outer orientation may be represented by dual quaternions (Daniilidis, 1999).

[0047] The point is then projected into the image plane by

[00002]$\begin{array}{cc}\left(\begin{array}{c}{x}_u\\ y_u\end{array}\right)=\frac{c}{z_c}\left(\begin{array}{c}{x}_c\\ y_c\end{array}\right),& \left(5\right)\end{array}$

with the chamber constant c of the lens. In an alternative embodiment of the invention, other camera models can be used instead (e.g., for telecentric lenses, Steger et al. [Section 3.9], 2018). Subsequently, to account for lens distortions, the undistorted point (x.sub.u, y.sub.u).sup.T is distorted to (x.sub.d, y.sub.d).sup.T. In a preferred embodiment of the invention, the division model (Lenz and Fritsch, 1990) is used to model lens distortion:

[00003]$\begin{array}{cc}\left(\begin{array}{c}{x}_u\\ y_u\end{array}\right)=\frac{1}{1+{\kappa r}_d^2}\left(\begin{array}{c}{x}_d\\ y_d\end{array}\right),& \left(6\right)\end{array}$

which models radially symmetric distortions by the parameter K. Thereby, r.sub.d.sup.2=x.sub.d.sup.2+y.sub.d.sup.2 applies. In another preferred embodiment of the invention, the polynomial model (Brown, 1971) is used, which models both radially symmetric and tangential distortions. In alternative embodiments of the invention, other alternative distortion models may be used in an obvious manner without departing from the scope of the invention.

[0048] Finally, the distorted point (x.sub.d, y.sub.d).sup.T is transformed into the image coordinate system:

[00004]$\begin{array}{cc}{p}_i=\left(\begin{array}{c}{x}_i\\ y_i\end{array}\right)=\left(\begin{array}{c}{x}_d/s_x+c_x\\ y_d/s_y+c_y\end{array}\right),& \left(7\right)\end{array}$

wherein represents the image main point and s.sub.x and s.sub.y describe the horizontal and vertical distances of neighboring sensor elements.

[0049] The six parameters for the division model (or ten parameters for the polynomial model) describe the internal orientation of the camera.

Calibration Model

[0050] In hand-eye calibration, the apparatus is moved to different apparatus poses. In the case of the moving camera robot example used for the invention description, the robot's tool is moved to n.sub.r different poses and a camera image is captured in each of these robot poses. In the case of calibration object-based calibration, the calibration object is placed at a fixed position in the robot's workspace (see FIG. 1). In the case of self-calibration, camera images are captured instead from a scene suitable for the extraction of salient image points. This can be, for example, the capturing of a sufficiently structured object or that of an arbitrary but structured background scene. Let .sup.tH.sub.b,j be the robot pose returned by the robot controller at the pose j (j=1, . . . n.sub.r). Further, let p.sub.k (k=1, . . . n.sub.w) be the 3D world points (given in WCS) of the calibration object or in the scene and their 2D projections into the image of the robot pose j be p.sub.j,k. Then the projection of a 3D point into the image is described by:

p.sub.j,k=π(.sup.cH.sub.t.sup.tH.sub.b,j.sup.bH.sub.wp.sub.k,i)  (8)

wherein π(p.sub.c, i) represents the projection of a point p.sub.c given in the CCS into the image. In the case of the division model, for example, π(p.sub.c, i) corresponds to the successive execution of Equation (5), the inverses of Equation (6) and Equation (7) using the inner orientation i.

[0051] Finally, the vector e.sub.t,j includes the transformation parameters of the robot poses .sup.tH.sub.b,j, the vector e.sub.c includes the transformation parameters of the unknown hand-eye pose .sup.cH.sub.t, and the vector e.sub.b, includes the transformation parameters of the unknown pose .sup.bH.sub.w (analogous to the outer orientation parameters of .sup.cH.sub.w described above). In a preferred embodiment of the invention, the transformation parameters are the three components of translation and the three Euler angles of rotation. In alternative embodiments, the rotational shares of the transformations are described by quaternions (4 parameters) or the transformations are described by dual quaternions (8 parameters). The invention further allows the use of other alternative parameterizations of the transformations, so that the invention can be easily integrated into existing methods. However, the number of parameters describing the robot poses .sup.tH.sub.b,j must at least correspond to the number of degrees of freedom of the robot. For example, while most industrial robots cover 6 degrees of freedom, there are special robot designs that allow fewer degrees of freedom. An example of this are SCARA robots, which have only 4 degrees of freedom (3 translations and 1 rotation) (Ulrich and Steger, 2016). In contrast, overparametrization, such as is present when using quaternions or dual quaternions, is possible. Other camera-guided apparatuses may also have constrained degrees of freedom. For example, some terrestrial robots can move in the plane, but have no possibility to leave that plane. Since this means they can only rotate around one axis and move in two directions, they have only 3 degrees of freedom (1 rotation and 2 translations).

[0052] In the case of a stationary camera, a calibration object (for calibration object-based calibration) or a sufficiently structured object (for self-calibration) is mounted on the end effector of the robot and thus moved along with the robot. The stationary camera then captures a camera image of the co-moved object at each approached robot pose. In the case of self-calibration, it can be helpful to ensure that the background is as homogeneous and structureless as possible so that the position of the object can be detected as robustly and automatically as possible in the camera images. The pose .sup.cH.sub.t in the moving case corresponds to the pose .sup.cH.sub.b in the stationary case. The pose .sup.bH.sub.w in the moving case corresponds to the pose .sup.tH.sub.w in the stationary case. The role of the poses .sup.cH.sub.w and .sup.tH.sub.b,j, however, remains unchanged.

Parameter Estimation in the Gauss-Markov Model

[0053] In the following, it is distinguished between the functional model and the stochastic model (Förstner and Wrobel, 2016). The functional model describes the relationships between the observations and the unknown parameters. In the stochastic model, the observations and the unknowns are treated as random variables with uncertainties, wherein the uncertainties are described by (co-)variances.

[0054] Assuming initially error-free robot poses, the hand-eye calibration problem can be formulated in the so-called Gauss-Markov model (Förstner and Wrobel [Chapter 4. 4], 2016), since all erroneous observations l can be expressed as a function l=ƒ(x) of the unknown parameters x, wherein ƒ describes the functional model and corresponds to Equation (8).

[0055] The vector contains the measured image points , i.e., the extracted image features in the captured camera images. In the case of calibration object-based calibration, in a preferred embodiment of the invention, the extracted image features represent the projection of the centers of circular marks on a calibration body (Steger and Ulrich, 2018). In an alternative embodiment of the invention, they represent the projection of the intersection points of a checkerboard pattern on a calibration body (OpenCV, 2021). In the case of self-calibration, the extracted image features represent salient image points that are computed using appropriate image processing operators in the images and mapped to each other over different images. Examples of such image processing operators are the Förstner point extractor (Förstner, 1994), the Harris point extractor (Harris and Stephens, 1988), and the SIFT point extractor (Lowe, 2004).

[0056] If a 3D point is not visible in a particular image, the corresponding entry in l is omitted. The sum of the measured image points over all images is denoted by n.sub.i, i.e. the number of observations n.sub.l is 2n.sub.i. For each observed image point, two equations of form (8) are given. Thus, 2n.sub.i equations are obtained. The structure of the vector x depends on the application scenario and can be determined according to the following set of rules: [0057] For all scenarios, x contains the hand-eye pose: x:=e.sub.c. For SCARA robots there is the peculiarity to consider that not all parameters of the hand-eye pose can be determined. For example, if the hand-eye pose is parameterized by 3 translation angles and 3 Euler angles, the robot pose parameter t.sub.z cannot be determined when calibrating SCARA robots. In this case, t.sub.z is set to 0 and excluded from the optimization (Ulrich and Steger, 2016). [0058] For a calibration object-based calibration, x must be extended by the unknowns e.sub.b: . In contrast, in the case of self-calibration, the 3D points can be reconstructed directly in the BCS. Therefore, in this case e.sub.b can be excluded from the unknowns and internally set to e.g. for the case that e.sub.b is described by 3 rotation parameters and 3 translation parameters. For other parameterizations, the parameters can be chosen to describe the unit image in the chosen parameterization. [0059] If the inner orientation of the camera is unknown and should be co-estimated at the same time, the vector with the unknowns is extended by the parameters of the inner orientation: . Note that u must typically be excluded from calibration to obtain a unique solution (Steger et al. [Chapter 3.9.4.2], 2018). [0060] In the case of self-calibration, x must be extended by the coordinates of the 3D points: .

[0061] Let n.sub.x be the final number of unknown parameters in x.

[0062] The stochastic model specifies the statistical properties of the observation process. In the present case, erroneous extracted image features are assumed. Assuming that the image points are uncorrelated and measured with the same accuracy, the n.sub.1×n.sub.1 weight coefficient matrix of the observations is set to the unit matrix: Q.sub.ll=I.

[0063] For linearization, the 2n.sub.1×n.sub.x Jacobian matrix A is set up, which includes the first derivatives of ƒ after the unknowns x, evaluated at the location of the initial values or current approximations for the unknowns x.sup.(0) (the determination of the initial values is described below):

[00005]$\begin{array}{cc}{A=\frac{\partial f\left(x\right)}{\partial x}.Math.}_{x=x^{\left(0\right)}}.& \left(9\right)\end{array}$

[0064] With Δl=l−ƒ(x.sup.(0)) and the weight matrix P.sub.ll=Q.sub.ll.sup.−1, corrections Δ{circumflex over (x)} for the unknowns can now be calculated. This is done by solving

(10)
after Δ{circumflex over (x)}, e.g. by Cholesky decomposition using the sparse matrices (the roof operator {circumflex over ( )} describes an estimated value in statistics). Thus, the improved unknowns result in {circumflex over (x)}.sup.(1)=x.sup.(0)+Δ{circumflex over (x)}. Finally, Equations (9) and (10) are applied repeatedly until convergence. This corresponds to minimizing the back-projection error of the 3D points into the camera images.

[0065] After convergence, the covariance matrix of the original observations is obtained by c.sub.H={circumflex over (σ)}.sub.0.sup.2Q.sub.ll, with the variance factor being , the residuals v=AΔ{circumflex over (x)}−Δl. and the redundancy r=n.sub.l−n.sub.x. The corrected observations are obtained by {circumflex over (l)}=l+{circumflex over (v)}. The covariance matrix of the estimated unknowns is obtained by variance propagation c.sub.{circumflex over (x)}{circumflex over (x)}={circumflex over (σ)}.sub.0.sup.2(A.sup.TP.sub.llA).sup.−1.

[0066] In the following, two alternative procedure models (parameter estimation in the Gauss-Helmert model and parameter estimation in the Gauss-Markov model with fictitious unknowns) are now described for the determination of the parameters of the hand-eye pose under the assumption of erroneous robot poses and erroneous extracted image features for the implicit consideration of erroneous camera poses:

Parameter Estimation in the Gauss-Helmert Model

[0067] To account for erroneous robot poses, they must be introduced as observations in addition to the image coordinates of the extracted image features. Therefore, the observations can no longer be expressed as a function of the unknowns. Thus, parameter estimation can no longer be readily performed in the Gauss-Markov model. In one embodiment of the invention, the parameter estimation is therefore performed in the Gauss-Helmert model (Förstner and Wrobel [Chapter 4.8], 2016). The functional model becomes ƒ(x,l)=0.

[0068] In this model, the vector l contains the erroneous observed image points of the extracted image features and the erroneous robot poses . The number of observation equations is still 2n.sub.i. However, the number of observations is now n.sub.l=2n.sub.i+6n.sub.r in for the case where the robot poses are represented by 3 translation parameters and 3 rotation parameters. In the case of an alternative representation of the robot poses, n.sub.l would change accordingly, e.g., would be n.sub.l=2n.sub.i+8n.sub.r when using dual quaternions, since dual quaternions contain 8 parameters. The vector x of the unknowns is identical to that of the Gauss-Markov model.

[0069] Compared to the Gauss-Markov model, the stochastic model must additionally account for the uncertainties in the robot poses. Tests on real systems have shown that the errors in the robot poses are mean-free and Gaussian distributed (Strobl and Hirzinger, 2006).

[0070] The following statistical modeling therefore includes the accuracies of the parameters describing the robot pose and the accuracies of the extracted image features, wherein the number of parameters describing the robot pose is at least equal to the number of degrees of freedom of the robot.

[0071] Even assuming uncorrelated observations, the relative accuracy between different sets of observations must be known in order to set up Q.sub.ll. For the case where the robot poses are represented by 3 translation angles and 3 Euler angles, the three observation groups are the image coordinates of the extracted image features, the Euler angles of the robot poses, and the translation components of the robot poses. Since this relative accuracy is not known in practice, approximate values are used to initialize Q.sub.ll. The actual variances are then estimated using the concept of variance components of the observations (Förstner and Wrobel, 2016). In one embodiment of the invention, the unit matrix is used to initialize Qi: Q.sub.ll: Q.sub.ll=I. However, this may lead to an unnecessarily high number of iterations to estimate the variance components. Therefore, in a preferred embodiment of the invention, the variances are initialized with more reasonable values. For example, one possibility when using Euler angles is to set the standard deviations of the image points to σ.sub.i=0.1 pixels, those of the Euler angles to σ.sub.a=0.1 degrees, and those of the translation components to σ.sub.t=1 min. Then Q.sub.ll=diag(rep(σ.sub.i.sup.2,2n.sub.i), rep([σ.sub.a.sup.2,σ.sub.a.sup.2,σ.sub.a.sup.2,σ.sub.t.sup.2,σ.sub.t.sup.2,σ.sub.t.sup.2],n.sub.r)), wherein the function generates a vector containing n copies of y.

[0072] The Jacobian matrix A is the same as in the Gauss-Markov model. In the Gauss-Helmert model, ƒ must be linearized also with respect to the observations. The derivations with respect to the observations are stored in the 2n.sub.i×n.sub.l Jacobian matrix B.

[00006]$\begin{array}{cc}{B=\frac{\partial f\left(x,l\right)}{\partial l}.Math.}_{x=x^{\left(0\right)}}.& \left(11\right)\end{array}$

[0073] The left upper 2n.sub.i×n.sub.l submatrix of B is a diagonal matrix in which all diagonal elements are set to −1.

[0074] With w=ƒ(x.sup.(0),I), the corrections for the unknowns can now be calculated. For this

must be solved to Δ{circumflex over (x)} (Förstner and Wrobel, 2016).

[0075] Accordingly, this model optimizes the parameters of the hand-eye pose by simultaneously minimizing a back-projection error of the image features in the captured camera images and the errors in the parameters describing the robot pose, taking into account the accuracies of the parameters describing the robot poses and the accuracies of the extracted image features.

[0076] After convergence of the optimization, the variance components for each observation group are estimated using the results of the optimization. When Euler angles are used, the variance component estimation yields corrected values for σ.sub.i, σ.sub.a and σ.sub.t and thus a corrected matrix Q.sub.ll. The procedure for estimating variance components is described in Förstner and Wrobel ([Section 4.2.4], 2016). The calculation of {acute over (v)} and {circumflex over (l)} in the Gauss-Helmert model can be looked up in Förstner and Wrobel (2016). Variance component estimation yields improved accuracies of the parameters describing the robot pose and of the extracted image features.

[0077] Finally, the optimization described above is performed again with the improved accuracies. Finally, the statistical modeling, optimization, and variance component estimation are repeated until the accuracies of the parameters converge. In practice, this is typically after 3 to 5 iterations.

[0078] Optionally, the covariance matrix of the observations is obtained by c.sub.ll={circumflex over (σ)}.sub.0.sup.2Q.sub.ll with the redundancy r=2n.sub.i−n.sub.x. The covariance matrix of the estimated unknowns is obtained by .

[0079] Another advantage of this approach is that the camera poses need not be assumed to be known and error-free. By allowing the camera poses to be expressed via the concatenation of erroneous observed or estimated transformations .sup.cH.sub.w=.sup.cH.sub.t .sup.tH.sub.b,j.sup.bH.sub.w, the approach implicitly accounts for erroneous camera poses.

Parameter Estimation in the Gauss-Markov Model with Fictitious Unknowns

[0080] Due to the computationally intensive matrix operations in the Gauss-Helmert model, in a preferred embodiment of the invention parameter estimation is performed in a more efficient variant of the Gauss-Markov model. This is equivalent to parameter estimation in the Gauss-Helmert model (Koch, 1999; Koch, 2007). Here, the robot poses are introduced as so-called fictitious unknowns.

[0081] The underlying idea here is to introduce the uncertain robot poses simultaneously as observations and as unknowns. The first part of the functional model is still l=ƒ(x). However, unlike the estimation in the Gauss-Markov model described above, 6 additional observation equations of the form e.sub.t,j=ƒ.sub.2(e.sub.t,j) for each robot pose j are now added for the case where the robot poses are represented by 3 translation parameters and 3 rotation parameters. For example, using quanternions for the rotations would add 7 additional observation equations per robot pose, while using dual quanternions would add, e.g., 8. Here, the function ƒ.sub.2 corresponds to the identity. This now gives 2n.sub.i+6n.sub.r equations for the case where the robot poses are represented by 3 translation parameters and 3 rotation parameters. Thus, the vector contains the observed image points and the robot poses analogous to the Gauss-Helmert model described above. At the same time, the robot poses are now also introduced as unknowns by extending the unknown vector accordingly: .

[0082] Since l is identical to that in the Gauss-Helmert model, the same stochastic model can be applied for the statistical modeling of the accuracies and for the case where the robot poses are represented by 3 translation parameters and 3 rotation parameters, Q.sub.ll=diag(rep(σ.sub.i.sup.2,2n.sub.i), rep([σ.sub.a.sup.2,σ.sub.a.sup.2,σ.sub.a.sup.2,σ.sub.t.sup.2,σ.sub.t.sup.2,σ.sub.t.sup.2],n.sub.r)) can be initialized.

[0083] Thus, even in the Gauss-Markov model with fictitious unknowns, the statistical modeling includes the accuracies of the parameters describing the robot pose and the accuracies of the extracted image features, wherein the number of parameters describing the robot pose is at least equal to the number of degrees of freedom of the robot.

[0084] The computation of the (2n.sub.i+6n.sub.r)×n.sub.x Jacobian matrix A (for the case where the robot poses are represented by 6 parameters) and of the corrections Δ{circumflex over (x)} is done by Equations (9) and (10). In this case, the lower left part of A contains zeros and the lower right part I.sub.6n.sub.i.sub.×6n.sub.r.

[0085] Thus, also this model optimizes the parameters of the hand-eye pose by simultaneously minimizing a back-projection error of the image features in the captured camera images and the errors in the parameters describing the robot pose while taking into account the accuracies of the parameters describing the robot poses and the accuracies of the extracted image features.

[0086] After convergence, the variance components for the observation groups are estimated using the results of the optimization as described in Förstner and Wrobel (2016) or Niemeier ([Chapter 9.3], 2008). Variance component estimation yields improved accuracies of the parameters describing the robot pose and of the extracted image features.

[0087] Finally, the optimization described above is performed again with the improved accuracies. Finally, the statistical modeling, optimization, and variance component estimation are repeated until the accuracies of the parameters converge. In practice, this is typically after 3 to 5 iterations.

[0088] Optionally, c.sub.ll={circumflex over (σ)}.sub.0.sup.2Q.sub.ll, {circumflex over (l)} and c.sub.{circumflex over (x)}{circumflex over (x)} are computed as in the parameter estimation in the Gauss-Markov model described above.

[0089] Like the previously described approach in the Gauss-Helmert model, also the approach of parameter estimation in the Gauss-Markov model with fictitious unknowns has the advantage that the camera poses do not have to be assumed to be known and error-free. By allowing the camera poses to be expressed via the concatenation of erroneous observed or estimated transformations .sup.cH.sub.w=.sup.cH.sub.t .sup.tH.sub.b,j.sup.bH.sub.w, the approach implicitly also accounts for erroneous camera poses.

Determination of Approximate Values for the Unknown Parameters

[0090] In a preferred embodiment of the invention, the initial values of the unknowns and in particular the approximate values for the parameters of the hand-eye pose are set by the following procedure: [0091] Approximate values for the hand-eye pose e.sub.c and for e.sub.b are obtained from any hand-eye calibration approach known from the literature and suitable for these purposes. In a preferred embodiment of the invention, a linear approach to hand-eye calibration is used for this purpose, e.g., the approach of Daniilidis (1999). In the approach of Daniilidis (1999), but also in some other approaches, this requires the prior determination of a camera pose for each robot pose based on the calibration marks on the calibration object extracted in the camera image. The determination of the approximate values for the hand-eye pose parameters is then performed by a hand-eye calibration approach using the camera poses and the robot poses. [0092] The initial values for the inner orientation are obtained from the data sheets of the camera (s.sub.x and s.sub.y) and the lens (C). The image principal point set to the center of the image and the distortion coefficients are set to 0. [0093] In the case of self-calibration, the determination of approximate values for the parameters is performed as follows: In a preferred embodiment of the invention, a SfM approach is performed on the captured camera images. A possible suitable SfM implementation is for example COLMAP (Schönberger and Frahm, 2016; Schönberger et al, 2016). The SfM approach provides the parameters of the inner orientation i, scaled reconstructed 3D points p.sub.k, for each image j the extracted 2D points of the image features p.sub.j,k, and a scaled outer orientation (i.e. camera pose) .sup.cH.sub.w,j.

[0094] In an alternative embodiment of the invention, any other suitable approaches known in the literature for determining initial values or additional knowledge from the specific application can be used to initialize the unknowns.

[0095] In the case of self-calibration, the inherently unknown scaling factor in the SfM approach must be determined in the initial values for p.sub.k and .sup.cH.sub.w,j. For this purpose, the normalization factor is first determined using the scaled camera poses and the robot poses. In a preferred embodiment of the invention, the camera motion .sup.c2H.sub.c1=.sup.cH.sub.w,j2 .sup.cH.sub.w,j1.sup.−1 and the tool motion .sup.t2H.sub.t1=.sup.tH.sub.b,j2.sup.tH.sub.b,j1.sup.−1 are calculated for all pairs of robot poses j.sub.1 and j.sub.2. Both rigid 3D transformations are then converted to the parameters of a screw (Daniilidis, 1999). The congruence theorem (Chen, 1991) states that the translation parameter d.sub.t of the screw of the tool motion is identical to the translation parameter of the screw of the camera motion. Thus, the ratio d.sub.t/d.sub.c reflects the unknown scale factor of the SfM approach and hence the sought normalization factor. In one embodiment of the invention, the average is calculated over the ratios of all pairs. In a preferred embodiment of the invention, the median of the ratios of all pairs is calculated for higher robustness, ignoring ratios for which d.sub.t or d.sub.c is below a noise threshold.

[0096] Finally, the determined normalization factor is used to normalize the scaled reconstructed 3D points p.sub.k and the scaled outer orientation (i.e., camera poses) .sup.cH.sub.w,j.

Determination of the Accuracy of the Robot

[0097] In both the Gauss-Helmert model and the Gauss-Markov model with fictitious unknowns, the results of optimizing the parameters of the hand-eye pose can be used to determine the accuracy of the robot. The estimation of the variance components allows to make a meaningful statement about the accuracy of the robot, which would normally require an involved robot calibration. The accuracy of the robot can be read directly from the matrix c.sub.ll resulting after the variance component estimation. In a preferred embodiment of the invention, in which the robot poses are described by 3 translation parameters and 3 rotation parameters, the variances of the translation parameters and the rotation parameters of the robot poses are averaged separately over all robot poses for this purpose, so that the accuracy of the robot can be presented in the form of two values.

Determination of Calibrated Robot Poses

[0098] In both the Gauss-Helmert model and the Gauss-Markov model with fictitious unknowns, it is possible to determine improved (calibrated) robot poses based on the results of optimizing the parameters of the hand-eye pose. By introducing the robot poses as observations, the vector contains the balanced robot poses in addition to the balanced image coordinates. These can be considered as corrected or calibrated robot poses. Thus, they can be used as a basis for simple and inexpensive robot calibration.

Advantages of the Invention

[0099] Explicitly modeling the uncertainty of an apparatus is advantageous for hand-eye calibration. It improves accuracy, provides calibrated apparatus poses, and gives information about the uncertainty of the apparatus. This is important, for example, in industrial robots used for tasks that require high accuracy. The parameter estimation in the Gauss-Markov model with fictitious unknowns combined with the estimation of variance components proposed in the invention provides a statistically sound representation of the problem. Different hand-eye calibration scenarios (e.g., calibration object-based calibration, self-calibration; calibration of different apparatuses, e.g., articulated-arm industrial robots, SCARA industrial robots, terrestrial exploration robots; unknown or known inner orientation) can be easily represented by adding or removing the appropriate parameters to or from the parameter vectors. Thus, a large number of applications can benefit from this invention.

REFERENCES

[0100] M. Abderrahim, A. Khamis, S. Garrido, and L. Moreno, “Accuracy and calibration issues of industrial manipulators”, in Industrial Robotics: Programming, Simulation and Application. IntechOpen, 2006, pp. 131-146. [0101] N. Andreff, R. Horaud, and B. Espiau, “Robot hand-eye calibration using structure-from-motion”, The International Journal of Robotics Research, vol. 20, no. 3, pp. 228-248, 2001. [0102] D. C. Brown, “Close-range camera calibration”, Photogrammetric Engineering, vol. 37, no. 8, pp. 855-866, August 1971. [0103] H. H. Chen, “A screw motion approach to uniqueness analysis of headeye geometry”, in Computer Vision and Pattern Recognition, 1991, pp. 145-151. [0104] K. Daniilidis, “Hand-eye calibration using dual quaternions”, International Journal of Robotics Research, vol. 18, no. 3, pp. 286-298, 1999. [0105] F. Dornaika and R. Horaud, “Simultaneous robot-world and hand-eye calibration”, IEEE Transactions on Robotics and Automation, vol. 14, no. 4, pp. 617-622, 1998. [0106] W. Förstner, “A Framework for Low Level Feature Extraction”, in: Jan-Olof Eklundh, eds.: Third European Conference on Computer Vision, Lecture Notes in Computer Science, vol. 801, pp. 383-394, Springer, 1994. [0107] W. Förstner and B. P. Wrobel, Photogrammetric Computer Vision: Statistics, Geometry, Orientation and Reconstruction. Springer International Publishing, 2016. [0108] C. Harris, M. Stephens, “A Combined Corner and Edge Detector”, Proceedings of the Fourth Alvey Vision Conference, pp. 147-151, 1988. [0109] A. Hofhauser, C. Steger, and N. Navab, “Perspective planar shape matching”, in Image Processing: Machine Vision Applications II, ser. Proc. SPIE 7251, K. S. Niel and D. Fofi, Eds., 2009. [0110] R. Horaud and F. Dornaika, “Hand-eye calibration”, International Journal of Robotics Research, vol. 14, no. 3, pp. 195-210, 1995. [0111] ISO 9283:1998, “Manipulating industrial robots—performance criteria and related test methods”, 1998. [0112] K.-R. Koch, Parameter Estimation and Hypothesis Testing in Linear Models. Berlin Heidelberg: Springer, 1999. [0113] K.-R. Koch, Introduction to Bayesian Statistics, 2nd ed. Heidelberg: Springer, 2007. [0114] K. Koide and E. Menegatti, “General hand—eye calibration based on reprojection error minimization”, IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 1021-1028, April 2019. [0115] R. Lenz and D. Fritsch, “Accuracy of videometry with CCD sensors”, ISPRS Journal of Photogrammetry and Remote Sensing, vol. 45, no. 2, pp. 90-110, 1990. [0116] D. G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision 60(2): pp. 91-110, 2004. [0117] W. Niemeier, “Ausgleichungsrechung-Statistische Auswertennethoden”, 2. überarbeitete and erweiterte Auflage, de Gruyter, Berlin, 2008. [0118] H. Nguyen and Q. Pham, “On the covariance of X in AX=XB”, IEEE Transactions on Robotics, vol. 34, no. 6, pp. 1651-1658, 2018. [0119] OpenCV, “Open Computer Vision”, Version 4.5.2, Open CV module calib3d, https://docs.opencv.org/4.5.2/d8/d0c/groups_calib3d.html, accessed 7.4.2021, 2021. [0120] M. Placzek and t. Piszczek, “Testing of an industrial robot's accuracy and repeatability in off and online environments”, Eksploatacja i Niezawodnosc—Maintenance and Reliability, vol. 20, no. 3, pp. 455-464, 2018. [0121] J. Schmidt and H. Niemann, “Data selection for hand-eye calibration: A vector quantization approach”, The International Journal of Robotics Research, vol. 27, no. 9, pp. 1027-1053, 2008. [0122] J. Schmidt, F. Vogt, and H. Niemann, “Robust hand-eye calibration of an endoscopic surgery robot using dual quaternions”, in Pattern Recognition, B. Michaelis and G. Krell, Eds. Springer Berlin Heidelberg, 2003, pp. 548-556. [0123] J. Schmidt, F. Vogt, and H. Niemann, “Calibration-free hand-eye calibration: A structure-from-motion approach”, in Pattern Recognition, W. G. Kropatsch, R. Sablatnig, and A. Hanbury, Eds. Springer Berlin Heidelberg, 2005, pp. 67-74. [0124] J. L. Schönberger and J. M. Frahm, “Structure-from-motion revisited”, in Conference on Computer Vision and Pattern Recognition (CVPR), 2016. [0125] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo”, in European Conference on Computer Vision (ECCV), 2016. [0126] P. Shiakolas, K. Conrad, and T. Yih, “On the accuracy, repeatability, and degree of influence of kinematics parameters for industrial robots”, International Journal of Modelling and Simulation, vol. 22, no. 4, pp. 245-254, 2002. [0127] K. H. Strobl and G. Hirzinger, “Optimal hand-eye calibration”, in 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2006, pp. 4647-4653. [0128] C. Steger, M. Ulrich, and C. Wiedemann, Machine Vision Algorithms and Applications, 2nd ed. Weinheim: Wiley-VCH, 2018. [0129] A. Tabb and K. M. A. Yousef, “Solving the robot-world handeye(s) calibration problem with iterative methods”, Machine Vision and Applications, vol. 28, no. 5, pp. 569-590, August 2017. [0130] R. Y. Tsai and R. K. Lenz, “A new technique for fully autonomous and efficient 3D robotics hand/eye calibration”, IEEE Transactions on Robotics and Automation, vol. 5, no. 3, pp. 345-358, June 1989. [0131] M. Ulrich and C. Steger, “Hand-eye calibration of SCARA robots using dual quaternions”, Pattern Recognition and Image Analysis, vol. 26, no. 1, pp. 231-239, 2016. [0132] M. Ulrich, C. Wiedemann, and C. Steger, “Combining scale-space and similarity-based aspect graphs for fast 3D object recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 10, pp. 1902-1914, October 2012. [0133] Z. Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, November 2000.