METHOD FOR ROBOT ASSISTED MULTI-VIEW 3D SCANNING MEASUREMENT BASED ON PATH PLANNING

Abstract

Robot assisted multi-view 3D scanning measurement based on path planning includes firstly, establishing a virtual simulation platform to complete the setting of measurement poses and measurement paths and perform the path evaluations of measurement paths. Then, completing the preliminary hand-eye calibration based on the properties of Kronecker product, and the preliminary hand-eye calibration is optimized by establishing a reprojection error cost function as the fitness function of the particle swarm optimization algorithm. Lastly, moving the robot to the measurement poses of the planned measurement paths, obtaining a single-view point cloud of the measured object and transforming it from the camera coordinate system to the robot base coordinate system to obtain a registered single-view point cloud based on the optimized hand-eye matrix. When registered single-view point clouds of all measurement poses are obtained, the points under the robot base coordinate system form a complete point cloud of the measured object.

Claims

1. A method for robot assisted multi-view 3D scanning measurement based on path planning, comprising: (1).establishing a virtual simulation platform 1.1).based on a real measurement environment, establishing a virtual simulation platform on a computer with an open source robot operating system (ROS), and importing the description file of a robot into the virtual simulation platform to create a robot which is the same as the robot of real measurement environment; at the same time, on the virtual simulation platform, installing a structured light measuring equipment and a depth camera, which are the same as that of real measurement environment on the fixture of the robot’s end flange; 1.2). simulating a measured object to obtain a simulated object, and placing the simulated object right in front of the robot and the structured light measuring equipment, determining multiple measurement surfaces of the simulated object and multiple measurement paths on each measurement surface for image taking of the simulated object by the structured light measuring equipment, where the i.sup.th measurement path of the k.sup.th measurement surface is denoted by .sub.kS.sub.i k = 1, 2, ..., K, i = 1,2,...,M.sub.k, K is the number of measurement surfaces, M.sub.k is the number of measurement paths of the k.sup.th measurement surface, and the j.sup.th measurement point, namely measurement pose of the i.sup.th measurement path of the k.sup.th measurement surface is denoted by .sub.kP.sub.ij, j =1, 2,..., N.sub.k, N.sub.k is the number of measurement poses of a measurement path of the k.sup.th measurement surface; where the view field of the structured light measuring equipment is a rectangle with a length of m cm and a width of n cm, the outer rectangle of the k.sup.th measurement surface is .sub.ka in length and .sub.kb in width, the measurement paths are vertical strips along the length, which satisfy the following constraints: any two adjacent measurement paths have a strip of common area, the number M.sub.k of measurement paths of the k.sup.th measurement surface is greater than or equal to .sub.ka/m, the number N.sub.k of measurement poses of a measurement path of the k.sup.th measurement surface is greater than or equal to .sub.kb/n, the j.sup.th measurement pose .sub.kP.sub.ij of the i.sup.th measurement path of the k.sup.th measurement surface is obtained according to the constraints; (2).planning measurement paths on the virtual simulation platform 2.1). on the virtual simulation platform, to the i.sup.th measurement path of the k.sup.th measurement surface, dragging the end of the robot to a measurement pose .sub.kP.sub.ij, where the measurement pose .sub.kP.sub.ij is (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij, .sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij), (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) is the position coordinate of the end of the robot, and (.sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) is the attitude coordinate of the end of the robot; 2.2).taking a RGB-D image of the simulated object at the measurement pose .sub.kP.sub.ij by the depth camera and converting the RGB-D image into a grayscale image, then filtering out the background of the RGB-D image by using threshold segmentation and extracting the contour from the RGB-D image to obtain a minimum circumscribed rectangle of the simulated object, then traversing the depths of all pixels of the RGB-D image within the minimum circumscribed rectangle to find out the shortest distance d.sub.min between the simulated object and the structured light measuring equipment and recording the pixel coordinate (u,v) that corresponds to the shortest distance d.sub.min on the RGB-D image, then judging whether the shortest distance d.sub.min satisfies the measurement range of the structured light measuring equipment: d.sub.min ∈ [D-δ,D+δ], where D is the focal length of the structured light measuring equipment for camera calibration, δ is the allowable measurement error range of the structured light measuring equipment, if yes, then going to step 2.4), otherwise going to step 2.3); 2.3). obtaining a corresponding spatial coordinate (x.sub.(u,v), y.sub.(u,v), z.sub.(u,v)) of the pixel coordinate (u,v) through a coordinate transformation, then determining a spatial line according to the spatial coordinate (x.sub.(u,v), y.sub.(u,v), z.sub.(u,v)) and the position coordinate (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) of the end of the robot: $\frac{x{-}_{k}P{x}_{ij}}{x_{\left(u,v\right)}{-}_{k}P{x}_{ij}}=\frac{y{-}_{k}P{y}_{ij}}{y_{\left(u,v\right)}{-}_{k}P{y}_{ij}}=\frac{z{-}_{k}P{z}_{ij}}{z_{\left(u,v\right)}{-}_{k}P{z}_{ij}}$ where (x,y,z) is a spatial coordinate on the spatial line; then, finding a position coordinate from the position coordinate (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) on and along the spatial line to obtain a position coordinate (.sub.k px.sub.ij, .sub.k py.sub.ij, .sub.k pz.sub.ij), which satisfies the following measurement conditions: $\left\{\begin{array}{l}{\tilde{d}}_{\min }=\sqrt{{\left({}_k\tilde{p}{x}_{ij}-x_{\left(u,v\right)}\right)}^2+{\left({}_k\tilde{p}{y}_{ij}-y_{\left(u,v\right)}\right)}^2+{\left({}_k\tilde{p}{z}_{ij}-z_{\left(u,v\right)}\right)}^2}\\ {\tilde{d}}_{\min }\in \left[D-\delta ,D+\delta \right]\end{array}\right)$ combining the position coordinate (.sub.k px.sub.ij, .sub.k py.sub.ij, .sub.k pz.sub.ij) with the attitude coordinate (.sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) to form a measurement pose (.sub.k px.sub.ij, .sub.k py.sub.ij, .sub.k pz.sub.ij, .sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) and updating the measurement pose .sub.kP.sub.ij with the measurement pose (.sub.k px.sub.ij, .sub.k py.sub.ij, .sub.k pz.sub.ij, .sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij); 2.4).recording the measurement pose .sub.kP.sub.ij, the going to step 2.1) for judging the next measurement pose .sub.kP.sub.i(j+1) until the shortest distances of all measurement poses of the measurement path .sub.kS.sub.i are judged, then, going to step 2.5); 2.5). evaluating the measurement path .sub.kS.sub.i 2.5.1). executing the measurement path .sub.kS.sub.i on the virtual simulation platform, where the robot will move continuously from the measurement pose .sub.kP.sub.i1 to the measurement pose .sub.kP.sub.iNk, then sampling and recording the position .sub.kṖ.sub.it in equal time during the movement, all positions .sub.kṖ.sub.it, t = 0, 1, ..., T form a motion path .sub.kS.sub.i, where t is a serial number of sampling, T is the number of samplings; calculating the shortest distance l.sub.1 between the starting position (.sub.kPx.sub.i1,, .sub.kPy.sub.i1, .sub.kPz.sub.i1) and the ending position (.sub.kPx.sub.iJ, .sub.kPy.sub.iJ, .sub.kPz.sub.iNk) of the measurement path .sub.kS.sub.i: $l_1=\sqrt{{\left({}_{k}P{x}_{i1}{-}_{k}P{x}_{i{N}_k}\right)}^2+{\left({}_{k}Py{}_{i1}{-}_{k}P{y}_{i{N}_k}\right)}^2+{\left({}_{k}P{z}_{i1}{-}_{k}P{z}_{i{N}_k}\right)}^2}$ calculating the motion distance between the starting position (.sub.kṖx.sub.i1, .sub.kṖy.sub.i1, .sub.kṖz.sub.i1) and the ending position (.sub.kṖx.sub.iT, .sub.kṖy.sub.iT, .sub.kṖz.sub.iT) of the motion path .sub.kS.sub.i : $\begin{array}{l}{l}_2=\\ {\displaystyle \underset{t=1}{\overset{T-1}{.Math.}}\sqrt{{\left({}_k\dot{P}{x}_{i\left(t+1\right)}{-}_k\dot{P}{x}_{it}\right)}^2+{\left({}_k\dot{P}y{}_{i\left(t+1\right)}{-}_k\dot{P}{y}_{it}\right)}^2+{\left({}_k\dot{P}{z}_{i\left(t+1\right)}{-}_k\dot{P}{z}_{it}\right)}^2}}\end{array}$ then obtaining a distance difference L, where L = l.sub.2 - l.sub.1, and creating an evaluation function f.sub.1(L), where f.sub.1 (L) = (δ.sub.1 - L) / δ.sub.1, 0 ≤ L ≤ δ.sub.1, δ.sub.1 is a maximum error threshold; 2.5.2). finding a position .sub.k P.sub.iw which has the shortest distance to the barycenter of the simulated object from the motion path .sub.kS.sub.i, where the shortest distance is denoted by l.sub.3, then creating an evaluation function f.sub.2(l.sub.3), where f.sub.2 (l.sub.3) = (l.sub.3 - δ.sub.2) /l.sub.3, δ.sub.2 is the shortest distance which guarantee no collision between the structured light measuring equipment and the simulated object; 2.5.3).creating an overall evaluation function .sub.kF.sub.i for the measurement path .sub.kS.sub.i, where .sub.kF.sub.i =(0.5f.sub.1(L)+0.5F(l.sub.3))*100, then evaluating the measurement path .sub.kS.sub.i according to the overall evaluation function .sub.kF.sub.i, if the overall evaluation function .sub.kF.sub.i > g, the measurement path .sub.kS.sub.i has passed the evaluation, then going to step 2.7), otherwise going to step 2.6), where g is a threshold which is determined by measurement operator according to an actual measurement scenario, and 0 < g <100; 2.6). traversing all measurement poses of the measurement path .sub.kS.sub.i to find a measurement pose .sub.kP.sub.is which has the shortest distance to position .sub.kP.sub.iw, then moving the robot to the measurement pose .sub.kP.sub.is, dragging the end of the robot to manually increase the shortest distance d.sub.min between the simulated object and the structured light measuring equipment, where the increase of the shortest distance d.sub.min needs to satisfies the measurement range of the structured light measuring equipment: d.sub.min ∈[D-δ,D+δ], replacing the measurement pose .sub.kP.sub.is with the measurement pose after the increase of the shortest distance d.sub.min to complete the correction of the measurement path .sub.kS.sub.i, then going to step 2.7); 2.7).for each measurement path of each measurement surface, performing step 2.1) to step 2.6) to complete the simulation of planning of the measurement paths, converting the planned measurement paths into communication messages that can be recognized by robot and sending the communication messages from the virtual simulation platform to the robot of real measurement environment; (3).measuring in real measurement environment 3.1).establishing a hand-eye calibration equation in the real measurement environment, installing a structured light measuring equipment on the fixture of a robot’s end flange, operating the robot to move to the i′.sup.th measurement pose and taking an image of a corner of a checkerboard, namely a calibration board to obtain a calibration board image, then recording the pose information of the robot and the position P.sub.i′ of the corner in camera coordinate system, for n′ measurement poses, obtaining n′ calibration board images, and their corresponding pose information of the robot and positions P.sub.i′ of the corner in camera coordinate system; obtaining the rotation matrix R.sub.ci′ and the translation vector T.sub.ci′ of the calibration board relative to the binocular camera of the structured light measuring equipment in each calibration board image according to the calibration method of Zhang, and then combining the rotation matrix R.sub.ci′ and the translation vector T.sub.ci’into an extrinsic parameter matrix H.sub.ci’, i′ = 1, 2,..., n′, meanwhile, obtaining the rotation matrix R.sub.gi′, and the translation vector T.sub.gi′ of the robot’s end flange relative to the base of the robot according to the measurement pose, then combining the rotation matrix R.sub.gi′ and the translation vector T.sub.gi′ into a robot pose matrix H.sub.gi′, i′ = 1, 2,..., n′,where: $H_{c{i}^{\prime }}=\left[\begin{array}{cc}{R}_{c{i}^{\prime }}& T_{c{i}^{\prime }}\\ 0& 1\end{array}\right],H_{g{i}^{\prime }}=\left[\begin{array}{cc}{R}_{g{i}^{\prime }}& T_{g{i}^{\prime }}\\ 0& 1\end{array}\right];$ establishing a hand-eye calibration equation based on the extrinsic parameter matrices H.sub.cu’, H.sub.cv′ and robot pose matrices H.sub.gu′, H.sub.gv′ of any two measurement poses: $\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right]H_{cg}=H_{cg}\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right]$ where u′, v′ are serial numbers of any two measurement poses, u′ ≠ v′ and: $\begin{array}{l}\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right]=\left[\begin{array}{cc}{R}_{g{u}^{\prime },v^{\prime }}& T_{g{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right],H_{cg}=\\ \left[\begin{array}{cc}{R}_{cg}& T_{cg}\\ 0& 1\end{array}\right]\text{and}\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right]=\left[\begin{array}{cc}{R}_{c{u}^{\prime },v^{\prime }}& T_{c{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right];\end{array}$ where R.sub.gu′,.sub.v′ is the rotation matrix of the matrix $\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right],$ T .sub.gu′,v′ is the translation vector of the matrix $\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right],R_{cg}$ is the rotation matrix of the hand-eye matrix H .sub.cg, T.sub.cg is the translation vector of the hand-eye matrix H.sub.cg, R.sub.cu′,v′ is the rotation matrix of the matrix $\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right],T_{c{u}^{\prime },v^{\prime }}$ is the translation vector of the matrix $\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right];$ 3.2). based on the properties of Kronecker product, transforming the hand-eye calibration equation into a least squares problem, and calculating the hand-eye matrix H.sub.cg by using a singular value decomposition establishing a linear equation set: (R.sub.gu′,v′ ⓍI-I ⓍR.sup.T .sub.cu′,v′ ) .Math. vec (R.sub.cg) = 0, where I is a unit matrix, Ⓧ is the operator of Kronecker product, vec is the operator of vectorization; placing all matrices (R.sub.gu′,v′ ⓍI-IⓍR.sup.T.sub.cu′v′) of all any two measurement poses by column to obtain a matrix R; performing a singular value decomposition on the matrix R to obtaining matrix V, namely the right singular matrix of the matrix R, and taking out the 9 elements of last column of matrix V to revert to a matrix R.sub.cg of 3×3; performing a singular value decomposition on the matrix R.sub.cg, i.e ${\overline{R}}_{cg}=U_R{\displaystyle {.Math.}_R{V}_R^T},$ where U .sub.R is the left singular matrix of the matrix R.sub.cg, Σ.sub.R is the singular value matrix of the matrix R.sub.cg, V.sub.R is the right singular matrix of the matrix R.sub.cg, then obtaining the rotation matrix R.sub.cg of the hand-eye matrix H.sub.cg: $R_{cg}=U_R{V}_R^T;$ placing all matrices (R.sub.gu′,v′ -I) of all any two measurement poses by column to obtain a matrix R.sub.g, placing all matrices (R.sub.cgT.sub.cu′,v′ - T.sub.gu′,v′) of all any two measurement poses by column to obtain a matrix T′, then calculating the translation vector of the hand-eye matrix H.sub.cg : T.sub.cg = R.sub.g .sup.-1T′. 3.3).based on minimizing a reprojection error, optimizing the hand-eye matrix by using a particle swarm optimization algorithm 3.3.1). creating a particle swarm with population size of K′, where the position and the velocity of the k′.sup.th particle are denoted by p.sub.k′ and v.sub.k′, respectively, then initializing the positions p.sub.k′ and the velocities v.sub.k′ of all K′ particles: $p_{k^{\prime }}=H_{cg}$ $v_{k^{\prime }}=rand\left(v_{\min },v_{\max }\right)$ where k′= 1,2, ..., K′, v.sub.min, v.sub.max are the upper threshold and the lower threshold of velocity at each iteration of particle, rand () is rand function; 3.3.2). establishing a reprojection error cost function as the fitness function of the particle swarm optimization algorithm: $\phi \left(p_{k^{\prime }}\right)={\displaystyle \underset{i^{\prime }=1,j^{\prime }\ne i^{\prime }}{\overset{n^{\prime }}{.Math.}}{‖P_{i^{\prime }}-\frac{1}{s_{i^{\prime }}}{\text{K}}^{\ast }\exp {\left[p_{k^{\prime }}{H}_{g{j}^{\prime }}{H}_{g{i}^{\prime }}^{-1}{p}_{k^{\prime }}{}^{-1}\right]}_{3\times 4}{P}_{j^{\prime }}‖}_2^2}$ where s.sub.i′ is the scale factor of the i′.sup.th measurement pose, K* is the intrinsic parameter matrix of the binocular camera, exp[.Math.].sub.3x4 is an operator of choosing a submatrix of 3×4 of the left upper corner of the matrix in square bracket, || ||.sub.2 is a square-normal operator, P.sub.j′ is the position of the corner of the checkerboard of any non-i.sup.th measurement pose in camera coordinate system; 3.3.3). according to the fitness function φ(p.sub.k′), finding the position of the current individual extremum min {φ(p.sub.k′), $\left(\phi \left(p_{k^{\prime }}^{\ast }\right)\right\}$ of each particle and taking it as the historical optimal position $p_{k^{\prime }}^{\ast },$ finding the position of the global extremum min {φ( p .sub.k′), k′ =1,2,...,K′}, namely the position of the particle of the minimum value of fitness function and taking it as the global optimal position g* : $\left\{\begin{array}{c}{p}_{k^{\prime }}^{\ast }\leftarrow \min \left\{\phi \left(p_{k^{\prime }}\right),\phi \left(p_{k^{\prime }}^{\ast }\right)\right\}\\ g^{\ast }\leftarrow \min \left\{\phi \left(p_{k^{\prime }}\right),k^{\prime }=1,2,.Math.,K^{\prime }\right\}\end{array}\right)$ 3.3.4). updating the velocity v.sub.k’and position p.sub.k’of each particle: $\left\{\begin{array}{c}{v}_{k^{\prime }}=\omega v_{k^{\prime }}+c_{1}rand\left(0,1\right)\times \left(p_{k^{\prime }}^{\ast }-p_{k^{\prime }}\right)+c_{2}rand\left(0,1\right)\times \left(g^{\ast }-p_{k^{\prime }}\right)\\ p_{k^{\prime }}=p_{k^{\prime }}+v_{k^{\prime }}\end{array}\right)$ where ω is a inertia factor, c.sub.1 and c.sub.2 are acceleration factors; returning step 3.3.3), until a termination condition is reached; step 3.3.5)taking the global optimal position g* as the optimized hand-eye matrix, which is denoted by H ${}_{cg}^{revised}$ ; 3.4).performing a 3D measurement 3.4.1).adjusting the binocular camera of the structured light measuring equipment so that it can clearly capture the measured object, the left and right cameras can be kept in the same horizontal position with a certain distance, and the binocular calibration can be completed; adjusting the robot so that it can carry the structured light measuring equipment to perform the 3D measurement and guarantee the full view of the measured object will be taken; 3.4.2).after receiving the planned measurement paths from the virtual simulation platform, moving the robot in turn to the measurement poses of the planned measurement paths; 3.4.3). for the measurement pose .sub.kP.sub.ij, calculating a pose matrix ${}_k{H}_{ij}^g$ from the coordinate system of the robot’s end flange to the robot base coordinate system according the measurement pose .sub.kP.sub.ij, then obtaining a rigid body transformation matrix: ${}_k{H}_{ij}^{pcl}{=}_k{H}_{ij}^g\text{*}{H}_{cg}^{revised}$ then obtaining a single-view point cloud .sub.kI.sub.ij of the measured object through the binocular camera of the structured light measuring equipment and transforming it from the camera coordinate system to the robot base coordinate system to obtain a registered single-view point cloud ${}_k{I}_{ij}^{homo}:$ ${}_k{I}_{ij}^{homo}{=}_k{H}_{ij}^{pcl}\text{*}{}_k{I}_{ij}$ when registered single-view point clouds of all measurement poses are obtained, the points under the robot base coordinate system form a complete point cloud of the measured object, the 3D the measurement is completed.

Description

BRIEF DESCRIPTION OF THE DRAWING

[0064] The above and other objectives, features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0065] FIG. 1 a flow diagram of a method for robot assisted multi-view 3D scanning measurement based on path planning in accordance with the present invention;

[0066] FIG. 2 is a diagram of determining measurement paths and measurement poses;

[0067] FIG. 3 is a flow diagram of measuring in real measurement environment in accordance with the present invention;

[0068] FIG. 4 is a diagram of establishing a hand-eye calibration equation;

[0069] FIG. 5 is a diagram of registering a single-view point cloud;

[0070] FIG. 6 is a diagram of a virtual simulation platform;

[0071] FIG. 7 is a comparison chart of the position distribution of calibration origins after and before optimization;

[0072] FIG. 8 is a diagram of the measurement poses and corresponding single-view point clouds on one measurement surface of the measured object in example 1;

[0073] FIG. 9 is a diagram of the setting of different measurement surfaces of the measured object and corresponding registered single-view point clouds of measurement surfaces in example 1;

[0074] FIG. 10 is an illustration of complete point cloud of the measured object in example 1;

[0075] FIG. 11(A) is an illustration of complete point cloud of not optimizing the hand-eye matrix of another measured object in example 2;

[0076] FIG. 11(B) is an illustration of complete point cloud of optimizing the hand-eye matrix of another measured object in example 2.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0077] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the similar modules are designated by similar reference numerals although they are illustrated in different drawings. Also, in the following description, a detailed description of known functions and configurations incorporated herein will be omitted when it may obscure the subject matter of the present invention.

[0078] In one embodiment of the present invention, as shown in FIG. 1, a method for robot assisted multi-view 3D scanning measurement based on path planning in accordance with the present invention comprises the following steps:

Step S1: Establishing a Virtual Simulation Platform

Step S1.1: Creating a Robot and Installing a Structured Light Measuring Equipment and a Depth Camera

[0079] Based on a real measurement environment, establishing a virtual simulation platform on a computer with an open source robot operating system (ROS), and importing the description file of a robot into the virtual simulation platform to create a robot which is the same as the robot of real measurement environment; at the same time, on the virtual simulation platform, installing a structured light measuring equipment and a depth camera, which are the same as that of real measurement environment on the fixture of the robot’s end flange.

[0080] Where the description file of a robot is URDF(Universal Robot Description Format description file, which includes the parameters of links, joints, kinematics and dynamic, visualization models, collision detection model.

Step S1.2: Determining Measurement Paths

[0081] FIG. 2 is a diagram of determining measurement paths and measurement poses. In one embodiment of the present invention, as shown in FIG. 2, in order to obtain the more complete, more accurate 3D shape characteristics of the measured object and guarantee the adjacent point clouds have an overlapping area, we need to perform a simulation on the measured object:

[0082] Simulating a measured object to obtain a simulated object, and placing the simulated object right in front of the robot and the structured light measuring equipment, determining multiple measurement surfaces of the simulated object and multiple measurement paths on each measurement surface for image taking of the simulated object by the structured light measuring equipment, where the i.sup.th measurement path of the k.sup.th measurement surface is denoted by .sub.kS.sub.i k = 1,2,...,K,i = 1, 2,...,M.sub.k, K is the number of measurement surfaces, M.sub.k is the number of measurement paths of the k.sup.th measurement surface, and the j.sup.th measurement point, namely measurement pose of the i.sup.th measurement path of the k.sup.th measurement surface is denoted by .sub.kP.sub.ij=1,2,...,N.sub.k, N.sub.k is the number of measurement poses of a measurement path of the k.sup.th measurement surface.

[0083] Where the view field of the structured light measuring equipment is a rectangle with a length of m cm and a width of n cm, the outer rectangle of the k.sup.th measurement surface is .sub.kα in length and .sub.kb in width, the measurement paths are vertical strips along the length, which satisfy the following constraints: any two adjacent measurement paths have a strip of common area, the number M.sub.k of measurement paths of the k.sup.th measurement surface is greater than or equal to .sub.ka/m, the number N.sub.k of measurement poses of a measurement path of the k.sup.th measurement surface is greater than or equal to .sub.kbln, the j.sup.th measurement pose .sub.kP.sub.ij of the i.sup.th measurement path of the k.sup.th measurement surface is obtained according to the constraints. As shown in FIG. 2, thus overlapping areas exist in horizontal direction and vertical direction.

Step S2: Planning Measurement Paths on the Virtual Simulation Platform

Step S2.1: Dragging the End of the Robot to a Measurement Pose

[0084] On the virtual simulation platform, to the i.sup.th measurement path of the k.sup.th measurement surface, dragging the end of the robot to a measurement pose .sub.kP i, where the measurement pose .sub.kP.sub.ij is (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij, .sub.kRx.sub.ij,.sub.kRy.sub.ij, .sub.kRz.sub.ij), (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) is the position coordinate of the end of the robot, and (.sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) is the attitude coordinate of the end of the robot.

Step S2.2: Detecting and Judging the Shortest Distance

[0085] Taking a RGB-D image of the simulated object at the measurement pose .sub.kP.sub.ij by the depth camera and converting the RGB-D image into a grayscale image, then filtering out the background of the RGB-D image by using threshold segmentation and extracting the contour from the RGB-D image to obtain a minimum circumscribed rectangle of the simulated object, then traversing the depths of all pixels of the RGB-D image within the minimum circumscribed rectangle to find out the shortest distance d.sub.min between the simulated object and the structured light measuring equipment and recording the pixel coordinate (u,v) that corresponds to the shortest distance d.sub.min on the RGB-D image, then judging whether the shortest distance d.sub.min satisfies the measurement range of the structured light measuring equipment: d.sub.min∈[D-δ,D+δ], where D is the focal length of the structured light measuring equipment for camera calibration, δ is the allowable measurement error range of the structured light measuring equipment, if yes, then going to Step S2.4, otherwise going to Step S2.3.

Step S2.3: Adjusting Measurement Pose

[0086] Obtaining a corresponding spatial coordinate (x.sub.(u,v), y.sub.(u,v), z.sub.(u,v)) of the pixel coordinate (u,v) through a coordinate transformation, then determining a spatial line according to the spatial coordinate (x.sub.(u,v), y.sub.(u,v), z.sub.(u,v)) and the position coordinate (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) of the end of the robot:

[00024]$\frac{x{-}_{k}P{x}_{ij}}{x_{\left(u,v\right)}{-}_{k}P{x}_{ij}}=\frac{y{-}_{k}P{y}_{ij}}{y_{\left(u,v\right)}{-}_{k}P{y}_{ij}}=\frac{z{-}_{k}P{z}_{ij}}{z_{\left(u,v\right)}{-}_{k}P{z}_{ij}}$

[0087] Where (x,y,z) is a spatial coordinate on the spatial line.

[0088] Then, finding a position coordinate from the position coordinate (.sub.kPx.sub.ij, .sub.kPy.sub.ij, .sub.kPz.sub.ij) on and along the spatial line to obtain a position coordinate (.sub.k p̃x.sub.ij, .sub.kp̃y.sub.ij, .sub.kp̃z.sub.ij), which satisfies the following measurement conditions:

[00025]$\left\{\begin{array}{l}{\tilde{d}}_{\min }=\sqrt{{\left({}_k\tilde{p}{x}_{ij}-x_{\left(u,v\right)}\right)}^2+{\left({}_k\tilde{p}{y}_{ij}-y_{\left(u,v\right)}\right)}^2+{\left({}_k\tilde{p}{z}_{ij}-z_{\left(u,v\right)}\right)}^2}\hfill \\ {\tilde{d}}_{\min }\in \left[D-\delta ,D+\delta \right]\hfill \end{array}\right)$

[0089] Combining the position coordinate (.sub.kp̃x.sub.ij, .sub.kp̃y.sub.ij, .sub.kp̃z.sub.ij) with the attitude coordinate (.sub.kRx.sub.ij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) to form a measurement pose (.sub.k p̃x.sub.ij, .sub.kp̃y.sub.ij, .sub.kp̃z.sub.ij, .sub.kR.sub.xij, .sub.kRy.sub.ij, .sub.kRz.sub.ij) and updating the measurement pose .sub.kP.sub.ij with the measurement pose (.sub.k p̃x.sub.ij, .sub.kp̃y.sub.ij, .sub.kp̃z.sub.ij, kR.sub.xij, .sub.kR.sub.ij, .sub.kRz.sub.ij).

Step S2.4: Repeating Steps S2.1- S2.3, Until All Measurement Poses of the Measurement Path are Judged

[0090] recording the measurement pose .sub.kP.sub.ij, the going to Step S2.1 for judging the next measurement pose .sub.kP.sub.i(j+1) until the shortest distances of all measurement poses of the measurement path .sub.kS.sub.i are judged, then, going to Step S2.5.

Step S2.5: Evaluating the Measurement Path .SUB.k.S.SUB.i

[0091] Step S2.5.1: executing the measurement path .sub.kS.sub.i on the virtual simulation platform, where the robot will move continuously from the measurement pose .sub.kP.sub.i1 to the measurement pose, then sampling and recording the position .sub.kṖ.sub.it in equal time during the movement, all positions .sub.kṖ.sub.it, t = 0,1, ..., T form a motion path .sub.kṠ.sub.i, where t is a serial number of sampling, T is the number of samplings.

[0092] Calculating the shortest distance l.sub.1 between the starting position (.sub.kPx.sub.i1, .sub.kPy.sub.i1, .sub.kPz.sub.i1) and the ending position (.sub.kPx.sub.iJ, .sub.kPy.sub.iJ, .sub.kPz.sub.iNk) of the measurement path .sub.kS.sub.i :

[00026]$l_1=\sqrt{{\left({}_{k}P{x}_{i1}{-}_{k}P{x}_{i{N}_k}\right)}^2+{\left({}_{k}P{y}_{i1}{-}_{k}P{y}_{i{N}_k}\right)}^2+{\left({}_{k}P{z}_{i1}{-}_{k}P{z}_{i{N}_k}\right)}^2}$

[0093] Calculating the motion distance between the starting position (.sub.kṖx.sub.i1, (.sub.kṖy.sub.i1, .sub.kṖz.sub.i1) and the ending position (.sub.kṖx.sub.iT, .sub.kṖy.sub.iT, .sub.kṖz.sub.iT) of the motion path .sub.kṠ.sub.i:

[00027]$\begin{array}{l}{l}_2=\\ {\displaystyle \underset{t=1}{\overset{T-1}{.Math.}}\sqrt{{\left({}_k\dot{P}{x}_{i\left(t+1\right)}{-}_k\dot{P}{x}_{it}\right)}^2+{\left({}_k\dot{P}{y}_{i\left(t+1\right)}{-}_k\dot{P}{y}_{it}\right)}^2+{\left({}_k\dot{P}{z}_{i\left(t+1\right)}{-}_k\dot{P}{z}_{it}\right)}^2}}\end{array}$

[0094] Then obtaining a distance difference L, where L = l.sub.2 — l.sub.1, and creating an evaluation function f.sub.1(L), where f.sub.1 (L) = (δ.sub.1 - L) / δ.sub.1, 0 ≤ L ≤ δ.sub.1, δ.sub.1 is a maximum error threshold.

[0095] Step S2.5.2: finding a position .sub.kṖ.sub.iw which has the shortest distance to the barycenter of the simulated object from the motion path .sub.kṠ.sub.i, where the shortest distance is denoted by l.sub.3, then creating an evaluation function f.sub.2(l.sub.3), where f.sub.2 (l.sub.3) = (l.sub.3 — δ.sub.2) / l.sub.3, δ.sub.2 is the shortest distance which guarantee no collision between the structured light measuring equipment and the simulated object.

[0096] Step S2.5.3: creating an overall evaluation function .sub.kF.sub.i for the measurement path .sub.kS.sub.i, where .sub.kF.sub.i = (0.5f.sub.1(L)+0.5F(l.sub.3))*100, then evaluating the measurement path .sub.kS.sub.i according to the overall evaluation function .sub.kF.sub.i, if the overall evaluation function .sub.kF.sub.i > g, the measurement path .sub.kS.sub.i has passed the evaluation, then going to Step S2.7, otherwise going to Step S2.6, where g is a threshold which is determined by measurement operator according to an actual measurement scenario, and 0 < g < 100.

Step S2.6: Correcting the Measurement Path

[0097] traversing all measurement poses of the measurement path .sub.kS.sub.i to find a measurement pose .sub.kP.sub.is which has the shortest distance to position .sub.kṖ.sub.iw, then moving the robot to the measurement pose .sub.kP.sub.is, dragging the end of the robot to manually increase the shortest distance d.sub.min between the simulated object and the structured light measuring equipment, where the increase of the shortest distance d.sub.min needs to satisfies the measurement range of the structured light measuring equipment: d.sub.min ∈ [D-δ,D+δ], replacing the measurement pose .sub.kP.sub.is with the measurement pose after the increase of the shortest distance d.sub.min to complete the correction of the measurement path .sub.kS.sub.i,, then going to Step S2.7.

Step S2.7: Repeating Step S2.1 - Step S2.6 for Each Measurement Path, Then Sending the Planned Measurement Paths to the Robot of Real Measurement Environment

[0098] For each measurement path of each measurement surface, performing Step S2.1 to Step S2.6 to complete the simulation of planning of the measurement paths, converting the planned measurement paths into communication messages that can be recognized by robot and sending the communication messages from the virtual simulation platform to the robot of real measurement environment.

Step S3: Measuring in Real Measurement Environment

[0099] As shown in FIG. 3, the measuring in real measurement environment comprises the following steps:

Step S3.1: Establishing a Hand-Eye Calibration Equation

[0100] In the real measurement environment, as shown in FIG. 4, installing a structured light measuring equipment on the fixture of a robot’s end flange, operating the robot to move to the i′.sup.th measurement pose and taking an image of a corner of a checkerboard, namely a calibration to obtain a board images, then recording the pose information of the robot and the position P.sub.i′ of the corner in camera coordinate system, for n′ measurement poses, obtaining n′ calibration board images, and their corresponding pose information of the robot and positions P.sub.i′ of the corner in camera coordinate system.

[0101] Obtaining the rotation matrix R.sub.ci′ and the translation vector T.sub.ci′ of the calibration board relative to the binocular camera of the structured light measuring equipment in each calibration board image according to the calibration method of Zhang, and then combining the rotation matrix R.sub.ci′ and the translation vector T.sub.ci′ into an extrinsic parameter matrix H.sub.ci′,i′ = 1,2,..., n′, meanwhile, obtaining the rotation matrix R.sub.gi′ and the translation vector T.sub.gi′ of the robot’s end flange relative to the base of the robot according to the measurement pose, then combining the rotation matrix R.sub.gi′ and the translation vector T.sub.gi′ into a robot pose matrix H.sub.gi′,i′ = 1, 2,..., n′,where:

[00028]$H_{c{i}^{\prime }}=\left[\begin{array}{cc}{R}_{c{i}^{\prime }}& T_{c{i}^{\prime }}\\ 0& 1\end{array}\right],H_{g{i}^{\prime }}\left[\begin{array}{cc}{R}_{g{i}^{\prime }}& T_{g{i}^{\prime }}\\ 0& 1\end{array}\right]$

[0102] For the reason that the relative pose relation between the base of the robot and the checkerboard (calibration board) is constant and the relative pose relation between the binocular camera and the robot’s end flange is constant, combining the coordinate transformation relation between the extrinsic parameter matrix of the binocular camera and the robot pose matrix, we can obtain the following relation for any two measurement poses:

[00029]$H_{g{u}^{\prime }}{H}_{c{g}^{\prime }}{H}_{c{u}^{\prime }}=H_{BW}=H_{BW}=H_{g{v}^{\prime }}{H}_{cg}{H}_{c{v}^{\prime }}$

[0103] Namely:

[00030]$H_{g{u}^{\prime }}{H}_{cg}{H}_{c{u}^{\prime }}=H_{g{v}^{\prime }}{H}_{cg}{H}_{c{v}^{\prime }}$

[0104] Performing a matrix transformation, namely multiplying the front of left side by matrix

[00031]$H_{g{v}^{\prime }}^{-1}$

and the rear of right side by matrix

[00032]$H_{c{u}^{\prime }}^{-1}$

we can obtain the following equation:

[00033]$H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}{H}_{cg}=H_{cg}{H}_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}$

[0105] Namely, establishing a hand-eye calibration equation based on the extrinsic parameter matrices H.sub.cu′, H.sub.cv′ and robot pose matrices H.sub.gu′, H.sub.gv′ of any two measurement poses:

[00034]$\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right]H_{cg}=H_{cg}\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right]$

where u′, v′ are serial numbers of any two measurement poses, u′ ≠ v′ and:

[00035]$\begin{array}{l}\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right]=\left[\begin{array}{cc}{R}_{g{u}^{\prime },v^{\prime }}& T_{g{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right],H_{cg}=\left[\begin{array}{cc}{R}_{cg}& T_{cg}\\ 0& 1\end{array}\right]\text{and}\\ \left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right]=\left[\begin{array}{cc}{R}_{c{u}^{\prime },v^{\prime }}& T_{c{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right];\end{array}$

where R.sub.gu′,v′ is the rotation matrix of the matrix

[00036]$\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right],T_{g{u}^{\prime },v^{\prime }}$

is the translation vector of the matrix

[00037]$\left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right],R_{cg}$

is the rotation matrix of the hand-eye matrix H.sub.cg, T.sub.cg is the translation vector of the hand-eye matrix H.sub.cg, R.sub.cu′,v′ is the rotation matrix of the matrix

[00038]$\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right],T_{c{u}^{\prime },v^{\prime }}$

is the translation vector of the matrix

[00039]$\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right].$

[0106] Letting

[00040]$A=H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }},B=H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1},X=H_{cg},$

then hand-eye calibration equation can be expressed as:

[00041]$AX=XB$

[00042]$C_n^2$

hand-eye calibration equation can be established by the n′ pluralities of calibration data.

Step S3.2: Calculating the Hand-Eye Matrix by Using a Singular Value Decomposition

[0107] based on the properties of Kronecker product, transforming the hand-eye calibration equation into a least squares problem, and calculating the hand-eye matrix H.sub.cg by using a singular value decomposition

[0108] Based on the properties of Kronecker product, expanding the hand-eye calibration AX=BX, a homogeneous equation can be obtained :

[00043]$\begin{array}{c}AX=XB\\ \left[H_{g{v}^{\prime }}^{-1}{H}_{g{u}^{\prime }}\right]H_{cg}=H_{cg}\left[H_{c{v}^{\prime }}{H}_{c{u}^{\prime }}^{-1}\right]\\ \left[\begin{array}{cc}{R}_{g{u}^{\prime },v^{\prime }}& T_{g{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_{cg}& T_{cg}\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}{R}_{cg}& T_{cg}\\ 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_{c{u}^{\prime },v^{\prime }}& T_{c{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right]\\ \left[\begin{array}{cc}{R}_{g{u}^{\prime },v^{\prime }}{R}_{cg}& R_{g{u}^{\prime },v^{\prime }}{T}_{cg}+T_{c{u}^{\prime },v^{\prime }}\\ 0& 1\end{array}\right]=\left[\begin{array}{cc}{R}_{cg}{R}_{c{u}^{\prime },v^{\prime }}& R_{cg}{T}_{c{u}^{\prime },v^{\prime }}+T_{cg}\\ 0& 1\end{array}\right]\end{array}$

[0109] Namely Formula 1 and formula 2:

[00044]$\begin{array}{cc}\left\{\begin{array}{c}{R}_{g{u}^{\prime },v}{R}_{cg}=R_{cg}{R}_{c{u}^{\prime },v^{\prime }}\\ R_{g{u}^{\prime },v^{\prime }}{T}_{cg}+T_{g{u}^{\prime },v^{\prime }}=R_{cg}{T}_{c{u}^{\prime },v^{\prime }}+T_{cg}\end{array}\right)& \text{­­­Formula 1 Formula 2}\end{array}$

[0110] Where R.sub.cu′,v′, R.sub.cg, R.sub.gu′,v′ are rotation matrices and belong to a special orthogonal group and are closed in multiplication.

[0111] Performing the following transformations on Formula 1:

[00045]$\begin{array}{c}{R}_{g{u}^{\prime },v^{\prime }}{R}_{cg}I=I{R}_{cg}{R}_{c{u}^{\prime },v^{\prime }}\\ vec\left(R_{g{u}^{\prime }.v^{\prime }}{R}_{cg}I\right)=vec\left(I{R}_{cg}{R}_{c{u}^{\prime },v^{\prime }}\right)\\ \left(R_{g{u}^{\prime },v^{\prime }}.Math.I^T\right)vec\left(R_{cg}\right)=\left(I.Math.R^T{}_{c{u}^{\prime },v^{\prime }}\right)vec\left(R_{cg}\right)\\ \left(R_{g{u}^{\prime },v^{\prime }}.Math.I-I.Math.R^T{}_{c{u}^{\prime },v^{\prime }}\right)vec\left(R_{cg}\right)=0\end{array}$

[0112] Then obtained:

[00046]$\left(R_{g{u}^{\prime },v^{\prime }}.Math.I-I.Math.R^T{}_{c{u}^{\prime },v^{\prime }}\right)vec\left(R_{cg}\right)=0$

[0113] Where I is a unit matrix, .Math. is the operator of Kronecker product, vec is the operator of vectorization.

[0114] So a linear equation of Cx=0 is obtained, thus we can transform the linear equation into a least squares problem by using singular value decomposition (SVD). For:

[00047]$\begin{array}{l}\left(R_{gu,v}.Math.I-I.Math.R^T{}_{cu,v}\right)=U{\displaystyle .Math.V^T},U^{T}U=V^{T}V=I,\\ \text{Σ}=diag\left({\sigma }_1,{\sigma }_2,.Math.,{\sigma }_9\right),{\sigma }_1\ge {\sigma }_2\ge .Math.\ge {\sigma }_9\ge 0\end{array}$

we can obtain the following least squares problem:

[00048]$\begin{array}{l}\min {‖\left(R_{gu,v}.Math.I-I.Math.R^T{}_{cu,v}\right)\cdot vec\left(R_{cg}\right)‖}_2^2=\\ \min {‖U{\displaystyle .Math.V^{T}vec\left(R_{cg}\right)}‖}_2^2=\min {‖{\displaystyle .Math.V^{T}vec\left(R_{cg}\right)}‖}_2^2\\ s.t.{‖vec\left(R_{cg}\right)‖}_2=1\end{array}$

[0115] Then obtained:

[00049]$\begin{array}{c}\min {‖{\displaystyle .Math.\cdot V^{T}vec\left(R_{cg}\right)}‖}_2^2\\ s.t.{‖vec\left(R_{cg}\right)‖}_2=1\end{array}$

[0116] Now the primal problem is converted into:

[00050]$\min {‖{\left(V^{T}vec\left(R_{cg}\right)\right)}^T{\displaystyle {.Math.}_{}^T{\displaystyle .Math.\left(V^{T}vec\left(R_{cg}\right)\right)}}‖}_2^2=\min {‖y^T{\displaystyle {.Math.}_{}^T{\displaystyle .Math.y}}‖}_2^2$

[0117] Where

[00051]$y^T{\displaystyle {.Math.}_{}^T{\displaystyle .Math.y}}={\sigma }_1^2{y}_1^2+{\sigma }_2^2{y}_2^2+$

[00052]$+{\sigma }_9^2{y}_9^2,{\sigma }_1\ge {\sigma }_2\ge .Math.\ge {\sigma }_9\ge 0.$

[0118] For there has a constraint of ||y||.sub.2 = 1, the minimal solution is y = [0 0 ... 1].sup.T, namely, the solution of Cx=0 is the last column of the matrix V:

[00053]$vec\left(R_{cg}\right)=Vy=\left[\begin{array}{cccc}{v}_1& v_2& .Math.& v_9\end{array}\right]\left[\begin{array}{c}0\\ 0\\ .Math.\\ 1\end{array}\right]=v_9$

[0119] Therefore, the calculation of the hand-eye matrix H.sub.cg in the present invention is: [0120] Firstly, establishing a linear equation set: (R.sub.gu′,v′ .Math.I-I.Math. R.sup.T.sub.cu′,v′) .Math. vec(R.sub.cg) = 0, where I is a unit matrix, .Math. is the operator of Kronecker product, vec is the operator of vectorization; [0121] placing all matrices (R.sub.gu′,v′ .Math.I-I.Math. R.sup.T.sub.cu′,v′) of all any two measurement poses by column to obtain a matrix R; [0122] performing a singular value decomposition on the matrix R to obtaining matrix V, namely the right singular matrix of the matrix R, and taking out the 9 elements of last column of matrix V to revert to a matrix R̅.sub.cg of 3×3; [0123] performing a singular value decomposition on the matrix where U.sub.R is the left singular matrix of the matrix R̅.sub.cg, Σ.sub.R is the singular value matrix of the matrix R̅.sub.cg, V.sub.R is the singular matrix of the matrix R̅.sub.cg, then obtaining the rotation matrix R.sub.cg of the hand-eye matrix

[0124] So the rotation matrix R.sub.cg can be calculated from Formula 1. In order to guarantee the Unit orthogonality, the rotation matrix R.sub.cg has been Orthogonalized by adopting Rodrigues’ rotation formula, as to eliminating the influence of measurement noise. Among Rodrigues’ rotation formula, Σ.sub.R can reflect the quality of the calibration to a certain extent. For good calibration, all elements of Σ.sub.R should be very close, even completely equal.

[0125] Lastly, through above-mentioned Formula 2, we can obtain:

[00056]$\begin{array}{l}{R}_{g{u}^{\prime },v^{\prime }}{T}_{cg}+T_{g{u}^{\prime },v^{\prime }}=R_{cg}{T}_{c{u}^{\prime },v^{\prime }}+T_{cg}\\ \left(R_{g{u}^{\prime },v^{\prime }}-I\right)T_{cg}=R_{cg}{T}_{c{u}^{\prime },v^{\prime }}-T_{g{u}^{\prime },v^{\prime }}\end{array}$

[0126] Placing all matrices (R.sub.gu′,v′ - I) of all any two measurement poses by column to obtain a matrix R.sub.g, placing all matrices (R.sub.cgT.sub.cu′,v′ - T.sub.gu’,v′) of all any two measurement poses by column to obtain a matrix T′, then calculating the translation vector of the hand-eye matrix H.sub.cg : T.sub.cg = R.sub.g .sup.-1T′ .

[0127] Thus, the rotation and translation matrix of the binocular camera coordinate system to the robot’s end flange coordinate system, the hand-eye matrix H.sub.cg is calculated and taken as the initial position of the subsequent particle swarm optimization algorithm.

Step S3.3: Based on Minimizing a Reprojection Error, Optimizing the Hand-Eye Matrix by Using a Particle Swarm Optimization Algorithm

Step S3.3.1: Creating a Particle Swarm and Taking the Hand-Eye Matrix as Initial Positions of All Particles

[0128] Creating a particle swarm with population size of K′, where the position and the velocity of the k′.sup.th particle are denoted by p.sub.k′ and v.sub.k′, respectively, then initializing the positions p.sub.k′ and the velocities v.sub.k′ of all K′ particles:

[00057]$p_{k^{\prime }}=H_{cg}$

[00058]$v_{k^{\prime }}=rand\left(v_{\min },v_{\max }\right)$

where k′ = 1,2,..., K′, v.sub.min, v.sub.max are the upper threshold and the lower threshold of velocity at each iteration of particle, rand () is rand function;

Step S3.3.2: Establishing a Reprojection Error Cost Function as the Fitness Function of the Particle Swarm Optimization Algorithm

[00059]$\phi \left(p_{k^{\prime }}\right)={\displaystyle \underset{i^{\prime }=1,j\ne i^{\prime }}{\overset{n^{\prime }}{.Math.}}{‖P_{i^{\prime }}-\frac{1}{s_{i^{\prime }}}{\text{K}}^{\ast }\exp {\left[p_{k^{\prime }}{H}_{g{j}^{\prime }}{H}_{g{i}^{\prime }}^{-1}{p}_{k^{\prime }}{}^{-1}\right]}_{3\times 4}{P}_{j^{\prime }}‖}_2^2}$

where s.sub.i′ is the scale factor of the i′.sup.th measurement pose, K* is the intrinsic parameter matrix of the binocular camera, exp[.Math.].sub.3×4 is an operator of choosing a submatrix of 3×4 of the left upper corner of the matrix in square bracket, || ||.sub.2 is a square-normal operator, P.sub.j′ is the position of the corner of the checkerboard of any non-i.sup.th measurement pose in camera coordinate system.

[0129] P = [x, y, z].sup.T is the position of a corner of the checkerboard (calibration board) in the world coordinate system W, B is the robot base coordinate system, P.sub.j′ is the position of the corner of the checkerboard of the j′.sup.th measurement pose in camera coordinate system. Their relationship can be express as:

[00060]$\left[\begin{array}{c}{P}_{j^{\prime }}\\ 1\end{array}\right]=H_{c{j}^{\prime }}\left[\begin{array}{c}P\\ 1\end{array}\right]$

[0130] As shown in FIG. 2, P.sub.i′ is the position of the corner of the checkerboard of the i′.sup.th measurement pose in camera coordinate system. Thus we can transform the position P.sub.j′ into the position P.sub.i′j′ under the i′.sup.th measurement pose, and according to the following hand-eye relationship:

[00061]$\begin{array}{c}{H}_{g{i}^{\prime }}X{H}_{c{i}^{\prime }}=H_{c{j}^{\prime }}X{H}_{g{j}^{\prime }}\\ H_{c{i}^{\prime }}{H}_{c{j}^{\prime }}^{-1}=X{H}_{g{j}^{\prime }}{H}_{g{i}^{\prime }}^{-1}{X}^{-1}\end{array}$

[0131] We can obtain the following expression of the position P.sub.i′j′:

[00062]$\left[\begin{array}{c}{P}_{i^{\prime }{j}^{\prime }}\\ 1\end{array}\right]=H_{c{i}^{\prime }}{H}_{c{j}^{\prime }}^{-1}\left[\begin{array}{c}{P}_{j^{\prime }}\\ 1\end{array}\right]=X{H}_{g{j}^{\prime }}{H}_{g{i}^{\prime }}^{-1}{X}^{-1}\left[\begin{array}{c}{P}_{j^{\prime }}\\ 1\end{array}\right]$

where

[00063]$X=H_{cg}.$

[0132] We can project the position P.sub.i′j′ to the imagine from the i′.sup.th measurement pose by using the calibration parameters of the binocular camera and obtain a image coordinate

[00064]${P^{\prime }}_{i^{\prime }{j}^{\prime }}.$

Namely:

[00065]$\begin{array}{c}{P^{\prime }}_{i^{\prime }{j}^{\prime }}={\frac{1}{s}}_{i^{\prime }}{\text{K}}^{\ast }{P}_{i^{\prime }{j}^{\prime }}\\ {P^{\prime }}_{i^{\prime }{j}^{\prime }}={\frac{1}{s}}_{i^{\prime }}{\text{K}}^{\ast }\exp {\left[XB{X}^{-1}\right]}_{3\times 4}\left[\begin{array}{c}{P}_{j^{\prime }}\\ 1\end{array}\right]\end{array}$

where s.sub.i′ is the scale factor of the i′.sup.th measurement pose, K* is the intrinsic parameter matrix of the binocular camera, exp[.Math.].sub.3×4 is an operator of choosing a submatrix of 3×4 of the left upper corner of the matrix in square bracket.

[0133] Denoting the reprojection error of the position P.sub.i′j′ as err, the reprojection error of the position P.sub.i′j′ is:

[00066]$err={‖P_{i^{\prime }}-{P^{\prime }}_{i^{\prime }{j}^{\prime }}‖}_2^2$

[0134] Substituting the expression of the position P.sub.i′j′, we can obtain:

[00067]$err={‖P_{i^{\prime }}-{\frac{1}{s}}_{i^{\prime }}{\text{K}}^{\ast }\exp {\left[XB{X}^{-1}\right]}_{3\times 4}{P}_{j^{\prime }}‖}_2^2$

[0135] Generalized to the n′ pluralities of calibration board images, we can obtain a reprojection error cost function:

[00068]$er{r}^{*}={\displaystyle \underset{i=1,j\ne i}{\overset{n}{.Math.}}{‖P_i-{\frac{1}{s}}_i\text{K}\exp {\left[XB{X}^{-1}\right]}_{3\times 4}{P}_j‖}_2^2}$

[0136] Taking the reprojection error cost function err* as the fitness function φ(p.sub.k′) of the particle swarm optimization algorithm to perform an iteration, we can obtain the optimized hand-eye matrix

[00069]$H_{cg}^{revised}.$

[0137] Step S3.3.3: according to the fitness function φ(p.sub.k′), finding the position of the current individual extremum min

[00070]$\left\{\phi \left(p_{k^{\prime }}\right),\phi \left(p_{k^{\prime }}^{\ast }\right)\right\}$

of each particle and taking it as the historical optimal position

[00071]$p_{k^{\prime }}^{\ast },$

finding the position of the global extremum min {φ(p.sub.k′), k′ = 1, 2,..., K′}, namely the position of the particle of the minimum value of fitness function and taking it as the global optimal position g* :

[00072]$\left\{\begin{array}{c}{p}_k^{\ast }\leftarrow \min \left\{\phi \left(p_k\right),\phi \left(p_k^{\ast }\right)\right\}\\ {\text{g}}^{\ast }\leftarrow \min \left\{\phi \left(p_k\right),k=1,2,.Math.,K\right\}\end{array}\right)$

[0138] Step S3.3.4: updating the velocity v.sub.k′ and position p.sub.k′ of each particle:

[00073]$\left\{\begin{array}{c}{v}_{k^{\prime }}=\omega v_{k^{\prime }}+c_{1}rand\left(0,1\right)\times \left(p_{k^{\prime }}^{\ast }-p_{k^{\prime }}\right)+c_{2}rand\left(0,1\right)\times \left(g^{\ast }-p_{k^{\prime }}\right)\\ p_{k^{\prime }}=p_{k^{\prime }}+v_{k^{\prime }}\end{array}\right)$

where ω is a inertia factor, c.sub.1 and c.sub.2 are acceleration factors;

[0139] returning Step S3.3.3, until a termination condition is reached;

[0140] Step S3.3.5: taking the global optimal position g* as the optimized hand-eye matrix, which is denoted by

[00074]$H_{cg}^{revised};$

Step S3.4: Performing a 3D Measurement

[0141] Step S3.4.1: adjusting the binocular camera of the structured light measuring equipment so that it can clearly capture the measured object, the left and right cameras can be kept in the same horizontal position with a certain distance, and the binocular calibration can be completed; adjusting the robot so that it can carry the structured light measuring equipment to perform the 3D measurement and guarantee the full view of the measured object will be taken;

[0142] Step S3.4.2: after receiving the planned measurement paths from the virtual simulation platform, moving the robot in turn to the measurement poses of the planned measurement paths;

[0143] Step S3.4.3: as shown in FIG. 5, for the measurement pose .sub.kP.sub.ij, calculating a pose matrix

[00075]${}_k{H}_{ij}^g$

from the coordinate system of the robot’s end flange to the robot base coordinate system according themeasurement pose .sub.kP.sub.ij, then obtaining a rigid body transformation matrix:

[00076]${}_k{H}_{ij}^{pcl}{=}_k{H}_{ij}^g\text{*}{H}_{cg}^{revised}$

then obtaining a single-view point cloud .sub.kI.sub.ij of the measured object through the binocular camera of the structured light measuring equipment and transforming it from the camera coordinate system to the robot base coordinate system to obtain a registered single-view point cloud

[00077]${}_k{I}_{ij}^{homo}:$

[00078]${}_k{I}_{ij}^{homo}{=}_k{H}_{ij}^{pcl}\text{*}{}_k{I}_{ij}$

when registered single-view point clouds of all measurement poses are obtained, the points under the robot base coordinate system form a complete point cloud of the measured object, the 3D the measurement is completed.

Example 1

[0144] The measured object is a 600 mm×450 mm×200 mm cuboid. Firstly, establishing a 1:1 virtual simulation platform on a computer, which has been installed an Ubuntu operating system and an open source robot operating system (ROS) according to a real measurement environment. In the example, the virtual simulation platform is shown in FIG. 6. A structured light measuring equipment and a depth camera, which are the same as that of real measurement environment are installed on the fixture of the robot’s end flange. The robot is a six degrees of freedom (DOF) robot. Then, planning measurement paths on the virtual simulation platform according to the method described in Step 2. After a measurement surface is completed, another measurement surface is selected to perform according to the method described in Step 2, until all measurement paths of all measurement surfaces are planned. The messages of the planned measurement paths are sent to the six DOF robot of real measurement environment.

[0145] In the real measurement environment, the checkerboard (calibration board) is 8 × 11 with square size of 15 mm.

[0146] In the example, Firstly, operating the six DOF robot to carry the binocular camera of the structured light measuring equipment to 9 different measurements poses and taking 9 calibration board images with resolutions of 3000 × 4096. Then, according to the calibration method of Zhang, calculating 9 extrinsic parameter matrices H.sub.ci′, i′ = 1,2,...,9 of the 9 calibration board images, and recording the pose information of the six DOF robot and the position P.sub.i′ of the corner in camera coordinate system. Meanwhile, calculating 9 robot pose matrix H.sub.gi′, i′ = 1, 2,...,9 . Thus 36 hand-eye calibration equations to be solved are established.

[0147] Then, based on the properties of Kronecker product, transforming the hand-eye calibration equation into a least squares problem, and calculating the hand-eye matrix H.sub.cg by using a singular value decomposition. In the example, the hand-eye matrix H.sub.cg is:

[00079]$\begin{array}{l}{H}_{cg}=\left[\begin{array}{cc}{R}_{cg}& T_{cg}\\ 0& 1\end{array}\right]=\\ \left[\begin{array}{cccc}0.0429661& 0.9988763& 0.0200075& 20.1263\\ \text{-}0.9990677& 0.0428734& 0.0050404& 166.462\\ 0.0041772& \text{-}0.0201986& 0.9997872& 361.342\\ 0& 0& 0& 1\end{array}\right]\end{array}$

[0148] Where the rotation matrix of the hand-eye matrix H.sub.cg is the matrix of 3 × 3 of the left upper corner, namely:

[00080]$R_{cg}=\left[\begin{array}{ccc}0.0429661& 0.9988763& 0.0200075\\ \text{-}0.9990677& 0.0428734& 0.0050404\\ 0.0041772& \text{-}0.0201986& 0.9997872\end{array}\right]$

the translation vector of the hand-eye matrix H.sub.cg is the column vector of 3×1 of the right upper corner, namely:

[0149] the rotation matrix of the hand-eye matrix H.sub.cg is the matrix of 3×3 of the left upper corner, namely:

[00081]$T_{cg}=\left[\begin{array}{c}20.1263\\ 166.462\\ 361.342\end{array}\right]$

[0150] The calculated hand-eye matrix H.sub.cg is taken as the initial position of the subsequent particle swarm optimization algorithm. In the example, the all positions of K′ particles in initial population are the calculated hand-eye matrix H.sub.cg .

[0151] In the example, the optimized hand-eye matrix

[00082]$H_{cg}^{revised}$

is:

[00083]$H_{cg}^{revised}=\left[\begin{array}{cccc}0.043376& 0.998763& 0.0203564& 19.6059\\ \text{-}0.998952& 0.0452435& \text{-}0.00692214& 162.861\\ \text{-}0.00599257& \text{-}0.0206492& 0.9997669& 350.159\\ 0& 0& 0& 1\end{array}\right]$

[0152] As shown in FIG. 7, after optimizing the hand-eye matrix H.sub.cg through the particle swarm optimization algorithm, we can calculate the position of calibration origin under the robot base coordinate system. Comparing the position distribution of calibration origins of n′ measurement poses, we can see that calibration origins (black dots) after optimization are relatively concentrated and almost overlapped; however calibration origins (hollow dots) before optimization are relatively dispersed. The comparison proves the improvement of hand-eye calibration, and the effect will be verified in subsequent 3D measurement.

[0153] After the optimized hand-eye matrix

[00084]$H_{cg}^{revised}$

is obtained, the six DOF robot carries the structured light measuring equipment to a measurement pose according to the planned measurement paths to taking imagines of the measured object, then a single-view point cloud of the measured object is obtained. The measurement poses and corresponding single-view point clouds on the 1.sup.st measurement surface (k=1) of the measured object are shown in FIG. 8, and registered single-view point clouds of measurement surfaces in robot base coordinate system are shown in FIG. 9. For FIG. 8 and FIG. 9, we can see that the single-view point clouds of the same measurement surface can be registered. As shown in FIG. 10, a complete point cloud of the measured object is obtained through the present invention.

Example 2

[0154] In the example, the measured object is complex mechanical component. As shown in FIGS. 11, the complete point cloud after optimizing the hand-eye matrix can embody more shape characteristics, which means that the complete point cloud after optimizing the hand-eye matrix can represent more abundant surface information and is beneficial to perform the subsequent process of point cloud high efficiently.

[0155] While illustrative embodiments of the invention have been described above, it is, of course, understand that various modifications will be apparent to those of ordinary skill in the art. Such modifications are within the spirit and scope of the invention, which is limited and defined only by the appended claims.