VISUO-HAPTIC SENSOR

Abstract

The present disclosure presents a visuo-haptic sensor, which is based on a passive, deformable element whose deformation is observed by a camera. In particular, the improved simplified sensor may determine the force and/or torque applied to a point of the sensor in multiple spatial dimensions.

Claims

1. An apparatus, comprising: a haptic element comprising at least one passive elastically deformable element deformable in multiple dimensions; at least one camera external to the haptic element and operable to capture images of the elastically deformable element; and a processor operable to determine a change of pose of the elastically deformable element by visual tracking and determine a measurement of force and torque applied to the elastically deformable element based on the captured images and the determined change of pose, wherein the measurement includes at least three components comprising forces and/or torques.

2. The apparatus of claim 1 wherein the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions.

3. The apparatus of claim 1, wherein the haptic element further comprises a rigid base element and a rigid front element coupled via the elastically deformable element; and wherein the processor is further operable to determine the change of pose of the rigid front element with respect to the base element.

4. The apparatus of claim 3, wherein the deformable element comprises a beam and/or a tool which is mounted to the front element.

5. The apparatus of claim 3, wherein the rigid base element and rigid front element comprise planar structures.

6. The apparatus of claim 1, wherein the change of pose is determined by observing at least two points on the haptic element.

7. The apparatus of claim 1, wherein the change of pose is determined by observing three or more points on the haptic element.

8. The apparatus of claim 1, further comprising a visual tracker to determine the change of pose.

9. The apparatus of claim 1, wherein the haptic element is mechanically attached to a movable robotic element and the movable robotic element is arranged in a field of view of the camera.

10. A method, comprising: providing a haptic element comprising at least one passive elastically deformable element deformable in multiple dimensions; capturing images of the elastically deformable element with at least one camera; determining a change of pose of the elastically deformable element by visual tracking; and determining a measurement of force applied to the elastically deformable element based on the captured images and the determined change of pose, wherein the measurement of force includes at least three dimensions including both force and torque components.

11. The method of claim 10, wherein the measurement comprises a 6-axis force and/or torque vector comprising forces along all three spatial dimensions and torques along all three spatial dimensions.

12. The method of claim 10, wherein the step of determining the change of pose comprises observing at least two points on the haptic element.

13. The method of claim 10, wherein the step of determining the change of pose comprises observing three or more points on the haptic element.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0035] FIG. 1a illustrates a haptic/tactile sensor according to the prior art.

[0036] FIG. 1b illustrates an exemplary haptic sensor comprising a haptic element mounted between a robot arm and a gripper.

[0037] FIG. 2a illustrates an exemplary apparatus 1 with a guided spring 31 and a metal rod/beam 32 as deformable elements.

[0038] FIG. 2b illustrates a sensor according to FIG. 2a when an external force and/or torque 10 is applied to the tool 9.

[0039] FIG. 3 illustrates a sensor according to FIG. 2a with a bit used as tool 9.

[0040] FIG. 4a illustrates a 6D force and/or torque sensor with an unguided spring 31 and a beam 32 as deformable elements.

[0041] FIG. 4b illustrates the sensor according to FIG. 4a when an external force 10 is applied to the tool 9

[0042] FIG. 5a illustrates a 6D force and/or torque sensor with an unguided spring 3 as the deformable element.

[0043] FIG. 5b illustrates the sensor according to FIG. 5a when an external force and torque 10 are applied to the tool 9.

[0044] FIG. 6 illustrates a 6D force and/or torque sensor based on two disks that are connected by deformable elements.

[0045] FIG. 7 illustrates the accuracy of the beam-based sensor being tested by applying a force with a Kuka LWR arm along F. Error bars indicate the standard deviation and are scaled by a factor of 2 for F.sub.z and 5 otherwise.

[0046] FIG. 8 illustrates the opening of a crown cap with a bottle opener used as tool 9 and mounted on a beam-based visuo-haptic sensor. Planar structures 80, 81 are used to determine the pose change of deformable element 31.

[0047] FIG. 9 illustrates a sensor according to FIG. 3 mounted on a robot arm to measure forces and/or torques while mounting a screw.

DETAILED DESCRIPTION

Combined Detector and Tracker

[0048] A combined tracker/detector for templates on planar structures 80,81,82 is used to find and follow a texture with an image or video provided by a camera 4. It is based on scale-invariant feature transform (SIFT) for feature detection and visual template tracking such as efficient second-order minimization (ESM) for tracking and refinement. Thus, the exact 6D pose of objects can be determined in realtime, as long as a model of the templates, i.e., an image with scale information, is available. The scale is determined from the dpi (dots-per-inch) value of the image file. The system detects/tracks multiple arbitrary planar templates such as printed designs or natural photos, simultaneously. Each template model is initialized on startup, i.e. SIFT features are extracted, and the tracker performs pre-calculations. Many artificial objects can be tracked directly using their printed texture as a model. Also, it is straight-forward to create real objects from template models at the exact scale with a standard printer. These templates are easily attached anywhere on a robot or an object. They offer more flexibility, ease of use and a higher tracking quality than 3D models or the pre-defined markers from the open source computer tracking library ARtoolkit.

[0049] The system relies on a GPU implementation of SIFT to search for the template models in the image. This step is too slow for processing at a framerate of 30 Hz on mid-range GPUs and should be avoided, if possible. Tracking is performed with ESM, which is either initialized from SIFT or from the pose of the previous frame. The latter case is preferred, since it allows skipping SIFT detection. Poses from ESM are much more accurate and less noisy than from SIFT matches. Tracking is always done with the “original” model, such that there is no drift, even if performed over long periods of time. Processing is split in multiple threads, running at different rates. Thread A performs SIFT detection and matching, if there is currently any template whose pose is unknown. Matching provides an estimate of the homography. The ESM runs in thread B at framerate and tracks all templates, for which initial poses are available. If required, each tracker may run in a separate thread to reduce latency on multi-core CPUs.

[0050] The real-world poses of the templates with respect to the camera 4 is obtained by homography decomposition, given the camera parameters.

Modeling Deformation of Beams

[0051] The deformation of beams or rods under load is described by the Euler-Bernoulli beam theory for small deflections. Here, a force and/or moment 10 perpendicular to the beam is applied at its front element 7, see FIG. 2b.

[0052] The other end of the beam 32 is considered to be fixed (“clamped”). In principle, the beam can be made of elastic material such as spring steel or plastic. Yet, no plastic (i.e. permanent) deformations should occur within the relevant range of deformations. We use a beam 32 made of spring steel with a diameter of about 2 mm and a length of 10-20 cm.

[0053] The deflection curve w(x) in 2D describes the deformation of the beam 32 along the y-axis over the x-axis:

[00001]$\begin{array}{cc}\frac{d^2}{{\mathrm{dx}}^2}.Math.\left(\mathrm{EI}.Math.\frac{d^2.Math.w}{{\mathrm{dx}}^2}\right)={}_{\mathrm{EI}=\mathrm{const}}.Math.\mathrm{EI}.Math.\frac{d^4.Math.w}{{\mathrm{dx}}^4}=q\left(x\right)& \left(1\right)\end{array}$

[0054] This is the commonly-used approximation for small deflection, which ignores the shortening of the deformed beam 32 along the x-axis. The values for the elastic modulus E and the second moment of area I are constants for uniform and homogeneous beams. For a circular cross section of the beam 32 with radius r,

[00002]$I=\frac{\pi }{4}.Math.r^4.$

There is no distributed load, such that q(x)=0. Quadruple integration of (1) yields four integration constants, which are used to fulfil the boundary conditions. Due to the clamping at x=0, w(0)=w′(0)=0. The derivatives of w have a distinct physical meaning—specifically the moment is M=−EIw″, and the shear force is Q=−EIw′. Therefore, when a force F 10 is applied perpendicular to the beam at its end, boundary conditions are:

[00003]${w^{″}}^{\prime }.Math.{{.Math.}_{x=L}.Math.=-\frac{F}{\mathrm{EI}}.Math..Math.\mathrm{and}.Math..Math.w^{″}.Math.}_{x=L}=0$

[0055] The force 10 is applied at x=L, where L is the length of the beam 32. With four boundary conditions, a unique solution wF can be given for Eqn. (1). Similarly, for a moment or torque M applied at point x=L, a solution wM is determined using the boundary conditions:

[00004]${w^{″}}^{\prime }.Math.{{.Math.}_{x=L}.Math.=0.Math..Math.\mathrm{and}.Math..Math.w^{″}.Math.}_{x=L}=-\frac{M}{\mathrm{EI}}$

[0056] Since the differential equation is linear, the two solutions can be superimposed. For a force and moment applied at ξ=1, with

[00005]$\xi =\frac{x}{L}$

and clamping at x=0, we obtain:

[00006]$\begin{array}{cc}w\left(\xi \right)=w_F+w_M=\frac{{\mathrm{FL}}^3\left(3.Math.{\xi }^2-{\xi }^3\right)}{6.Math..Math.\mathrm{EI}}+\frac{{\mathrm{ML}}^2.Math.{\xi }^2}{2.Math..Math.\mathrm{EI}}=-\frac{{\mathrm{FL}}^3}{6.Math..Math.\mathrm{EI}}.Math.{\xi }^3+\left(\frac{{\mathrm{FL}}^3}{2.Math..Math.\mathrm{EI}}+\frac{{\mathrm{ML}}^2}{2.Math..Math.\mathrm{EI}}\right).Math.{\xi }^2& \left(2\right)\end{array}$

[0057] The deflection of the real beam is observed by a camera 4 at one or multiple points of the curve w. Extension of the discussed 2D case to 3D is straight-forward by separation along y and z, yielding deformation curves wy,wz, see FIG. 2b. Forces along or moments around the x-axis cause only very small deformations, which cannot be observed with a simple beam. A model according to Eqn. (2) is fitted to the observations and solved for F,M to obtain the causing force and torque. In the following, it is assumed that w and w′ are known only at a single point ξ.sub.1ε|0, 1. By rearranging Eqn. (2) and the derivate thereof, a linear expression w=W. f is obtained in matrix form, with w=[w(ξ.sub.1)w′(ξ.sub.1)].sup.T and f=[F M].sup.T. Given an observation w, the acting force/moment f is obtained by inversion of W:

[00007]$\begin{array}{cc}f=W^{-1}.Math.w,W^{-1}=\frac{2.Math..Math.\mathrm{EI}}{L^2.Math.{\xi }^2}\left[\begin{array}{cc}\frac{6}{L.Math..Math.\xi }& -\frac{3}{L}\\ \frac{3.Math.\left(\xi -2\right)}{\xi }& -\left(\xi -3\right)\end{array}\right]={}_{\xi =1}.Math.\frac{2.Math..Math.\mathrm{EI}}{L^2}\left[\begin{array}{cc}\frac{6}{L.Math.}& -\frac{3}{L}\\ -3& 2\end{array}\right]& \left(3\right)\end{array}$

[0058] If multiple observations are available, we obtain an overdetermined system [w|.sub.ξ.sub.1; w|.sub.ξ.sub.2: . . . =[W|.sub.ξ.sub.1; W|.sub.ξ.sub.2: . . . ].Math.f, which is solved for f by least-square minimization. From Eqn. (3), note that F is significantly more sensitive to changes in w than M for the typical case L<1. Also, there is a strong coupling of the two components.

Beam-Based 6D Force and/or Torque Sensor

[0059] FIG. 2a illustrates a design of a beam-based sensor. It is based on the guided spring 31 and the round beam 32 made of spring steel as the deformable element 3. The guiding limits the motion of the spring to compression along/torsion around the x-axis, and is implemented via guiding element 39, gliding in a pipe. Element 39 also connects spring 31 and beam 32. FIG. 2b shows the deflection of the beam/rod resulting from applied forces and/or torque 10. Force/moments along the x-axis deform the spring 31. A camera 4 observes the entire structure and tracks templates T.sub.R,A,B on the planar structures 80,81. Many applications only require a force reading at a single point, but in multiple spatial dimensions—ideally as a full 6D force and/or torque reading. Typically, such sensors are mounted between the end-effector of a robot arm or another actuator and a tool 9—such as a gripper, screwdriver or specialized manipulator. Upon contact, forces and torques 10 imposed by the tool 9 onto an object (or vice versa) must be measured in up to six dimensions. Also, a human could guide the arm by imposing forces onto the tool. Therefore, the present disclosure proposes another design of a visuo-haptic sensor based on a haptic element consisting of metal beam 32 and spring 31 which together deform in 6D when a force and/or torque 10 is applied. Again, the passive component is measured by an external camera 4, which also observes the surrounding scene. It allows measuring forces and/or torque in multiple dimensions at a single point—much like commercial force and/or torque sensor modules. Note that the beam theory can be applied to elongated elements of various inner structure by adaptation of the shape constant I. Therefore, a deformable connection element within the arm—such as the link between the last two joints—could be used as the deformable element 3.

[0060] A photo based on this structure is shown in FIG. 3. The sensor according to FIG. 2a consists of a round rod of spring steel (the beam 32) between guiding element 39 and front element 7, as well as a guided spring 31 between base element 6 and 39. The base element 6 of the sensor is mounted to the actuator, which is typically a robot arm. The tool 9 is mounted on the other end 7. A camera 4 observes the entire sensor from 6 to 7, as well as the scene around it. Forces or moments/torques 10 applied at 7 along or around axes Y,Z lead to a deflection of the beam 32 as described in the above background section describing modeling deformation of beams. The deflections are separable into w.sub.Y and w.sub.Z and then processed as in the 2D case. Note, that w.sub.Y represents forces along Y and moments around Z. Due to the guidance via element 39 the spring 31 is not affected by these forces/moments. Forces or moments along or around X hardly deform the beam 32, since it is very stiff along this direction. Instead, a force along X results in a compression of the spring 31 between 6 and 39 according to Hooke's law. Similarly, a moment around X results in a torsion of the spring 31. The corresponding deformation models are straight-forward, and the stiffness constants are adjusted independently of the beam 32. Using this design, a full 6D force/moment can be read from the beam/spring deformation.

[0061] Observations of the deflection curve of the beam w may be obtained in different ways: An edge tracker may be used to track the two contours along the beam and provide a dense sampling of w. This approach provides a high robustness to occlusion, due to the large number of tracked points. However, point tracking does not provide any reliable depth information, basically limiting force measurements to 1D. Simple features along the beam 32—such as colored spheres—would allow for easier detection, yet still with very limited depth resolution. A template tracker/detector, see the above background section describing a combined detector and tracker, on the other hand, provides a full 6D pose, including accurate depth values and the full rotation, which corresponds to the derivatives of w′.sub.Y,Z. Planar textured templates are attached along the beam 32 at a single fixture point via planar structures 80,81 respectively for this tracker. They may be integrated into the design of the robot case. The pose of the beam 32 at the fixture point and the template are related by a constant rigid transformation determined by the mounting structure. Only a low number of templates can be attached along the rod, resulting in a limited number of observations of w and w′. The acting force/moment 10 is calculated from these observations using (3) as outlined there.

[0062] As indicated in FIGS. 2a and 4a, multiple templates are attached to the sensor in our implementation via planar structures 80,81,82. The reference template .sub.R on structure 80 on the base element 6 provides a local reference frame .sub.R in which all sensor-related transformations are expressed.

[0063] Like that, the pose of the sensor may vary, and the camera 4 can move relative to the sensor. In principle, the reference frame may be fixed to the camera frame or calculated from a robot model. Yet, errors can be reduced significantly, if a reference frame is obtained within the image, close to where measurements are performed. Two templates .sub.A,B are attached along the beam 32, at fixed relative positions ξ.sub.1,2. Since the templates on 81,82 are not aligned perfectly, and the beam 32 might exhibit some plastic deformation, a zeroing procedure is performed at startup, with zero force/moment 10 applied to the sensor. The corresponding resting poses .sub.A0,B0.sup.OO are expressed relative to the frame .sub.R.

[0064] Measurements of the template poses are performed for each video frame and expressed relative to their (constant) resting poses, which in turn are based on the current (variable) reference frame .sub.R. Thus, the poses .sub.A.sup.(A0),.sub.B.sup.(B0) are zero as long as the beam 32 is not deformed—regardless of the current sensor orientation. Since the Y and Z components are processed separately, the respective translational components of .sub.x.sup.(*0 correspond directly to samples of w.sub.Y,Z. The slope is obtained from the 3D rotation matrix R, projected to the respective 2D plane, such that w′.sub.Y=R.sub.1,0/R.sub.0,0 and w′.sub.Z=R.sub.2,0/R.sub.0,0. Finally, the compression and torsion of the spring 31 is obtained from the X-component of the translation and the respective rotation angle. Force and translation—respectively moment and torsion—are related by the spring 31 stiffness constant.

6D Force and/or Torque Sensor with Unconstrained Beam/Spring

[0065] Another embodiment of the 6D force and/or torque sensor based on two passive elastic elements 31,32 is built according to FIG. 4a. FIG. 4b shows the structure of FIG. 4a in a deformed state, when a force F.sub.Y 10 is applied. This design is similar to the aforementioned embodiment i.e. FIG. 2a, except for the fact that the spring 31 is not constrained by a linear guiding. The spring 31 deforms in all six dimensions, when a 3D force and/or a 3D torque 10 is applied to the end-point or tool 9. At the same time, the beam 32 also deforms in multiple dimensions. Since the beam 32 is very stiff along axis X, it effectively deforms only in response to forces and/or torques in four dimensions, namely for F.sub.Y,Z and T.sub.Y,Z. The two deformable elements 31,32 are therefore coupled and deform simultaneously for 4 out of 6 force and/or torque dimensions. Compared to aforementioned beam-based embodiment, this embodiment of the present disclosure advantageously allows for more compact sensors with an even simpler mechanical structure. Also, the deformations of the two elements 31,32 are decoupled in the aforementioned beam-based embodiment, which allows for a separate adjustment of the respective stiffness parameters.

[0066] As shown in FIG. 4a, a spring 31 is mounted at its base to the base element 6, which is also used as a mount. A flexible beam 32 is mounted to the other end of the spring 31 at point 38. The beam 32 is connected to front element 7. A tool 9 is also mounted to the front element 7. It is possible to build both the spring 31 and the beam 32 from one single wire. The material and geometric parameters of the flexible elements are chosen according to design criteria, such as desired force/torque range applied to 9, maximal acceptable deformation and measurement accuracy.

[0067] Templates or any other kind of visual feature that allows for visual 6D pose estimation are placed on planar structures 80,81,82 attached at or near 6, 38 and 7. The 6D poses (position and rotation) of these locations are known from the feature pose and the constant offset between the location and its respective feature. Accurate 6D poses can be obtained, for instance, with a combined tracker/detector, as described above. Placing the features close to the given locations ensures a high pose accuracy. Additional features may be placed anywhere along the flexible elements to improve the accuracy of pose estimation. A camera is placed such that it observes all these features with an image resolution sufficient for the desired force and/or torque resolution. It may be fixed to the base 6, as for camera 42, or fixed to another structure, as for camera 41. In the former case, if the connection between 42 and 6 is known, feature 83 may be omitted. Otherwise, 6 serves as a local reference frame, which the other poses are expressed in.

[0068] The deformations of 31 and 32 are determined individually from the pose change between 6 and 38, or 38 and 7, respectively. Applied forces and/or torques 10 in 6D are determined using a deformation model, as outlined above. Compression of the spring 31 along its major axis caused by F.sub.X, as well as torsion around this axis caused by T.sub.X may be modeled according to Hooke's law. Deformation caused by F.sub.Y,Z and T.sub.Y,Z is modeled by the beam theory, applied individually for both elastic elements 31 and 32. In that case, two measurements are available, which improves measurement accuracy or robustness to partial occlusions. Deformation models may also be calibrated/learned using a generic model by applying test forces and/or torques to the sensor.

6D Force and/or Torque Sensor with Unconstrained Spring

[0069] In another embodiment, see FIG. 5a, the flexible structure consists only of a single element, the spring 3. Compared to embodiment illustrated in FIGS. 2a and 4a, the flexible element 32 is missing. The tool 9 is connected to the front element 7 at the end of the spring—either directly, or with a rigid structure. Since the end of the spring deforms in 6D, a 6D force and/or torque 10 applied to 9 is mapped to a 6D deformation of the spring 3, which is expressed by a 6D transformation of 7 with respect to the non-deformed state. The force and/or torque 10 is derived by a deformation model as outlined with respect to FIGS. 4a and 4b, except that there is no second measurement from 32. The structure is depicted in a deformed state in FIG. 5b, when a force F.sub.Y and a torque T.sub.Z are applied simultaneously.

6D Force and/or Torque Sensor Based on Disks Connected by Deformable Elements

[0070] FIG. 6 illustrates a 6D force and/or torque sensor based on two disks 61, 71 that are connected by deformable or elastic elements 33. In this implementation, disk 61 serves both as the base element 6 and as the planar structure 80. Disk 71 serves as the front element 7 and the planar structure 81. The two disks are connected via elastic elements 33. Assuming that the sensor is fixed on 61, a 6D force and/or torque 10 applied to disk 71 or to tool 9 attached to 71 is measured by the change of pose of 71 with respect to 61. The external structure resembles that of commercial 6D force and/or torque sensors such as the JR3, and can thus replace corresponding sensors in existing setups of robotic systems. In typical applications, the sensor is mounted to a robotic arm or the end of a kinematic chain at 61. A gripper, manipulator or tool 9 is mounted to the disk 71.

[0071] The two disks are made of a rigid material and serve as mounting plates. They can be of arbitrary shapes, whereby planar structures are most feasible. The disks are connected by one or several elastic elements 33, such as springs, rods of rubber or rubber foam. The elements should exhibit an elastic deformation behavior. The dimensions, arrangement and stiffness of these elements are chosen according to the required 6D sensitivity and measurement range of the sensor. There is a direct relationship between pose changes (in 6D) and a 6D force and/or torque 10 applied to 71, which is referred to as deformation model. This model can be obtained analytically, by simulation or by calibration. In the latter case, known force and/or torque values are applied, and the observed deformation is stored.

[0072] In case of a single beam-like deformable element, the relationship is expressed as discussed in the above background section describing modeling deformation of beams. The 6D poses of disks 61 and 71 are determined by at least one camera, such as cameras 41,42, based on features, templates or patterns on the disks 61,71. For instance, the pose of an arbitrary texture may be found as described in the above background section describing a combined detector and tracker. An even simpler approach is to find at least four uniquely identifiable features on each disk. If a depth camera is used, the poses can also be derived from the 3D surfaces or structure. A sufficient part of both disk must be observable for pose estimation.

[0073] In case of a camera position similar to camera 41, disk 71 could be made smaller, or it could be made partly transparent. In the latter case, parts of disk 61 that are seen through 71 are refracted. Displacements caused by refraction must be compensated. In case of camera arrangement according to camera 42, the inner surfaces of both disks 61,71 can be observed, except for parts that are occluded by the deformable elements 33. The pose of 71 is always expressed in the coordinate frame of 61, which allows both the cameras 41,42 and the entire sensor to move. A reference or zero pose is obtained when no external force 10 is applied to 71 or the tool 9. When an external force or torque 10 is applied, the pose of 71 changes from this reference pose. This change is converted to a force and/or torque based on the above-mentioned deformation model.

[0074] In preferred implementations, the sensor is built according to FIG. 2a and FIG. 6.

Experiments with the Beam-Based Sensor

Measurement Precision

[0075] The accuracy of the beam-based sensor is analyzed in an experiment with the sensor depicted in FIG. 3. The mechanics of the sensor are build according to FIG. 2a, whereby the spring steel beam 32 between point 39 and 7 exhibits a length of 12 cm and a diameter of 1.5 mm. Its material parameters for Eqn. (2) are E=206 GPa; I=1/4×r.sup.4=2:48. 10.sup.−13 m.sup.4. The planar structures 80 with a template of 4×4 cm is attached to the base element 6 as a reference, and two more planar structures 81,82 are attached to the beam 32 at a single point. The frame is fixed at base element 6, and a robot arm pushes against the sensor tip at the front element 7 along F.sub.Y. The applied force is increased stepwise from zero to a maximum value, and then again decreased in the same fashion. Force and position are recorded using precise arm sensors as a reference. A webcam with 1600×896 px observes the templates on planar structures 80,81,82 from above (along −Z) and determines 6D force and/or torque 10 values according to Eqn. (3).

[0076] The obtained results are shown in FIG. 7. Since the force 10 is applied only along F.sub.Y, it is expected that all other force and torque components 103 remain zero. Indeed, crosstalk to other components is small. The moment M.sub.Z, which is mathematically strongly coupled with F.sub.Y, deviates from zero only for large forces. The force F.sub.X exhibits a low level of coupling, since deformations along Y also shorten the beam 32 along the X-direction. This effect is currently not modeled. Also, a small systematic error, i.e. a deviation from the ideal line, is observed. This may be attributed to incorrect material parameters and to the approximations used in beam theory. A compensation of this error is straight-forward using a calibration process. Contrary to foam-based sensors, there is no hysteresis effect when increasing the force, curve 101, or decreasing the force, curve 102. As expected, sensor noise increases with increasing camera distance. However, the noise level is very low, except for the F.sub.z component. In order to increase accuracy for F.sub.Z—which should be done for larger distances—either a depth/stereo camera and tracker could be used, or observations from two cameras at different locations could be combined.

Robotic Manipulation

[0077] Opening a crown cap, as used on many beverage bottles, is a good example for a manipulation operation which relies on joint visuo-haptic perception. For this experiment, a standard bottle opener is mounted onto a beam-based sensor according to FIG. 2a, see FIG. 8. The sensor is mounted on the end-effector of a Kuka LWR arm, together with a webcam, camera 4. Beam deformations caused by forces or torques 10 applied to the crown cap are observed using the template pictures on planar structures 80,81. Furthermore, the webcam observes the scene near the tool 9 in order to search for the bottle. The visuo-haptic sensor is the only sensor used in this system—the dedicated force and/or torque sensors of the robot arm are not used.

[0078] Bottles are recognized and tracked using the texture on their crown caps with a template-based detector/tracker. As soon as the user selects the desired bottle, the robot scans for it within its working space. Once the object has been detected and localized, the arm moves the tool 9 above the object and rotates it as required for the uncapping operation. The accuracy of this rotation movement is limited by two factors: Since the wrist joint of the arm is relatively far from the tool 9, a large motion is required in the joint space. This goes along with a larger positioning error (which could, however, be compensated by visual servoing). Yet, the used 2D tracker has only a limited depth accuracy—the height of the crown cap cannot be determined exactly. The alignment of the tool 9 on the crown cap must be very accurate in all dimensions in order to perform the uncapping successfully.

[0079] This level of precision is impossible using pure vision, especially for the depth direction. The exact position of the opener is determined haptically, by measuring the forces or moments with the visuo-haptic sensor. Refinement is first performed along the z-axis (height), the x-axis and finally they-axis. Once the tool 9 is aligned, the arm rotates the tool 9 around its center point. If the crown cap has been removed successfully, the tracker/detector will no longer detect its texture on top of the bottle.