Grasp modelling

Abstract

A method of generating a configuration of a robotic hand for automatically grasping a first object, the robotic hand comprising a plurality of parts, is provided. The method comprises: receiving data representing the first object; receiving a plurality of first models generated based upon an example grasp of a second object, the example grasp being based upon a configuration of the robotic hand for grasping the second object in which a plurality of parts of said hand contact said second object, each of said plurality of first models representing a relationship between a respective part of the robotic hand and a property of the second object associated with said part of the robotic hand; and processing the data representing the first object based upon the plurality of first models to determine said configuration of the robotic hand for automatically grasping the first object.

Claims

1. A method of generating a configuration of a robotic hand for automatically grasping a first object, the robotic hand comprising a plurality of parts, the method comprising: receiving data representing the first object; receiving a plurality of first models generated based upon an example grasp of a second object, the example grasp being based upon a configuration of the robotic hand for grasping the second object in which a plurality of parts of said hand contact said second object, each of said plurality of first models representing a relationship between a respective part of the robotic hand and a property of the second object associated with said part of the robotic hand, wherein the relationship is independent of any other part of the robotic hand; receiving a second model generated based upon the example grasp, the second model representing a relationship between the plurality of parts of the robotic hand; and processing the data representing the first object based upon the plurality of first models to determine said configuration of the robotic hand for automatically grasping the first object.

2. A method according to claim 1, wherein processing the data representing the object based upon the plurality of first models comprises: determining a relationship between each of said plurality of first models and said data representing the first object associated with a plurality of locations of said first object; wherein said configuration of the robotic hand for automatically grasping the first object is determined based upon the determined relationships for the plurality of first models.

3. A method according to claim 2, wherein determining a relationship between each of the plurality of first models and the data representing the first object comprises, for each of said first models: determining correspondence between the property of the second object associated with the respective part of the robotic hand and a property of each of the plurality of locations of the first object; wherein said configuration of the robotic hand for automatically grasping the first object is determined based upon the determined correspondence.

4. A method according to claim 1, wherein the property of the second object is a property generated based upon data representing the second object associated with a region of the second object that has a predetermined spatial relationship with the respective part of the robotic hand when the second object is grasped in the example grasp.

5. A method according to claim 4, wherein the predetermined spatial relationship is based upon contact with the respective part of the robotic hand.

6. A method according to claim 1, wherein each of the received plurality of first models represents a relationship between a spatial arrangement of the respective part of the robotic hand relative to the second object and the property of the second object associated with the part of the robotic hand.

7. A method according to claim 1, wherein each of the received plurality of first models comprises a probability density function.

8. A method according to claim 1, wherein the property is a property of a surface of the region of the object, wherein the property is based upon curvature of the surface of the region of the object.

9. A method according to claim 1, wherein processing the data representing the first object comprises determining a relationship between at least one of said plurality of first models and a location associated with said first object based upon said second model.

10. A method according to claim 9, wherein determining a relationship between at least one of said plurality of first models and a location associated with said first object based upon said second model comprises: determining a location associated with a selected one of said first models based upon a relationship between the selected one of said plurality of first models and said data representing the first object; determining a configuration of the robotic hand based upon the determined location associated with the selected one of the first models; and determining a location associated with the at least one of said plurality of first models based upon the determined configuration.

11. A method according to claim 10, further comprising generating a score associated with said determined configuration of the robotic hand, said score being determined based upon a relationship between said at least one of said plurality of first models and said data representing the first object associated with said determined location.

12. A method according to claim 11, further comprising: determining a location associated with each of said plurality of first models based upon the determined configuration; wherein said score is determined based upon a relationship between each of said plurality of first models and the associated locations.

13. A method according to claim 11, further comprising generating a plurality of scores based upon a plurality of determined configurations of the robotic hand, each configuration of the robotic hand being based upon a determined location associated with a different selected one of the first models; wherein said configuration of the robotic hand for automatically grasping the first object is determined based upon said plurality of scores.

14. A method according to claim 1, wherein the relationship between the plurality of parts of the robotic hand represented by the second model comprises a relationship between the plurality of parts of the robotic hand when the robotic hand is in contact with the second object during the example grasp.

15. A method according to claim 1, wherein the relationship between the plurality of parts of the robotic hand represented by the second model comprises a relationship between the plurality of parts of the robotic hand prior to the robotic hand contacting the second object during the example grasp.

16. A method according to claim 1, wherein the data representing the first object comprises image data.

17. A method according to claim 1, wherein said received plurality of first models are generated based upon a plurality of example grasps of a second object, wherein each of said plurality of example grasps being associated with a respective second object.

18. A method according to claim 1, wherein at least two of said plurality of first models represent a relationship between the same part of the robotic hand and a property of the second object associated with said part of the robotic hand and wherein said relationship between a respective part of the robotic hand and a property of the second object associated with said part of the robotic hand is based upon a volume defined relative to the part of the robotic hand, and wherein processing the data representing the first object based upon the plurality of first models comprises combining data associated with a plurality of first models using a product of mixtures approach, and wherein said plurality of models for which data is combined is generated using machine learning techniques.

19. A non-transitory computer program comprising computer readable instructions configured to cause a computer to carry out a method of generating a configuration of a robotic hand for automatically grasping a first object, the robotic hand comprising a plurality of parts, the method comprising: receiving data representing the first object; receiving a plurality of first models generated based upon an example grasp of a second object, the example grasp being based upon a configuration of the robotic hand for grasping the second object in which the plurality of parts of said hand contact said second object, each of said plurality of first models representing a relationship between a respective part of the robotic hand and a property of the second object associated with said part of the robotic hand, wherein the relationship is independent of any other part of the robotic hand; receiving a second model generated based upon the example grasp, the second model representing a relationship between the plurality of parts of the robotic hand; and processing the data representing the first object based upon the plurality of models and the second model to determine said configuration of the robotic hand for automatically grasping the first object.

20. A computer apparatus for generating a configuration of a robotic hand for automatically grasping a first object, the robotic hand comprising a plurality of parts, the apparatus comprising: a memory storing processor readable instructions; and a processor arranged to read and execute instructions stored in said memory; wherein said processor readable instructions comprise instructions arranged to control the computer to carry out a method comprising: receiving data representing the first object; receiving a plurality of first models generated based upon an example grasp of a second object, the example grasp being based upon a configuration of the robotic hand for grasping the second object in which the plurality of parts of said hand contact said second object, each of said plurality of first models representing a relationship between a respective part of the robotic hand and a property of the second object associated with said part of the robotic hand, wherein the relationship is independent of any other part of the robotic hand; receiving a second model generated based upon the example grasp, the second model representing a relationship between the plurality of parts of the robotic hand; and processing the data representing the first object based upon the plurality of first models and the second model to determine said configuration of the robotic hand for automatically grasping the first object.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of various aspects of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

(2) FIG. 1 is a schematic diagram of a system for automatically grasping an object in accordance with an embodiment of the invention;

(3) FIG. 2 is a schematic diagram of a computer of the system of FIG. 1;

(4) FIG. 3 is a flowchart showing processing to generate a configuration of a robotic hand for automatically grasping a novel object in accordance with the invention;

(5) FIG. 4 is a point cloud representing an object suitable for use with the invention;

(6) FIG. 5 is a schematic illustration of a part of a robotic hand in contact with a region of an object;

(7) FIG. 6 is a flowchart showing processing to perform Monte Carlo integration; and

(8) FIG. 7 is a flowchart showing processing to evaluate a plurality of grasps in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

(9) Referring first to FIG. 1 a system for controlling a robotic hand 1 is shown. The robotic hand 1 comprises a plurality n of parts H.sub.i, i=1 . . . n, that are moveable relative to one another and that can be controlled to grasp a novel object 2. The plurality of parts H.sub.i that are moveable relative to one another comprise two or more grasping parts moveable relative to one another to grasp an object, as well as other parts that allow the two or more grasping parts to be moved relative to the novel object 2. For example, the robotic hand 1 may comprise a hand attached to an arm. The hand may comprise a palm with one or more degrees of freedom that allow parts of the palm to be moved relative to one another and two or more fingers with one or more degrees of freedom. For example the robotic hand 1 may comprise a plurality of parts that are generally arranged to simulate the movement of a human arm and hand.

(10) A computer 3 is arranged to receive example grasp data 4 associated with a grasp of the training object 5. The example grasp data 4 may for example be data associated with grasping the training object that is generated based upon manual manipulation of the robotic hand 1 to grasp the training object 5 or alternatively the training object 5 may be a virtual training object and the grasp data may be generated by simulating a grasp of the virtual training object using appropriate computer software.

(11) The computer 3 is arranged to process the example grasp data 4 to generate model data 6. The model data 6 comprises a plurality of contact models M.sub.i each contact model M.sub.i corresponding to a respective part of the robotic hand H.sub.i, each contact model M.sub.i representing a relationship between the part of the robotic hand H.sub.i and a property of the training object. For example, the model data may comprise a model for each of the plurality of parts of the robotic hand that contact the training object 5 in the grasp used to generate the grasp data 4, and each model may model a statistical distribution of features of a surface region of the training object associated with a region (that may be modelled for example as a point) at which the hand contacts the training object 5. It will of course be appreciated that more than one training object may be used and the model data may model a relationship between the respective part of the robotic hand and a property of each of the training objects.

(12) The model data may additionally comprise a configuration model C that encodes the configuration of all of the parts of the robotic hand H.sub.i for the grasp of the training object 5 when the object is grasped and additionally the configuration of the parts H.sub.i as the robotic hand approaches the configuration used to grasp the training object.

(13) The model data 6 is processed together with object data 7 associated with the novel object 2 that is to be grasped by a computer 8 to generate object grasp data 9 for grasping the novel object 2 using the robotic hand 1. The computer 8 may be the same computer as computer 3 or alternatively computers 3 and 8 may be separate computers and the model data 6 may be provided from computer 3 to computer 8 in any convenient way, for example using an appropriate communications network.

(14) In general terms and as described in detail below, the object data 7 provides data associated with the property that is modelled by the model data 6. The computer processes the models with respect to a plurality of features of the novel object determined from the object data 7 to determine regions of the novel object that are similar to regions of the training object 5 that have a predetermined spatial relationship with the robotic hand 1 in the grasp used to generate the grasp data 4. That is, the models are used to determine regions of the novel object 2 for which the property that is modelled by the model data 6 has a high correspondence with the property of the region of the training object 5. For example, the predetermined spatial relationship may be based upon contact with the object in the grasp used to generate the grasp data 4. The configuration model is used to reduce the search space across the plurality of models based upon the configuration of the plurality of parts of the robotic hand in the grasp used to generate the grasp data 4.

(15) FIG. 2 shows the computer 3 in further detail, although it will be appreciated that where computer 8 is a different computer to computer 3 it will typically have the same general structure. It can be seen that the computer comprises a CPU 3a which is configured to read and execute instructions stored in a volatile memory 3b which takes the form of a random access memory. The volatile memory 3b stores instructions for execution by the CPU 3a and data used by those instructions. For example, in use, data such as example grasp data 4, model data 6, object data 7 and object grasp data 9 may be stored in volatile memory 3b.

(16) The computer 3 further comprises non-volatile storage in the form of a hard disc drive 3c. Data such as example grasp data 4, model data 6, object data 7 and object grasp data 9 may be stored on hard disc drive 3c and processed together. The computer 3 further comprises an I/O interface 3d to which are connected peripheral devices used in connection with the computer 3. More particularly, a display 3e is configured so as to display output from the computer 3 such as data used in the simulation of a grasp. Input devices are also connected to the I/O interface 3d. Such input devices may include a keyboard 3f and a mouse 3g which allow user interaction with the computer 3. It will be appreciated that the computer may have other input interfaces, for example robotic hand 1 may be arranged in communication with computer 3 to provide computer 3 with data associated with the configuration of the plurality of parts of the robotic hand during a manually controlled training grasp. A network interface 3h allows the computer 3 to be connected to an appropriate communications network so as to receive and transmit data from and to other computers such as computer 8. The CPU 3a, volatile memory 3b, hard disc drive 3c, I/O interface 3d, and network interface 3h, are connected together by a bus 3i.

(17) Referring to FIG. 3, processing to generate a configuration of a robotic hand for automatically grasping a novel object is shown. At step S1 data representing the novel object to be grasped O is received. The data O comprises data that allows a property of the novel object to be determined at a plurality of locations. The property may be any physical property that can be determined for objects from which correspondence between points of objects can be determined and is generally a property of points of the surface of the object.

(18) For example the data O may comprise a point cloud of at least part of the surface of the novel object, the point cloud providing a three-dimensional representation of the surface of the novel object and being generated based upon image data representing the object. The property may be any property that can be determined from the point cloud such as for example principal curvatures of points p of the surface, the principal curvatures indicating curvature in a direction in a plane tangential to the surface of the object and curvature in a direction in a plane perpendicular to the surface normal. An example point cloud of a cup is shown in FIG. 3 in which each dot represents a point on the surface of the cup. FIG. 3 also shows rays representing surface normals of a plurality of points and principal curvatures of those points.

(19) At step S2 model data is received, the model data being generated based upon a training grasp of a training object. As described above, the model data comprises a plurality of contact models M.sub.i, and may additionally comprise a configuration model C, which are described in further detail below.

(20) In general terms each contact model M.sub.i models the distribution of features of a region of the received object data that corresponds to a region of the object that is contacted by the part H.sub.i during the training grasp, and allows the model to be processed with respect to object data associated with a novel object to identify regions of the object that are similar to the region that is contacted during the training grasp.

(21) In more detail, the model data comprises a plurality of contact models M.sub.i, i.Math.{1 . . . n}, each contact model representing a relationship between a respective part of the robotic grasp device H.sub.i and a property of the training object associated with the part H.sub.i for the training grasp. A contact model may for example be generated for each part H.sub.i that contacts the object during the training grasp.

(22) The property of the training object associated with the part H.sub.i is based upon the location of the part H.sub.i relative to the training object when the training object is grasped in the training grasp. For example, each contact model may be a probability density function and may be approximated by (1):

(23) $\begin{array}{cc}{M}_i\left(u,r\right)\underset{l}{.Math.}{w}_{\mathrm{il}}{N}_3\left(p.Math.{}_{p_{\mathrm{il}}},{}_{p_i}\right)\left(q.Math.{}_{q_{\mathrm{il}}},{}_{q_i}\right)N_K\left(r.Math.{}_{r_l},{}_r\right)& \left(1\right)\end{array}$
where:

(24) u=(p,q) is a pose of the part H.sub.i relative to a feature frame, defined by a point in three-dimensional space p and an orientation in three-dimensional space represented by quaterniona four-dimensional real vector q;

(25) r is a K-dimensional vector defining the property of the training object to be modelled;

(26) l is an index to a point in the region of the received object data that is to be modelled for part H.sub.i, the feature having pose v.sub.l and property to be modelled r.sub.l;

(27) $N_k\left(x.Math.,\right)=C\exp \left(-\frac{{.Math.x-.Math.}^2}{{}^2}\right)$
is a k-variate isotropic Gaussian kernel with mean , bandwidth and normalisation constant C;

(28) $\left(q.Math.,\right)=C\frac{\exp \left({}^{T}q\right)+\exp \left(-{}^{T}q\right)}{2}$
corresponds to a pair of von Mises Fisher distributions which forms a Gaussian-like distribution on SO(3), with mean , dispersion parameter and normalisation constant C;

(29) (.sub.p.sub.il, .sub.q.sub.il)=u.sub.il is the centre of a kernel computed based upon the geometric relation between v.sub.l and s.sub.i such that s.sub.i=v.sub.lu.sub.il, where s.sub.i=(p,q) is a pose of the part H.sub.i relative to a feature frame v.sub.l. s.sub.i is associated with the hand part and remains fixed for all processing, defined by a point in three-dimensional space p and an orientation in three-dimensional space q.;

(30) (.sub.p.sub.i, .sub.q.sub.i, .sub.r) are real value constants which define kernel bandwidth;

(31) w.sub.il is a weight applied to point l based upon the distance between the point l and the part H.sub.i and may be determined based upon (2) below, where

(32) $\underset{l}{.Math.}{w}_{\mathrm{il}}=1.$

(33) As described above, the weight w.sub.il may be determined according to (2):
w.sub.il=exp(v.sub.la.sub.il.sup.2) (2)
where:

(34) a.sub.il is a point on the surface of part H.sub.i that is closest to point l and is a predetermined constant for the contact model.

(35) The various components described above are illustrated in FIG. 5, which shows a part of the robotic hand 15 having an associated pose u in contact with an object at a region 16 of the object. A point 17 of the point cloud of the object corresponding to a point l described above is shown with associated components used in (1) above illustrated between the point l and the part of the robotic hand 15. In particular, as described above the point l has an associated pose represented by v and comprising a location in three-dimensional space p and an orientation q, together with an associated property r (not shown). As described above, a weight w.sub.l is determined for the point l based upon a distance between the shortest distance between the point l and the part of the robotic hand 15.

(36) Each model M.sub.i generated according to (1) encodes the distribution of features in a region surrounding the part H.sub.i, with the contribution of each point in the region surrounding the part H.sub.i being inversely proportional to the distance of the point any part of the point H.sub.i that contacts the object during the sample grasp.

(37) At step S3 grasp data is generated based upon the object data received at step S1 and the model data received at step S2. In general terms the grasp data is generated by determining a correspondence between properties of the regions of the training object that is modelled by the contact models M.sub.i and properties of regions of the novel object to be grasped. The correspondence over poses of the object (v,r) O and points of the model (u,r) M.sub.i. where v, u are poses of the form (p,q) for some position p and orientation q and r is the property of the point, can be described probabilistically by a query density Q.sub.i(s) where s is the set of all possible position and orientation combinations of part H.sub.i by (3):
Q.sub.i(s)Z.sub.iP.sub.i(s|v,u)O(v,r)M.sub.i(u,r)dvdudr(3)
where:

(38) Z.sub.i is a normalization constant;

(39) P.sub.i(s|v,u) is the conditional density and represents pose multiplications s=vu; and

(40) O and M.sub.i contribute to the weight of pose s and through the common property variable r and satisfy (4) below.
O(v,r)dvdr=1M.sub.i(v,r)dudr=1(4)
Using Bayes rule the integral (3) can be rewritten as (5):
Q.sub.i(s)Z.sub.iP.sub.i(s|v,u)O(r|v)O(v)M.sub.i(r)dvdudr(5)

(41) The training object and the novel object are represented as a collection of surface features without uncertainties such that O(r|v) becomes a mapping r(v):v.fwdarw.r , which for any choice of v uniquely determines r, effectively removing integration over r such that (5) can be approximated as (6):
Q.sub.i(s)P.sub.i(s|v,u)O(v)M.sub.i(u|r(v))M.sub.i(r(v)dvdu(6)
Q.sub.i(s) can be computed directly using Monte Carlo integration in any convenient way, for example as described below with reference to FIG. 6.

(42) At step S5 the point cloud representing the novel object O is sampled to randomly select a plurality of points, the plurality of points defining a surface feature {circumflex over (v)}O with corresponding feature vector {circumflex over (r)} and at step S6 a conditional distribution M.sub.i(u|{circumflex over (r)}) is generated based upon the feature vector {circumflex over (r)}. At step S7 a plurality of points are sampled from M.sub.i, M.sub.i(u|{circumflex over (r)}). At step S8 a centre of a kernel corresponding to a possible position and orientation of the hand part H.sub.i is determined based upon the sampled values and {circumflex over (v)} according to ={circumflex over (v)} and at step S9 a weight w is determined for the kernel according to w=M.sub.i ({circumflex over (r)}). At step S10 Q.sub.i(s) is approximated according to (7) below.

(43) $\begin{array}{cc}{Q}_i\left(s\right)\underset{l}{.Math.}{w}_{\mathrm{il}}{N}_3\left(p.Math.{}_{{\hat{p}}_{\mathrm{il}}},{}_{p_i}\right)\left(q.Math.{}_{{\hat{q}}_{\mathrm{il}}},{}_{q_i}\right)& \left(7\right)\end{array}$
where:

(44) (.sub.{circumflex over (p)}.sub.il,.sub.{circumflex over (q)}.sub.il)=.sub.il is the centre of the l.sup.th kernel computed at step S8;

(45) (.sub.p.sub.i,.sub.q.sub.i) are real value constants which define kernel bandwidth;

(46) $\underset{l}{.Math.}{w}_{\mathrm{il}}=1$
are normalised weights.

(47) A query density Q.sub.i(s) may be determined according to the above for each model M.sub.i. Suitable grasps can then be determined by combining the query densities to determine a configuration of parts H.sub.i on the object for which the correspondence between regions of the training object grasped during the training grasp and regions of the novel object is greatest and for which the configuration of parts H.sub.i is possible given the constraints upon possible configuration of the parts H.sub.i.

(48) As described above, the model data may additionally include a configuration model C that encodes the configuration of all of the parts of the robotic hand H.sub.i for the grasp of the training object both during and prior to the grasp of the object. The configuration model C can be used to restrict the search space for the query densities as described below.

(49) The configuration model may for example encode a set of configurations of the joints of the robotic hand h R.sup.D, where D is the number of parts of the hand, for an example grasp. The configuration model C may additionally encode configurations of the robotic hand as the hand moves towards the training object for the training grasp to allow the configuration model to provide improved flexibility for the search space. The configuration model C may for example be constructed using kernel density estimation according to (8):

(50) $\begin{array}{cc}C\left(x\right)\underset{\left[-,\right]}{.Math.}w\left(h\left(\right)\right)N_D\left(x.Math.h\left(\right)\right)& \left(8\right)\end{array}$
where w(h())=exp(h()h.sup.g.sup.2) and a is a positive constant; and

(51) h() is a set of configurations interpolated between h.sup.t and h.sup.g according to (9):
h()=(1)h.sup.g+h.sup.t (9)
where:

(52) h.sup.g is the joint angles of the robotic hand when the hand is grasping the training object during the training grasp;

(53) h.sup.t is the joint angles at a predetermined distance before the hand grasps the training object during the training grasp; and

(54) R and for all <0, configurations h() are beyond h.sup.g.

(55) The configuration model may be used to determine a set of grasps by maximising (10):

(56) $\begin{array}{cc}h=\underset{h_w,h_j}{\arg \max }C\left(h_j\right)\underset{i}{.Math.}{Q}_i\left(k_i^{\mathrm{for}}\left(h_w,h_j\right)\right)& \left(10\right)\end{array}$
where:

(57) C(h.sub.j) is the configuration model;

(58) Q.sub.i is a query density corresponding to hand part H.sub.i; and

(59) k.sub.i.sup.for(h.sub.w,h.sub.j) is a forward kinematic function of the robotic hand with wrist pose h.sub.w and joint configuration h.sub.j that allows the position of the hand parts H.sub.i having corresponding densities Q.sub.i upon the novel object to be estimated based upon pose of the robotic hand h=(h.sub.w,h.sub.j) modelled by the configuration model C.

(60) Maximisation of (10) can be effected using a simulated annealing procedure as will now be described with reference to FIG. 7. At step S10 a part H.sub.i for which a corresponding density Q.sub.i exists (for example, a part that contacts the novel object during the training grasp) is selected at random and at step S11 a pose s.sub.i for part H.sub.i is determined by sampling from Q.sub.i. At step S12 a configuration h.sub.j is selected from C(h.sub.j) at random, the configuration h.sub.j representing a configuration of the joints of the robotic hand in which hand part H.sub.i has pose s.sub.i. Given that the configuration model C(h.sub.j) represents configurations in the approach to the object to be grasped the configuration h.sub.j may not be a configuration when the object is grasped and as such, at step S13, poses of the hand parts when the object is grasped are determined from the configuration h.sub.j using k.sub.i.sup.for(h.sub.w,h.sub.j) to determine a robotic hand pose h representing a configuration of the hand parts when the object is grasped based upon the initial pose s.sub.i.

(61) At step S14 the configuration determined at step S13 is evaluated to determine a value for the robotic hand pose h based upon (10) by determining the product of the correspondence values determined based upon the models Q.sub.i of each hand part H.sub.jthat contacts the novel object when the robotic hand grasps the object with robotic hand pose h.

(62) At step S15 it is determined if more configurations are to be selected based upon the pose s.sub.i determined at step S11 and if more configurations are to be selected the processing of steps S12 to S15 are repeated to select a further robotic hand pose h and associated value, for example by selecting a configuration of the robotic hand that is at a different point in the training grasp movement towards the object. The further configuration may for example be selected based upon a previous configuration that has a highest associated value. The processing of steps S12 to S15 may for example be repeated 200 times, although it will of course be appreciated that the processing of steps S12 to S15 may be repeated any suitable predetermined number of times. The processing of steps S12 to S15 provides a greatest value for a configuration h.sub.j given starting pose s.sub.i selected at step S11.

(63) At step S16 it is determined whether more hand parts are to be processed as a starting point for the processing of FIG. 7 and if it is determined that more hand parts are to be processed a further hand part is selected at step S10 and the processing of FIG. 7 is repeated. The processing of FIG. 7 will typically be repeated based upon selection of each hand part that has a predetermined spatial relationship with the training object during the training grasp, for example that contacts the training object during the training grasp, multiple times, with a different pose s.sub.i being selected at step S11 each time the hand part is selected such that each hand part is evaluated for a plurality of configurations based upon a set of poses that provide maximal correspondence between the novel object, as determined using the models Q.sub.i. For example, a large number of values may be determined for configurations h.sub.j given starting pose s.sub.i selected at step S11, for example 1000 or more, for different starting hand part and configuration combinations. Combinations that are not possible may be identified and removed, for example based upon known reachability and workspace constraints and the remaining best scoring configuration may be used to generate a grasp trajectory for grasping the novel object.

(64) Whilst it is described above that a configuration model may be used to constrain the search space it will be appreciated that the search space can be constrained in any convenient way, for example based upon the movement of the hand towards the object.

(65) It is described above that the model data comprises a plurality of contact models M.sub.i, i .Math.{1 . . . n}, each contact model representing a relationship between a respective part of a robotic grasp device H.sub.i and a property of the training object associated with the part H.sub.i for the training grasp. In some embodiments each part of the robotic grasp device H.sub.i may have one or more associated receptive fields defining a three-dimensional space volume rigidly attached to a particular hand link, the receptive field being denoted by R.sub.i, i=1, . . . , N. Each receptive field can extend a distance R.sup.+ from a given part of a robotic grasp device H.sub.i, for example as illustrated in FIG. 5.

(66) Where receptive fields are associated with parts H.sub.i each contact model may represent a relationship between a receptive field and a property of the training object associated with the receptive field in a similar manner to that described above. For example, a contact model may be generated for each receptive field that intersects the object to be grasped during the training grasp. Each model may model a statistical distribution of features of a surface region of the training object associated with a region (that may be modelled for example as a point) at which the receptive field intersects the training object 5. Each contact model may be generated based upon a probability density model approximated according to (1) as described above, with weights W.sub.ilthat exponentially decrease based upon the distance to the surface of the hand feature determined according to (2) as described above.

(67) In this way, contact models may be generated based upon a relationship between the object to be grasped and receptive fields R.sub.i that are indicative of neighbourhoods important for grasping of an object. For example, a receptive field can be defined such that a greater importance is given to objects in a space in three dimensions in a particular direction relative to a hand part. Suitable receptive fields may be determined in any convenient way. For example receptive fields may be input by a user based upon user knowledge of important three dimensional spaces for a particular hand part, or alternatively receptive fields may be determined for an object using machine learning techniques as will be known to a person skilled in the art. Additionally, whilst it has been described above that weights are generated in a similar manner to that described above, it will be appreciated that weights may also be generated for receptive fields using machine learning techniques.

(68) A grasp may be determined based upon receptive fields as described above with reference to FIG. 10. Alternatively, in some embodiments receptive fields may be arranged into collections M.sub.mand a hand pose h may be determined based upon collections of receptive fields using a product of mixtures approach according to (11):

(69) $\begin{array}{cc}h=\underset{h_w,h_j}{\arg \max }C\left(h_j\right)\underset{m}{.Math.}\underset{M_m}{.Math.}{Q}_i\left(k_{{}_l\left(i\right)}^{\mathrm{for}}\left(h_w,h_j\right)\right)& \left(11\right)\end{array}$
where:

(70) C(h.sub.j) is the configuration model as above;

(71) Q.sub.iis a query density corresponding to receptive field R.sub.i; and

(72) k.sub.l(i).sup.for(h.sub.w,h.sub.j) is a forward kinematic function of the robotic hand with wrist pose h.sub.w and joint configuration h.sub.i that allows the position of a hand part h.sub.l(i), where .sub.l(i) is a mapping from receptive fields to hand parts given that each hand part may have a plurality of associated receptive fields, to be estimated based upon pose of the robotic hand h=(h.sub.w,h.sub.j) modelled by the configuration model C.

(73) The optimisation of (11) therefore maximises a product of the sum of query density values of each collection. By maximising over collections in this way, a relationship between associated receptive fields can be included in the evaluation such that the effect of associated receptive fields can be combined.

(74) The optimisation of (11) may be carried out using a simulated annealing procedure as described above with reference to FIG. 7 but with receptive fields being selected in place of hand parts, or other suitable numerical optimisation procedure as will be known to a person skilled in the art.

(75) Although specific embodiments of the invention have been described above, it will be appreciated that various modifications can be made to the described embodiments without departing from the spirit and scope of the present invention. That is, the described embodiments are to be considered in all respects exemplary and non-limiting. In particular, where a particular form has been described for particular processing, it will be appreciated that such processing may be carried out in any suitable form arranged to provide suitable output data.