System and method for determining the material loading condition of a bucket of a material moving machine

Abstract

A system is provided for determining the material loading condition of a bucket of a material moving machine. The system includes an arrangement for generating a surface model of the surface of the material loaded into the bucket and comparing the surface model with a predefined preferred surface shape associated with the bucket.

Claims

1. A system comprising: a perception assembly adapted to: generate a surface model of a surface of a material loaded into a loader bucket of a loader, wherein the loader bucket is adapted to be moved relative to a remaining portion of the loader to thereby be loaded with the material; determine a bucket location of the bucket; determine a type of material located at the bucket location; determine a type of material loaded into the bucket based on the type of material located at the bucket location; determine a predefined preferred surface shape associated with the loader bucket based on the type of material loaded into the bucket; compare the surface model with the predefined preferred surface shape; determine a material loading condition of the loader bucket based on the comparison of the surface model with the predefined preferred surface shape; and issue a signal indicative of the material loading condition.

2. The system according to claim 1, wherein the surface model comprises a three-dimensional model of the surface of the material loaded into the loader bucket.

3. The system according to claim 1, wherein the predefined preferred surface shape comprises information indicative of a nominal angle of repose for the material loaded into the bucket.

4. The system according to claim 1, wherein the system is further adapted to determine at least one of a spillage parameter indicative of a risk of spilling or a fill factor parameter indicative of a material fill factor.

5. The system according to claim 1, wherein the signal indicative of the material loading condition is further indicative of whether or not the bucket is appropriately loaded.

6. The system according to claim 1, wherein the perception assembly comprises at least one of a camera or a laser sensor.

7. The system according to claim 6, wherein the surface model comprises a three-dimensional model of the surface of the material loaded into the bucket, and wherein the perception assembly further comprises at least one of a time-of-flight camera, a stereo camera, a structured light camera or an actuated laser range finder.

8. The system according to claim 6, wherein at least a portion of the perception assembly is adapted to be attached to a portion of the material moving machine.

9. The system according to claim 1, wherein the perception assembly is further adapted to identify the loader bucket by: generating a model of the loader bucket; and comparing the model of the loader bucket with a stored set of bucket models.

10. The system according to claim 1, wherein the system is further adapted to: use a global positioning system (GPS) device to determine the bucket location of the loader bucket; and access a database to determine the type of material located at the bucket location.

11. A loader comprising: a loader bucket that is movable relative to a remaining portion of the material moving machine, wherein movement of the loader bucket causes the loader bucket to be loaded with a material, and a system comprising a perception assembly adapted to: generate a surface model of a surface of the material loaded into the bucket; and compare the surface model with a predefined preferred surface shape associated with the bucket, the comparison based on a determination of a volume of the material exceeding the predefined preferred surface shape, wherein the system is further adapted to: determine a material loading condition of the bucket based on the comparison of the surface model with the predefined preferred surface shape; and issue a signal indicative of the material loading condition.

12. The loader according to claim 11, wherein the surface model comprises a three-dimensional model of the surface of the material loaded into the bucket.

13. The loader according to claim 11, further comprising a cabin comprising a cabin top, wherein at least the portion of the perception assembly is attached to the cabin top.

14. A method comprising: moving a loader bucket of a loader relative to a remaining portion of the loader to load the loader bucket with a material; generating a surface model of the surface of the material loaded into the loader bucket; comparing the surface model with a predefined preferred surface shape associated with the loader bucket, the comparing further comprising determining a volume of the material exceeding the predefined preferred surface shape; determining a material loading condition of the loader bucket based on the comparison of the surface model with the predefined preferred surface shape; and issuing a signal indicative of the material loading condition.

15. The method according to claim 14, wherein the predefined preferred surface shape comprises information indicative of a nominal angle of repose for the material loaded into the loader bucket.

16. The method according to claim 15, wherein the loader bucket comprises a bucket bottom and a bucket opening rim with at least three rim points, wherein the predefined preferred surface shape is generated by forming a set of rim point planes by: generating a rim point plane for each pair of adjacent rim points, wherein the rim point plane intersects each one of the pair of adjacent rim points and extends towards the opposing side of the loader bucket with a plane angle (a) indicative of the nominal angle of repose, and assembling the set of rim point planes to form the predefined preferred surface shape, the predefined preferred surface shape comprising a rim point plane floor portion of each rim point plane a rim point plane floor portion having a location such that no other plane of the set of rim point planes is located between the rim point plane floor portion and the bucket bottom.

17. The method according to claim 16, wherein the bucket opening rim comprises at least three rim corners, the rim points comprising the rim corners.

18. The method according to claim 14, wherein comparing the surface model with a predefined preferred surface shape associated with the loader bucket further comprises determining the volume of the material exceeding the predefined preferred surface shape.

19. The method according to claim 16, wherein comparing the surface model with a predefined preferred surface shape associated with the loader bucket further comprises determining the volume of the material exceeding the predefined preferred surface shape, and wherein the feature of comparing the surface model with a predefined preferred surface shape associated with the loader bucket comprises determining the volume of the material that exceeds each rim point plane floor portion.

20. The method according to claim 18, wherein the feature of comparing the surface model with a predefined preferred surface shape associated with the loader bucket comprises: determining a plurality of surface model points; for each one of the surface model points that exceeds of the predefined preferred surface shape, determining the smallest distance between the surface model point and the predefined preferred surface shape.

21. The method according to claim 20, wherein the method further comprises determining a parameter value indicative of at least one of an average value (m) or a variance (.sup.2) of the smallest distance between the surface model point and the predefined preferred surface shape for all of the surface model points that are located outside of the predefined preferred surface shape.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) With reference to the appended drawings, below follows a more detailed description of embodiments of the invention cited as examples.

(2) In the drawings:

(3) FIG. 1 is a schematic perspective view of a material moving machine;

(4) FIG. 2 is a schematic side view of an image of a bucket filled with material;

(5) FIG. 3 is a schematic side view of a surface model of a material surface;

(6) FIG. 4-FIG. 6 illustrate an example of a predefined preferred surface shape;

(7) FIG. 7 illustrates an implementation of comparing a surface model with a predefined preferred surface shape;

(8) FIG. 8 is a schematic perspective view of a material moving machine comprising alternative implementations of perception assemblies, and

(9) FIG. 9 is a flow chart of a method.

DETAILED DESCRIPTION

(10) The invention will be described in the following for a material moving machine in the form of a wheel loader 10 such as the one illustrated in FIG. 1. The wheel loader 10 should be seen as an example of a material moving machine which could comprise a system according to the present invention and/or for which the method of the present invention may be used. However, it is also envisaged that the present invention may be implemented in another type of material moving machine, such as an excavator (not shown) or a bulldozer (not shown). Moreover, it should be emphasized that although the invention is described hereinbelow with reference to a movable material moving machine, the invention is equally applicable on, and consequently also comprises, stationary material moving machines.

(11) The FIG. 1 material moving machine 10 comprises a bucket 12. Moreover, the material moving machine 10 comprises an actuating arrangement 14 for lifting and/or tilting the bucket 12. Purely by way of example, the actuating arrangement 14 may comprise a hydraulic actuator and/or an electric actuator (not shown in FIG. 1).

(12) Purely by way of example, the bucket 12 may be adapted to be moved, relative to the ground and/or relative to the remaining portion of the material moving machine, to thereby be loaded with material.

(13) Moreover, FIG. 1 illustrates that the bucket 12 is filled with material 16. Purely by way of example, the material may comprise sand, gravel, soil, pebbles, rocks or the like.

(14) FIG. 1 further illustrates a system 18 for determining the material loading condition of the bucket 12 of the material moving machine 10. In FIG. 1, the material moving machine 10 comprises the system 18. However, it is also envisaged that the system 18 may be spatially separated from the material moving machine 10. Purely by way of example, the system 18, or at least parts of it, may be located in a stationary or moving object spatially separated from the material moving machine 10.

(15) The system 18 comprises means for generating a surface model of the surface of the material loaded into the bucket.

(16) To this end, though purely by way of example, FIG. 2 illustrates an image of the FIG. 1 bucket 12 and material 16, which image has been generated by the system 18. Purely by way of example, the FIG. 2 model may be generated by a point cloud measurement of the bucket 12 and the material 16.

(17) Furthermore, FIG. 3 illustrates a surface model 17 of the surface of the material 16. The FIG. 3 surface model 17 has been generated on the basis of the FIG. 2 image. Purely by way of example, the FIG. 3 surface model 17 may be generated by extrapolating the data, e.g. points, obtainable from the FIG. 2 image.

(18) It should be noted that in FIG. 2 and FIG. 3, the image and the surface model 17 are three-dimensional models, which are preferred implementations of the image and surface model. However, it is also envisaged that embodiments of the system 18 instead generate a two-dimensional image and/or surface model. Furthermore, although FIG. 2 and FIG. 3 illustrate an alternative in which the surface model is generated in two steps, viz firstly generating an image and thereafter generating a surface model 17, it is envisaged that the surface model may be generated in a single step, e.g. by generating the surface model on the basis of a set of points of at least the material surface, or by using more steps than two.

(19) The system 18 may be adapted to identify the type of bucket 12 in the FIG. 3 image. Purely by way of example, the system may be adapted to read a bucket identifier (not shown) such as a bar code or any other type of identification information. It should be noted that the identification information need not necessarily contain visually readable information but may instead contain information that may be read using e.g. a radio wave sensor, a magnetic sensor or the like. Instead of, or in addition to, reading a bucket identifier, the system 18 may be adapted to generate a model of the bucket 12, e.g. based on the image of the bucket 12 and material 16, and compare the model of the bucket with a stored set of models. As a further option, the type of bucket 12 may be manually inputted into the system 18, e.g. by an operator of the material moving machine 10.

(20) Irrespective of how the type of bucket 12 is identified, the system may be adapted to remove portions, e.g. points, of the model of the material 16 loaded into the bucket 12 such that only the portion of the material 16 that is located inside a bounding box of a bucket model may be regarded when assessing the loading condition.

(21) Once the relevant portions of the material have been identified, the system 18 may generate a surface model 17, for instance by generating a mesh of triangular elements wherein a node in a triangular element corresponds to a point the image of the material. The mesh thus generated may constitute the surface model 17. Once the surface model 17, and its position and orientation in relation to the bucket, have been identified, entities such as the volume of the material 16 loaded into the bucket 12 may be determined.

(22) Such a volume determination may be performed in a plurality of ways. As a non-limiting example, a geometrical model of the bucket 12 may be combined with the surface model 17 and the volume enclosed by the above two models may be determined by integration.

(23) FIG. 3 illustrates an example of such an integration in which the interior of the bucket 12 has been modelled by a plurality of bucket nodes. The bucket nodes and the nodes of the surface model 17 are combined triangulate elements that are generated between adjacent nodes. For each triangular element, a tetrahedron element may be generated, three corners of which are the nodes of the triangular element and the fourth corner is the centroid of the volume enclosed by the bucket nodes and the nodes of the surface model 17. Purely by way of example, the location of such a centroid may be calculated by calculating the mean value of the location the bucket nodes and the nodes of the surface model 17.

(24) Furthermore, the system 18 may be adapted to determine a predefined preferred surface shape associated with the bucket 12, viz the bucket type thus determined. Purely by way of example, the system 18 may be adapted to determine the predefined preferred surface shape by receiving data from a database. The database may form part of the system 18. Alternatively, the database may be separate of the system 18 and the system 18 may comprise means for communicating with the database.

(25) Irrespective of how the predefined preferred surface shape is determined, the system 18 is adapted to compare the surface model with a predefined preferred surface shape associated with the bucket 12.

(26) The predefined preferred surface shape need not exclusively be based on the relevant bucket 12. It is envisaged that the predefined preferred surface shape may also be dependent on the type of material loaded into the bucket 12.

(27) The preferred surface shape of the material loaded into a bucket may vary with the type of material that is loaded into the bucket 12. To this end, the system 18 is preferably adapted to receive information indicative of the type of material that is to be loaded or has been loaded into the bucket. Purely by way of example, the system 18 may be adapted to receive input from an operator indicative of the type of material. As another option, the system 18 may be able to determine the type of material by e.g. reading the weight of the material loaded into the bucket, for instance using one or more weight sensors (not shown), determining the volume of the material loaded into the bucket, e.g. using the above-discussed image and/or the above-discussed surface model, determining the density of the material and from the density thus determined establish the type of material.

(28) As a further option, the system 18 may be able to determine the type of material by determining the location of the bucket, e.g. using a GPS system or the like, and using data of the material at that location, e.g. using a database, a look-up table, and thus determine the type of material.

(29) Regardless of how the type of material is determined, the system 18 is preferably adapted to determine a predefined preferred surface shape associated with the bucket 12, viz the bucket type thus determined, and also the type of material thus determined. Purely by way of example, the predefined preferred surface shape, based on the above two parameters of bucket type and material type, may be determined by receiving data from a database. Instead of, or in addition to, receiving data from a database, the system 18 may be adapted to calculate the predefined preferred surface shape based on the bucket and material types.

(30) A parameter that may be relevant when determining a predefined preferred surface shape for a material type is the nominal angle of repose, viz the steepest angle relative to a horizontal plane to which a material can be piled without slumping, for the material loaded into the bucket 12. Based on the nominal angle of repose for the type of material and the type of bucket 12, it is possible to determine a predefined preferred surface shape. Purely by way of example, such a predefined preferred surface shape comprises a volume with an inclined top surface, such as a pile, wherein shape of the volume is such that an appropriately large volume of material may be loaded into the bucket 12 but wherein the inclination of the volume's top surface is equal to or below the above-discussed nominal angle of repose for that material type.

(31) Based on the surface model thus generated and the predefined preferred surface shape thus determined, the system 18 may be adapted to determine a spillage parameter indicative of the risk of spilling and/or a fill factor parameter indicative of the material fill factor.

(32) Purely by way of example, the system 18 may be adapted to issue a signal indicative of the material loading condition. For instance, the system 18 may be adapted to issue a signal indicative of whether or not the bucket 12 is appropriately loaded.

(33) As a non-limiting example, the signal indicative of whether or not the bucket 12 is appropriately loaded may comprise information as regards the volume of the material in the bucket that exceeds the predefined preferred surface shape. As a further non-limiting example, the signal may comprise information as regards the location of the volume exceeding the predefined preferred surface shape.

(34) FIGS. 4 to 6 illustrate how a predefined preferred surface shape may be generated for a bucket 12 and a nominal angle of repose for a material.

(35) The implementation of the bucket 12 illustrated in FIG. 4 comprises a bucket bottom 19 and a bucket opening rim 20. The bucket opening rim 20 comprises at least three rim points. The bucket implementation illustrated in FIG. 4 comprises four rim corners 22, 24, 26, 28 that also constitute the rim points. However, it is also envisaged that at least one rim point may be located in another rim position than in a rim corner.

(36) Based on the rim points 22, 24, 26, 28, a set of rim point planes is formed by generating a rim point plane for each pair of adjacent rim points 22, 24, 26, 28, wherein the rim point plane intersects each one of the pair of adjacent rim points and extends towards the opposing side of the bucket 12 with a plane angle indicative of the nominal angle of repose.

(37) FIG. 4 illustrates two rim point planes 30, 32 and FIG. 5 illustrates two further rim point planes 34, 36.

(38) With reference to FIG. 4, a first rim point plane 30 is generated for adjacent rim points, viz a first rim point 22 and a second rim point 24. The first rim point plane intersects each one of the pair of adjacent rim points 22, 24 and extends towards the opposing side of the bucket 12 with a plane angle indicative of the nominal angle of repose. Purely by way of example, the plane angle may be a function, e.g. proportional to, the nominal angle of repose. As a non-limiting example, the plane angle may be the same, or at least substantially the same, as the nominal angle of repose.

(39) A second rim point plane 32 is generated for another set of adjacent rim points, viz a third rim point 26 and a fourth rim point 28 in FIG. 4.

(40) FIG. 5 illustrates a third rim plane 34 and a fourth rim plane 36 for the two other sets of adjacent rim points, viz the first 22 and third 26 rim point as well as the second 24 and the fourth 28 rim point.

(41) FIG. 6 illustrates that the predefined preferred surface shape 38 may be generated by

(42) assembling the set of rim point planes. FIG. 6 illustrates that the predefined preferred surface shape 38 comprises a rim point plane floor portion 30, 32, 34, 36 of each rim point plane. A rim point plane floor portion has a location such that no other plane of the set of rim point planes is located between the rim point plane floor portion and the bucket bottom.

(43) As has been intimated hereinabove, the signal indicative of whether or not the bucket 12 is appropriately loaded may comprise information as regards the volume of the material in the bucket that exceeds the predefined preferred surface shape.

(44) FIG. 7 illustrates a loading condition in which a portion 16 of the material 16 in the bucket 12 exceeds the predefined preferred surface shape 38. As used herein the expression exceeds the predefined preferred surface shape indicates a position on the side of the predefined preferred surface shape 38 which faces away from the bucket bottom 19.

(45) As such, embodiments of the system 18 and/or a method may comprise determining the volume of the material exceeding the predefined preferred surface shape 38. The magnitude of the volume may provide appropriate information as to whether or not the bucket is appropriately filled. Purely by way of example, a loading condition having a large volume that exceeds the predefined preferred surface shape 38 may have a high risk of spilling and may therefore be less preferred.

(46) In the load condition illustrated in FIG. 7, the portion 16 of the material 16 in the bucket 12 is located in the second rim point plane floor portion 32. In accordance with an embodiment of the system 18 and/or the method, the feature of comparing the surface model 17 with a predefined preferred surface shape associated with the bucket 12 comprises determining the volume of the material that exceeds each rim point plane floor portion 30, 32, 24, 36.

(47) The feature of comparing the surface model with a predefined preferred surface shape associated with the bucket may comprise determining a plurality of surface model points. Purely by way of example, the surface model points could be related to a point cloud measurement of the bucket 12 and the material 16, such as the point cloud measurement procedure presented hereinabove with reference to FIG. 2. As such, as a non-limiting example, the surface model points could be at least a subset of the points of the point cloud.

(48) However, it is also envisaged that the surface model points are generated on the basis of a surface model of the surface of the material 16 loaded into the bucket 12.

(49) Irrespective of how the surface model points are generated, the feature of comparing the surface model with a predefined preferred surface shape associated with the bucket may further comprise, for each one of the surface model points that exceeds the predefined preferred surface shape, determining the smallest distance d between the surface model point and the predefined preferred surface shape.

(50) Purely by way of example, the smallest distance d may be determined by assuming a plane defined by a point c (for instance chosen to be a rim point and or a rim corner), a normal n, and a surface model point p.

(51) The smallest distance d may then be computed by the following equation:
d(p,c,n)=(pc).Math.nEquation 1

(52) The variance of a set of points P={p.sub.1, . . . , p.sub.n} is computed as

(53) $^{2} = \frac{1}{n + 1} {.Math.}_{i = 1}^{n} {d (p_{i}, c, n)}^{2} .$

(54) In this case, however, we are only interested in points outside of the model volume; i.e., points with a positive distance d. A more relevant measure, then, is the variance around the mean positive distance m. Let Q={p.sub.1, . . . , p.sub.k} be the set of points whose distance d(p,c,n) is positive.

(55) It is then possible to determine the average value and/or standard deviation of the smallest distance between the surface model point and the predefined preferred surface shape for all of the surface model points that are located outside of the predefined preferred surface shape.

(56) The average value m of the smallest distance d may be determined in accordance with the following:

(57) $\begin{matrix} m = \frac{1}{k} {.Math.}_{i = 1}^{k} d (p_{i}, c, n) & Equation 2 \end{matrix}$

(58) and the variance 2 of the smallest distance d by be determined by:

(59) $\begin{matrix} ^{2} = \frac{1}{k + 1} {.Math.}_{i = 1}^{k} {(d (p_{i}, c, n) - m)}^{2} & Equation 3 \end{matrix}$

(60) Instead of, or in addition to, determining the average value and/or variance of the smallest distance d, the smallest distance d for a set of surface model points may be presented in relation to the predefined preferred surface shape. Purely by way of example, the smallest distance d for a set of surface model points may be presented for each rim point plane floor portion 30, 32, 24, 36.

(61) As such, each rim point plane floor portion comprises a reference point c, a normal n and two plane vectors a, b each one of which extending in the plane of the relevant rim point plane floor portion.

(62) To this end, reference is again made to FIG. 7 in which the above-discussed geometrical entities c, n, a, b are illustrated for the first rim point plane floor portion 30. However, it should be noted that each one of the first rim point plane floor portions comprises these entities.

(63) The first rim point plane floor portion 30 may be divided into a plurality of portion areas 30k, or portion cells, each one of which having an extension in directions parallel to the a and b vectors. For each one portion cells 30k, the average value m and/or the variance 2 may be determined in accordance with Equation 2 or Equation 3, respectively. The thus determined average value m and/or the variance 2 may then be presented for each portion cell 30k, for instance by colour coding each portion cell with a colour corresponding to the determined value(s) or by presenting histograms with bars for each portion cell.

(64) In order to determine the surface model, the means for generating a surface model of the system 18 may comprise a perception assembly for generating the surface model. As non-limiting examples, the perception assembly may comprise at least one of a camera and a laser sensor. In the event that the system 18 is adapted to generate a three-dimensional surface model, the perception assembly may comprise at least one of a time-of-flight camera, a stereo camera, a structured light camera or an actuated laser range finder.

(65) FIG. 8 illustrates the FIG. 1 material moving machine 10 and two alternative implementations of the perception assembly 40. In the first alternative implementation, a perception assembly 40 is attached to the material moving machine 10. Purely by way of example, and as is indicated in FIG. 8, the material moving machine 10 may comprise a cabin 42 with a cabin top 44, wherein at least the portion of the perception assembly 40 of the first alternative is attached to the cabin top 44. However, it is also envisaged that the first alternative implementation of the perception assembly 40 may be attached to another portion of the material moving machine 10, such as the actuating arrangement 14.

(66) In the second alternative implementation of the perception assembly, a perception assembly 40 is not attached to the material moving machine 10. Moreover, though purely by way of example, the perception assembly 40 in accordance with the second alternative implementation is spatially separated from the material moving machine 10. As non-limiting examples, the perception assembly 40 in accordance with the second alternative implementation may be located on a stationary object, such as a mast (not shown) or the like, or may be located on a moving object. By way of example only, the moving object may be an object that is capable of flying and that has one or more rotors, e.g. a helicopter or a quadcopter.

(67) Finally, FIG. 9 illustrates a flow chart illustrating a method in accordance with a third aspect of the present invention. The third aspect relates to a method for determining the material loading condition of a bucket of a material moving machine. The bucket may be adapted to be moved to thereby be loaded with the material.

(68) As may be gleaned from FIG. 9, the method comprises

(69) generating a surface model of the surface of the material loaded into the bucket, and

(70) comparing the surface model with a predefined preferred surface shape associated with the bucket.

(71) It is to be understood that the present invention is not limited to the embodiments described above and illustrated in the drawings; rather, the skilled person will recognize that many changes and modifications may be made.

System and method for determining the material loading condition of a bucket of a material moving machine

Assignee

Inventors

Cpc classification

Classification Explorer

G05D1/0251

PHYSICS

Classification Explorer

B60R2300/60

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

B60R2300/302

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

G06F30/20

PHYSICS

Classification Explorer

E02F9/2029

FIXED CONSTRUCTIONS

Classification Explorer

G06V20/653

PHYSICS

Classification Explorer

B60Y2200/412

PERFORMING OPERATIONS; TRANSPORTING

Classification Explorer

G06V20/56

PHYSICS

Classification Explorer

E02F3/34

FIXED CONSTRUCTIONS

Classification Explorer

E02F3/40

FIXED CONSTRUCTIONS

International classification

Classification Explorer

G06F30/20

PHYSICS

Classification Explorer

E02F3/34

FIXED CONSTRUCTIONS

Classification Explorer

G06K9/00

PHYSICS

Classification Explorer

E02F3/40

FIXED CONSTRUCTIONS

Classification Explorer

E02F9/20

FIXED CONSTRUCTIONS

Abstract

Claims

Description