METHOD AND SYSTEM OF CONFIGURING A MACHINE CONTROL UNIT OF A CONSTRUCTION MACHINE
20240060275 · 2024-02-22
Assignee
Inventors
Cpc classification
G05D1/0251
PHYSICS
E02F9/262
FIXED CONSTRUCTIONS
E02F3/7609
FIXED CONSTRUCTIONS
E02F3/841
FIXED CONSTRUCTIONS
E02F3/7668
FIXED CONSTRUCTIONS
E02F3/765
FIXED CONSTRUCTIONS
E01C19/004
FIXED CONSTRUCTIONS
International classification
Abstract
A system and method for configuring a machine control unit of a construction machine for performing an earth-moving operation, the earth-moving operation comprising a multitude of phases that are to be performed consecutively, the method comprising, by a computing unit: receiving three-dimensional measuring data of a terrain in a surrounding of the construction machine in at least a first detection range, detecting elements of a previous phase of the earth-moving operation based on the three-dimensional measuring data, and configuring, based at least on the detected elements, the machine control unit to at least partially perform the next phase of the earth-moving operation automatically.
Claims
1. A system for configuring a machine control unit of a construction machine for performing an earth-moving operation, the earth-moving operation comprising a multitude of phases that are to be performed consecutively, the system comprising a measuring system configured for capturing three-dimensional measuring data of a terrain in a surrounding of the construction machine in at least a first detection range; a user interface configured for receiving user input from the operator of the construction machine; and a computing unit operatively coupled at least with the measuring system, the user interface and the machine control unit, wherein the computing unit is configured for detecting elements of a previous phase of the earth-moving operation based on the three-dimensional measuring data; and configuring, based at least on the detected elements, the machine control unit to at least partially perform the next phase of the earth-moving operation automatically.
2. The system according to claim 1, wherein the measuring system comprises at least one measuring unit at the construction machine, each measuring unit being configured for capturing 3D point cloud data, wherein each measuring unit comprises: at least one laser scanner, a plurality of ToF cameras, a millimetre wave radar system, and/or one or more stereo camera systems.
3. The system according to claim 1, comprising a context camera having a known position relative to the measuring system and/or the three-dimensional measuring data and being configured for capturing context image data of the terrain within the first detection range, wherein the user interface is configured for displaying at least one context image based on the context image data to an operator of the construction machine, the computing unit is configured for overlaying the detected elements on the displayed context image and for receiving input from the operator related to a next phase of the earth-moving operation; and configuring the machine control unit is also based on the input from the operator, particularly wherein the user interface comprises a touch-sensitive display on which the context image is displayed and on which the user input is received; and/or the input from the operator comprises a selection of one or more of the detected elements.
4. The system according to claim 1, wherein the construction machine is a motor grader and the earth-moving operation comprises construction a ditch, wherein the phases are passes of the motor grader along a ditch line, including a plurality of cutting passes; and the detected elements comprise an edge and/or surface produced by a previous cutting pass, particularly wherein the earth-moving operation comprises V ditching, and/or the input from the operator comprises selecting an edge and/or surface produced by a previous cutting pass; selecting a cutting depth for the next pass; selecting to maintain a parallel cut for the next pass; and/or selecting an edge and a surface produced by a previous cutting pass and selecting to adjust a cross slope of the surface for the next pass while maintaining the selected edge as a vertex of a plane to be shaped during the next pass.
5. The system according to claim 1, wherein the computing unit is configured for receiving operator input indicating a desired line, along which line the earth moving operation is desired to be performed, localizing the construction machine based on the three-dimensional measuring data, positioning the tool at the desired line, and performing a first phase of the multitude of phases, thereby producing detectable elements, particularly wherein the construction machine is localized relative to the desired line; the computing unit is configured for steering the construction machine to the desired line; and/or the desired line is a spline.
6. The system according to claim 5, wherein operator input comprises an offset of the desired line from a given line and the computing unit is configured to calculate the desired line based on the offset, particularly wherein the given line is a spline and the computing unit is configured to calculate a spline as the desired line based on the offset.
7. The system according to claim 1, wherein the measuring system is configured for continuously capturing the three-dimensional measuring data and for continuously providing the captured three-dimensional measuring data to the computing unit; the computing unit is configured for continuously evaluating the three-dimensional measuring data for detecting the elements and for continuously configuring the machine control unit while at least partially automatically performing a phase of the earth-moving operation.
8. A construction machine, comprising a tool for performing an earth-moving operation, in particular for digging a ditch or trench; a control unit for at least partially controlling the earth-moving operation; and a system according to any one of the preceding claims, the system being operatively coupled with the control unit or comprising the control unit.
9. The construction machine according to claim 8, wherein the construction machine is a grader, a dozer or an excavator.
10. A computer-implemented method for configuring a machine control unit of the construction machine according to claim 8, for performing an earth-moving operation, the earth-moving operation comprising a multitude of phases that are to be performed consecutively, the method comprising, by a computing unit, receiving three-dimensional measuring data of a terrain in a surrounding of the construction machine in at least a first detection range; detecting elements of a previous phase of the earth-moving operation based on the three-dimensional measuring data; and configuring, based at least on the detected elements, the machine control unit to at least partially perform the next phase of the earth-moving operation automatically.
11. The method according to claim 10, comprising capturing context image data of the terrain within the first detection range using a context camera having a known position relative to the three-dimensional measuring data and/or to a measuring system capturing the three-dimensional measuring data; displaying at least one context image based on the context image data to an operator of the construction machine, overlaying the detected elements on the displayed context image; receiving input from the operator related to a next phase of the earth-moving operation, particularly wherein the input from the operator comprises a selection of one or more of the detected elements; and configuring the machine control unit also based on the input from the operator.
12. The method according to claim 10, wherein the construction machine is a motor grader and the earth-moving operation comprises constructing a ditch, wherein the phases are passes of the motor grader along a ditch line, including a plurality of cutting passes; and the detected elements comprise an edge and/or surface produced by a previous cutting pass, particularly wherein the earth-moving operation comprises V ditching; the input from the operator comprises selecting an edge and/or surface produced by a previous cutting pass; the input from the operator comprises selecting to maintain a parallel cut for the next pass; and/or the input from the operator comprises selecting a cutting depth for the next pass.
13. The method according to claim 10, comprising receiving operator input indicating a desired line, along which line the earth moving operation is desired to be performed, localizing the construction machine based on the three-dimensional measuring data, particularly relative to the desired line, positioning the tool at the desired line, and performing a first phase of the multitude of phases, thereby producing detectable elements.
14. The method according to claim 13, wherein the method comprises automatically steering the construction machine to the desired line; the desired line is a spline; and/or the operator input comprises an offset of the desired line from a given line and the method comprises calculating the desired line based on the offset.
15. The method according to claim 10, wherein the three-dimensional measuring data is captured continuously and evaluated continuously for detecting the elements, and the machine control unit is configured continuously while at least partially automatically performing a phase of the earth-moving operation.
16. The method according to claim 10, wherein the earth-moving operation comprises constructing a ditch, the desired line is a ditch line, particularly a ditch edge line, the first phase is a marking pass along the ditch line, and the produced elements produced by the marking pass comprise an edge and/or surface.
17. The method according to claim 16, wherein the construction machine is a motor grader; the earth-moving operation comprises V ditching; and/or the ditch line is a ditch edge line, a ditch spline or a ditch edge spline.
18. A computer program product comprising program code which is stored on a non-transitory machine-readable medium, and having computer-executable instructions for performing the method according to claim 10.
19. A computer program product comprising program code which is stored on a non-transitory machine-readable medium, and having computer-executable instructions for performing the method according to claim 17.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0057] By way of example only, preferred embodiments will be described more fully hereinafter with reference to the accompanying figures, wherein:
[0058]
[0059]
[0060]
[0061]
[0062]
[0063]
DETAILED DESCRIPTION
[0064]
[0065] In the construction machines 1 shown here, i.e. in graders and crawlers, current automation systems often comprise grade control systems that can be configured to maintain a desired cross slope. This is a simple functionality that can be used by operators during road construction and ditch construction. Typical cross slope systems only control one axis of a tool motion. They act as kinematic constraints, requiring the operator to control a depth of the tool 11, navigate the machine 1, and control other tool motions, e.g. associated with material management.
[0066]
[0067] The operator of a motor grader starts with a light marking cut (see also
[0068]
[0069] As shown in
[0070] As shown in more detail in
[0071] In context, an RCD in particular comprises some sensor arrangement to capture a 3D point cloud of the environment using a non-contact method. Typically, this includes determining distances to points in the environment using time-of-flight (ToF) measurements based on light waves (e.g. in the infrared spectrum), radio waves, or ultrasonic sound waves. However, stereoscopic vision sensors can provide point-cloud measurements of the environment using disparity, or the difference in position of data in the left and right images. For instance, each RCD may comprise at least two cameras and optionally other sensors, such as a laser scanner or an arrangement of ToF cameras. Reality-capture devices 3a,b of one or more measuring units 2 may be configured to contribute to generating the same 3D point cloud of the terrain, to detect obstacles and/or to track persons or animals entering a danger zone around the machine. Measuring units 2 may be provided on many parts of the construction machine, for instance connected to the tool 11, on the chassis 12 and/or on the cab 14.
[0072] The system for configuring the machine control unit comprises a computing unit and a user interface. Preferably, the user interface may be provided at or inside the cab 14 so that it may be used by an operator of the excavator 1 during operation. The user interface comprises a display for displaying live images and/or a graphical user interface (GUI). In the exemplary embodiment shown in
[0073] The user interface comprises input devices that preferably are provided inside the cab 14e.g. comprising a touch-sensitive display (touchscreen) and optionally a stylus for use with the touchscreen. The computing unit can use the measuring data generated by the measuring unit (RCD data)e.g. LiDAR data from a LiDAR scanner and image data from a multitude of camerasfor generating a 3D model, e.g. of the construction site or parts thereof, and optionally also for obstacle detection.
[0074] The construction site measuring system additionally may comprise at least one of the following components, which optionally may be provided in a common housing together with the measuring unit 2 or in a common housing together with the computing unit and/or the user interface: [0075] a GNSS-antenna configured for generating position data, [0076] an Inertial Measurement Unit (IMU) configured for generating IMU data, and [0077] a cellular unit configured for transmitting any data to a remote station or other vehicles, e.g. construction machines or haul trucks on the same construction site.
[0078] For instance, if the measuring data comprises LiDAR data and image data, the image data can be used for colouring the LiDAR data and/or for optimizing a referencing of the LiDAR data by matching them with an image-based generated point cloud (e.g. generated by a visual simultaneous localization and mapping (VSLAM) algorithm). Also a feature tracking and/or feature recognition algorithm can help combining the LiDAR data to a consistent and well-referenced global point cloud. Similarly, the position data gained with the GNSS-antenna and the IMU data from the IMU can be used for a sensor fusion to obtain a higher precision when building a 3D model of the terrain. A VSLAM point cloud generation can also be supported on LiDAR data, in particular in such a way that the LiDAR data introduce scale and thereby increase stability of the algorithm. The LiDAR scanners may be configured for generating the LiDAR data while the two rotation axes of each scanner rotate faster than 0.1 Hz, particularly faster than 1 Hz, with a point acquisition rate of at least 300,000 points per second, particularly at least 500,000 points per second.
[0079] LiDAR scannersas well as ToF camerasmay be capable to capture a 3D representation of the surrounding at a very fast pace. Therefore, with a moving construction machine it is possible to generate a coherent 3D point cloud based on a SLAM (Simultaneous Localization and Mapping) algorithm that uses the LiDAR data or the ToF dataeither alone or in combination with image data from the cameras. Such localization and mapping is specifically advantageous if the construction machine is operating under a bridge or some other place shadowed from GNSS signals. The SLAM algorithm may be supported by at least one IMU providing IMU data that may be processed to stabilize the algorithm. In particular, all such fused sensor data can be processed by a Kalman filter.
[0080] Optionally, the system for configuring the machine control unit may also comprise at least one context camera, e.g. as a part of the construction site measuring system. Said context camera is any camera device that generates an image that provides context of the environment. In particular, the context camera need not provide any data for use in a sensor system. However, images provided by a stereoscopic camera are suitable for use as a context camera in addition to their use as a sensor through the calculation of a disparity map. The context camera may be provided in a common housing together with the RCD 3a,b of the measuring unit 2, at the semi-transparent display 4 or in a common housing together with the computing unit and the user interface. The context image data may be related to the RCD data of the construction site measuring system through the use of extrinsic parameters, which characterize the relative position of the context camera and RCDs as they are mounted on the construction machine 1. Images taken by the context camera are displayable to the operator of the machine on the semi-transparent display 4, on a touchscreen and/or on any other display of the user interface, particularly as a live video.
[0081] The use of one or more RCDs 3a,b mounted to the construction machine enables the entire work plan to be localized to the site based on the RCD data. If the RCDs are used for navigation or mapping (e.g. some variant of SLAM), then a design file can be localized to that navigation frame. If the machine is outfitted with GNSS or another global localization system, then the RCD data can be localized to global coordinates and the design file can be localized to the world frame as well. The design file can then be saved and output for reference or later use.
[0082]
[0083] The system may further comprise a machine control unit 75 operatively coupled with the computing unit 71. The machine control unit 75 is configured to aid the operator in performing earth-moving operations. In particular, this includes at least partially automatically supervising coordinates of a tool of the construction machine, for instance so that the blade of the motor grader of
[0084] The system 70 may also comprise one or more reality capture devices (RCD) 82 and at least one context camera 83 that are operatively coupled with the computing unit 71. An image of the context camera 83 is displayable on a screen, e.g. a touchscreen, of the user interface 73. Alternatively, the RCD 82 and context camera 83 may be provided separately on the construction machine and be connectable to the system 70 so that they are operatively coupled with the computing unit 71.
[0085] The computing unit 71 is configured to receive, from the RCD 82, point cloud data of a surface of a construction site, particularly of a surface of the surrounding of the construction machine. The computing unit 71 is configured to receive, from the user interface 73, user input regarding a planned earth-moving operation on the construction site. The computing unit 71 determines 3D coordinates of the planned earth-moving operation and programmes the machine control unit 75 accordingly.
[0086]
[0087] In order to construct a road with this cross section using the described system, it is necessary for the system to provide for spline offsetting. This allows the operator to use the inner edge of the ditch 30 to define the working direction of the road and then offset the shoulder edge and crown 40 of the road laterally in order to ensure they are parallel to the ditch line. The system should also be capable of generating and following curved splines for steering. This is illustrated in
[0088] A similar methodology can be used when slot grading with a crawler (as shown in Figure lb). The operator performs an initial marking cut and then configures the system to maintain the desired slope and navigate along the desired trench. The operator is then only responsible for managing vehicle load and shifting it from forward to reverse and vice versa until a windrow spreading operation is necessary. In a slot grading approach the operator sets a marking line and then uses the system to generate a windrow to either side of the crawler in order to develop a trench. The system then steers between the two windrows.
[0089]
[0090] Aside from V-ditching, other embodiments of the method and system are can also be used for other applications that comprise iterative passes over the same route to construct a structure, such as road building or slot grading. Aside from motor graders, the disclosed method and workflow can be used with other construction machines, such as crawlers and wheel loaders.
[0091] During performing the method, a three-dimensional point cloud of a 3D surface is captured, for instance using one or more reality-capture devices (RCD) mounted on a construction machine to capture the surface surrounding the machine. This allows localizing the construction machine as well as recognizing features in the surrounding. In particular, the 3D point cloud may be captured continuously. Optionally, a context camera having a known position relative to the 3D point cloud and/or to a measuring system capturing the 3D point cloud captures context image data of the surrounding of the machine.
[0092] The illustrated method starts with performing a marking or registration pass, where the soil is cut just enough to provide a registration line to aid steering on subsequent passes. The operator can either perform 101 the marking pass manually, or start an assistance procedure 110 to be assisted by the machine control unit of the construction machine and/or a system.
[0093] In the latter case, i.e. during the assistance procedure 110, the operator indicates a desired line for the marking pass (ditch line), for instance a straight line or a spline. This indication may comprise an input of coordinates or a line drawn in a context image displayed on a touchscreen. Also, the operator may select a feature in the context image, e.g. a road or similar structure, and indicate an offset for the desired line from the feature. Alternatively or additionally, a data file comprising the coordinates can be uploaded to the system.
[0094] This indication of the desired ditch line is received 111 as input by the system. Since the RCDs mounted to the construction machine enable localization 112 of the construction machine, optionally the automation system can be set to steer 113 to the desired ditch line and/or set to position 114 the tool, e.g. a toe of a mould board, at the designated ditch line and maintain this position, e.g. at least a height and inclination, while the construction machine follows the ditch lineeither steered by the operator, or controlled by the automation system. The marking pass is thus performed 115 fully or partly automatically. In the latter case, for instance, the automation system actuates blade side shift in order to keep the tool, e.g. the toe of the mould board, at the desired position while the operator controls the steering, circle shift and circle angles.
[0095] After the marking pass has been made (manually 101 or using the assistance procedure 110), a following pass can begin, e.g. as shown here a cutting procedure 120 that includes making a first deep cut. Using the RCDs mounted to the construction machine, the construction site measuring system can automatically detect 121 edges and surfaces of the previous pass, i.e. of the marking pass or a previous cutting pass. These edges and surfaces may be detected 121, e.g., in image data and/or point cloud data captured by the RCDs.
[0096] The system can then present these detected elements in a display, for instance by overlaying 122 them in a context image displayed on a touchscreen, and allow the operator to select previously shaped edges and/or surfaces in order to configure the automation systems for a present cutting pass. For instance, the operator can select the edge of the previous cut and configure the system to cut 25 mm below it for the next pass. The operator can select the previous surface and configure the system to maintain a parallel cut for the following pass. The cut might also be intended to change the cross slope of the surface but use the previously sloped edge as a vertex of the plane being shaped on the subsequent cutting pass. These selections are received 123 as operator input on the user interface, e.g. on a touchscreen showing a context image, to configure 124 the automation system for fully or partly automatically performing 125 the cutting pass. This cutting procedure 120 can be repeated until the last pass has been made and the construction of the ditch is completed.
[0097] Optionally, if a user interface displays a context image on a touchscreen that is also used for receiving the operator input (e.g. in steps 111 and 123), the input may include a selection of pixels of the context image or be interpreted as such a selection of pixels. Since the context camera capturing the context image data has a known position relative to the measuring system and/or the 3D measuring data, the selection of pixels (e.g. representing a desired ditch line) can be mapped to the 3D measuring data, e.g. to a surface of a 3D terrain model that is generated based on the 3D measuring data. Based on the mapping, it is then possible to determine 3D coordinates on the surface that can be provided to the machine control unit for at least partially controlling the earth-moving operation. This is described in detail in the European patent application No. 22180149.1.
[0098] The data captured by the RCDs can also be used to monitor the shed material as it forms a windrow. The volume of material in the windrow or the height of the windrow can be used to indicate to the operator when it is necessary to perform a spreading pass (see e.g. steps 7 and 8 in
[0099] Optionally, the method may include storing information regarding previously identified lines and surfaces in a memory and retrieving this information in a subsequent pass. For instance, the method may comprise passes for cleaning a bottom of a ditch (see e.g. steps 9 and 10 in
[0100] In some embodiments, the system may be configured to self-configure. In this mode, the operator may set the cross-slope and height for the marking pass, incrementing the height until the correct depth is reached for a marking pass. The localization capability of the RCDs enables a globally consistent height to be maintained as the vehicle navigates the site. The operator can then use an increment button to tell the system the desired depth of each cutting pass. The construction site measuring system can use its localization capabilities to determine when it was at the beginning position of the job. It can then combine the localization data with the configuration for the previous cutting pass in order to automatically detect and identify the edges and surfaces cut by the machine. It can then automatically set the desired steering spline or mould-board toe line based on the previous pass and operator indicated depth.
[0101] According to some embodiments, a method and system can also be used to monitor the construction of a V-ditch. Since the system knows the workflow, it can be used to track the steps used in the construction as well as the system configuration and work product for each pass. This data provides a real-time status for the construction job and can also be used to train new operators, using the system to designate the desired workflow for inexperienced operators.
[0102] Optionally, the system may include a mechanism for sensing a load of the construction machine. This may include sensing engine torque, watching engine speed changes, hydrostatic loop pressures and or displacements, transmission output torque, torque converter torque, etc. In this case, a depth control of the control unit may be set to a load dependent state where the system can automatically adjust a height of the tool based on the detected load.
[0103] Although aspects are illustrated above, partly with reference to some preferred embodiments, it must be understood that numerous modifications and combinations of different features of the embodiments can be made. All of these modifications lie within the scope of the appended claims.