SYSTEM AND CONTROL METHOD FOR PREVENTING ERRONEOUS OPERATION OF WORK MACHINE, AND EXCAVATOR

Abstract

A camera captures an image of a region including at least a portion of an operating member and generates image data indicative of the image. A controller acquires the image data from the camera. The controller determines whether an operation of the operating member by an operator is an intentional operation or an unintentional operation based on the image. When the operation of the operating member by the operator is determined to be the intentional operation, the controller controls a work machine according to the operation of the operating member. When the operation of the operating member by the operator is determined to be the unintentional operation, the controller invalidates the operation of the operating member.

Claims

1. A system for preventing an erroneous operation of a work machine, the system comprising: an operating member configured to be operated by an operator; a camera configured to capture an image of a region including at least a portion of the operating member and generate image data indicative of the image; and a controller configured to acquire the image data from the camera, the controller being configured to determine whether an operation of the operating member by the operator is an intentional operation or an unintentional operation based on the image, control the work machine according to the operation of the operating member when the operation of the operating member by the operator is determined to be the intentional operation, and invalidate the operation of the operating member when the operation of the operating member by the operator is determined to be the unintentional operation.

2. The system according to claim 1, wherein the operating member is configured to be operated by a hand of the operator, and the controller is configured to determine that the operation of the operating member by the operator is the intentional operation when the image shows that the operating member is held by the hand of the operator.

3. The system according to claim 2, wherein the controller is configured to determine that the operation of the operating member by the operator is the unintentional operation when the image shows that the operating member is touched by a portion other than the hand of the operator.

4. The system according to claim 2, wherein the controller is configured to determine that the operation of the operating member by the operator is the unintentional operation when the image shows that a portion of clothes of the operator is caught on the operating member.

5. The system according to claim 2, wherein the operating member is a lever.

6. The system according to claim 2, wherein the operating member is a steering wheel.

7. The system according to claim 1, wherein the controller includes a trained image recognition model of artificial intelligence, and is configured to apply the image to the image recognition model to determine whether the operation of the operating member by the operator is the intentional operation or the unintentional operation.

8. The system according to claim 1, wherein the work machine is an excavator.

9. A control method of a work machine for preventing an erroneous operation of the work machine that includes an operating member, the control method comprising: acquiring image data indicative of an image of a region including at least a portion of the operating member; determining whether an operation of the operating member by an operator is an intentional operation or an unintentional operation based on the image; controlling the work machine according to the operation of the operating member when the operation of the operating member by the operator is determined to be the intentional operation; and invalidating the operation of the operating member when the operation of the operating member by the operator is determined to be the unintentional operation.

10. The control method according to claim 9, wherein the operating member is configured to be operated by a hand of the operator, the control method further comprising: determining that the operation of the operating member by the operator is the intentional operation when the image shows that the operating member is held by the hand of the operator.

11. The control method according to claim 10, further comprising: determining that the operation of the operating member by the operator is the unintentional operation when the image shows that the operating member is touched by a portion other than the hand of the operator.

12. The control method according to claim 10, further comprising: determining that the operation of the operating member by the operator is the unintentional operation when the image shows that a portion of clothes of the operator is caught on the operating member.

13. The control method according to claim 10, wherein the operating member is a lever.

14. The control method according to claim 10, wherein the operating member is a steering wheel.

15. The control method according to claim 9, further comprising: applying the image to a trained image recognition model of artificial intelligence to determine whether the operation of the operating member by the operator is the intentional operation or the unintentional operation.

16. The control method according to claim 9, wherein the work machine is an excavator.

17. An excavator comprising: a traveling body; a rotating body rotatably attached to the traveling body; a work implement attached to the rotating body; a cab disposed on the rotating body; an operating member disposed in the cab and configured to be operated by an operator to operate at least one of the traveling body, the rotating body, or the work implement; a camera configured to capture an image of a region including at least a portion of the operating member and generate image data indicative of the image; and a controller configured to acquire the image data from the camera, the controller being configured to determine whether an operation of the operating member by the operator is an intentional operation or an unintentional operation based on the image, control at least one operation of the traveling body, the rotating body, or the work implement according to the operation of the operating member when the operation of the operating member by the operator is determined to be the intentional operation, and invalidate the operation of the operating member when the operation of the operating member by the operator is determined to be the unintentional operation.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a side view of a work machine.

[0012] FIG. 2 is a block diagram illustrating a configuration of a control system of the work machine.

[0013] FIG. 3 is a perspective view illustrating an inside of a cab.

[0014] FIG. 4 is a flowchart illustrating processes for detecting an erroneous operation.

[0015] FIG. 5 is a diagram illustrating a configuration of an image recognition model of artificial intelligence.

[0016] FIG. 6 is a diagram illustrating a configuration of the image recognition model of artificial intelligence.

[0017] FIG. 7 is a view illustrating an example of an image of an operating state in which a first operating member is held.

[0018] FIGS. 8A-8D are views illustrating examples of images of various ways the first operating member is held in the operating state.

[0019] FIG. 9 is a view illustrating an example of an image of the operating state in which the first operating member is touched by a leg of the operator.

[0020] FIG. 10 is a view illustrating an example of an image of the operating state in which the first operating member is touched by a body of the operator.

[0021] FIG. 11 is a view illustrating an example of an image of the operating state in which the first operating member is touched by clothing of the operator.

[0022] FIG. 12 is a block diagram illustrating a configuration of the control system of the work machine according to a modified example.

[0023] FIG. 13 is a view illustrating an example of an image of the operating state in which the first operating member is operated by hand.

[0024] FIG. 14 is a view illustrating an example of an image of the operating state in which the first operating member is operated by the operator while standing.

[0025] FIG. 15 is a view illustrating an example of an image of the operating state in which the first operating member is touched by an arm of the operator.

[0026] FIG. 16 is a view illustrating an example of an image of the operating state in which the first operating member is not visible in an image captured by a camera.

DESCRIPTION OF EMBODIMENTS

[0027] Hereinafter, a control system of a work machine 1 according to an embodiment will be described with reference to the drawings. FIG. 1 is a side view of the work machine 1. In the present embodiment, the work machine 1 is a hydraulic excavator.

[0028] As illustrated in FIG. 1, the work machine 1 includes a vehicle body 2 and a work implement 3. The work implement 3 is attached to the front part of the vehicle body 2. The vehicle body 2 includes a rotating body 4, a traveling body 5, and a cab 6. The rotating body 4 is rotatably attached to the traveling body 5. The cab 6 is disposed on the rotating body 4. The traveling body 5 includes crawler belts 6a and 6b. The work machine 1 travels due to the rotation of the crawler belts 6a and 6b.

[0029] The work implement 3 includes a boom 11, an arm 12, and a bucket 13. The boom 11 is attached to the rotating body 4 so as to be movable up and down. The arm 12 is movably attached to the boom 11. The bucket 13 is movably attached to the arm 12. The work implement 3 includes a boom cylinder 14, an arm cylinder 15, and a bucket cylinder 16. The boom cylinder 14, the arm cylinder 15, and the bucket cylinder 16 are hydraulic cylinders and driven by hydraulic fluid supplied from a hydraulic pump 22 described later. The boom cylinder 14 actuates the boom 11. The arm cylinder 15 actuates the arm 12. The bucket cylinder 16 actuates the bucket 13.

[0030] FIG. 2 is a block diagram illustrating a configuration of a control system of the work machine 1. As illustrated in FIG. 2, the work machine 1 includes an engine 21, a hydraulic pump 22, a power transmission device 23, and a controller 24. The engine 21 is controlled by command signals from the controller 24. The hydraulic pump 22 is driven by the engine 21 to discharge hydraulic fluid. The hydraulic fluid discharged from the hydraulic pump 22 is supplied to the boom cylinder 14, the arm cylinder 15, and the bucket cylinder 16.

[0031] The work machine 1 includes a rotation motor 25. The rotation motor 25 is a hydraulic motor and driven by hydraulic fluid from the hydraulic pump 22. The rotation motor 25 causes the rotating body 4 to rotate. Although one hydraulic pump 22 is illustrated in FIG. 2, a plurality of hydraulic pumps may be included.

[0032] The hydraulic pump 22 is a variable displacement pump. A pump control device 26 is connected to the hydraulic pump 22. The pump control device 26 controls the tilt angle of the hydraulic pump 22. The pump control device 26 includes, for example, an electromagnetic valve and is controlled by command signals from the controller 24. The controller 24 controls the pump control device 26, thereby controlling the displacement of the hydraulic pump 22.

[0033] The work machine 1 includes a control valve 27. The hydraulic pump 22, the cylinders 14 to 16, and the rotation motor 25 are connected to each other by means of a hydraulic circuit via the control valve 27. The control valve 27 is controlled by command signals from the controller 24. The control valve 27 controls the flow rate of the hydraulic fluid supplied from the hydraulic pump 22 to the cylinders 14 to 16 and the rotation motor 25. The controller 24 controls the control valve 27, thereby controlling the operation of the work implement 3. The controller 24 controls the control valve 27, thereby controlling the rotation of the rotating body 4.

[0034] The power transmission device 23 transmits driving force of the engine 21 to the traveling body 5. The crawler belts 6a and 6b are driven by the driving force from the power transmission device 23 to cause the work machine 1 to travel. The power transmission device 23 may be, for example, a torque converter or a transmission having a plurality of transmission gears. Alternatively, the power transmission device 23 may be another type of transmission, such as a hydro static transmission (HST) or a hydraulic mechanical transmission (HMT).

[0035] The controller 24 is programmed to control the work machine 1 based on acquired data. The controller 24 controls the engine 21, the traveling body 5, and the power transmission device 23, thereby causing the work machine 1 to travel. The controller 24 controls the hydraulic pump 22 and the control valve 27, thereby causing the work implement 3 to operate. The controller 24 controls the hydraulic pump 22 and the control valve 27, thereby causing the rotating body 4 to rotate.

[0036] The controller 24 includes a processor 31, such as a CPU. The processor 31 executes processes for controlling the work machine 1. The controller 24 includes a storage device 32. The storage device 32 includes a memory, such as a RAM or a ROM, and an auxiliary storage device, such as a hard disk drive (HDD) or a solid state drive (SSD). The storage device 32 stores data and programs for controlling the work machine 1.

[0037] The work machine 1 includes a first operating member 33, a second operating member 34, a third operating member 35, and a fourth operating member 36. FIG. 3 is a perspective view illustrating an inside of the cab 6. As illustrated in FIG. 3, the first operating member 33, the second operating member 34, the third operating member 35, and the fourth operating member 36 are disposed in the cab 6. A seat 37 is disposed in the cab 6. The first operating member 33 is disposed at one side of the seat 37. The second operating member 34 is disposed at the other side of the seat 37. The first operating member 33 and the second operating member 34 are operated by a hand of an operator.

[0038] The first operating member 33 is a lever. The first operating member 33 is tiltable in the front-back and left-right directions from a neutral position. The first operating member 33 outputs a signal indicative of the operating direction and operating amount of the first operating member 33. The controller 24 receives the signal from the first operating member 33. The controller 24 causes the work implement 3 to operate according to the operation of the first operating member 33 by the operator. Alternatively, the controller 24 causes the rotating body 4 to rotate according to the operation of the first operating member 33 by the operator.

[0039] The second operating member 34 is a lever. The second operating member 34 is tiltable in front-back and left-right directions from a neutral position. The second operating member 34 outputs a signal indicative of the operating direction and operating amount of the second operating member 34. The controller 24 receives the signal from the second operating member 34. The controller 24 causes the work implement 3 to operate according to the operation of the second operating member 34 by the operator.

[0040] The third operating member 35 is disposed in front of the seat 37. The third operating member 35 is a lever. The third operating member 35 is tiltable in the front-back direction. The third operating member 35 outputs a signal indicative of the operating direction and operating amount of the third operating member 35. The controller 24 receives the signal from the third operating member 35. The controller 24 causes the work machine 1 to travel according to the operation of the third operating member 35 by the operator.

[0041] The fourth operating member 36 is a pedal. The fourth operating member 36 is coupled to the third operating member 35. The fourth operating member 36 operates integrally with the third operating member 35. The controller 24 causes the work machine 1 to travel according to the operation of the third operating member 35 or the fourth operating member 36 by the operator.

[0042] The work machine 1 includes a locking member 38. The locking member 38 is disposed in the cab 6. The locking member 38 is disposed at a side of the seat 37. The locking member 38 is movable between a locked position and a released position. When the locking member 38 is in the locked position, the controller 24 invalidates the operation of the first operating member 33 and the second operating member 34. That is, when the locking member 38 is in the locked position, the controller 24 prohibits the operation of the work implement 3 regardless of the operation of the first operating member 33 and the second operating member 34. When the locking member 38 is in the locked position, the controller 24 prohibits the rotation of the rotating body 4 regardless of the operation of the first operating member 33.

[0043] For example, in a case where the control valve 27 is an electric pilot type, the controller 24 does not output a command signal to the control valve 27 regardless of the operation of the first operating member 33 and the second operating member 34 when the locking member 38 is in the locked position. Alternatively, in a case where the control valve 27 is a hydraulic pilot type, the controller 24 stops supplying the pilot pressure to the control valve 27 when the locking member 38 is in the locked position.

[0044] When the locking member 38 is in the released position, the controller 24 controls the work implement 3 or the rotating body 4 according to the operation of the first operating member 33 and the second operating member 34. That is, when the locking member 38 is in the released position, the controller 24 causes the work implement 3 to operate according to the operation of the first operating member 33 and the second operating member 34. When the locking member 38 is in the released position, the controller 24 causes the rotating body 4 to rotate according to the operation of the first operating member 33.

[0045] The work machine 1 includes a camera 39. The camera 39 captures an image of a region including the first operating member 33, the second operating member 34, and the seat 37 in the cab 6. The number of cameras 39 is not limited to one and a plurality of cameras may be disposed in the cab 6. The camera 39 generates image data indicative of the captured image. The camera 39 communicates with the controller 24 by wire or wirelessly. The controller 24 receives the image data from the camera 39. The image indicated by the image data may be a still image or a moving image.

[0046] The controller 24 detects an erroneous operation of the first operating member 33 and the second operating member 34 by the operator based on the image. Hereinafter, processes for detecting an erroneous operation executed by the controller 24 will be described. In the following description, a case where the first operating member 33 is operated will be described. However, the same processes may be executed in a case where the second operating member 34 is operated.

[0047] FIG. 4 is a flowchart illustrating processes for detecting an erroneous operation. In step S101, the controller 24 determines whether the lock is released. When the locking member 38 is in the locked position, the controller 24 determines that the lock is not released. When the lock is not released, the controller 24 maintains the lock in step S106. When the locking member 38 is in the released position, the controller 24 determines that the lock is released. When the lock is released, the process proceeds to step S102.

[0048] In step S102, the controller 24 acquires the image data. The controller 24 acquires the image data indicative of an image including the first operating member 33 from the camera 39.

[0049] In step S103, the controller 24 determines whether the operation of the first operating member 33 is performed. The controller 24 determines whether the operation of the first operating member 33 is performed based on a signal from the first operating member 33. When the operation of the first operating member 33 is not performed, the lock is maintained in step S106. When the operation of the first operating member 33 is performed, the process proceeds to step S104.

[0050] In step S104, the controller 24 determines whether the operation of the first operating member 33 by the operator is an intentional operation or an unintentional operation. The controller 24 performs a determination based on the image indicated by the image data.

[0051] The controller 24 determines whether the operation shown in the image is an intentional operation or an unintentional operation by using image recognition technology that uses artificial intelligence (AI). As illustrated in FIG. 5, the controller 24 includes an image recognition model 41 that is trained. The image recognition model 41 is implemented on the controller 24. The image recognition model 41 is an artificial intelligence model for image analysis. The image recognition model 41 analyzes image data D11 that is input and determines whether an image showing a specific operation is included in the images indicated by the image data D11.

[0052] The image recognition model 41 performs image analysis using deep learning. The image recognition model 41 includes a neural network illustrated in FIG. 6. For example, the image recognition model 41 includes a deep neural network, such as a convolutional neural network (CNN). As illustrated in FIG. 6, a neural network 120 includes an input layer 121, an intermediate layer 122, and an output layer 123. The layers 121, 122 and 123 include one or more neurons. The neurons of adjacent layers are coupled together and weights are set for each coupling. The number of neuron couplings may be set as appropriate. Threshold are set for each neuron and output data D12 of each neuron is determined according to whether the sum of the products of the input values to each neuron and the weights exceed the threshold.

[0053] The image data D11 is input to the input layer 121. The output data D12 indicative of a classification of the operation detected in the image is output to the output layer 123. The classification includes an intentional operation and an unintentional operation. The image recognition model 41 is trained to output the output data D12 indicative of the classification of the operation detected in the image when the image data D11 is input. Trained parameters of the image recognition model 41 acquired by training are stored in the controller 24. The trained parameters include, for example, the number of layers of the neural network, the number of neurons in each layer, the coupling relationships among the neurons, the weights of the couplings among neurons, and the thresholds of each neuron.

[0054] For the image showing that the first operating member 33 is held by a hand of an operator 100 as illustrated in FIG. 7, the image recognition model 41 is trained to output the output data D12 indicative of the intentional operation. Therefore, when the image captured by the camera 39 shows that the first operating member 33 is held by the hand of the operator 100, the image recognition model 41 outputs the output data D12 indicative of the intentional operation. In this case, the controller 24 determines that the operation by the operator 100 is the intentional operation.

[0055] For the images of various ways of holding the first operating member 33 by the operator 100 as illustrated in FIGS. 8A-8D, the image recognition model 41 is trained to output the output data D12 indicative of the intentional operation. For example, an image 51 in FIG. 8A shows a state in which the first operating member 33 is held by some fingers with the other fingers untouched. An image 52 in FIG. 8B shows a state in which the first operating member 33 is held by the entire hand. An image 53 in FIG. 8C shows a state in which the first operating member 33 is pushed by a palm. An image 54 in FIG. 8D shows a state in which the first operating member 33 is touched by finger tips. Therefore, while the image captured by the camera 39 in FIGS. 8A-8D show various ways of holding, the controller 24 can still appropriately determine that the operation by the operator 100 is the intentional operation.

[0056] On the other hand, for the image showing that the operating member is touched by a portion other than the hand of the operator 100 as illustrated in FIGS. 9 to 11, the image recognition model 41 is trained to output the output data D12 indicative of the unintentional operation. Therefore, when the image captured by the camera 39 shows that the operating member is touched by a portion other than the hand of the operator 100, the image recognition model 41 outputs the output data D12 indicative of the unintentional operation. In this case, the controller 24 determines that the operation by the operator 100 is the unintentional operation.

[0057] For example, for the image showing that the first operating member 33 is touched by a foot of the operator 100 as illustrated in FIG. 9, the image recognition model 41 is trained to output the output data D12 indicative of the unintentional operation. For the image showing that the first operating member 33 is touched by a body of the operator 100 as illustrated in FIG. 10, the image recognition model 41 is trained to output the output data D12 indicative of the unintentional operation. For the image showing that a portion of clothes of the operator 100 is caught on the first operating member 33 as illustrated in FIG. 11, the image recognition model 41 is trained to output the output data D12 indicative of the unintentional operation. Therefore, when the image captured by the camera 39 shows that the operating member is touched by a portion other than the hand of the operator 100, the controller 24 can appropriately determine that the operation by the operator 100 is the unintentional operation.

[0058] When it is determined in step S104 that the operation by the operator 100 is the intentional operation, the process proceeds to step S105. In step S105, the controller 24 allows the operation of the first operating member 33. That is, the controller 24 causes the work implement 3 or the rotating body 4 to operate according to the operation of the first operating member 33.

[0059] When it is determined in step S104 that the operation by the operator 100 is the unintentional operation, the process proceeds to step S106. In step S106, the controller 24 maintains the lock. That is, the controller 24 invalidates the operation of the first operating member 33 and does not cause the work implement 3 or the rotating body 4 to operate regardless of the operation of the first operating member 33.

[0060] According to the control system of the work machine 1 according to the present embodiment described above, it is determined whether the operation of the first operating member 33 by the operator 100 is the intentional operation or the unintentional operation based on the image of the region including at least a portion of the first operating member 33. Therefore, it is possible to detect an erroneous operation with high accuracy regardless of a way of operating the first operating member 33 by the operator 100. Further, it is possible to quickly determine whether the operation of the first operating member 33 by the operator 100 is a normal operation. Therefore, the deterioration of the operability of the work machine 1 during normal operation can be reduced. The same effect as described above can be achieved in a case where the second operating member 34 is operated.

[0061] Although an embodiment of the present invention has been described so far, the present invention is not limited to the above embodiment and various modifications can be made without departing from the gist of the invention.

[0062] The work machine 1 is not limited to the hydraulic excavator and may be another type of work machine, such as a wheel loader, a bulldozer, or a motor grader. The configuration of the work machine 1 is not limited to that as mentioned above and may be changed. For example, the rotation motor 25 may be an electric motor.

[0063] The first to fourth operating members 33 to 36 are not limited to those of the above embodiment and may be modified. For example, the first to fourth operating members 33 to 36 are not limited to levers and may be switches. A portion of the first to fourth operating members 33 to 36 may be omitted or changed. Alternatively, another operating member, such as a steering wheel, may be provided. The controller 24 may execute the same processes for detecting an erroneous operation on the steering wheel as described above. The work machine 1 may include a steering mechanism. The controller 24 may steer the work machine 1 according to the operation of the operating member by the operator.

[0064] The field of view of the camera 39 may include only one of the first operating member 33 or the second operating member 34. The field of view of the camera 39 may not include the seat 37. A camera may be provided individually for each of the first operating member 33 and the second operating member 34. The field of view of the camera 39 may include the third operating member 35 or the fourth operating member 36. The controller 24 may execute the same processes for detecting an erroneous operation on the third operating member 35 or the fourth operating member 36 as described above.

[0065] The controller 24 may include a plurality of processors, such as a CPU or a GPU. The above processes may be distributed and executed among the plurality of processors 31. The controller 24 is not limited to one unit and the above processes may be distributed and executed among the plurality of controllers. For example, FIG. 12 is a diagram illustrating the control system of the work machine 1 according to a modified example.

[0066] As illustrated in FIG. 12, the control system of the work machine 1 may include a first controller 24a and a second controller 24b. The first controller 24a has the same configuration as the controller 24 of the above embodiment. The second controller 24b includes a processor 31b and a storage device 32b in the same manner as the first controller 24a. The second controller 24 may have a processing capacity suitable for the image recognition using AI. Among the above-mentioned processes, the processes for determining the operation with the image recognition may be executed by the second controller 24b. The first controller 24a may execute processes for controlling the work machine 1 such as outputting the command signals to the control valve 27.

[0067] The order of the above-mentioned processes may be changed. Some of the above-mentioned processes may be changed or omitted. For example, the determination between the intentional operation and the unintentional operation may be performed by another image recognition technology using AI such as a support vector machine, instead of deep learning. Alternatively, the determination between the intentional operation and the unintentional operation is not limited to AI and may be performed by a rule-based image recognition technology such as pattern matching.

[0068] As illustrated in FIG. 13, the controller 24 may determine that the operation of the operator 100 is the intentional operation when the operator 100 is trying to hold the first operating member 33 by the hand. Accordingly, the operability of the work machine 1 can be further improved.

[0069] As illustrated in FIG. 14, when the image captured by the camera 39 shows that the operator 100 holds the first operating member 33 while standing, the controller 24 may determine that the operation by the operator 100 is the intentional operation. For example, the operator 100 may operate the first operating member 33 while standing in order to confirm the surrounding conditions of the work machine 1. Therefore, the controller 24 can appropriately determine that the operation of the operator as illustrated in FIG. 13 is the intentional operation.

[0070] As illustrated in FIG. 15, when the first operating member 33 is not held by a hand of the operator 100 and is touched by an arm, the controller 24 may determine that the operation of the operator 100 is the unintentional operation. As illustrated in FIG. 16, when the operating member does not appear in the image because it is hidden by a wearable object 101 of the operator 100, such as a helmet, the controller 24 may determine that the determination is impossible. Alternatively, when the operating member does not appear in the image because it is hidden by the body of the operator 100, the controller 24 may determine that the determination is impossible. In this case, the controller 24 may maintain the lock.

[0071] According to the present disclosure, it is possible to detect an erroneous operation of the work machine with high accuracy.