METHOD FOR THE COLLISION-FREE MOVEMENT OF A CRANE

Abstract

In a method for a collision-free movement of a crane in a crane lane, a sensor captures a first training data set of raw data during a movement of the crane outside a crane operation in the crane lane. The first training data set is evaluated while teaching a first neural network based on the captured raw data and first training data are determined from the evaluated first training data set. The sensor captures current sensor data during a movement of the crane during the crane operation in the crane lane. The current sensor data is compared with the first training data, and an anomaly is detected between the current sensor data and the first training data.

Claims

1.-16. (canceled)

17. A method for a collision-free movement of a crane in a crane lane, said method comprising: capturing with a sensor a first training data set of raw data during a movement of the crane outside a crane operation in the crane lane; evaluating the first training data set while teaching a first neural network based on the captured raw data; determining first training data from the evaluated first training data set; capturing with the sensor current sensor data during a movement of the crane during the crane operation in the crane lane; comparing the current sensor data with the first training data; and detecting an anomaly between the current sensor data and the first training data.

18. The method of claim 17, wherein the sensor is an optical sensor.

19. The method of claim 17, further comprising: at least partially assigning the first neural network to a central IT infrastructure; and sending the raw data of the first training data set for evaluating the first training data set to the central IT infrastructure.

20. The method of claim 19, further comprising sending the first training data from the central IT infrastructure to a detection module assigned to the crane.

21. The method of claim 17, wherein the sensor is designed as a camera, and further comprising capturing lane markings in a region of the crane lane by the camera.

22. The method of claim 17, further comprising checking a plausibility of a detection of the anomaly through a confidence estimation of the first neural network.

23. The method of claim 17, further comprising: providing second training data from a second training data set of a second neural network; comparing the current sensor data with the second training data; and detecting an object in the current sensor data.

24. The method of claim 23, wherein the object is detected at a same time as the anomaly is detected.

25. The method of claim 23, wherein the object is detected in a detection module assigned to the crane.

26. The method of claim 23, further comprising checking a plausibility of a detection of the object through a confidence estimation of the second neural network.

27. The method of claim 23, further comprising stopping the crane after detection of the anomaly and/or detection of the object.

28. The method of claim 17, further comprising moving the crane in an automated manner in the crane lane.

29. The method of claim 28, wherein the crane is moved completely in an automated manner in the crane lane.

30. A control unit, comprising at least one of a digital logic module, GPU and AI accelerator for carrying out a method as set forth in claim 17.

31. The control unit of claim 30, wherein the digital logic module is a microprocessor, a microcontroller or an ASIC (application-specific integrated circuit).

32. A computer program product, comprising a computer program embodied in a non-transitory computer readable medium, wherein the computer program, when loaded into a control unit as set forth in claim 31 and executed by the control unit, causes the control unit to perform the steps of: capturing with a sensor a first training data set of raw data during a movement of the crane outside a crane operation in the crane lane; evaluating the first training data set while teaching a first neural network based on the captured raw data; determining first training data from the evaluated first training data set; capturing with the sensor current sensor data during the movement of the crane during the crane operation in the crane lane; comparing the current sensor data with the first training data; and detecting an anomaly between the current sensor data and the first training data.

33. A safety system, comprising: a sensor designed to capture a first training data set of raw data during a movement of a crane outside a crane operation in a crane lane; and a control unit communicating with the sensor to receive the first training data set of raw data and designed to carry out a method as set forth in claim 17.

34. The safety system of claim 33, wherein the sensor is an optical sensor.

35. A crane, comprising a safety system, said safety system comprising a sensor designed to capture a first training data set of raw data during a movement of a crane outside a crane operation in a crane lane, and a control unit communicating with the sensor to receive the first training data set of raw data and designed to carry out a method as set forth in claim 17.

36. The crane of claim 35, designed as a gantry crane and further comprising a further said sensor, said crane being movable in at least two directions of travel, in particular opposite directions of travel, with the directions of travel being assigned to the sensor and further sensor, respectively, which have each a detection area in a corresponding one of the directions of travel.

Description

[0029] The invention is described and explained in more detail hereinafter with reference to the exemplary embodiments in the figures.

[0030] FIG. 1 A diagrammatic view of a gantry crane,

[0031] FIG. 2 A flow chart of a first method for the movement of a crane in an automated manner,

[0032] FIG. 3 A flow chart of a second method for the movement of a crane in an automated manner,

[0033] FIG. 4 A flow chart of a third method for the movement of a crane in an automated manner,

[0034] FIG. 5 A flow chart of a fourth method for the movement of a crane in an automated manner,

[0035] FIG. 6 A flow chart of an image evaluation in a detection module,

[0036] FIG. 7 A first example image with a lane marking and

[0037] FIG. 8 A second example image with a lane marking.

[0038] The exemplary embodiments explained hereinafter are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention which are to be considered independently of one another and which in each case also develop the invention independently of one another and are therefore also to be regarded as part of the invention individually or in a combination other than that shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0039] The same reference characters have the same meaning in the various figures.

[0040] FIG. 1 shows a diagrammatic view of a crane 2 which can be moved in a crane lane 4 in a first direction of travel 6 and in a second direction of travel 8. The crane 2 is designed, by way of example, as a rubber-tired gantry crane, in particular a container crane, which has supports 10 which are connected via a crane bridge 12. A spreader and a trolley are not shown in FIG. 1 for reasons of clarity. During a crane operation, the crane 2 is moved in an automated manner by means of lane markings 16 for loading and/or unloading loads 14 designed as containers. The lane markings 16 are designed, for example, as hatched areas. Alternatively, the crane 2 is moved in an automated manner on rails. For the movement of the crane 2 in an automated manner in the crane lane 4, at least one sensor 18, in particular an optical sensor, is used. By way of example, the crane 2 in FIG. 1 has two sensors 18 in each case for a direction of travel 6, 8. For example, the sensors 18 are designed as cameras for capturing the lane markings 16 in the region of the crane lane 4, one of the two cameras for the respective direction of travel 6, 8 being mounted on one of the supports 10 of the crane 2 and having a detection area 20 in the respective direction of travel 6, 8. In particular, the cameras are arranged in a weatherproof housing with a sun roof and are installed downwards at an angle of 20? to 30?, in particular 25??2?, in order to minimize impairment of image capture by weather conditions, for example by sun and rain, even over a longer period of time. In particular, the four sensors 18 designed as cameras, for example for the purpose of remote control and/or for automatic driving of the crane 2, called ASA (Auto Steering Assistance System), are already installed on the crane 2, obviating the need for any additional sensor hardware. The data captured by the sensors 18 is transmitted to a detection module 22 for video evaluation. The detection module 22 comprises crane automation, which is also called crane PLC, and a control unit 23 for controlling the method. Depending on the direction of travel 6, 8 an evaluation for the respective camera side is carried out. In particular, the cameras of the respective camera side are simultaneously evaluated and run through the same detection process in parallel. The detection module 22 is connected to a central IT infrastructure 26 via a digital data connection 24, A central IT infrastructure 26 is, for example, at least one local computer system, which is not assigned to the crane, and/or a cloud. The central IT infrastructure 26 provides storage space, computing power and/or application software. In the cloud storage space, computing power and/or application software are provided as a service via the Internet. Such a cloud environment is the MindSphere, for example. Data transmission, in particular digital data transmission, takes place, for example, wirelessly. In particular, the data is transmitted via WLAN. The central IT infrastructure 26 comprises a first neural network 28 in FIG. 1. In particular, the detection module 22 assigned to the crane 2 comprises a first neural network 28, which is provided via the central IT infrastructure 26.

[0041] FIG. 2 shows a flow chart of a first method for the movement of a crane 2 in an automated manner, the crane 2, for example, being designed as in FIG. 1. The method comprises capturing 30 a first training data set of temporally successive raw data by means of at least one sensor 18, in particular an optical sensor, during a movement of the crane 2 outside crane operation in the crane lane 4. In addition, further training data can be collected during operation for subsequent optimization. In particular, the first training data set is captured when the crane 2 is moved in the first direction of travel 6 and in the second direction of travel 8. The raw data is embodied, for example, as camera images which are read cyclically during the movements of the crane 2 and made available to the detection module 22. In particular, lane markings 16 are captured in the region of the crane lane 4 by means of at least one camera.

[0042] An evaluation 32 of the first training data set is then carried out while teaching a first neural network 28 on the basis of the captured raw data. The raw data comprises, for example, image sequences of day and night times as well as different weather conditions of the crane lane 4 in a normal situation or target situation, In particular, additional information, which is stored, for example, in an additional text file, is assigned to the image sequences manually or in an automated manner. The additional information includes, for example, label information. Label information contains information about where a search pattern is located in an image. As different lane markings 16 are used in terminals, for example, as yet unknown types of lane markings 16, which are in particular called object classes, can be trained when teaching the first neural network 28. For example, the first neural network 28 is at least partially assigned to the central IT infrastructure 26, the raw data for evaluating 32 the first training data set being sent to the central IT infrastructure 26 as high GPU/CPU capacities are required for this purpose. For example, it is placed on an already trained first neural network 28, this being trained to recognize new, in particular project-specific, lane markings 16.

[0043] First training data is then determined 34 from the evaluated first training data set, the first training data being sent from the central IT infrastructure 26 to the detection module 22 of the crane 2. The teaching described by means of the first neural network 28 takes place, for example, during commissioning of the crane 2 and can be expanded as required during a project phase.

[0044] During actual crane operation, capturing 36 of current sensor data takes place by means of the at least one, in particular optical, sensor 18 during a movement of the crane 2 in a direction of travel 6, 8 in the crane lane 4, a comparison 38 of the current sensor data with the first training data thereupon taking place.

[0045] If an object, for example, a person or an object, is located in the region of the crane lane 4 and is captured by at least one sensor 18 during crane operation, detection 40 of an anomaly takes place between the current sensor data and the first training data. Detecting 40 the anomaly takes place independently of the kind, the shape and the type of object as it is not possible to predict which object may be located in the region of the crane lane 4 and whether this represents an obstacle for the crane.

[0046] For example, after detecting 40 the anomaly, an alarm is triggered and/or the complete loading process is automatically stopped. In particular, evaluation images which triggered the alarm and/or led to the stop can be archived. The evaluation images can be displayed, for example at an operator operating station.

[0047] FIG. 3 shows a flow chart of a second method for the movement of a crane 2 in an automated manner. After detecting 40 the anomaly, checking 42 the plausibility of the detection of the anomaly takes place by means of a confidence estimation of the first neural network 28. The further embodiment of the method in FIG. 3 corresponds to that in FIG. 2.

[0048] FIG. 4 shows a flow chart of a third method for the movement of a crane 2 in an automated manner. The third method comprises providing 44 second training data from a second training data set of a second neural network 48. In particular, the second neural network 46 is pretrained for object detection. Pretrained objects are, for example, persons, cars, transport vehicles, lifting tools and/or containers.

[0049] A comparison 48 of the current sensor data with the second training data is then carried out. In particular, for the comparison 48 with the second training data, essentially at the same time, the same current sensor data is used for the comparison with the first training data 38. Furthermore, the same at least one sensor 18 is used for both comparisons. If an object is located in the region of the crane lane 4 and is captured by at least one sensor 18 during crane operation, capturing 50 the object takes place in the current sensor data. In particular, detecting 50 the object takes place essentially at the same time as detecting 40 the anomaly, the greatest possible stability of the system being achieved by a combination of the results of both detection methods, anomaly and object detection.

[0050] Stopping 52 the crane 2 then takes place after detecting 40 the anomaly and/or detecting 50 the object. Alternatively, an alarm is triggered. If required, the crane 2 is stopped manually. The further embodiment of the method in FIG. 4 corresponds to that in FIG. 3.

[0051] FIG. 5 shows a flow chart of a fourth method for the movement of a crane 2 in an automated manner. After detecting 50 the object, checking 54 the plausibility of the detection of the object takes place by means of a confidence estimation of the second neural network 46. The further embodiment of the method in FIG. 5 corresponds to that in FIG. 3.

[0052] FIG. 6 shows a flow chart of an image evaluation in a detection module, the provision 56 of current sensor data taking place by means of the four cameras shown in FIG. 1. The four image sequences 58, 60, 62, 64 respectively captured by a camera each comprise label information 66 in the region of the lane markings 16 for marking the desired image sections. Depending on the direction of travel 6, 8, in each case two of the image sequences 58, 60, 62, 64 are relevant for further evaluation. Comparing the current sensor data with the respective training data results in the detection 40 of an anomaly 68 and the detection 50 of an object 70. Stopping 52 of the crane 2 thereupon takes place.

[0053] FIG. 7 shows a first example image 72 with a lane marking 16, which is designed as hatched areas and is suitable, for example, for a rubber-tired gantry crane.

[0054] FIG. 8 shows a second example image 74 with a lane marking 16, which is designed as rails for a crane 2 which can be moved on rails. Furthermore, FIG. 8 shows label information 66 for marking the desired image section.

[0055] In summary, the invention relates to a method for collision-free movement of a crane 2 in a crane lane 4, In order to achieve the highest level of reliability possible, it is proposed that the method comprises the Wowing steps: capturing 30 a first training data set of temporally successive raw data by means of at least one sensor 18, in particular an optical sensor, during a movement of the crane 2 outside crane operation in the crane lane 4; evaluating 32 the first training data set, teaching a first neural network 28 on the basis of the captured raw data; determining 34 first training data from the evaluated first training data set; capturing 36 current sensor data by means of the at least one sensor 18, in particular an optical sensor, during a movement of the crane 2 during crane operation in the crane lane 4; comparing 38 the current sensor data with the first training data and detecting 40 an anomaly between the current sensor data and the first training data.