Method for determining stacking state, controller, and material handling equipment
12492109 ยท 2025-12-09
Assignee
Inventors
Cpc classification
G05D1/86
PHYSICS
B66F9/0755
PERFORMING OPERATIONS; TRANSPORTING
International classification
B66F9/075
PERFORMING OPERATIONS; TRANSPORTING
B66F9/06
PERFORMING OPERATIONS; TRANSPORTING
G05D1/86
PHYSICS
Abstract
Disclosed are a method for determining a stacking state, a controller, and material handling equipment. The technical solution includes: acquiring target data of a first stacking object and a second stacking object; extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object; extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data; and calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state. The present disclosure may help improve efficiency and accuracy of a determining process of a stacking state.
Claims
1. A material handling equipment, comprising a controller, wherein the controller is configured to execute program instructions to implement following steps: acquiring target data of a first stacking object and a second stacking object by using a first sensor; extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object; extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data; and calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first stacking object relative to the second stacking object; wherein the extracting the first boundary of the first stacking object based on the first target data, and extracting the second boundary of the second stacking object based on the second target data comprises: projecting the first target data and the second target data onto a two-dimensional plane to obtain a first projection; and performing contour extraction on the first projection to obtain the first boundary of the first stacking object and the second boundary of the second stacking object in the first projection.
2. The material handling equipment according to claim 1, wherein the controller is further configured to implement following steps: extracting, from the target data, third target data of the first stacking object and fourth target data of the second stacking object; extracting a third boundary of the first stacking object based on the third target data, and extracting a fourth boundary of the second stacking object based on the fourth target data; calculating a width of a second gap between the first stacking object and the second stacking object based on the third boundary and the fourth boundary, and comparing the width of the second gap with the first threshold to determine a second stacking state of the first stacking object relative to the second stacking object; and determining, based on the first stacking state and the second stacking state, a third stacking state of the first stacking object relative to the second stacking object.
3. The material handling equipment according to claim 2, wherein before the acquiring target data of a first stacking object and a second stacking object by using a first sensor, the controller is further configured to implement following steps: controlling the material handling equipment to move to align the first stacking object with the second stacking object in an XOY plane.
4. The material handling equipment according to claim 3, wherein the determining, based on the first stacking state and the second stacking state, the third stacking state of the first stacking object relative to the second stacking object comprises: when the width of the first gap is less than or equal to the first threshold, and the width of the second gap is less than or equal to the first threshold, determining the third stacking state to be sensing-based stacking success; and when the width of the first gap is greater than the first threshold, or the width of the second gap is greater than the first threshold, determining the third stacking state to be sensing-based stacking failure.
5. The material handling equipment according to claim 4, wherein the controller is further configured to implement following steps: when the third stacking state is the sensing-based stacking success, controlling the material handling equipment to end a stacking process.
6. The material handling equipment according to claim 4, wherein the controller is further configured to implement following steps: when the third stacking state is the sensing-based stacking failure, controlling the material handling equipment to lift the first stacking object; re-acquiring the target data of the first stacking object and the second stacking object by using the first sensor; controlling the material handling equipment to adjust a pose; and re-determining the third stacking state.
7. The material handling equipment according to claim 6, wherein the controlling the handling equipment to lift the first stacking object comprises: controlling the handling equipment to lift the first stacking object to a stacking preparation height.
8. The material handling equipment according to claim 6, wherein the controller is further configured to implement following steps: when the third stacking state is the sensing-based stacking failure, controlling the material handling equipment to move to a stacking preparation position.
9. The material handling equipment according to claim 6, wherein the controller is further configured to implement following steps: counting a quantity of the sensing-based stacking failures, and outputting an alarm prompt when the quantity of the sensing-based stacking failures is greater than a first preset quantity of failures.
10. The material handling equipment according to claim 4, wherein the controller is further configured to implement following steps: when the third stacking state is the sensing-based stacking failure, controlling the handling equipment to lift the first stacking object; and re-determining the third stacking state.
11. The material handling equipment according to claim 4, wherein the controller is further configured to implement following steps: acquiring comprehensive pressure data by using a second sensor when the third stacking state is the sensing-based stacking success, and determining, based on the comprehensive pressure data, a fourth stacking state of the first stacking object relative to the second stacking object.
12. The material handling equipment according to claim 11, wherein the determining, based on the comprehensive pressure data, the fourth stacking state of the first stacking object relative to the second stacking object comprises: when the comprehensive pressure data is less than or equal to a second threshold, determining the fourth stacking state to be pressure-sensing stacking success; when the comprehensive pressure data is greater than the second threshold, determining the fourth stacking state to be pressure-sensing stacking failure.
13. The material handling equipment according to claim 12, wherein the controller is further configured to implement following steps: when the fourth stacking state is the pressure-sensing stacking success, controlling the material handling equipment to end a stacking process.
14. The material handling equipment according to claim 12, wherein the controller is further configured to implement following steps: when the fourth stacking state is the pressure-sensing stacking failure, controlling the material handling equipment to lift the first stacking object; re-acquiring the target data of the first stacking object and the second stacking object by using the first sensor; re-acquiring, by the second sensor, the comprehensive pressure data; controlling the material handling equipment to adjust a pose; and re-determining the fourth stacking state.
15. The material handling equipment according to claim 14, wherein the controller is further configured to implement following steps: counting a quantity of the pressure-sensing stacking failures, and outputting an alarm prompt when the quantity of the pressure-sensing stacking failures is greater than a second preset quantity of failures.
16. The material handling equipment according to claim 14, wherein the controller is further configured to implement following steps: counting a quantity of sensing-based stacking failures and a quantity of the pressure-sensing stacking failures; and outputting an alarm prompt when a sum of the quantity sensing-based stacking failures and the quantity of the pressure-sensing stacking failures is greater than a third preset quantity of failures.
17. The material handling equipment according to claim 2, wherein the first target data is data of a first target region of the first stacking object, and the second target data is data of a second target region of the second stacking object; and the first target region and the second target region are respectively located on a same side of the first stacking object and the second stacking object; and the third target data is data of a third target region of the first stacking object, and the fourth target data is data of a fourth target region of the second stacking object; and the third target region and the fourth target region are respectively located on another same side of the first stacking object and the second stacking object.
18. The material handling equipment according to claim 17, wherein the first target region and the third target region are respectively located on opposite sides of the first stacking object, and the second target region and the fourth target region are respectively located on opposite sides of the second stacking object.
19. The material handling equipment according to claim 18, wherein the first target region comprises a bottom edge on a side of the first stacking object, the second target region comprises a top edge on a side of the second stacking object, the third target region comprises a bottom edge on the other side of the first stacking object, and the fourth target region at least comprises a top edge on the other side of the second stacking object.
20. The material handling equipment according to claim 2, wherein the extracting the third boundary of the first stacking object based on the third target data, and extracting the fourth boundary of the second stacking object based on the fourth target data comprises: projecting the third target data and the fourth target data onto a two-dimensional plane to obtain a second projection; and performing contour extraction on the second projection to obtain the third boundary of the first stacking object and the fourth boundary of the second stacking object in the second projection.
21. The material handling equipment according to claim 2, wherein the calculating the width of the second gap between the first stacking object and the second stacking object based on the third boundary and the fourth boundary comprises: determining a second width between the third boundary and the fourth boundary based on the third boundary and the fourth boundary, wherein the second width is a largest width between the third boundary and the fourth boundary; and determining the second width to be the width of the second gap.
22. The material handling equipment according to claim 1, wherein the calculating the width of the first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary comprises: determining a first width between the first boundary and the second boundary based on the first boundary and the second boundary, wherein the first width is a largest width between the first boundary and the second boundary; and determining the first width to be the width of the first gap.
23. A controller, configured to execute program instructions, to implement following steps: acquiring target data of a first stacking object and a second stacking object by using a first sensor; extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object; extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data; and calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first stacking object relative to the second stacking object; wherein the extracting the first boundary of the first stacking object based on the first target data, and extracting the second boundary of the second stacking object based on the second target data comprises: projecting the first target data and the second target data onto a two-dimensional plane to obtain a first projection; and performing contour extraction on the first projection to obtain the first boundary of the first stacking object and the second boundary of the second stacking object in the first projection.
24. A method for determining a stacking state, comprising: acquiring target data of a first stacking object and a second stacking object by using a first sensor; extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object; extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data; and calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first stacking object relative to the second stacking object; wherein the extracting the first boundary of the first stacking object based on the first target data, and extracting the second boundary of the second stacking object based on the second target data comprises: projecting the first target data and the second target data onto a two-dimensional plane to obtain a first projection; and performing contour extraction on the first projection to obtain the first boundary of the first stacking object and the second boundary of the second stacking object in the first projection.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) To describe the technical solutions in the embodiments of the present disclosure or in the conventional technology more clearly, the following briefly describes the accompanying drawings required for describing the embodiments. Apparently, the accompanying drawings in the following description only show some embodiments of the present disclosure, and those skilled in the art may still derive other drawings from these accompanying drawings without creative efforts.
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)
DETAILED DESCRIPTION OF THE EMBODIMENTS
(18) The following clearly and completely describes the technical solutions in embodiments of the present disclosure with reference to the accompanying drawings in embodiments of the present disclosure. Apparently, the described embodiments are merely some but not all of embodiments of the present disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of the present disclosure shall fall within the protection scope of the present disclosure.
(19) Terms used in embodiments of the present disclosure are merely intended to describe particular embodiments but are not intended to limit the present disclosure. The singular forms of a/an, said, and the used in embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly implies otherwise.
(20) It should be understood that, the term and/or used in this specification is merely an association relationship that describes associated objects, and represents that there may be three relationships. For example, A and/or B may represent three cases: only A exists, both A and B exist, and only B exists. In addition, the character / in this specification generally indicates an or relationship between the associated objects. In addition, the term based on used in this specification is not limited to being based on an object only. For example, determining B based on A may indicate: determining B based on A only, or determining B partially based on A.
(21) In an existing warehousing scenario, when material handling equipment stacks one stacking object onto another stacking object, due to an external environment (such as an uneven ground) and an error of the material handling equipment, the stacking may fail. Therefore, it is necessary to determine whether stacking of two stacking objects is successful.
(22) In view of this, the present disclosure provides a new idea, and proposes a method for determining a stacking state, a controller, and material handling equipment.
(23) The method provided in embodiments of the present disclosure may be applied to material handling equipment shown in
(24) The material handling equipment 1 in embodiments of the present disclosure may be an automated guided forklift, a pallet truck, a crane truck, an Automated Guided Vehicle (AGV), an Autonomous Mobile Robot (AMR), a humanoid robot, or the like. A corresponding stacking execution component 20 may be a fork, a robotic arm, or the like.
(25) Generally, the material handling equipment 1 receives a handling task from a Robot Control System (RCS), and after receiving the handling task, the material handling equipment 1 may be controlled by the controller 40 to perform the handling task. For example, the material handling equipment 1 is controlled to stack a first stacking object A onto a second stacking object B.
(26) The RCS may be integrated into the material handling equipment 1, and forms a whole with the material handling equipment 1, or may be separately used as a device different from the material handling equipment 1. For example, the RCS may be a device such as a tablet computer, a notebook computer, a Personal Computer (PC), a local server, or a cloud server. This is not limited in embodiments of the present disclosure.
(27) The sensor 30 may be in a form of a sensor module, and may include radar for collecting point cloud data, for example, Lidar. The Lidar may be installed on the stacking execution component 20, to facilitate collection of point cloud data. Further, the Lidar may be a three-dimensional Lidar. The sensor 30 may also include a camera for collecting image data. The camera may be installed on the stacking execution component 20, to facilitate collection of image data. The sensor 30 may also include a pressure sensor for collecting pressure data, and the pressure sensor may be installed on the stacking execution component 20.
(28) The memory in the controller 40 may be further configured to store data collected by the sensor 30, such as point cloud data, image data, or pressure data.
(29) The following describes the main terms in the present disclosure.
(30) A processor is configured to be responsible for executing core functions such as computation, control, and decision-making. It may receive data from a sensor, run control algorithms, and command an actuator to complete a task. Common processor types may include a central processing unit (CPU), a digital signal processor (DSP), a microcontroller unit (MCU), and so on. A processor in this specification refers to a collection of processors for performing the same or different tasks.
(31) A memory is configured to store data or a program. The memory in this specification refers to a collection of memory used for performing the same or different tasks.
(32) A controller typically includes a processor and memory at a hardware level. Optionally, the controller may also include input/output interfaces, a mainboard, peripheral circuits, and components. At a software level, the controller usually includes a control algorithm, an operating system, communication protocols, and so on. The controller in this specification refers to a set of controllers for performing the same or different tasks.
(33) Material handling equipment refers to equipment that is capable of automatically or semi-automatically performing handling a handling task. Common forms of the material handling equipment include a forklift, an Automated Guided Vehicle (AGV), an Autonomous Mobile Robots (AMR), a humanoid robot, a robotic arm, and the like.
(34) An Automated Guided Forklift (AGF) refers to an intelligent industrial vehicle that integrates forklift technology and Automated Guided Vehicle (AGV) technology. It can automatically complete tasks such as material handling and stacking.
(35) Stacking refers to arranging and piling up some objects vertically according to a specific rule.
(36) A stacking object refers to an object involved in stacking, and specifically may be goods itself or goods with a simple package such as a wrapping film, or may be a container that can hold and carry goods, for example, a material cage, a wooden box, a plastic box, or a pallet.
(37) A stacking process refers to a process in which material handling equipment lifts a first stacking object to make the first stacking object move close to a second stacking object, and then adjusts a pose of the material handling equipment to align the first stacking object with the second stacking object, and thus places the first stacking object on the second stacking object, so as to complete stacking.
(38) The first stacking object refers to a stacking object located above another. The second stacking object refers to a stacking object located below.
(39) A stacking state refers to a relative position state of two stacking object in a stacking process.
(40) Alignment means a state in which two or more stacking objects are arranged in a vertical direction, and at least part of border lines of a stacking object are parallel to or overlapped with another stacking object. The vertical direction refers to a Z-axis direction (namely, a height direction of the material handling equipment) in a coordinate system of material handling equipment.
(41) In embodiments of the present disclosure, still referring to
(42) Each of a first target region and a third target region refers to a specific region that is on the first stacking object and that is used for detection and analysis, for example, a key structural region such as a boundary or an edge target point of the first stacking object.
(43) Each of a second target region and a fourth target region refers to a specific region that is on the second stacking object and that is used for detection and analysis, for example, a key structural region such as a boundary or an edge target point of the second stacking object.
(44)
(45) In addition, the first target region and the third target region may alternatively be located on two adjacent sides of the first stacking object, and the second target region and the fourth target region may alternatively be located on two adjacent sides of the second stacking object.
(46) Target data refers to data acquired by a sensor, including point cloud data, image data, and the like.
(47) First target data is data, acquired by a sensor, of the first target region of the first stacking object, and includes point cloud data and image data. The first target data may be used for calculating a pose, a stacking state, and the like of the first stacking object.
(48) Third target data is data, acquired by a sensor, of the third target region of the first stacking object, and includes point cloud data and image data. The third target data may be used for calculating a pose, a stacking state, and the like of the first stacking object.
(49) Second target data is data, acquired by a sensor, of the second target region of the second stacking object, and includes point cloud data and image data. The second target data may be used for calculating a pose, a stacking state, and the like of the second stacking object.
(50) Fourth target data is data, acquired by a sensor, of the fourth target region of the second stacking object, and includes point cloud data and image data. The fourth target data may be used for calculating a pose, a stacking state, and the like of the second stacking object.
(51) First stacking state refers to a relative position state of the first target region of the first stacking object and the second target region of the second stacking object in a stacking process.
(52) Second stacking state refers to a relative position state of the third target region of the first stacking object and the fourth target region of the second stacking object in a stacking process.
(53) Third stacking state refers to a relative position state, determined based on a first sensor, of the first stacking object and the second stacking object in a stacking process.
(54) Fourth stacking state refers to a relative position state, determined based on a second sensor, of the first stacking object and the second stacking object in a stacking process.
(55) The first sensor may include but is not limited to a three-dimensional Lidar, a depth camera, or the like.
(56) The second sensor may include but is not limited to a pressure switch.
(57)
(58) Step S301: acquiring target data of a first stacking object and a second stacking object by using a first sensor.
(59) Step S302: extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object.
(60) Step S303: extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data.
(61) Step S304: calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first stacking object relative to the second stacking object.
(62) It may be learned from the foregoing procedure that, in the present disclosure, the first stacking state of the first stacking object relative to the second stacking object is determined based on a width of a gap between the first stacking object and the second stacking object. Implementation of this solution is not affected by light and has strong adaptability to an environment and low dependence on data without requiring a large amount of labeled data, so that efficiency and accuracy of a stacking determining process may be improved.
(63) The following describes in detail steps in the foregoing procedure and effects that can be further generated with reference to the embodiment. Terms such as first and second in this disclosure do not have limitations on a size, a sequence, a quantity, or the like, but are merely used to distinguish between names, for example, first stacking object and second stacking object are used to distinguish between two stacking objects, first target region, second target region, third target region and fourth target region are used to distinguish between four target regions, first target data, second target data, third target data and fourth target data are used to distinguish between four target data, and first gap and second gap are used to distinguish between two gaps.
(64) The acquiring target data of the first stacking object and the second stacking object by using the first sensor in Step S301 is described in detail with reference to embodiments.
(65) The first sensor in the embodiment of the present disclosure may include but is not limited to Lidar, such as a 3D Lidar and a depth camera.
(66) Specifically, when the controller controls a fork of material handling equipment to stack the first stacking object stacking onto the second stacking object, Lidar of the material handling equipment is configured to scan the first stacking object and the second stacking object to obtain the target data of the first stacking object and the target data of the second stacking object. Then, the Lidar stores collected target data in the memory in the controller, and the controller acquires the target data from the memory. Alternatively, the Lidar may directly send collected target data to the controller.
(67) The target data herein may be point cloud data, and the point cloud data includes 3D geometric information of the first stacking object and the second stacking object.
(68) In addition, in an actual application process, after the controller aligns the first stacking object and the second stacking object vertically by controlling a pose of the material handling equipment, it is necessary to first lower the fork of the material handling equipment to a fork-out height, allowing the first stacking object to be stacked onto the second stacking object. In this case, the Lidar is configured to scan the first stacking object and the second stacking object to obtain the target data, so that the stacking state may be determined based on the target data.
(69) The extracting, from the target data, the first target data of the first stacking object and the second target data of the second stacking object in Step S302 is described below in detail with reference to the embodiment.
(70) After acquiring the target data, the controller is required to extract the first target data of the first stacking object and the second target data of the second stacking object from the target data.
(71) Optionally, the first target data may be data of a first target region of the first stacking object, the second target data may be data of a second target region of the second stacking object, and the first target region and the second target region are respectively located on a same side of the first stacking object and the second stacking object.
(72) In the embodiment of the present disclosure, the first target region at least includes a bottom edge on a side of the first stacking object, and the second target region at least includes a top edge on a side of the second stacking object.
(73) In a stacking process, along the X-axis, it may be observed that, in an implementation, as shown in
(74) The first target region and the second target region herein are only required to be located on a same side of the first stacking object and the second stacking object respectively and at least include a bottom edge of the first stacking object and a top edge of the second stacking object respectively, and specific positions thereof are not limited in the embodiment of the present disclosure.
(75) Specifically, the controller may obtain pre-stored specifications (such as size and shape) of the first stacking object and the second stacking object from the memory, and then determine the first target data of the first target region and the second target data of the second target region from the target data according to the specifications.
(76) Optionally, after the target data is obtained, preprocessing such as filtering, outlier removal, data cleaning, and data transformation may be performed on the target data. The preprocessing may be performed to remove a noise point from the target data, and reserve valid data for future use. In this way, accuracy and reliability of the target data may be further improved.
(77) The extracting the first boundary of the first stacking object based on the first target data, and extracting the second boundary of the second stacking object based on the second target data in Step S303 is described below in detail with reference to the embodiment.
(78) In an implementation, as shown in
(79) Step S3031: projecting the first target data and the second target data onto a two-dimensional plane to obtain a first projection.
(80) Step S3032: performing contour extraction on the first projection to obtain the first boundary of the first stacking object and the second boundary of the second stacking object.
(81) The two-dimensional plane herein may include but is not limited to a plane for projecting and analyzing the target data in a coordinate system of the material handling equipment.
(82) When the first target region includes a bottom edge on a left (or right) side of the first stacking object, and the second target region includes a top edge on a left (or right) side of the second stacking object, the two-dimensional plane is an XOZ-plane in a coordinate axis centered on the material handling equipment, where an X-axis represents a travel direction (namely, a first direction of the material handling equipment body 10) of the material handling equipment, a Y-axis represents a lateral direction (namely, a second direction of the material handling equipment body 10) of the material handling equipment, and a Z-axis represents a height direction of the material handling equipment, which are consistent with directions of the coordinate axes in
(83) When the first target region includes a bottom edge on a front (or rear) side of the first stacking object, and the second target region includes a top edge on a front (or rear) side of the second stacking object, the two-dimensional plane is a YOZ plane in a coordinate axis centered on the material handling equipment.
(84) A projection direction is not limited to a direction perpendicular to the XOZ-plane or the YOZ-plane, and any projection direction perpendicular to the Z-axis may be used as the projection direction in the embodiment.
(85) The calculating the width of the first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with the first threshold to determine the first stacking state of the first stacking object relative to the second stacking object in Step S304 is described below in detail with reference to the embodiment.
(86) In the embodiment of the present disclosure, the first stacking state includes a first safe state and a first unsafe state, where the first safe state refers to that the width of the first gap is less than or equal to the first threshold.
(87) A target region configuration manner in
(88) As shown in
(89) As shown in
(90) Specifically, when the width of the first gap between the first stacking object and the second stacking object is calculated based on the first boundary and the second boundary, if the width of the gap formed between the first boundary and the second boundary is uneven, the width of the first gap is determined based on a maximum width of a gap formed between the first boundary and the second boundary.
(91) Exemplarily, as shown in
(92) Step S3041: determining a first width between the first boundary and the second boundary based on the first boundary and the second boundary, where the first width is a largest width between the first boundary and the second boundary.
(93) Step S3042: determining the first width to be the width of the first gap.
(94) The first threshold in embodiments of the present disclosure may be obtained through automatic learning based on historical data, or may be set according to an actual requirement, which is not limited in embodiments of the present disclosure.
(95) In addition, before determining the first stacking state of the first stacking object relative to the second stacking object, the controller is required to control movement of the material handling equipment, so as to ensure that the first stacking object and the second stacking object are aligned in the XOY plane. In other words, projections of the first stacking object and the second stacking object on the XOY plane are aligned. In this way, a success rate in subsequent stacking may be improved, and damage or a safety accident caused by misalignment may be reduced. Such precise alignment is also a basis for subsequent measurement of the gap width and determining of a stacking state, ensuring efficiency and accuracy of an entire stacking process.
(96) Exemplarily, as shown in
(97) Step S300: controlling movement of material handling equipment to align the first stacking object and the second stacking object in an XOY plane. Exemplarily, the material handling equipment is controlled to move to align orthographic projections of the first stacking object and the second stacking object on a horizontal plane.
(98) In the embodiment of the present disclosure, a first stacking state of a first stacking object relative to a second stacking object is determined based on a width of a gap formed on any side when the first stacking object is stacked onto the second stacking object, thereby improving efficiency and accuracy of a stacking determining process.
(99) Based on a same idea, an embodiment of the present disclosure further provides a method for determining a stacking state. As shown in
(100) Step S601: acquiring target data of a first stacking object and a second stacking object by using a first sensor.
(101) Step S602: extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object.
(102) Step S603: extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data.
(103) Step S604: calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state where the first stacking object is stacked onto the second stacking object.
(104) Step S605: extracting, from the target data, third target data of the first stacking object and fourth target data of the second stacking object.
(105) Step S606: extracting a third boundary of the first stacking object based on the third target data, and extracting a fourth boundary of the second stacking object based on the fourth target data.
(106) Step S607: calculating a width of a second gap between the first stacking object and the second stacking object based on the third boundary and the fourth boundary, and comparing the width of the second gap with the first threshold to determine a second stacking state of the first stacking object relative to the second stacking object.
(107) The second stacking state includes a second safe state and a second unsafe state, and the second safe state refers to that the width of the second gap is less than or equal to the first threshold.
(108) Step S608: determining, based on the first stacking state and the second stacking state, a third stacking state of the first stacking object relative to the second stacking object.
(109) Specifically, when the width of the second gap between the first stacking object and the second stacking object is calculated based on the third boundary and the fourth boundary, if the width of the gap formed between the third boundary and the fourth boundary is uneven, the width of the second gap is determined based on a maximum width of a gap formed between the third boundary and the fourth boundary.
(110) Content of Step S601 to Step S604 is the same as that of Step S301 to Step S304, and details are not described herein again.
(111) Step S605 to step S607 are merely different from Step S302 to Step S304 in that: in Step S302 to Step S304, the first stacking state is determined based on the first target data of the first target region and the second target data of the second target region; while in Step 605 to Step 607, the second stacking state is determined based on the third target data of the third target region and the fourth target data of the fourth target region. Specific implementation principles are the same, and thus details are not described herein again.
(112) Exemplarily, as shown in
(113) Step S6071: determining a second width between the third boundary and the fourth boundary based on the third boundary and the fourth boundary, where the second width is a largest width between the third boundary and the fourth boundary.
(114) Step S6072: determining the second width to be the width of the second gap.
(115) In an implementation, as shown in
(116) Step S6061: projecting the third target data and the fourth target data onto a two-dimensional plane to obtain a second projection.
(117) Step S6062: performing contour extraction on the second projection to obtain the third boundary of the first stacking object and the fourth boundary of the second stacking object in the second projection.
(118) The two-dimensional plane herein may include but is not limited to a plane for projecting and analyzing the target data in a coordinate system of the material handling equipment.
(119) The determining, based on the first stacking state and the second stacking state, a third stacking state of the first stacking object relative to the second stacking object in Step S608 is described below in detail with reference to the embodiment.
(120) Specifically, when the first stacking state is the first safe state, and the second stacking state is a second safe state, the third stacking state is determined to be sensing-based stacking success; when the first stacking state is a first unsafe state and/or the second stacking state is a second unsafe state, the third stacking state is determined to be sensing-based stacking failure.
(121) In other words, as shown in
(122) Step S6081: when the width of the first gap is less than or equal to the first threshold, and the width of the second gap is less than or equal to the first threshold, determining the third stacking state to be sensing-based stacking success.
(123) Step S6082: when the width of the first gap is greater than the first threshold, or the width of the second gap is greater than the first threshold, determining the third stacking state to be sensing-based stacking failure.
(124) In the embodiment of the present disclosure, a stacking state of the first stacking object relative to the second stacking object is determined jointly based on gap widths on opposite sides when the first stacking object is stacked onto the second stacking object, thereby improving efficiency and accuracy of a determining process of the stacking state.
(125) Further, as shown in
(126) Step S609: when the third stacking state is the sensing-based stacking success, controlling the material handling equipment to end a stacking process. For example, when the material handling equipment is an automated guided forklift, the controller may control the automated guided forklift to perform a fork-out operation.
(127) In addition, when the third stacking state is the sensing-based stacking failure, that is, the first stacking object fails to be stacked on the second stacking object, it may be considered that there is an obvious obstacle on the second stacking object. Thus, when the third stacking state is the sensing-based stacking failure, the material handling equipment may be controlled to lift the first stacking object. In this case, the obstacle may be removed, and the controller may re-determine the third stacking state.
(128) A retry mechanism used when the third stacking state is the sensing-based stacking failure is further proposed in the embodiment of the present disclosure. As shown in
(129) Step S610: when the third stacking state is the sensing-based stacking failure, controlling the material handling equipment to lift the first stacking object.
(130) Step S611: re-acquiring the target data of the first stacking object and the target data of the second stacking object by using the first sensor.
(131) Step S612: controlling the material handling equipment to adjust a pose based on re-acquired target data.
(132) Step S613: re-determining the third stacking state of the first stacking object relative to the second stacking object.
(133) In addition to the retry mechanism, the controller may further control the material handling equipment to move to a stacking preparation position, that is, a starting position returned after stacking determined as failed.
(134) Exemplarily, as shown in
(135) Step S6083: controlling the material handling equipment to move to a stacking preparation position.
(136) In addition, to improve reliability of a stacking operation and avoid a failure attempt of an infinite cycle due to an abnormal condition, an alarm mechanism is further proposed in the embodiment of the present. When the third stacking state is the sensing-based stacking failure, the method for determining the stacking state according to the embodiment of the present disclosure may further include the following steps.
(137) Step S614: counting a quantity of the sensing-based stacking failures, and outputting an alarm prompt when the quantity of the sensing-based stacking failures is greater than a first preset quantity of failures. Once the quantity of the sensing-based stacking failures exceeds a first preset quantity of failures, an alarm prompt is output. This mechanism is designed to alert an operator or an automatic control system of a potential problem in a timely manner, so as to take an appropriate intervention measure to ensure continuity and safety of a stacking operation.
(138) Based on a same idea, an embodiment of the present disclosure further provides a method for determining a stacking state. As shown in
(139) Step S701: acquiring target data of a first stacking object and a second stacking object by using a first sensor.
(140) Step S702: extracting, from the target data, first target data of the first stacking object and second target data of the second stacking object.
(141) Step S703: extracting a first boundary of the first stacking object based on the first target data, and extracting a second boundary of the second stacking object based on the second target data.
(142) Step S704: calculating a width of a first gap between the first stacking object and the second stacking object based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first stacking object relative to the second stacking object.
(143) Step S705: extracting, from the target data, third target data of the first stacking object and fourth target data of the second stacking object.
(144) Step S706: extracting a third boundary of the first stacking object based on the third target data, and extracting a fourth boundary of the second stacking object based on the fourth target data.
(145) Step S707: calculating a width of a second gap between the first stacking object and the second stacking object based on the third boundary and the fourth boundary, and comparing the width of the second gap with the first threshold to determine a second stacking state of the first stacking object relative to the second stacking object.
(146) Step S708: determining, based on the first stacking state and the second stacking state, a third stacking state of the first stacking object relative to the second stacking object.
(147) Step S709: acquiring comprehensive pressure data by using a second sensor when the third stacking state is the sensing-based stacking success, and determining, based on the comprehensive pressure data, a fourth stacking state of the first stacking object relative to the second stacking object.
(148) Content of Step S701 to Step S708 is the same as that of Step S601 to Step S608, and details are not described herein again.
(149) The acquiring comprehensive pressure data by using a second sensor when the third stacking state is the sensing-based stacking success, and determining, based on the comprehensive pressure data, a fourth stacking state of the first stacking object relative to the second stacking object in Step S709 is described below in detail with reference to the embodiment.
(150) In the embodiment of the present disclosure, only when the third stacking state is the sensing-based stacking success, to further ensure accuracy of a determined stacking state, the comprehensive pressure data may be further acquired by using the second sensor, and thus the fourth stacking state is determined based on the comprehensive pressure data. When the third stacking state is the sensing-based stacking failure, there is no need to acquire the comprehensive pressure data by using the second sensor.
(151) Specifically, as shown in
(152) Step S7091: when the comprehensive pressure data is less than or equal to a second threshold, determining the fourth stacking state to be pressure-sensing stacking success.
(153) Step S7092: when the comprehensive pressure data is greater than the second threshold, determining the fourth stacking state to be pressure-sensing stacking failure.
(154) In the embodiment of the present disclosure, the third stacking state of the first stacking object relative to the second stacking object is determined based on gap widths on opposite sides when the first stacking object is stacked onto the second stacking object. When the third stacking state is the sensing-based stacking success, the fourth stacking state is further determined with reference to comprehensive pressure data. The fourth stacking state is pressure-sensing stacking success or pressure-sensing stacking failure. In other words, in this embodiment of the present disclosure, a stacking state of the first stacking object relative to the second stacking object may be jointly determined with reference to data collected by a plurality of sensors, so as to ensure accuracy of a stacking determining process.
(155) Further, as shown in
(156) Step S710: when the fourth stacking state is the pressure-sensing stacking success, controlling material handling equipment to end a stacking process. When the material handling equipment is an automated guided forklift, the ending process of stacking operation may be a fork-out operation.
(157) In addition, a retry mechanism used when the fourth stacking state is the pressure-sensing stacking failure is further proposed in the embodiment of the present disclosure, and the retry mechanism may further include the following steps.
(158) Step S711: when the fourth stacking state is the pressure-sensing stacking failure, controlling the material handling equipment to lift the first stacking object.
(159) Step S712: re-acquiring the target data of the first stacking object and the target data of the second stacking object by using the first sensor.
(160) Step S713: re-acquiring comprehensive pressure data by using the second sensor.
(161) Step S714: controlling the material handling equipment to adjust a pose based on re-acquired target data.
(162) Step S715: re-determining the fourth stacking state.
(163) In addition, to improve reliability of a stacking operation and avoid a failure attempt of an infinite cycle due to an abnormal condition, an alarm mechanism is further proposed in the embodiment of the present disclosure.
(164) In an implementation, the method for determining the stacking state according to the embodiment of the present disclosure may further include the following steps.
(165) Step S716: counting a quantity of the pressure-sensing stacking failures, and outputting an alarm prompt when the quantity of the pressure-sensing stacking failures is greater than a second preset quantity of failures. Once the quantity of the pressure-sensing stacking failures exceeds a second preset quantity of failures, the alarm prompt is output. This mechanism is designed to alert an operator or an automatic control system of a potential problem in a timely manner, so as to take an appropriate intervention measure to ensure continuity and safety of a stacking operation.
(166) In another implementation, the method for determining the stacking state according to the embodiment of the present disclosure may further include the following steps.
(167) Step S717: counting a quantity of the sensing-based stacking failures and a quantity of the pressure-sensing stacking failures, and outputting an alarm prompt when a sum of the quantity of the sensing-based stacking failures and the quantity of the pressure-sensing stacking failures is greater than a third preset quantity of failures.
(168) The first preset quantity of failures, the second preset quantity of failures, and the third preset quantity of failures may have a same value or different values. The first preset quantity of failures, the second preset quantity of failures, and the third preset quantity of failures having a same value is used as an example. For example, the first preset quantity of failures, the second preset quantity of failures, and the third preset quantity of failures are all set to 5. That is, when the quantity of the sensing-based stacking failures exceeds 5, or the quantity of the pressure-sensing stacking failures exceeds 5, or a sum of the quantity of the sensing-based stacking failures and the quantity of the pressure-sensing stacking failures exceeds 5, an alarm prompt is output. A specific quantity of failures in the embodiment of the present disclosure is not limited, which may be adjusted according to an actual situation.
(169) The following describes, with reference to an actual application scenario, an implementation of the method for determining a stacking state provided in the embodiment of the present disclosure. As shown in
(170) Step S800: controlling material handling equipment to move to align the first material cage with the second material cage in an XOY plane.
(171) Step S801: controlling the material handling equipment to lower a fork of the material handling equipment to a fork-out height, and collecting point cloud data of the first material cage and point cloud data of the second material cage by using a first sensor.
(172) Step S802: preprocessing the point cloud data, where the preprocessing includes performing filtering processing on the point cloud data, removing a noise point from the point cloud data to reserve valid point clouds.
(173) Step S8031: extracting, from preprocessed point cloud data, first target data of a first target region of the first material cage and second target data of a second target region of the second material cage.
(174) Step S8032: extracting, from preprocessed point cloud data, third target data of a third target region of the first material cage and fourth target data of a fourth target region of the second material cage.
(175) Step S8041: extracting a first boundary of the first material cage based on the first target data, and extracting a second boundary of the second material cage based on the second target data.
(176) Step S8042: extracting a third boundary of the first material cage based on the third target data, and extracting a fourth boundary of the second material cage based on the fourth target data.
(177) Step S8051: calculating a width of a first gap between the first material cage and the second material cage based on the first boundary and the second boundary, and comparing the width of the first gap with a first threshold to determine a first stacking state of the first material cage relative to the second material cage.
(178) Step S8052: calculating a width of a second gap between the first material cage and the second material cage based on the third boundary and the fourth boundary, and comparing the width of the second gap with a first threshold to determine a second stacking state of the first material cage relative to the second material cage.
(179) Step S806: determining a third stacking state of the first material cage relative to the second material cage based on the first stacking state and the second stacking state. When the third stacking state is sensing-based stacking success, Step S807 is performed; otherwise, Step S809 is performed.
(180) Step S807: acquiring comprehensive pressure data by using a second sensor, and determining, based on the comprehensive pressure data, a fourth stacking state of the first material cage relative to the second material cage. When the fourth stacking state is pressure-sensing stacking success, Step S808 is further performed; when the fourth stacking state is pressure-sensing stacking failure, Step S809 is performed.
(181) Step S808: controlling the material handling equipment to perform a fork-out operation. A process ends.
(182) Step S809: counting a sum of a quantity of the sensing-based stacking failures and a quantity of the pressure-sensing stacking failures, performing Step S810 when the sum is greater than a third preset quantity of failures and performing Step S811 when the sum is not greater than a third preset quantity of failures.
(183) Step S810: outputting an alarm prompt. A process ends.
(184) Step S811: controlling the material handling equipment to lift the fork and adjust a pose of the fork, so as to adjust stacking positions of the first material cage and the second material cage, and then returning to Step S800.
(185) The foregoing describes specific embodiments of this specification. Other embodiments fall within the scope of the appended claims. In some cases, actions or steps recorded in the claims may be performed in an order other than those described in the embodiments, while still bringing desired results. In addition, the desired results do not require a process illustrated in the drawings to necessarily follow the shown specific order or sequential order. In some implementations, multitasking and parallel processing are also possible or may be advantageous.
(186) According to an embodiment of another aspect, a controller is provided. The controller is configured to executes program instructions to implement any method for determining a stacking state disclosed in embodiments of the present disclosure.
(187) According to an embodiment of another aspect, material handling equipment is further provided. The material handling equipment includes a controller, configured to executes program instructions to implement any method for determining a stacking state disclosed in embodiments of the present disclosure.
(188) Embodiments in this specification are all described in a progressive manner, for same or similar parts in embodiments, mutual reference may be made. Each embodiment focuses on what is different from other embodiments. Especially, The controller embodiment or the material handling equipment embodiment is basically similar to a method embodiment, and therefore is described briefly; and for related parts, reference may be made to partial descriptions in the method embodiment. The controller embodiments or the material handling equipment embodiment described above are merely an example. The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, that is, may be located at one position, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to an actual need to achieve the objectives of the solutions of the embodiments. A person of ordinary skill in the art may understand and implement embodiments of the present disclosure without creative efforts.
(189) In addition, an embodiment of the present disclosure further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program. When the program is executed by a processor, the method in any one of the foregoing method embodiments is implemented.
(190) The present disclosure further provides a computer program product, including a computer program. When the computer program is executed by a processor, the method in any one of the foregoing method embodiments is implemented.
(191) According to specific embodiments provided in the present application, the present application discloses the following technical effects.
(192) Firstly, in the present application, a stacking state of the first stacking object relative to the second stacking object is determined based on a width of a gap between the first stacking object and the second stacking object. Because a large quantity of labeled data is not required, dependence on data is reduced, thereby improving efficiency and accuracy of a determining process of a stacking state.
(193) Secondly, in the present application, a stacking state of the first stacking object relative to the second stacking object is determined based on a width of a gap formed on any side when the first stacking object is stacked onto the second stacking object, thereby improving efficiency and accuracy of a determining process of a stacking state.
(194) Thirdly, in the present application, a stacking state of the first stacking object relative to the second stacking object is determined jointly based on gap widths on opposite sides when the first stacking object is stacked onto the second stacking object, thereby improving efficiency and accuracy of a determining process of a stacking state.
(195) Fourthly, in the present application, comprehensive pressure data is further introduced. When sensing-based stacking success is determined, a stacking state of the first stacking object relative to the second stacking object is further determined based on the comprehensive pressure data, thereby improving accuracy of stacking state evaluation.
(196) Fifthly, in the present application, an effective boundary extraction method is provided by projecting target data into a two-dimensional plane and extracting a contour, thereby improving accuracy of stacking state evaluation.
(197) Sixthly, in the present application, when it is determined that stacking of the first stacking object and the second stacking object are successful, a fork-out operation is performed; when it is determined that stacking of the first stacking object and the second stacking object stacking are failed, a stacking position of the first stacking object and the second stacking object is re-adjusted, thereby implementing intelligent control on material handling equipment, optimizing a stacking procedure, reducing manual intervention, improving operation efficiency, and also reducing a safety risk caused by an improper operation.
(198) Seventhly, in the present application, a fork is controlled to lower to a fork-out height, and then a sensor is controlled to perform scanning. Such operation may ensure that the sensor collects data at a best position, improving data quality and ensuring data accuracy.
(199) Eighthly, in the present application, data accuracy and reliability are improved by preprocessing target data, which provides high-quality data for subsequent stacking state evaluation.
(200) Lastly, in the present application, a safety mechanism is provided by counting a quantity of stacking failures and outputting an alarm prompt, thereby ensuring safety and continuity of a stacking operation.
(201) Certainly, any invention of the present application is not necessarily required to achieve all of the advantages described above.
(202) Based on the foregoing descriptions of the implementations, a person skilled in the art may clearly understand that the present disclosure may be implemented by software in addition to a necessary universal hardware platform. Based on such an understanding, the technical solutions in the present disclosure essentially, or the part contributing to the conventional technology may be implemented in a form of a computer program product. The computer program product may be stored in a storage medium, for example, a ROM/RAM, a magnetic disk, or an optical disc, and includes several instructions for instructing a computer device (which may be a personal computer, a server, a network device, or the like) to perform the methods in all embodiments or some of the embodiments of the present disclosure.
(203) The technical solutions provided in the present disclosure are described above in detail. The principles and implementations of the present disclosure are described herein by using specific examples. The descriptions about embodiments of the present disclosure are merely provided to help understand the methods and core ideas of the present disclosure. In addition, a person of ordinary skill in the art can make variations to the present disclosure in terms of the specific implementations and disclosure scopes according to the ideas of the present disclosure. Therefore, the content of this specification shall not be construed as a limitation on the present disclosure.