SYSTEMS AND METHODS FOR PROCESSING OBJECTS, INCLUDING AUTOMATED RADIAL PROCESSING STATIONS
20230322490 · 2023-10-12
Inventors
- Thomas Wagner (Concord, MA, US)
- Kevin Ahearn (Nebo, NC, US)
- John Richard Amend, Jr. (Belmont, MA, US)
- Benjamin Cohen (Somerville, MA, US)
- Michael Dawson-Haggerty (Pittsburgh, PA, US)
- William Hartman Fort (Stratham, NH, US)
- Christoper GEYER (Arlington, MA, US)
- Jennifer Eileen King (Oakmont, PA, US)
- Thomas Koletschka (Cambridge, MA, US)
- Michael Cap Koval (Mountain View, CA, US)
- Kyle Maroney (North Attleboro, MA, US)
- Matthew T. Mason (Pittsburgh, PA, US)
- William Chu-Hyon McMahan (Cambridge, MA, US)
- Gene Temple Price (Cambridge, MA, US)
- Joseph Romano (Arlington, MA, US)
- Daniel Smith (Canonsburg, PA, US)
- Siddhartha Srinivasa (Seattle, WA, US)
- Prasanna Velagapudi (Pittsburgh, PA, US)
- Thomas Allen (Reading, MA, US)
Cpc classification
B07C3/008
PERFORMING OPERATIONS; TRANSPORTING
G06Q10/087
PHYSICS
G06K7/10861
PHYSICS
B65G1/1373
PERFORMING OPERATIONS; TRANSPORTING
B65G1/1376
PERFORMING OPERATIONS; TRANSPORTING
B25J9/0093
PERFORMING OPERATIONS; TRANSPORTING
International classification
B65G1/137
PERFORMING OPERATIONS; TRANSPORTING
G06Q10/087
PHYSICS
B25J9/00
PERFORMING OPERATIONS; TRANSPORTING
G06K7/10
PHYSICS
B07C5/34
PERFORMING OPERATIONS; TRANSPORTING
Abstract
A method of processing objects using a programmable motion device is disclosed. The method includes the steps of perceiving identifying indicia representative of an identity of a plurality of objects and directing the plurality of objects toward an input area from at least one input conveyance system, acquiring an object from the plurality of objects at the input area using an end effector of the programmable motion device, and moving the acquired object toward an identified processing location using the programmable motion device, said identified processing location being associated with the identifying indicia and said identified processing location being provided as one of a plurality of processing locations that are radially spaced from the programmable motion device.
Claims
1-34. (canceled)
35. A processing system for processing objects comprising: a circulating input conveyor for conveying objects to be processed, said circulating input conveyor providing that the objects may be conveyed in a circuit, a plurality of processing stations in communication with the input conveyance system via an input area conveyor of each processing station that is adapted to receive objects from the input conveyor, each of the plurality of processing stations including: a perception unit for perceiving identifying indicia representative of an identity of an object within the input area; a plurality of processing bins provided in a plurality of semicircular rows, each of the plurality of semicircular rows being provided at a different elevation; and a programmable motion device for grasping the object from the input area and for moving to object to a selected processing bin of the plurality of processing bins.
36. The processing system as claimed in claim 35, wherein the input area includes two input stations, one on either side of the programmable motion device.
37. The processing system as claimed in claim 35, wherein a stationary base of the programmable motion device is positioned on a floor below the input area.
38. The processing system as claimed in claim 35, wherein a stationary base of the programmable motion device is suspended from above with respect to the input area.
39. The processing system as claimed in claim 35, wherein said processing system is provided with a plurality of processing systems, each of which is in communication with the at least one input conveyance system.
40. The processing system as claimed in claim 35, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum risk factor.
41. The processing system as claimed in claim 35, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum time factor.
42. The processing system as claimed in claim 35, wherein each processing bin includes at least one associated indicator light.
43. The processing system as claimed in claim 35, wherein the plurality of semicircular rows of processing bins are arranged between the input conveyance system and an output conveyance system.
44. A processing system for processing objects comprising: an input conveyor for conveying objects to be processed, and a plurality of processing stations in communication with the input conveyance system via an input area conveyor of each processing station that is adapted to receive objects from the input conveyor, each of the plurality of processing stations including: a perception unit for perceiving identifying indicia representative of an identity of an object within the input area; a plurality of processing bins provided in a plurality of semicircular rows; and a programmable motion device for grasping the object from the input area and for moving the object to a selected processing bin of the plurality of processing bins, said plurality of processing bins at each of the plurality of processing rows each being tilted toward the programmable motion device.
45. The processing system as claimed in claim 44, wherein the input area includes two input stations, one on either side of the programmable motion device.
46. The processing system as claimed in claim 44, wherein a stationary base of the programmable motion device is positioned on a floor below the input area.
47. The processing system as claimed in claim 44, wherein a stationary base of the programmable motion device is suspended from above with respect to the input area.
48. The processing system as claimed in claim 44, wherein said processing system is provided with a plurality of processing systems, each of which is in communication with the at least one input conveyance system.
49. The processing system as claimed in claim 44, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum risk factor.
50. The processing system as claimed in claim 44, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum time factor.
51. The processing system as claimed in claim 44, wherein each processing bin includes at least one associated indicator light.
52. The processing system as claimed in claim 44, wherein the plurality of semicircular rows of processing bins are arranged between the input conveyance system and an output conveyance system.
53. A processing system for processing objects comprising: an input conveyor for conveying objects to be processed, a plurality of processing stations in communication with the input conveyance system via a pair of input area conveyors of each processing station that are adapted to receive objects from the input conveyor, each of the plurality of processing stations including: a perception unit for perceiving identifying indicia representative of an identity of an object within either of the pair of the input areas; a plurality of processing bins provided in a plurality of semicircular rows, the plurality of semicircular rows being provided at a plurality of elevations; and a programmable motion device for grasping the object from either of the pair of the input areas and for moving the object to a selected processing bin of the plurality of processing bins, said pair of input areas being provided on either side of the programmable motion device.
54. The processing system as claimed in claim 53, wherein a stationary base of the programmable motion device is positioned on a floor below the input area.
55. The processing system as claimed in claim 53, wherein a stationary base of the programmable motion device is suspended from above with respect to the input area.
56. The processing system as claimed in claim 53, wherein said processing system is provided with a plurality of processing systems, each of which is in communication with the at least one input conveyance system.
57. The processing system as claimed in claim 53, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum risk factor.
58. The processing system as claimed in claim 53, wherein the processing system includes a trajectory planning system for selecting a path to the selected processing bin with an associated minimum time factor.
59. The processing system as claimed in claim 53, wherein each processing bin includes at least one associated indicator light.
60. The processing system as claimed in claim 53, wherein the plurality of semicircular rows of processing bins are arranged between the input conveyance system and an output conveyance system.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0015] The following description may be further understood with reference to the accompanying drawings in which:
[0016]
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029] The drawings are shown for illustrative purposes only.
DETAILED DESCRIPTION
[0030] In accordance with an embodiment, the invention provides a method of processing objects using a programmable motion device. The method includes the steps of perceiving identifying indicia representative of a plurality of objects and directing the plurality of objects toward an input area from at least one input conveyance system. The method also includes the steps of acquiring an object from the plurality of objects at the input area using an end effector of the programmable motion device, and moving the acquired object toward an identified processing location using the programmable motion control device. The identified processing location is associated with the identifying indicia, and the identified processing location is provided as one of a plurality of processing locations.
[0031] Generally, objects need to be identified and conveyed to desired object specific locations. The systems reliably automate the identification and conveyance of such objects, employing in certain embodiments, a set of conveyors, a perception system, and a plurality of destination bins. In short, applicants have discovered that when automating sortation of objects, there are a few main things to consider: 1) the overall system throughput (objects sorted per hour), 2) the number of diverts (i.e., number of discrete locations to which an object can be routed), 3) the total area of the sortation system (square feet), and 4) the capital and annual costs to purchase and run the system.
[0032] Processing objects in a break-pack distribution center is one application for automatically identifying and processing objects. As noted above, in a break-pack distribution center, objects commonly arrive in trucks, are conveyed to sortation stations where they are sorted according to desired destinations into boxes (or packages) that are then then loaded in trucks for transport to, for example, shipping or distribution centers or retail stores. In a shipping or distribution center, the desired destination is commonly obtained by reading identifying information printed on the box or package. In this scenario, the destination corresponding to identifying information is commonly obtained by querying the customer's information system. In other scenarios, the destination may be written directly on the box, or may be known through other means such as by assignment to a vendor bin.
[0033] The system also requests specific bins of objects from a storage system, which helps optimize the process of having desired objects be delivered to specific singulator cells in an efficient way without simply letting all bins of objects appear at each singulator cell in a purely random order.
[0034]
[0035] The plurality of processing locations 52 are provided for example, as a plurality of sets of boxes 54, 56 that are generally provided in a plurality of semicircles of varying height and radial distances from a central region. At the central region, is provided a base of a programmable motion device (e.g., a robotic articulated arm) 58 that includes an end effector for grasping objects from the input areas 46, 48, and is programmed to move each object to one of the boxes of the plurality of sets of boxes 54, 56. When a box is complete, an indicator 60 such as a light, is illuminated, indicating that the associated box may be removed (e.g., by a human) to an output conveyor 62. The completed box (e.g., 64) may be sealed and labeled at the processing station 30, or may be sealed and labeled at a later station.
[0036] It is assumed that the vendor bins that contain objects are marked in one or more places on their exterior of the vendor bin with a visually distinctive mark such as a barcode (e.g., providing a UPC code) or radio-frequency identification (RFID) tag or mailing label so that they may be sufficiently identified with a scanner for processing. The type of marking depends on the type of scanning system used, but may include 1D or 2D code symbologies. Multiple symbologies or labeling approaches may be employed. The types of scanners employed are assumed to be compatible with the marking approach. The marking, e.g. by barcode, RFID tag, mailing label or other means, encodes a identifying indicia (e.g., a symbol string), which is typically a string of letters and/or numbers. The symbol string associates the vendor bin with a specific set of homogenous objects.
[0037] The operations of the system described above are coordinated with a central control system 70 as shown in
[0038] In accordance with another embodiment, the system may employ a programmable motion device that is suspended from above, and processes decanted vendor bins from an in-feed conveyor, moving objects to processing locations as discussed above. With reference to
[0039] As discussed above, systems of embodiments include a perception system (e.g., 50, 52) that is mounted above a bin of objects to be processed, looking down into the bin (e.g., 44). The system 50, for example and as shown in
[0040]
[0041] If an object cannot be fully perceived by the detection system, the perception system considers the object to be two different objects, and may propose more than one candidate grasp of such two different objects. If the system executes a grasp at either of these bad grasp locations, it will either fail to acquire the object due to a bad grasp point where a vacuum seal will not occur (e.g., on the right), or will acquire the object at a grasp location that is very far from the center of mass of the object (e.g., on the left) and thereby induce a great deal of instability during any attempted transport. Each of these results is undesirable.
[0042] If a bad grasp location is experienced, the system may remember that location for the associated object. By identifying good and bad grasp locations, a correlation is established between features in the 2D/3D images and the idea of good or bad grasp locations. Using this data and these correlations as input to machine learning algorithms, the system may eventually learn, for each image presented to it, where to best grasp an object, and where to avoid grasping an object.
[0043] As shown in
[0044]
[0045] The invention provides therefore in certain embodiments that grasp optimization may be based on determination of surface normal, i.e., moving the end effector to be normal to the perceived surface of the object (as opposed to vertical or gantry picks), and that such grasp points may be chosen using fiducial features as grasp points, such as picking on a barcode, given that barcodes are almost always applied to a flat spot on the object.
[0046] In accordance with various embodiments therefore, the invention further provides a sortation system that may learn object grasp locations from experience (and optionally human guidance). Systems designed to work in the same environments as human workers will face an enormous variety of objects, poses, etc. This enormous variety almost ensures that the robotic system will encounter some configuration of object(s) that it cannot handle optimally; at such times, it is desirable to enable a human operator to assist the system and have the system learn from non-optimal grasps.
[0047] The system optimizes grasp points based on a wide range of features, either extracted offline or online, tailored to the gripper's characteristics. The properties of the suction cup influence its adaptability to the underlying surface, hence an optimal grasp is more likely to be achieved when picking on the estimated surface normal of an object rather than performing vertical gantry picks common to current industrial applications.
[0048] In addition to geometric information the system uses appearance based features as depth sensors may not always be accurate enough to provide sufficient information about graspability. For example, the system can learn the location of fiducials such as barcodes on the object, which can be used as an indicator for a surface patch that is flat and impermeable, hence suitable for a suction cup. One such example is the use of barcodes on consumer products. Another example is shipping boxes and bags, which tend to have the shipping label at the object's center of mass and provide an impermeable surface, as opposed to the raw bag material, which might be slightly porous and hence not present a good grasp.
[0049] By identifying bad or good grasp points on the image, a correlation is established between features in the 2D/3D imagery and the idea of good or bad grasp points; using this data and these correlations as input to machine learning algorithms, the system can eventually learn, for each image presented to it, where to grasp and where to avoid.
[0050] This information is added to experience based data the system collects with every pick attempt, successful or not. Over time the robot learns to avoid features that result in unsuccessful grasps, either specific to an object type or to a surface/material type. For example, the robot may prefer to avoid picks on shrink wrap, no matter which object it is applied to, but may only prefer to place the grasp near fiducials on certain object types such as shipping bags.
[0051] This learning can be accelerated by off-line generation of human-corrected images. For instance, a human could be presented with thousands of images from previous system operation and manually annotate good and bad grasp points on each one. This would generate a large amount of data that could also be input into the machine learning algorithms to enhance the speed and efficacy of the system learning.
[0052] In addition to experience based or human expert based training data, a large set of labeled training data can be generated based on a detailed object model in physics simulation making use of known gripper and object characteristics. This allows fast and dense generation of graspability data over a large set of objects, as this process is not limited by the speed of the physical robotic system or human input.
[0053] The robotic system may also employ motion planning using a trajectory database that is dynamically updated over time, and is indexed by customer metrics. The problem domains contain a mix of changing and unchanging components in the environment. For example, the objects that are presented to the system are often presented in random configurations, but the target locations into which the objects are to be placed are often fixed and do not change over the entire operation.
[0054] One use of the trajectory database is to exploit the unchanging parts of the environment by pre-computing and saving into a database trajectories that efficiently and robustly move the system through these spaces. Another use of the trajectory database is to constantly improve the performance of the system over the lifetime of its operation. The database communicates with a planning server that is continuously planning trajectories from the various starts to the various goals, to have a large and varied set of trajectories for achieving any particular task. In various embodiments, a trajectory path may include any number of changing and unchanging portions that, when combined, provide an optimal trajectory path in an efficient amount of time.
[0055]
[0056] Once the articulated arm has acquired an object and is positioned at the base location, the paths to each of the plurality of sets of destination bins 54, 56 are not changing. In particular, each destination bin is associated with a unique destination bin location, and the trajectories from the base location to each of the destination bin locations individually is not changing. A trajectory, for example, may be a specification for the motion of a programmable motion device over time. In accordance with various embodiments, such trajectories may be generated by experience, by a person training the system, and/or by automated algorithms. For a trajectory that is not changing, the shortest distance is a direct path to the target destination bin, but the articulated arm is comprised of articulated sections, joints, motors etc. that provide specific ranges of motion, speeds, accelerations and decelerations. Because of this, the robotic system may take any of a variety of trajectories between, for example, base locations and destination bin locations.
[0057]
[0058] The risk factor may be determined in a number of ways including whether the trajectory includes a high (as pre-defined) acceleration or deceleration (linear or angular) at any point during the trajectory. The risk factor may also include any likelihood that the articulated arm may encounter (crash into) anything in the robotic environment. Further, the risk factor may also be defined based on learned knowledge information from experience of the same type of robotic arms in other robotic systems moving the same object from a base location to the same destination location.
[0059] As shown in the table at 96 in
[0060]
[0061]
[0062] The choice of fast time vs. low risk factor may be determined in a variety of ways, for example, by choosing the fastest time having a risk factor below an upper risk factor limit (e.g., 12 or 14), or by choosing a lowest risk factor having a maximum time below an upper limit (e.g., 1.0 or 1.2). Again, if the risk factor is too high, valuable time may be lost by failure of the robotic system to maintain acquisition of the object. An advantage of the varied set is robustness to small changes in the environment and to different-sized objects the system might be handling: instead of re-planning in these situations, the system iterates through the database until it finds a trajectory that is collision-free, safe and robust for the new situation. The system may therefore generalize across a variety of environments without having to re-plan the motions.
[0063] Overall trajectories therefore, may include any number of changing and unchanging sections. For example. networks of unchanging trajectory portions may be employed as commonly used paths (roads), while changing portions may be directed to moving objects to a close-by unchanging portion (close road) to facilitate moving the object without requiring the entire route to be planned. For example, the programmable motion device (e.g., a robot) may be tasked with orienting the grasped object in front of an automatic labeler before moving towards the destination. The trajectory to sort the object therefore, would be made up of the following trajectory portions. First, a grasp pose to a home position (motion planned). Then, from home position to an auto-labeler home (pulled from a trajectory database). Then, from the auto-labeler home to a labelling pose (motion planned). Then, from the labelling pose to an auto-labeler home (either motion planned or just reverse the previous motion plan step). Then, from the auto-labeler home to the intended destination (pulled from the trajectory database). A wide variety of changing and unchanging (planned and pulled from a database) portions may be employed in overall trajectories. In accordance with further embodiments, the object may be grasped from a specific pose (planned), and when the object reaches a destination bin (from the trajectory database), the last step may be to again place the object in the desired pose (planned) within the destination bin.
[0064] In accordance with further embodiments, the motion planning may also provide that relatively heavy items (as may be determined by knowing information about the grasped object or by sensing weight—or both—at the end effector) may be processed (e.g., moved in trajectories) and placed in boxes in very different ways than the processing and placement of relatively light objects. Again, the risk verses speed calculations may be employed for optimization of moving known objects of a variety of weights and sizes as may occur, for example, in the processing of a wide variety of consumer products.
[0065] In accordance with a further embodiment of the present invention, and with reference to
[0066] The system, therefore, provides means that interface with the customer's outgoing object conveyance systems. When a bin (or package) is full as determined by the system (in monitoring system operation), a human operator may pull the bin from the processing area, and place the bin in an appropriate conveyor. When a bin is full, it gets removed to the closed/labelled, another empty bin is immediately placed in the location freed up by the removed full bin, and the system continues processing as discussed above.
[0067] In accordance with a specific embodiment, the invention provides a user interface that conveys all relevant information to operators, management, and maintenance personnel. In a specific embodiment, this may include lights indicating bins that are about to be ejected (as full), bins that are not completely properly positioned, the in-feed hopper content level, and the overall operating mode of the entire system. Additional information might include the rate of object processing and additional statistics. In a specific embodiment, the system may automatically print labels and scan labels before the operator places the packages on an output conveyor. In accordance with a further embodiment, the system may incorporate software systems that interface with the customer's databases and other information systems, to provide operational information to the customer's system, and to query the customer's system for object information.
[0068] Those skilled in the art will appreciate that numerous modifications and variations may be made to the above disclosed embodiments without departing from the spirit and scope of the present invention.