Method of Sorting

Abstract

A method of sorting is described, and which includes providing a product stream formed of individual objects of interest having feature aspects which can be detected; generating multiple images of each of the respective objects of interest; classifying the feature aspects of the objects of interest; identifying complimentary images by analyzing some of the multiplicity of images; fusing the complimentary images to form an aggregated region representation of the complimentary images; and sorting the respective objects of interest based at least in part upon the aggregated region representation which is formed.

Claims

1. A method of sorting, comprising: providing a product stream formed of individual objects of interest each having a feature aspect which can be detected; generating multiple images each having different perspectives of the respective objects of interest having the detectable feature aspect; classifying the feature aspect of each of the objects of interest as being either a parent or a child feature; identifying a complementary region of the images of the respective objects of interest by analyzing at least some of the previously identified parent and/or child features identified in some of the multiplicity of images; fusing the complementary regions of the images which are identified so as to form a resulting aggregated region representation of each of the objects of interest; and sorting the respective objects of interest based, at least in part, upon the aggregated region representation which is formed.

2. A method as claimed in claim 1, and further comprising: providing an inspection station and orienting the inspection station along the product stream so that the product stream passes through the inspection station; and providing a multiplicity of detection devices, and positioning the respective detection devices in a predetermined, known position, relative to the inspection station, and further orienting the respective detection devices so that each of the detection devices has a predetermined field of view of the product stream passing through the inspection station.

3. A method as claimed in claim 2, and wherein the step of generating multiple images further comprises: providing a controller for selectively actuating the respective detection devices in a predetermined order, and in real-time so as to selectively synchronize the operation of the respective detection devices and facilitate the generation of the multiple images, each having synchronized portions, and different perspectives, and further having the feature aspects which can be detected; and generating a detection device signal by selectively actuating the respective detection devices by means of the controller, and delivering the respective detection device signals to the controller so as to generate, at least in part, the multiple images having different perspectives.

4. A method as claimed in claim 3, and after the step of actuating the respective detection devices, the method further comprises: assigning a real-time, time stamp to each synchronized portion of the individual images formed from the detection devices having the multiple different perspectives.

5. A method as claimed in claim 4, and further comprising: detecting each of the respective objects of interest in the individual images by means of a known background signal which is detected in each of the images which are generated.

6. A method as claimed in claim 5, and after the step of assigning the real-time, time stamp, the method further comprises: calculating a vertical position of the respective objects of interest, relative to the individual detection devices, by utilizing the real-time, time stamp; and calculating the horizontal position of the respective objects of interest, in each of the images generated by the respective detection devices, by employing a known position of the respective detection devices.

7. A method as claimed in claim 6, and further comprising: after the step of classifying a feature aspect of each of the objects of interest, identifying parent features acquired from the different perspective images based, at least in part, upon one or more of a group of factors selected from the group comprising a position; a size; an orientation; and a classification of each of the parent features.

8. A method as claimed in claim 7, and further comprising: establishing, by means of the controller, a given region which is formed around each of the objects of interest, and which are seen in the multiple images formed by each of the respective detection devices; and assigning an absolute position to the given region by utilizing the calculated vertical and horizontal positions of the individual objects of interest.

9. A method as claimed in claim 8, and further comprising: calculating the given region surrounding each of the parent features based, at least in part, upon one or more of a group of factors selected from the group comprising a size; a shape; and an orientation of the parent feature; and identifying, if present, at least one child feature, of each parent feature, within each given region.

10. A method as claimed in claim 9, and wherein the step of identifying a complementary region further comprises: associating the identified, complementary regions by employing an absolute position of the given region, and any similarities, or differences identified in the previously identified parent and/or child features; and fusing the complementary regions taken from the different detection devices so as to facilitate the formation of the aggregate region representation of the multiple perspective images formed of the respective objects of interest.

11. A method as claimed in claim 10, and wherein the step of fusing regions with the complementary images further comprises: forming the aggregated region representations of the multiple perspective regions bounding each of the individual objects of interest with the controller, based, at least in part, upon similarities and differences identified in the parent features, and wherein the aggregated region representation of each of the individual objects of interest is formed from the multiple images which were derived from the different detection devices; and further forming with the controller, a real-time, multiple-aspect representation of the individual objects of interest travelling in the product stream by utilizing at least a portion of the detection device signals generated by at least some of the detection devices, and wherein the multiple-aspect representations which are formed by the controller each has a plurality of features depicting a multitude of characteristics possessed by each of the objects of interest; identifying within each aggregated region representation all the parent and child features; calculating a population of the objects of interest by associating the parent features within each of the aggregated region representations; and calculating a population of child features within each of the aggregated region representations by associating individual child features into sets based upon similarities, and differences of the respective child features identified in each of the aggregated region representations.

12. A method as claimed in claim 11, and further comprising: performing the sorting step of the individual objects of interest based, at least in part, upon the identified parent and/or child feature aspects.

13. A method as claimed in claim 12, and further comprising: quantifying from the aggregated region representations, by using the controller, a number of defects which are present in each of the objects of interest while minimizing the likelihood of quantifying the same defect more than once.

14. A method as claimed in claim 13, and further comprising: preparing sorting statistics for the respective objects of interest moving along the product stream by the collection of the parent and child features identified from the multiple images which were taken across the respective objects of interest.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Preferred embodiments of the invention are described, below, with reference to the following accompany drawings.

[0009] FIG. 1 is a highly simplified view of the overall methodology of the present invention.

[0010] FIG. 2 is a greatly simplified view of an apparatus or other arrangement which can implement at least some of the steps which form the methodology of the present invention.

[0011] FIG. 3 is a greatly simplified flow diagram which shows at least some of the steps of the present methodology as implemented by the present invention.

[0012] FIG. 4 is a greatly simplified, partial, flow diagram which illustrates at least some of the steps in the methodology of the present invention as seen in FIG. 3.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] This disclosure of the invention is submitted in furtherance of the constitutional purposes of the U.S. Patent Laws to promote the progress of science and useful arts (Article 1, Section 8).

[0014] As discussed earlier in this application, one of the many problems confronting the developers of high-speed, mass flow, food sorting devices is to collect increasing amounts of information relative to the objects of interest being sorted, and thereafter, by utilizing this information, make accurate sorting decisions to remove undesirable objects and/or products having defects from the product stream which is moving rapidly through a predetermined inspection station, or make other sorting decisions to implement a sorting strategy for a product stream so as to produce different grades of a product for end-user consumption, or further processing. One of the problems which has received increasing amounts of research concerns the propensity for the previously described sorting devices to make inaccurate sorting decisions based, at least in part, upon sensor or other detector information, which identifies the same object defect in the product stream multiple times. As should be appreciated, when an associated controller operates using sensor or detector information which identifies the same defect, multiple times, the controller may cause the sorting device to inappropriately eject an object from a product stream and/or generate inaccurate statistics or information about the product stream, from the multiple sensors and/or detectors when, in fact, only one defect is actually present in the object or product which has been ejected. This so-called double counting of defects in an object can result in incorrectly ejected products, and inaccurate sorting statistics. In the food industry these inaccurate ejections of products can result in increased financial losses for a food processor.

[0015] Consequently, the production or generation of accurate sorting statistics regarding defects which are seen or detected in the objects of interest which are present in a moving product stream is paramount in maintaining not only the quality of the resulting sorted product, but the profitability for processors which use such high-speed, mass-flow, sorting equipment. As was described with reference to the aforementioned earlier U.S. patent, sorting devices of this level of sophistication or complexity, often employ multiple cameras or other detection devices which, when selectively operated, create resulting data which is then utilized to build or otherwise construct what is termed a multi-dimensioned feature space and from which sorting decisions are then made. This multi-dimensioned feature space is discussed in detail in U.S. Pat. No. 9,517,491, the teachings of which are incorporated by reference, herein. It is, of course, important to recognize in sorting food products, such as green beans, potato strips (french fries), and the like, that all exterior surfaces (and some interior or hidden surfaces) of the products or objects being sorted may be seen, viewed, or detected by the cameras, or other sensors, and thereafter, a decision must be made by the controller, as provided, as to whether the object, and/or product that has been observed, or detected by the cameras, or other detectors, by utilizing visual or invisible bands of electromagnetic radiation, should be removed from the product stream because it has a defect, or the object which was observed or detected was unacceptable for further processing.

[0016] Referring now to the drawings, the method of sorting is generally depicted in the drawings, and is indicated by the numeral 10 in FIG. 1, and following. In this regard the methodology 10 includes a first step of providing a product stream 11 which is formed of individual objects of interest 12. The individual objects of interest 12 each have a main body 13, which is formed of multiple sides 14. As seen in the drawings, the main body 13 may have a feature aspect, portion, or region 15, which is acceptable for further processing. On the other hand, the main body 13 may have an unacceptable feature aspect, defect, region or portion 16, and which renders that portion of the main body 13 unsuitable for further processing. For purposes of the present application, the individual objects of interest 12, are here depicted, as elongated potato strips which have been cut in a fashion for further processing into French fries.

[0017] As seen in FIG. 1, the step of providing the product stream 11 includes transporting the product stream by way of a conveyor assembly 20 to an inspection station so that the objects may be individually imaged, as described, below. The conveyor assembly 20 is of traditional design, and has a first, intake end 21, and a second, discharge end 22. While a continuous conveyor belt is illustrated, those skilled in the art will readily recognize that any other traditional conveying device such as an excited-frame vibratory conveyor, or a delivery chute cooperating with such a conveying device or similar arrangement could be employed with equal success. The product stream 11 is delivered to the conveyor assembly 20 by way of a product delivery device 23 which is typically located in a position which is elevationally above, and in product discharging relation relative to, the first intake end 21 of the conveyor. The individual objects of interest 12 then travel to the second, discharge or exhaust end 22 of the conveyor 20, and wherein, upon arriving at this location, the individual objects of interest 12 travel, typically, under the influence of gravity, along a vertically, downwardly disposed path of travel 24. As will be appreciated a chute (not shown) could be employed to support the movement of the objects of interest as they individually move along the path of travel under the influence of gravity. The path of travel of the product stream 11, as seen in FIG. 1, includes a first portion where the objects of interest are physically supported on the conveyor assembly 20 or an accompanying chute (not shown); and a second, unsupported, and substantially vertically oriented portion 25 as seen in FIG. 1. The path of travel 24, having the path portion which is unsupported 25, permits the viewing of the various sides 14 of each of the objects of interest 12, and which are moving along in the product stream 11. In the present invention, the method 10 includes another step of providing an inspection station 26, and orienting the inspection station 26 so that the product stream 11 moving along the predetermined path of travel 24 passes through the inspection station 26. The step of providing the inspection station is generally indicated (in one instance) by the box labeled 26, in FIG. 1. However, it should be recognized that the inspection station 26, is a general region in which an optical viewing, or sensor detection step may take place of the respective objects of interest 12 traveling in the product stream 11, as the individual objects of interest move along the path of travel 24, and which includes the unsupported path portion 25, during a predetermined inspection step. As seen in FIG. 1, the inspection station may include a region where the objects of interest 12 may be inspected from only one perspective, (such as when the objects of interest 12 are supported on an underlying surface), and only a limited number of surfaces can be seen. In the present methodology the invention includes another step of providing a field of view, and/or line-of-sight 27 relative to the product stream 11, and which is passing through the inspection station 26. The field of view 27 is best understood, and appreciated, by a study of FIG. 2, and where the field of view is also identified by the abbreviation FOV. As seen in FIG. 1, the arrangement which is generally depicted shows a first inspection station 26, which is oriented so as to permit the imaging of the individual objects of interest 12 which are supported on, and transported by the conveyor assembly 20, as well as a second, inspection station which is located elevationally, below, the conveyor 20, and which allows the inspection of the objects of interest 12 as they move in the unsupported portion of the product stream 25. This arrangement as seen in FIG. 1 permits the present methodology to inspect the various sides 14 of the respective objects of interest 12.

[0018] Referring again to FIG. 1, the methodology 10 of the present invention includes a step of providing a plurality of selectively energizable electromagnetic radiation emitters which are generally indicated by the numeral 30. The respective emitters 30, when selectively energized, emit predetermined beams 31 of visible or invisible electromagnetic radiation which are individually oriented, or otherwise directed towards the inspection station 26, and along the FOV and/line-of-sight 27. As seen in the drawings, the selectively energizable electromagnetic radiation emitters 30 may utilize a traditional optical focusing element 32, and which allows the formation of the predetermined, given beams 31, and which are precisely directed at, and along, the predetermined field of view and/or line-of-sight 27. As seen in the drawings, electrical conduits 33 are provided, and which permit or facilitate the selective energizing of the respective electromagnetic radiation emitters 30. Still further, and as seen in FIG. 1, the present methodology includes a step of providing a selectively energizable background element 34 which can be selectively energized so as to provide, or generate, a predetermined visible and/or invisible, background optical signal 35, and which will be useful in the formation of images, and the recognition of defects in the objects of interest 12 as seen in these subsequently produced images. These several aspects of the current methodology will be described in greater detail, hereinafter.

[0019] The methodology of the present invention 10 further includes a step of providing a plurality of detection devices 40, and positioning the respective detection devices which are here indicated as first, second and third devices 41, 42 and 43, respectively, at different perspectives or spatial positions, 44, relative to the inspection station 26, and the product stream 11 which is passing through the inspection station 26. The plurality of detection devices 40 are further located in a given, known position, and further, each of the respective detection devices 41, 42 and 43 are oriented so as to share, at least in part, a predetermined field of view 27 relative to at least one of the path portions 24 or 25 of the product stream 11. The respective detection devices 40, when actuated, generate device signals which are generally indicated by the numeral 46 (FIG. 2), and which is further utilized in the fashion, as described, hereinafter, to enhance the ability for the present methodology 10 to achieve the superior sorting results, and performance, as will be described, hereinafter.

[0020] The methodology 10 of the present invention includes another step of providing a controller 50 for selectively actuating 51 the respective detection devices 40 in a predetermined order, and in real-time, so as to selectively synchronize the operation of the respective detection devices 40, discussed, above (FIG. 3). Still further, the step of selectively actuating the respective detection devices 41, 42 and 43, with the controller 50, in real-time, so as to enhance the operation of the respective detection devices 40 which are actuated, by avoiding, for example, a destructive interference which might occur if the respective detectors 40 were operated substantially, simultaneously. This destructive interference is discussed in the aforementioned US patent. The method of sorting 10, further includes still another step 52, of generating multiple images (FIG. 2), each having different perspectives or angular orientations of the respective objects of interest 12 having detectible feature aspects 15, and 16, respectively. In this regard, the device signals 46 (FIG. 2) which are generated by the plurality of detection devices 40 comprise, at least in part, the individual images having multiple, different perspectives 52, of each of the objects of interest 12, and which are moving along the product stream 11. The methodology of the present invention 10 includes still another step 53 (FIG. 3) of delivering the respective device signals to the controller 50 (FIG. 3), for further processing. As seen in FIG. 3, the controller 50 implements a further step 60 of acquiring multiple, different perspective images 52 relative to a given, and predetermined background signal 35, and detecting 61 the respective objects of interest 12 relative to the background signal 35 which is recognized, and present within each of the perspective images which have been acquired 52 from the selectively actuated, and synchronized detection devices 40.

[0021] Still referring to FIG. 3 the methodology of the present invention 10 includes a step 70 of discriminating a feature aspect 15, and/or 16 of the individual objects of interest 12 from within each of the different, perspective images 52 which have been acquired 60. The methodology 10 includes, still another step 72, of classifying the respective feature aspects of the individual objects of interest 12 into either a parent 73, or a child 74 feature and/or class (FIG. 2), from each of the different perspective images 52 which have been acquired 60. In addition to the foregoing, the methodology includes still another step 80 of associating a set of parent features 73 which were derived from the different and acquired, perspective images 52 which have been acquired in the step 60, based, at least in part, upon one or more of a group of factors selected from the group comprising the individual object of interest's 12 position; size; shape; orientation; and classification within each of the different perspective images 52 which have been acquired 60. Still further the method 10 includes another step 90 (FIG. 3) of calculating a given region within which to set a bound, boundary, bounding box 180 (FIG. 2) and/or contour surface surrounding the identified parent features 73, and which is based, at least in part, upon one or more of a group of factors selected from the group comprising the size; the shape; and the orientation of the individual parent features 73. The methodology includes still another step 100, of identifying one or more child features 74, of each parent feature 73, and which is identified within the given region that was defined in step 90, above. The method includes yet another step 110 of associating one or more child features or sets of child features 74 within each given region identified in step 90, above, and which is identified, based, at least in part, upon one or more of a group of factors comprising the location of one or more of the child features within the given region 90; the size of one or more of the child features; and the classification of one or more of the child features.

[0022] The step 120 of forming, with the controller 50, in real time, a plurality of multiple-aspect aggregated region representations 160 of the individual objects of interest 12 which are traveling in the production steam 11 is accomplished by means of utilizing, at least a portion, of the device signals 46, and which are further generated by each of the detection devices 40 (FIGS. 2 and 3). In this step, 120, the respective multiple-aspect aggregated region representations 160 each have a plurality of features 15 and/or 16, respectively, and which are classified as parent 73 and/or child features 74 within each of the aggregated region representations. Each aggregated region representation further depicts a multitude of characteristics (which may be considered, for example, physical structures or regions; chemical or biological compounds; visually discernible features, or aspects; and/or humanly invisible features or aspects) possessed by each of the objects of interest 12. The method 10 includes still another step 130, of identifying within each of the aggregated region representations 120, all parent and/or child features (73 and 74), and feature sets formed from same. The method 10 includes still another step 140, of calculating a population of the objects of interest 12 by associating the parent and child object features 73 and 74, which were previously identified. Finally, the methodology 10 of the present invention 10 includes a step 150 of sorting the individual objects of interest 12, based, at least in part, upon the aggregated region representations 120, which are formed of the multiple perspective views or images 52, of each of the objects of interest 12, and the multiple aspect region representations 120, which are formed by the controller 50, in real-time, as the individual objects of interest 12 move along in the product stream 11, and through the inspection station 12.

[0023] The method of the present invention includes yet another step (FIG. 4), which is performed after the step of actuating 51 the respective detection devices 40, and acquiring the multiple perspective images 52, of assigning a real-time, time stamp 162, to each of the individual images which are formed by the detection devices 40 having the multiple different perspectives 52. Further, after the step of assigning the real-time time stamp 162, and detecting the objects of interest 12 relative to a known background signal 62; the method includes another step of calculating 163 a vertical position of the respective objects of interest 12 relative to the individual detection devices 40, by utilizing the real-time, time stamp. The method includes still another step 164 of calculating the horizontal position of the respective objects of interest 12 in each of the images generated by the respective detection devices 52, by employing the known position of the respective detection devices 40.

Operation

[0024] The operation of the described methodology for implementing the present invention 10 is believed to be readily apparent, and is briefly summarized at this point.

[0025] In its broadest aspect the method of sorting 10 of the present invention includes the steps of providing a product stream 11, formed of individual objects of interest 12, and each having one or more feature aspects 15 and/or 16 which can be detected. The invention 10 includes another step of generating multiple images each having different perspectives 52, of the respective objects of interest 12, and which include the detectible feature aspects 15, 16, 73 or 74. The method 10 includes another step 72 of classifying the respective feature aspects 15 and/or 16 of each of the objects of interest 12, as being either a parent 73 or a child feature 74. The method 10 includes yet another step of identifying complimentary regions or views 190 of the respective objects of interest 12 by analyzing at least some of the previously identified parent and child features 73, and 74, respectively, and which are identified in some of the multiplicity of images 52. The method of the present invention 10 includes still another step 200 of fusing the complimentary regions or views 190 which are identified so as to form in step 120 a resulting aggregated region representation 160 of the multiple, complimentary regions or views 190. Finally the methodology 10, in its broadest aspect, includes a step 150 (FIG. 1) of sorting the respective objects of interest 12 based, at least in part, upon the aggregated region representations 120 which are formed by means of the steps which are broadly recited, above.

[0026] Therefore it will be seen that the present invention 10 provides a convenient means whereby accurate sorting 150, of a stream of products 11, and which are formed of multiple objects of interest 12, can take place, and defects 16 can be readily identified in the respective objects of interest 12, while minimizing, to the extent possible, a misidentification and/or erroneous quantification of the defects in the respective objects of interest 12 which are being sorted, and which might be occasioned by the same defect 16 being seen, and identified in the respective objects of interest 12, by more than one detection device 40 which is employed to inspect the product stream 11.

[0027] In compliance with the statute the invention has been described in language more or less specific as to structural and methodological features. It should be understood, however, that the invention is not limited to the specific features shown and described since the means herein disclosed comprise preferred forms of putting the invention into effect. The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended Claims appropriately interpreted in accordance with the Doctrine of Equivalence.