System and Method for Part Identification Using 3D Imaging

Abstract

The system and method deal with three-dimensional (3D) scanning technology which produces object representations which permit rapid, highly-accurate part identification which is not afforded by traditional two-dimensional (2D) camera imaging. The system and method are applicable to any field wherein repair/replacement parts are needed, such as the plumbing, automotive, fastener, marine, window, door, etc. fields

Claims

1. A system for use in identifying any particular object or part not otherwise identifiable, by using 3D imaging, the system comprising: a first 3D imaging device for imaging substantially all available 3D models of the any particular object or part and creating a database including the 3D imaging scans and associated identifying information and parameters, a second 3D imaging device which provides a 3D imaging scan of one model of the particular object or part which is not otherwise identifiable, a computer having access to the database and being programmed to assess the available models against the 3D image scan and parameters from the second 3D imaging device, a communication network between the second 3D imaging device and the computer with its associated database which compares the 3D image scan from the second 3D imaging device to those available in the associated database and provides information to a user of the second 3D imaging device for at least one model of the any particular object or part which fits the parameters of the object or part scanned by the second 3D imaging device.

2. The system of claim 1 wherein the database includes not only 3D scans or images but also associated feature representations, including but not limited to: object or part measurements, dimensions, geometric shape descriptors, etc., which are stored as digital profiles.

3. The system of claim 1 wherein the database can be populated by scanning physical objects or parts, or by other means, including but not limited to: obtaining and inputting 3D CAD drawings.

4. A method for identifying any particular part not otherwise identifiable, by using 3D imaging, using a system comprising: a first 3D imaging device for imaging substantially all available 3D models of the any particular object or part and creating a database including the 3D imaging scans and associated identifying information and parameters, a second 3D imaging device which provides a 3D imaging scan of one model of the particular object or part which is not otherwise identifiable, a computer having access to the database and being programmed to assess the available models against the 3D image scan and parameters from the second 3D imaging device, a communication network between the second 3D imaging device and the computer with its associated database which compares the 3D image scan from the second 3D imaging device to those available in the associated database and provides information to a user of the second 3D imaging device for at least one model of the any particular object or part which fits the parameters of the object or part scanned by the second 3D imaging device, the method including the steps of: creating the database of substantially all available 3D models of the any particular object or part using a 3D imaging device, creating a 3D image of the otherwise unidentifiable object or part using the second 3D imaging device, associating a computer preprogrammed to assess or compare a 3D image or scan of the otherwise unidentifiable object or part provided from the second 3D imaging device to the database of models, using the communication network to send the 3D image of the otherwise unidentifiable object or part to the computer for comparison to the 3D models in the associated database, and, providing at least one 3D model of the particular object or part which fits the parameters of the object or part scanned by the second 3D imaging device.

5. The method of claim 4 wherein 3D imaging devices comprising sensors or scanners are used in the field to scan objects or parts that need to be identified, producing measurements in real-world scale units, and extracting feature profiles commensurate with those stored in the database.

6. The method of claim 5 wherein field scanned object or part information is transmitted to the preprogrammed computer and compared as a query profile to the information stored in the database to identify most likely object or part matches.

7. The method of claim 5 wherein candidate database object or part matches are ranked based on similarity to the query information provided.

8. The method of claim 7 wherein resulting ranked matches are provided to the user, along with images, text descriptions, pricing, and other associated database information.

9. The method of claim 5 wherein the computer can also dispatch results to a 3rd party inventory management website or software service.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0034] FIG. 1 presents a system level logic flow diagram used by the computerized system and method of the present invention.

[0035] FIG. 2 presents a logic flow diagram for creation of a database of particular parts.

[0036] FIG. 3 presents a logic flow diagram of database creation in connection with a part matching process of the present invention.

[0037] FIG. 4 shows a customer bringing in a part for identification by the system and method of the present invention, in this example a plumbing cartridge.

[0038] FIG. 5 shows the cartridge of FIG. 4 being placed on a turntable for scanning and identification.

[0039] FIG. 6 shows the cartridge being scanned by a countertop 3D scanner, as an example, creating a 3D point cloud model.

[0040] FIG. 7 shows the 3D point cloud model created by the scan of the part and the model is compared to all the cartridge models and their parameters stored in the system database.

[0041] FIG. 8 shows a screenshot of potential matching parts retrieved from the system, in order of match percentage.

[0042] FIG. 9 shows a screenshot of the highest ranking part selected and shows a button for visualization in all dimensions, if desired.

[0043] FIG. 10 shows what appears when the button of FIG. 9 is activated.

[0044] FIG. 11 shows activation of a measuring tool of the system which brings up point to point measurements for comparison with the actual part.

[0045] FIG. 12 shows a screenshot of an order page for the correct part, which also provides a list of related parts which may also potentially be needed.

DESCRIPTION OF THE PREFERRED EMBODIMENT

[0046] Turning to FIG. 1, there is illustrated therein a low level system level logic flow diagram 10 wherein a user has a physical object (part) that he/she wishes to identify, e.g., for the purpose of replacement or re-ordering. Some parts may be easily identified by visual inspection 12 from information imprinted on the surface, prior experience, distinctive shape, size, etc. If such is present, and the part is visually identifiable 14, then one merely verifies that the part is correct 16, and takes action, ordering the proper part at 18. However, many are not readily identifiable, perhaps due to uncommon brand, outdated product, damaged/dirty condition, or unfamiliarity. In such a case, the user captures the part parameters with a 3D scanning device at 20. The resulting scan data is transmitted at 22 to a computerized recognition system that resides either locally or on a remote network.

[0047] The data is then processed to extract a profile consisting of geometric and photometric descriptors 24 suitable for part identification. The computerized recognition system compares this profile against a corresponding database 26 of previously acquired profiles, eliminates very unlikely matches 28, and ranks surviving candidates according to match likelihood. The associated identifiers (e.g., part numbers) for the best matches, along with their likelihoods, are returned and presented to the user via graphical display 30.

[0048] If the system logic 10 returns a single match at 32, the part is verified at step 16 and ordered at step 18. If the logic 10 returns more than one match at 32, then the user can interactively filter the results at 34 via the display based on manually-entered attributes and based on visual comparison of the physical part with images, 3d models, and textual descriptions of the candidates. Finally, when the candidate list has been narrowed to a single matching part at 32, the user verifies the correct part at 16 and can act on the resulting identifier, for example, order a replacement directly from the catalog website at 18.

[0049] Turning now to FIG. 2, the corresponding database 40 of the computerized system 10 used to recognize the particular part is created offline, and contains a set of part identifiers, such as, for example, for a plumbing valve and may be captured by any suitable means such as those at steps 42, 44 or 46. Associated with each identifier is created a geometric and photometric profile 48 compatible with that extracted from the scanned part, as well as other information such as photographs, textual descriptions, and human-readable attributes that can be used for manual filtering by the user. The database may be populated by scanning physical objects, or by other means such as acquiring CAD models from manufacturers. Profiles are then stored at 50 to the database 40 and are derived from the initial 3d representation in the same manner as from the user-scanned part. Definitions of terms used herein are listed below for ease of comprehension:

part: a specific physical object with [mostly] rigid 3d shape; for instance, a faucet stem or mounting bracket.
user: the operator of the system; for instance, a retail store employee wishing to identify a part, or a consumer wishing to find a replacement.
scanner: a device that can create a digital 3d model of a presented object by measuring distance and appearance, e.g., using cameras, projectors, and/or lasers.
features: attributes of an object such as local and global 3d shape, physical dimensions, and visual appearance represented as a series of numeric values.
profile: a digital representation of an object that encodes one or more features.
identifier: a unique name or tag uniquely identifying a particular part, such as a product number or SKU.
database: a digital catalog or repository of parts, along with their identifiers and profiles, that can be queried according to various attributes.
match: a correspondence between the user-scanned part and a candidate from the database.
likelihood: a score or probability that a particular match is correct, or a degree of similarity between the query and the match.
recognition: the search for and retrieval of the most likely matches for a particular part from a database, as well as the likelihood of each match.
query: a particular part instance that the user wishes the system to recognize.

[0050] The system distinguishes itself from competition and prior art in a number of ways, some of which are denoted at present in FIGS. 1 and 2 by red dotted boxes. Foremost among these is the use of 3D information at all stages, including database creation, part query, part matching, and user interaction. While other methods may share similar overall logic structure, they rely solely on digital images instead, which are inherently 2D projections of an object's true 3D shape.

[0051] Particular Items of Note: [0052] (1) A digital 3D representation of the query part is acquired (e.g., via turntable scanner or mobile device) and used for matching by the system against the database information stored in memory. This representation captures the part's true size and shape in real-world units. Other methods acquire 2d images only. [0053] (2) The recognition system operates using 3D data, producing geometric shape-based (and optionally appearance-based) features that directly encode local surface structure independent of environment and imaging conditions. Other methods exclusively use appearance-based features, which are derived from 2d image projections and thus (a) have no notion of true scale; (b) have no notion of “depth” or 3d surface structure; and (c) are affected dramatically by illumination, shadows, camera settings, viewpoint, and environment. [0054] (3) Results are presented to the user in a multitude of formats that consist of traditional images and text annotations (these are the only formats returned by other methods), but additionally present rotatably viewable 3d models stored in the database. [0055] (4) Retrieving 3D models, and encoding their true dimensions, allow the user to further narrow the search results via interactive inspection (e.g., manipulating the candidate models in a 3D viewer) and via manual data entry (e.g., specifying part length and diameter). Other methods allow only for visual inspection of static images and textual descriptions. [0056] (5) As with queries, models acquired for database generation are also inherently 3D. This allows alternate “virtual” or “ideal” sources such as CAD files to populate the database without needing to obtain physical parts. Furthermore, only a single (partial or complete) scan suffices to represent each part. Other methods require imaging physical parts, and often involve dozens or hundreds of examples for training. [0057] (6) Also as with queries, the system extracts geometric shape-based (and optionally appearance-based) features when populating the database. These features are highly commensurate with those extracted for queries because they rely on exactly the same underlying geometry. With image-based methods, matching query to database information is substantially more difficult and requires much more extensive imaging of each part in order to sufficiently capture expected variations in size, viewpoint, and illumination.

[0058] Turning now to FIG. 3, the figure depicts the computerized process 60 for (a) database creation and its use in (b) matching in more detail. In both pipelines, the input data is pre-processed at 62 to properly scale and orient the part, and to create a low-level representation (e.g., 3d point cloud and differential orientation at each point); also in both pipelines, these low-level representations are used to extract both global features at 64 and local features at 66. Global or holistic features describe the entire part and provide compact, coarse characteristics such as overall shape and dimensions, while local features encode finer shape detail localized to small regions on the part's surface, with both being stored in database 40. All features are indexed via efficient hashing and either stored to the database 40 or used to query the database 40.

[0059] The recognition process begins by considering all possible database models for a particular query at 70 and then applies a sequence of progressively more complex (and more selective) filters. The first of these filters, global feature extraction 72, consider only holistic query descriptors, providing a coarse classification at 74 that can be compared very quickly to those in the database 40 so as to immediately eliminate most possible candidates, resulting in set A. Next, local descriptors 76 extracted from the query part are hashed at 78 and used to efficiently scan the database's feature index, further reducing possible matches to set B. Finally, the system evaluates each surviving candidate in greater detail at 80 to determine its complete surface-to-surface similarity to the query; this process produces the final list of candidates, along with similarity scores that can be used for ranking and user presentation.

[0060] FIG. 4 shows a customer 100 bringing in a part 102 for identification by the system and method of the present invention, in this example a plumbing cartridge 102. Although the example throughout deals with a plumbing part this should not be construed as limiting inasmuch as the method and system may be used in any other field where repair/replacement parts are required, such as for example in the automotive part industry, etc.

[0061] FIG. 5 shows the part 102, in the form of the cartridge 102 of FIG. 4, being placed on a turntable 104 for 3D scanning and identification by the system and method of the present invention.

[0062] FIG. 6 shows the cartridge 102 being scanned by a countertop 3D scanner 106, as an example, creating a 3D point cloud model of same. Although a countertop scanner is exemplified this should not be construed as limiting inasmuch as any electronic device capable of capturing a 3D image could be used, even a future cellular phone with 3D imaging capability.

[0063] FIG. 7 shows the 3D point cloud model 108 created by the scan of the part and the model and its parameters are compared to all the cartridge models and their parameters stored in the particular plumbing cartridge database 40 created for the system and stored at 50 in the memory thereof. Again, this should not be construed as limiting to the versatility of the system and method.

[0064] FIG. 8 shows a screenshot 110 of potential matching parts retrieved from a search through the database 40 in the system memory, preferably, in the preferred embodiment, ranking down from the highest match percentage.

[0065] FIG. 9 shows a screenshot 112 of the highest ranking part 114 selected and shows a link button 116 for use in presenting rotatable visualization in all dimensions, if desired.

[0066] FIG. 10 shows what appears in the next screenshot 118 when the link button 116 of FIG. 9 is activated to provide rotatability to the view of the part 102.

[0067] FIG. 11 shows a screenshot 120 including point to point measurements 122 which may be elicited from the system and method upon activation of a measuring tool 124 of the system which brings up a point to point measurement table 122 of measurements along each axis for comparison with the dimensions of the actual part 102.

[0068] FIG. 12 shows a screenshot 128 of an order page for the selected, highest ranking part 102, which order page may also provide a list of related parts 130 which may also potentially be needed.

[0069] As described above, the system and method of the present invention provide a number of advantages, some of which have been described above and others of which are inherent in the invention.

[0070] Also, modifications may be proposed without departing from the teachings herein. For example, although the description deals with repair and/or replacement parts, the system and method may be used in identification of new parts as well. Accordingly, the scope of the invention is only to be limited as necessitated by the accompanying claims.