METHOD AND APPARATUS FOR SEARCHING A DATABASE OF 3D ITEMS USING DESCRIPTORS
20170337733 · 2017-11-23
Assignee
Inventors
Cpc classification
G06V10/451
PHYSICS
G06V20/647
PHYSICS
G06T17/20
PHYSICS
International classification
Abstract
A method and apparatus for searching a database of 3D items using descriptors created for each item and for the 3D computer-generated model or the 3D physical object. The descriptor comprises a vector string comprising each of the feature vectors extracted from the images, and dimension-related features each representing a dimensional feature. The descriptor is created by obtaining 2D rendered images of the model by rotating the model about each different axis independently, and extracting a feature vector representing features. The model is rotated about the axes in angular steps which may be the same for each axis. The axes may comprise three mutually-perpendicular axes, where each angular step is in the range from 30° to 45°.
Claims
1. A computer-implemented method of creating a descriptor for a three-dimensional (3D) computer-generated model, comprising: obtaining a plurality of two-dimensional (2D) rendered images of the model from a fixed viewpoint by rotating the model fully about each of a plurality of different axes independently; and extracting from each 2D rendered image a feature vector representing features of each 2D rendered image, wherein the descriptor comprises a vector string comprising each of feature vectors extracted.
2. A method as claimed in claim 1, wherein the descriptor further comprises one or more dimension-related features each of the one or more dimension-related features representing a dimensional feature of the model.
3. A method as claimed in claim 1, wherein the model is rotated about one or more of the axes in angular steps of magnitude greater than zero.
4. A method as claimed in claim 3, wherein an angular step is a same for each axis.
5. A method as claimed in claim 1, wherein the plurality of different axes comprises three mutually-perpendicular axes.
6. A method as claimed in claim 3, wherein the plurality of different axes comprises three mutually-perpendicular axes and each angular step is in a range from 30° to 45°.
7. A computer-implemented method of creating a descriptor for a three-dimensional (3D) object, comprising: obtaining at least one two-dimensional (2D) image of the 3D object; and extracting from the at least one 2D image a feature vector representing features of the 2D image; wherein the descriptor comprises one of the feature vector extracted, when there is only one 2D image, and a vector string comprising each of the feature vectors extracted, when there is more than one 2D image; and wherein the descriptor further comprises one or more dimension-related features each representing a dimensional feature of the 3D object.
8. A method as claimed in claim 1, wherein the at least one 2D feature vector is extracted from the 2D image using a deep neural network.
9. A computer-implemented searching method for searching a database of three-dimensional (3D) items for similarity with one of a specified 3D object and a 3D computer-generated model, which searching method comprises: a similarity assessment process in which one of a feature vector and feature vectors of a descriptor associated with the one of the specified 3D object and the model and feature vectors of descriptors associated with 3D items in the database are used to ascertain a degree of similarity between the one of the specified 3D object and the model and one or more of the 3D items in the database; wherein each of the descriptors associated with the 3D items in the database being created in accordance with the method of claim 1, and one of the descriptor associated with the specified 3D object being created in accordance with the method of claim 7 and the descriptor associated with the specified 3D model being created in accordance with the method of claim 1.
10. A method as claimed in claim 9, wherein the descriptors associated with the 3D items in the database being created in accordance with the method of claim 2, and the similarity assessment process further comprises using dimension-related features of the descriptors of the one of the specified 3D object and model and the 3D items in the database to identify any 3D item in the database which is not compatible in size with the one of the specified 3D object and model.
11. A method as claimed in claim 9, wherein the similarity assessment process comprises: using the one of the feature vector and vectors of the descriptor associated with the one of the specified 3D object and the model to create a feature matrix fM.sub.s for the one of the specified 3D object and model; using the feature vectors of the descriptors associated with the 3D items in the database to create a feature matrix fM.sub.d for each 3D item in the database; appending feature matrices fM.sub.d created for the 3D items in the database to create a database matrix dM; using the feature matrix fM.sub.s for the one of the specified 3D object and the model and the database matrix dM to compute a similarity matrix sM indicative of a distance between each feature vector of the descriptor associated with the one of the specified 3D object and the model and each feature vector of the descriptors associated with the 3D items in the database; and deriving a similarity vector sV from the similarity matrix sM, where the similarity vector sV has a length DS and a j-th element of the similarity vector represents a similarity between the one of the specified 3D object and the model and a j-th item in the database.
12. A method as claimed in claim 11, wherein: a number NR.sub.s of rows of the feature matrix fM.sub.s is a same as the number of feature vectors, the number NC.sub.s of columns of the feature matrix fM.sub.s is the same as the number of features in the feature vector or vectors, and an i-th row of the feature matrix fM.sub.s corresponds to an i-th feature vector of the descriptor; in each such feature matrix fM.sub.d the number NR.sub.d of rows of the matrix is the same as the number of feature vectors in the descriptor associated with a respective database 3D item, the number of columns NC.sub.d of the matrix is the same as the number of features in each feature vector in the descriptor associated with the respective database 3D item, and the i-th row of the matrix corresponds to the i-th feature vector of the descriptor associated with the database 3D item concerned; and the number of rows in the database matrix dM is DS*NR.sub.d, where DS is a total number of items in the database, and the number of columns in the database matrix dM is NC.sub.d.
13. A computer program which, when run on a computer, causes that computer to carry out the method of claim 1.
14. A method for use in manufacturing a product, the method comprising: providing a database of three-dimensional (3D) items which have been one of previously manufactured and prepared for manufacture, the database including descriptors associated with each item wherein each of the descriptors associated with the 3D items in the database being created in accordance with the method of claim 1; receiving information specifying a product to be manufactured, the information comprising geometric information specifying a geometry of the product; deriving descriptors from the geometric information of the product in accordance with the method of claim 1; using the searching method of claim 9 to search the database to ascertain a degree of similarity between items in the database and the product to be manufactured; determining which item in the database is most similar to the product to be manufactured; and retrieving stored manufacturing process information relating to a manufacturing process associated with manufacture of the item determined to be most similar to the product to be manufactured, and using that retrieved information to manufacture the product.
15. A descriptor creation apparatus configured to create a descriptor for a three-dimensional (3D) computer-generated model, which apparatus comprises: an image renderer configured to obtain a plurality of two-dimensional (2D) rendered images of the model from a fixed viewpoint by rotating the model fully about each of a plurality of different axes independently; a feature vector extractor configured to extract from each 2D rendered image a feature vector representing features of each 2D rendered image; and a descriptor assembler configured to form the descriptor by assembling a vector string comprising each of feature vectors extracted.
16. The apparatus as claimed in claim 15, wherein the descriptor assembler is configured to include in the descriptor one or more dimension-related features each representing a dimensional feature of the model.
17. The apparatus as claimed in claim 15, wherein the image renderer is operable to rotate the model about one or more of the axes in angular steps of magnitude greater than zero.
18. The apparatus as claimed in claim 17, wherein the angular step is a same for each axis.
19. The apparatus as claimed in claim 15, wherein the plurality of different axes comprises three mutually-perpendicular axes.
20. The apparatus as claimed in claim 17, wherein the plurality of different axes comprises three mutually-perpendicular axes and each angular step is in a range from 30° to 45°.
21. A descriptor creation apparatus configured to create a descriptor for a three-dimensional (3D) object, comprising: a feature vector extractor configured to extract from one or more two-dimensional (2D) images of the 3D object respective feature vectors each representing features of the 2D image; and a descriptor assembler configured to form the descriptor from one of a feature vector extracted, when there is only one 2D image, and to assemble a vector string comprising each of the feature vectors extracted, when there is more than one 2D image; the descriptor assembler being further configured to include in the descriptor one or more dimension-related features each representing a dimensional feature of the 3D object.
22. A searching apparatus configured to search a database of three-dimensional (3D) items for similarity with one of a specified 3D object and 3D computer-generated model, which searching apparatus comprises: a similarity assessor configured to use one of a feature vector and feature vectors of a descriptor associated with the one of the specified 3D object and the model and feature vectors of descriptors associated with 3D items in the database to assess a degree of similarity between the one of the specified 3D object and the model and one or more 3D items in the database; wherein each of the descriptors associated with the 3D items in the database was created by descriptor creation apparatus in accordance with claim 15, and one of the descriptor associated with the specified 3D object was created by descriptor creation apparatus in accordance with claim 21 and the descriptor associated with the specified 3D model was created by descriptor creation apparatus in accordance with claim 15.
23. The apparatus as claimed in claim 22, wherein the descriptors associated with the 3D items in the database were created in accordance with the apparatus of claim 15, and the similarity assessor is further configured to use dimension-related features of the descriptors of the one of the specified 3D object and the model and the 3D items in the database to identify any 3D item in the database which is not compatible in size with the specified 3D object/model.
24. A non-transitory computer readable storage medium for storing a method for controlling a computer as recited in claim 1.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Reference will now be made, by way of example, to the accompanying drawings, in which:
[0042]
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
DETAILED DESCRIPTION
[0055] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the like elements throughout. The embodiments are described below to explain the embodiments by referring to the figures.
[0056] The input for which a descriptor is required may be either: [0057] 1. a full 3D model of a physical object, given in a supported geometry format; or [0058] 2. only one or a few images (views), such as photographs or drawings, of a physical 3D object.
[0059] As illustrated in
[0060]
[0061]
[0062] The creation of a 3D descriptor for input comprising a full 3D model in accordance with an embodiment will now be described with reference to the flowchart of
[0063] After the 3D model is read into the processor 10 (step S31), a rendering engine (image renderer 11) is used to create one or more views (images) of the 3D model (step S32). A process of creating such images is shown in the flowchart of
[0064] The process of
[0065] The effect of the rotation process is both to increase the amount of information extracted from the 3D model and to ensure that the descriptor is fully or substantially invariant to object rotation. For many practical applications, this method is fully rotation invariant, since for most CAD models the orientation of the 3D model is aligned to one of the axes. The descriptor proposed in the present application would be fully rotation invariant in all cases if a very small rotation angle was used when performing the rendering, but this would require significant computational resources. Accordingly, in exchange for a small reduction in rotational invariance, larger rotation steps are used so that less computational resources are required. For 3D CAD models coming from mechanical design, it has been found that rotational increments or steps DX=DY=DX=45 degrees provides a good balance between computational requirements and accuracy; this results in twenty-two images per 3D model.
[0066] Returning to the process of
[0067] Feature extraction may follow the process shown in
[0068] Returning to
[0072] Finally, all the feature vectors of the 3D model and its dimensional features are assembled into one descriptor for the 3D model (step S35). An example of such a descriptor is shown in
[0073] When (as in most cases) the descriptor is rotation invariant, the order of the feature vectors for the images is not important. Also, for computational efficiency it may be preferable that the feature vectors for all images are combined and stored as one contiguous vector, while the dimensional features are stored as a separate vector.
[0074] By way of example, the creation of a descriptor in respect of a 3D CAD model of a screw will now be described. As shown in
[0075] For each image, a GoogLeNet deep neural network is used to extract feature vectors. Firstly, each input image is resized to 256×256 pixels, the pixel mean is extracted, and the data is placed on the input layer (data layer) of the neural network. The data is fed forward through the network and the values of the layer “pool5/7×7_s1” (the penultimate layer) are extracted as a vector of length 1024. Hence each image is transformed into a feature vector of 1024 values. The dimension of the bounding box (3 values) and the total surface area of the model are extracted as dimensional features. Thus, in total, the assembled descriptor (feature vectors of all images and dimensional features) has a length of 22*1024+4=22,532 values.
[0076] As described above, the method of
[0077] For a database of items comprising 3D computer-generated models the process shown in
[0078] Once descriptors for the 3D item database have been created, the database can be queried to find 3D items similar to an input model or object. Searching apparatus comprising a similarity assessor 30 as shown in
[0079] The first step of the search process of
[0080] Following the creation of a feature matrix fM for the input item as described above, a database matrix dM is created (step S72) by the database matrix generator 32. To create the database matrix, the feature matrix fM for each model in the database is created and then all the rows of those feature matrices fM are appended to create a matrix of size DS*(NX+NY+NZ) rows, where DS is the total number of models in the database, and N columns, where N represents the length of the feature vectors.
[0081] Then, the models in the database which are closest to the input item are found using the remaining steps of the search process of
[0082] Firstly, in step S73, a similarity matrix sM, where sM=1−fM*dM.sup.T, is computed by the similarity matrix generator 33. That is sM contains the cosine distance between the feature vectors of each view of the input model and the feature vectors of each view of each model in the database. In the formula for computing sM, the symbol “*” represents matrix multiplication and the superscript “T” represents matrix transposition. Note that this formula assumes that the feature vectors have been normalized.
[0083] Next, a reduction operation is performed (step S74) on the matrix sM by the similarity vector generator 34 to compute a similarity vector sV, as described below The vector sV is of length DS, with the j-th element storing the distance between the input model and the j-th model in the database. Here the distance between the i-th view of the input model and the j-th model in the database is defined as the minimum cosine distance between the feature vector corresponding to the i-th view of the input model (in other words, the i-th row of sM) and the feature vectors corresponding to all views of the j-th model in the database. The distance between the input model and j-th model in the database is defined as the summation over all the distances between the views of the input model and j-th model in the database.
[0084] After the similarity vector sV, i.e. the distance the input model and all models in the database, has been computed, the models from the database that do not satisfy the dimensionality criteria are removed (step S75) by the dimension filter 35. Lastly, an ordered vector of model IDs for the models which have been found to be similar is output (step S76), with the first one being that with the smallest distance (i.e. closest similarity) to the input model.
[0085] By way of example, a process for finding in a database 3D models similar to a particular input 3D model will be described with reference to
[0086] The feature matrix fM, containing the features of the input model, has 22 rows and 1024 columns while the database matrix has 8,000*22=176,000 rows and 1024 columns. The similarity matrix sM, which has 22 rows and 176,000 columns, is then computed. The similarity matrix sM shows the distance between the feature vectors of each of the 22 images of the input model and those of the feature vectors of all images of all 3D models in the database. Following the above-mentioned reduction operation, a similarity vector sV is obtained which has 8,000 elements that represent the distance between the input model and all models in the database. The results are then ordered by distance, those which show a relative difference in dimension features larger than 200% are removed, and (as shown at the bottom of
[0087] Embodiments can be applied in any field that requires searching for 3D data. Examples are mechanical design, 3D printing, etc. Embodiments can also be applied in fields where an input 3D model is not available, such as object recognition from live cameras.
[0088] A particular application of an embodiment is described below with reference to
[0089]
[0093] Information relating to previously-manufactured products is stored in a product database 30. Geometric information relating to each product is held in the database in the form of a descriptor which has been derived from the products using the above-described descriptor assembly process. The database 30, or a separate database, holds other information relating to the products in both text and numeric formats that is referred to hereafter as metadata. The metadata can be split into three categories: [0094] 1. Product specification metadata, which includes material properties, tolerances and any other information required to describe the product to a level that enables it to be manufactured [0095] 2. Manufacturing process metadata (i.e. accumulated manufacturing knowhow), which includes information regarding the manufacturing process to be used, such as the stages of the manufacturing process, machine parameters, energy usage, materials usage, etc. [0096] 3. Manufacturing cost, selling price, delivery time, etc. (which are likely to be dependent upon the manufacturing process that was used).
[0097] The system 10, upon receiving the required information from the user regarding the product to be manufactured, uses information extracted from similar previously-manufactured products found in the database 30 to generate one or both of the following outputs: [0098] A recommendation for the best manufacturing process to be used to manufacture that product [0099] Under the assumption that the recommended manufacturing process will be used, a quote with information such as cost, delivery time and so on.
[0100] In particular, the system 10 employs a searching method according to an embodiment to determine whether the database 30 contains any items which are similar to the product which is to be manufactured. In this case searching is carried out using a feature vector obtained by appending a feature vector or vectors derived from stored item metadata to feature vectors extracted from the stored item descriptors. If more than one item in the database is identified as being similar to the product, the items are ranked according to their degree of similarity with the product to determine which item is most similar to the product. The previously-stored details of the item considered to be similar (or most similar) to the product are used to provide the user of the system with the required information, i.e. a recommendation as to the best manufacturing process to use to make the product and/or a quotation as to how much the product will cost to manufacture and/or a delivery time.
[0101] Embodiments may be implemented in hardware, or as software modules running on one or more processors, or on a combination thereof. That is, those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functionality described above.
[0102] The embodiments may also be one or more device or apparatus programs (e.g. computer programs and computer program products) for carrying out part or all of the methods described herein. Such programs may be stored on computer-readable media, or could, for example, be in the form of one or more signals. Such signals may be data signals downloadable from an Internet website, or provided on a carrier signal, or in any other form.
[0103]
[0104] The computing device comprises a processor 993, and memory, 994. Optionally, the computing device also includes a network interface 997 for communication with other such computing devices, for example with other computing devices.
[0105] For example, an embodiment may be composed of a network of such computing devices. Optionally, the computing device also includes one or more input mechanisms such as keyboard and mouse 996, and a display unit such as one or more monitors 995. The components are connectable to one another via a bus 992.
[0106] The memory 994 may include a computer readable medium, which term may refer to a single medium or multiple media (e.g., a centralized or distributed database and/or associated caches and servers) configured to carry computer-executable instructions or have data structures stored thereon. Computer-executable instructions may include, for example, instructions and data accessible by and causing a general purpose computer, special purpose computer, or special purpose processing device (e.g., one or more processors) to perform one or more functions or operations. Thus, the term “computer-readable storage medium” may also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methods of the present disclosure. The term “computer-readable storage medium” may accordingly be taken to include, but not be limited to, solid-state memories, optical media and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media, including Random Access Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid state memory devices).
[0107] The processor 993 is configured to control the computing device and execute processing operations, for example executing computer program code stored in the memory 994 to implement the methods described with reference to
[0108] The display unit 995 may display a representation of data stored by the computing device and may also display a cursor and dialog boxes and screens enabling interaction between a user and the programs and data stored on the computing device. The input mechanisms 996 may enable a user to input data and instructions to the computing device.
[0109] The network interface (network I/F) 997 may be connected to a network, such as the Internet, and is connectable to other such computing devices via the network. The network I/F 997 may control data input/output from/to other apparatus via the network.
[0110] Other peripheral devices such as microphone, speakers, printer, power supply unit, fan, case, scanner, trackball etc may be included in the computing device.
[0111] Methods may be carried out on a computing device such as that illustrated in
[0112] A method may be carried out by a plurality of computing devices operating in cooperation with one another. One or more of the plurality of computing devices may be a data storage server storing at least a portion of the data.
[0113] The above-described embodiments may advantageously be used independently of any other of the embodiments or in any feasible combination with one or more others of the embodiments.
[0114] Although a few embodiments have been shown and described, it would be appreciated by those skilled in the art that changes may be made in these embodiments without departing from the principles and spirit of the embodiments, the scope of which is defined in the claims and their equivalents.