System for measuring objects in tally operations using computer vision object detection methodologies

Abstract

Stock management for wood and lumber products requires measuring and counting items individually on a continuous basis; considering a single lumber package alone can contain hundreds of pieces, it is a tedious task that is error prone when done manually. The invention provides a technology solution that involves taking a picture of products using a smart-phone, or a tablet's built-in camera, processing said picture data to detect individual items using Artificial Intelligence Object Detection methods, and utilizing special algorithms to measure and compute unit volume to present the user a detailed description, measure, count, and a summary. This process helps identify and take stock counts faster and with higher accuracy.

Claims

1. Computer Object Detection models and algorithms to particularly recognize individual lumber pieces in an electronic image, detect and measure their dimensions, including their widths; such models and algorithms not being limited to a particular platform, library, or tool set in the art of Artificial Intelligence (AI) computer vision object detection.

2. Computer Object Detection models and algorithms to recognize individual items of inventory consisting of three dimensions, via an electronic image, detect their dimensions, and measure volume and count for purposes of stock identification and control.

3. Algorithms and methods to convert and calculate detected object dimensions in claims 1 and 2, to real world actual measured values in imperial or metric units, i.e. inches or millimeters, or other units, compute individual and total volumes based on unit of measure in use.

4. Software application, providing the capability to take pictures, process the image data in detecting objects for measuring, identify, organize, and manage product images with type, quality, quantity data including ability to transmit, upload, and share its data; such software application running on various available platforms.

5. In the software application in claim 4, presentation of the measured and counted data in detail and summary form upon completion of the detections, on the mobile platform, as well as other computing platforms, in display on screen and printed format.

Description

DESCRIPTION OF DRAWINGS

[0019] The present invention is described in more detail, reference being made to the accompanying drawings, in which

[0020] FIG. 1. Illustrates the components of the invention; a device to snap a picture and a typical lumber pack.

[0021] FIG. 2. is the application screen where the detected objects have boxes drawn around each item, and the additional product details are displayed.

[0022] FIG. 3. Illustrates the process flow

DETAILED DESCRIPTION

[0023] FIG. 1. Outlines general components of the invention and the basic process. Facing the front of the lumber pack 1 user prepares to take a picture of the bundle to be measured. Using his/her smart phone, tablet, or other mobile device with a built-in camera 2, user snaps a picture of the bundle from the SnapTally application running on the device. If the picture is satisfactory, user confirms and the detection process starts. Next, user initiates the “Detect” function in the application against the picture taken, detection process runs and detected items are displayed on the screen 3.

[0024] FIG. 2. Illustrates the application screen 1 upon completion of detection process, containing the description of the product 2 (entered by user), all the detected items have bounding boxes drawn around them 3. User may inspect and further edit, add, or remove boxes if necessary. User initiates the “Product” function to specify the type of product, it's quality, and any other details related to this work. He/she then proceeds to run the actual measurement of the detections with “Measure” function. In measuring each item, the pixels in the image are evaluated, and based on the boxes drawn around each object, the actual measurement value is assigned to the pieces. The application displays product information and summary data resulting from the identification and measurement of the bundle 4. All the tally details are shown with thickness, length, width, number of pieces, and volume 5.

[0025] The application finally computes all the individual volumes, and a summary of the lumber bundle. This data is saved on the device, and can be managed, edited, re-detected later if necessary. Captured data can be uploaded, transmitted to a server system for further processing.

[0026] It should be emphasized that the term “comprises/comprising” when used in this specification is taken to specify the presence of stated features, steps or components but does not preclude the presence or addition of one or more other features, steps, components or groups thereof.

[0027] Description of the Invention

[0028] The foregoing disclosure and the claims describe the preferred embodiments of the present invention. Particularly with respect to the claims, it should be understood that changes may be made without departing from its essence. In this regard, it's intended that such changes would still fall within the scope of the present invention which may be accomplished. To the extent such revisions utilize the essence of the present invention, each naturally fall within the breadth of protection encompassed by this patent.

Objective

[0029] The present invention SnapTally addresses the need to take measurements of products quickly and accurately. The system utilizes a built-in high-definition camera to capture the image of the object; it processes the image, and produces data related to the measurements and counts within the image. Main objective is to provide measurement and count data of objects in the real-world by simply taking pictures.

[0030] Benefits

[0031] While a manual measuring operation is the most inefficient method, and other alternative solutions require apparatus and equipment that may not be practical to install, as well as being costly, the SnapTally invention can be employed by anyone with a smart phone, tablet, or another mobile device with a camera; it's practical to deploy and it produces results quickly.

[0032] Architecture and Methods

[0033] The SnapTally system is based on two major components; mobile device with a camera and SnapTally application. The application runs on smart phones, tablets, and other devices with Android operating systems; however other operating systems can also be available. A built-in camera is required to snap pictures, and network connectivity enables data transmission and processing.

[0034] General Specifications

[0035] Hardware and Equipment: Smartphone, Tablet, Rugged Handheld Device, or Mobile Device [0036] Built-in Camera [0037] Wi-Fi and/or GSM

[0038] Operating System: Android

[0039] Mobile Software: Sierra SnapTally to detect, measure, and manage related data

[0040] Process

[0041] The SnapTally mobile software captures an object's image using device's built-in camera, and saves the image on the device; a built-in flash or an external flash can be used to aid in improving image quality.

[0042] The object being measured is a package of lumber boards. The system recognizes and marks each individual board within the pack and measures its width. The lumber package for measurement is shown in FIG. 1.

[0043] The picture is snapped from the front end of the package of lumber, the face showing the widths of the boards are to be measured (see FIG. 1, label 1). The image is then submitted to be processed, either on the mobile device, or on a server running a model of a Machine Learning Library with neural networking and algorithms used in identifying pieces of objects contained in the picture. The model resolves each object and returns the data related to the detection performed. The image is displayed for the operator with all the object representations drawn as boxes around each piece. (See FIG. 2)

[0044] The user supplies primary data to convert and compute image information to actual measurement data; the system uses such data to compute product total volume and count. In most applications the length data for a package is a fixed value, and thickness is part of the product identification specified by user. FIG. 3 labels 4 and 5 show all measurements of individual pieces and totals displayed on the screen. The operator may also add or edit objects manually as required, enter product information, save and/or transfer the data for continued processing.

[0045] Comparison to Other Inventions

[0046] U.S. Pat. No. 5,307,294 “Automated End Tally System” is designed to perform the task of measuring lumber boards. This system requires a sophisticated mechanical and electronic equipment to be installed. In comparison, the present invention requires no such costly installations; the only equipment used primarily is a hand held device with a camera and the software for detections. The prior art also requires lifting and moving heavy bundles of lumber whereas the present invention performs all measuring on location without moving the objects.

[0047] U.S. Pat. No. 7,431,212 “Practical mobile data collection and processing system with bar-coded tally ruler” is another invention that measures and counts products. The system uses a bar-coded ruler to scan each individual piece whereas the present invention performs the measuring by detecting all the objects at once.

[0048] Development

[0049] The mobile application that drives the process of detection and management of data is built for the Android® platform, and Java language has been used to develop the system. The application utilizes a local database management system SQLite to store and manage data related to the application. The application is easily portable to other platforms such as Apple iPhone®. On the server platform, a Machine Learning library model that has been specifically trained to process object detection is configured to service the mobile application. The server accepts images from the mobile application and provides coordinates of objects upon detection. A compact version of the Machine Learning model is also available to process images without the server.

[0050] Object Detection

[0051] One of the components of the invention is the object detection process to identify the items in the image presented. The methods utilized are explained in the following sections:

[0052] Training a Custom Object Detection Model

[0053] An Object Detection Model is part of an Artificial Intelligence system that includes a deep-learning network. The model must be trained to detect specific types of objects. A custom model is generally based on a framework of available models that are further trained and/or customized.

[0054] Tensorflow is a well known and utilized open source platform with libraries and tools for machine learning that is provided by Google. Libraries contain various object detection models, as well as other features such as speech. Tensorflow is available for download from an open source platform “github”.

[0055] Among various object detection models available for implementation, the invention utilizes a specific model as described below:

[0056] Faster_RCNN: Faster Regional Convolutional Neural Network. This base model performs better for object detection of smaller objects.

[0057] Using this model does not limit the invention's technical architecture from using other models, combination of different models, or other computer vision technologies now or in the future. The Faster RCNN model is the current technology utilized in object detections at the time of the filing of the patent. [0058] 1. In order to train an object detection model, one must first annotate the objects being presented for training. This is one of the most painstaking and important tasks in object detection. Present invention contains a large collection of images taken on location at actual customer/user sites. Each individual object in the image is then “annotated”, a box drawn around it to identify it. A single image contains hundreds of objects, the training collection consists of thousands of images, both original and augmented, for performance improvements.

[0059] Once the annotations are completed the actual machine learning process starts. The data is presented to machines and software designed for learning about the data; this process involves testing, modifying parameters for improvements, presenting additional data, and training on multiple cycles until valid results are obtained. Fine tuning for a specific application, such as intended in the present invention, can take months and years. Therefore, while object detection has been a popular catch phrase and utilized in many different practical applications, in the present invention it represents a focused solution, and requires a highly specialized concentration, algorithms, effort, and know-how to provide a particular industrial application.

[0060] Serving the Model for Inference

[0061] Once satisfactory results are obtained from training and testing, a trained “model” is generated to serve and produce consistent detection of objects for the application. In the present invention, SnapTally, such a model is presented images and returns all of the objects contained within a particular image with coordinates. The platform of the service can vary depending on the desired objectives and configuration; this platform can include a mobile device.

[0062] Summary and Status

[0063] SnapTally system is continually being improved, additional capabilities and features may be added, existing algorithms may be optimized through developed versions of the solution. However, the objective of the invention and the method of producing real-world measurements from images remain the same, detecting all of the pieces and measuring them accurately. Major features included are, capturing images via camera, Object Detection, Measure and Compute, Data Management, Editing and Adding Objects, Uploading to Cloud, Label Printing.

CONCLUSION

[0064] Ability to measure, count, and manage product information, particularly for wood and lumber inventories present a unique challenge. A fast and accurate method is required to keep up with business demands as products are continually on the move. The SnapTally invention is unique and offers new methods by simply snapping a picture of a product to detect and measure objects within the image; in comparison to other image based detection systems, SnapTally does not require a special apparatus or equipment. The system works with smart phones, tablets, and other devices with built-in cameras. SnapTally empowers users and managers, brings an effective solution to the problem of measuring and counting inventories.