Cashier fraud detecting system and method and product image selection generation for artificial neural network learning related applications

Abstract

A group of inventions relates to artificial neural networks and their application for computer vision, in particular for the surveillance camera data processing systems and methods to automatically detect cashier fraud by verifying images using artificial neural networks. To detect cashier fraud, a POS system includes a barcode reader, memory, an image capture device, and a data processing module. The data processing module is configured to receive the data about the scanned product from the product database and to receive the video data from the image capture device. An automatic generation of product image set for artificial neural network learning contains stages when the barcode is read by placing an item against a barcode reader by a cashier, the barcode data signal then provides the data about the scanned product from product database, when the barcode data signal gives the image of the placed product from the image capture device, the received image of the placed item is saved with the data about the scanned product in the product database, then the abovementioned stages are repeated for each item placed against the barcode reader.

Claims

1. A POS system for detecting fraud comprising: a barcode reader configured to send a barcode data signal; a memory module configured to store a product database with product images corresponding to products; an image capture device configured to receive video data from a cashier barcode scanned area; and a data processing module configured to: receive data related to a first product from the product database corresponding to the barcode data signal from the barcode reader and receive data related to at least a second product from the product database corresponding to the barcode data signal from the barcode reader; receive video data from the image capture device; verify using an artificial neural network, wherein the verification comprises: comparing a first product image received from the image capture device at the moment of receiving a barcode data signal with at least one product image received from the product database corresponding to the first product corresponding to the barcode data and comparing at least a second product image received from the image capture device at the moment of receiving a barcode data signal with at least one product image received from the product database corresponding to the second product corresponding to the barcode data; updating the artificial neural network with the second product image; and comparing image descriptors of each scanned item and of the images for the scanned product barcode using Euclidean distance, wherein if the Euclidean distance is less than a pre-set value, then the data processing module determines using the artificial neural network that the image of the scanned product matches the image from the product database and if the Euclidean distance exceeds the pre-set value then the data processing module continues to compare the image descriptors of the scanned item and of other product images in the product database; and detect fraud when the verification result is negative, wherein data about each product comprises a product price; and wherein if the product price of the product is less than pre-set threshold, then the product is not verified.

2. The system of claim 1, wherein the data processing module is additionally configured for automatic generating or updating the product image selection for artificial neural network learning, and this selection contains sets of product images.

3. The system of claim 1, wherein the products in the product database are divided into the product classes.

4. The system of claim 3, wherein each product or each class of products has its own artificial neural network used for verification.

5. The system of claim 1, wherein data about each product includes the name of the product, its value, its barcode, a selection of product images.

6. The system of claim 5, wherein the verification runs in a real time mode.

7. The system of claim 6, wherein the detection of cashier fraud requires sending a notification of a pre-set particular user.

8. The system of claim 5, wherein verification runs through archived data obtained from the barcode reader and the image capture device stored in the memory.

9. The system of claim 8, wherein archive data verification automatically generates a report with all detected facts of a cashier fraud.

10. A method of detecting cashier fraud embodied by a POS system comprising at least one data processing module, wherein the method contains the stages: placing a first product against a barcode reader by a cashier to read a barcode, and developing a first barcode data signal; and placing a second product against the barcode reader by the cashier, and developing a second barcode data signal; wherein each barcode data signal gives the data about a corresponding scanned product from product database, wherein the product database contains constantly updated product images; wherein video data from an image capture device is received and verified by an artificial neural network taught by the constantly updated product database and by video data of multiple products, wherein the verification comprises: comparing, in parallel for each barcode data signal giving the data about the corresponding scanned product from the product database, and product images received from the image capture device at the moment of receiving the barcode data signal; comparing image descriptors of each scanned item and of the images for the scanned product barcode using Euclidean distance, wherein if the Euclidean distance is less than a pre-set value then the data processing module determines using the artificial neural network that the image of the scanned product matches the image from the product database and if the Euclidean distance exceeds the pre-set value then the data processing module continues to compare the image descriptors of the scanned item and of other product images in the product database; if verification result is negative, then at least one data processing module detects cashier fraud, wherein data about each product comprises a product price; and wherein if the product price of the product is less than pre-set threshold, then the product is not verified.

11. The method of claim 10, further comprising selecting or updating the product images using the data processing module; and teaching an artificial neural network and detecting cashier fraud in parallel.

12. The method of claim 10, wherein the products in the product database are divided into the product classes.

13. The method of claim 12, wherein each product or a class of products has its own artificial neural network used to verification.

14. The method of claim 10, wherein the data about each product comprises a product name, a product barcode, or a product image.

15. The method of claim 14, wherein the product image selection contains N latest uploaded images for this product where N is a positive integer pre-set by a user.

16. The method of claim 10, wherein the verification runs in a real time mode.

17. The method of claim 16, wherein the detection of cashier fraud requires sending a notification of a pre-set particular user.

18. The method of claim 10, wherein verification runs through archived data obtained from the barcode reader and the image capture device stored in a memory.

19. The method of claim 18, wherein archive data verification automatically generates a report with all detected facts of a cashier fraud.

20. Non-transitory computer readable medium storing instructions that, when executed by a computer, cause it to perform the method of detecting cashier fraud of claim 10.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

(1) FIG. 1 is a flow chart of a computer system detecting cashier fraud.

(2) FIG. 2 is a flow chart of a method for detection of cashier fraud.

(3) FIG. 3 is a flow chart of a method of automatic selection of product images to teach an artificial neural network.

DETAILED DESCRIPTION

(4) Possible embodiments for the inventions are described below. However, the group of inventions is not limited to these embodiments. It is apparent to those skilled in the art that the inventions include other embodiments.

(5) The technical solution in its various embodiments may be implemented as POS systems and methods implemented by various computational devices, as well as a computer-read data medium storing CPU-executed instructions.

(6) FIG. 1 gives a flow chart of a POS system for detecting—cashier fraud. The POS system includes a barcode reader (10), an image capture device (20), memory (30), and at least one data processing module (40, . . . , 4n).

(7) In this context, POS systems may be any computational systems based on interconnected hardware and software.

(8) A barcode reader may be a scanner of any manufacturer (of any type, for example, LED, laser, image scanner). A barcode is graphic information printed on the surface or the package of a product with the possibility to read it with devices, for example, scanners. In some embodiments the barcode may be a QR-code which is recently quite popular in trade.

(9) In this application, an image capture device may be a video camera.

(10) A data processing module may be a CPU, microprocessor, computer, PLC (programmable logic controller), or an integrated circuit configured to execute data processing commands (instructions, applications, etc.).

(11) A memory device configured to store data may be, without limitation, a hard disk drive (HDD), flash drive, ROM (read-only memory), solid-state drive (SSD), optical drives, etc.

(12) The computer system may include other compatible devices, for example, sensors, input and output devices, display devices, etc.

(13) An example of how the abovementioned POS system works to detect cashier fraud is provided below. The same principles may also be applied to the disclosed method of detecting a cashier fraud.

(14) Considering the operating principles of POS system embodiments in a real time mode, suppose that a customer with an item comes to the cashier at the checkout to buy the product.

(15) An image capture device, a video camera in our case, is positioned to receive video data from the cashier's barcode scanning area with no interruptions. The POS system may also include other surveillance cameras to control other zones in the checkout area, for example, in the area where cashier accepts the items from the customer or in the zone where the purchased items are picked up.

(16) When a cashier places a product against the barcode reader, the barcode reader generates a barcode data signal, which prompts the data processing module to retrieve the data for the scanned product barcode from the product database.

(17) The data about each product stored in the product database may include, but not limited to, the name of the product, its price, its barcode, product images, etc. The products in the product database may be subdivided into product classes. Examples of these classes may be, without limitation, box, bundle, glass bottle, plastic bottle, etc.

(18) The data processing module receives the video from the image capture device and identifies the image of the scanned item placed against the barcode reader. Then the data processing module verifies the image using an artificial neural network.

(19) The verification includes comparison of the item's image received from the image capture device at the time of receiving the barcode data signal and at least one product image from the product image data stored for the scanned product barcode.

(20) If the verification result is positive, that is, the item image received from the image capture device sufficiently matches at least one of the images for the scanned product barcode, then the product registration is considered successful. If the image of the scanned item matches at least one image from the images for the scanned product barcode, then the system stops the verification process, even if the images include images which have not been compared with the scanned item image. This saves the available resources of the system and speeds up the comparison.

(21) If the verification result is negative, then at least one data processing module detects the cashier fraud.

(22) In some POS system embodiments, this verification may be done both for the product data and for the product data class related to the scanned product barcode.

(23) Before the item image is transmitted to the artificial neural network during verification and during the learning process of the artificial neutral network, the image may be preprocessed for, for example, finding the borders of the item, scaling the item image, and by transforming the color characteristics of the item image.

(24) Then the artificial neural network identifies the key features of the item image. The result is a set of numbers called image descriptors. Verification compares the image descriptors of the scanned item and of the images for the scanned product barcode using, for example, Euclidean distance. In the context of the example, if the Euclidean distance is less than a pre-set value, then the data processing module determines using the ANN that the image of the scanned product matches the image from the product database. If the Euclidean distance exceeds the pre-set value, then the data processing module continues to compare the image descriptors of the scanned item and of the next product image. If all available images for a product do not sufficiently match the image of the scanned item received from the image capture device, the data processing module detects the cashier fraud.

(25) Additionally, this POS system is configured so that once cashier fraud is detected in a real time mode, a notification to a pre-set user of the system, for example, to a security service employee, is sent. The notification to the user could be sent via an SMS, MMS, or an email. This configuration helps the users to respond promptly to the detection of the fraud, to arrest the wrongdoers, and to hold them responsible.

(26) The disclosed POS system for detecting the fraud is more accurate in comparison with the known equipment because the comparison is made only of the scanned item image and the data for the scanned product, rather than the data for the entire product database. The advantages of this solution include a high rate of image processing and higher accuracy in detecting cashier fraud.

(27) This computer system is described above in the real time mode, although it can also be used for the verification of archived data by the artificial neural network. In such embodiment, the data from the barcode reader and the image capture device automatically get stored in the memory of the POS system. Thus, a security service employee with the access to the data may look through them and launch the verification at the data processing module at any convenient time. For example, to save the resources of the data processing module, a security service employee may run the verification once a day or once a week. This verification takes less time and is more efficient when cases of fraud are rare.

(28) The result of this verification to detect cashier fraud may be prepared as an automatically generated report at the display module or stored in the computer system memory or in the security system database. This report includes all detected facts of cashier fraud.

(29) Additionally, the mentioned POS system is configured so that the plurality of product images is generated automatically or updated to teach the artificial neural network.

(30) At least one data processing module may generate or update the product images, teach an artificial neural network, and detect cashier fraud in parallel. Once the data processing module receives an image of the scanned item, this image is used in parallel for verification and for generation or updating the product images. And if this image is added to the product images before the verification runs to the end, then, for this case, the data processing module is configured so that the image of the scanned item is not compared with the just added new product image.

(31) In some embodiments, a POS system contains a separate artificial neural network used for verification of each class of products or even for each product.

(32) Additionally, to update the data in the POS system, the image selection may contain only N latest uploaded product images for each product, where N is a user preset positive integer.

(33) If the information about a fraud concerning cheap items is not very important for the store in comparison with the information about more expensive goods, then the system may be configured to analyze the price of the scanned item before the verification process. If the price in the product data is below a preset threshold (for example, less than $10, or $100, or $500, etc.), then this product will not be verified. The user of the POS system can set this threshold and change it in particular situations.

(34) An example of a particular embodiment for the method of detecting the cashier fraud is shown in FIG. 2 as a flow-chart of one of the embodiments of the method of detecting cashier fraud.

(35) This method is embodied in a POS system with at least one data processing module. The method contains the stages when

(36) (100) once a cashier places an item against a barcode reader, a barcode is scanned, and a barcode data signal is generated;

(37) (200) the barcode data signal helps get the data about the scanned product from the product database, which database contains a continuously updated product image selection;

(38) (300) video data is received from an image capture device; and

(39) (400) verification is run using the artificial neural network taught by the updated product database, where the verification means the comparison of an item image from the image capture device with at least one product image from the data about the scanned product;

(40) (500) if the verification result is negative, then at least one data processing module detects the cashier fraud.

(41) This method may be implemented by the POS system disclosed above and may be extended and enhanced by all alternative elements disclosed for POS systems detecting cashier fraud.

(42) One method of forming set of images for teaching an artificial neural network used for verification to detect cashier fraud is disclosed below.

(43) FIG. 3 shows a flow chart of one embodiment of the method of automatic generation of product image set for teaching an artificial neural network.

(44) This method is implemented using a POS system disclosed above with at least one data processing module. The method contains the following stages:

(45) (1001) once a cashier places an item against the barcode reader, a barcode is scanned, and a barcode data signal is generated, as described above;

(46) (1002) the data about the scanned product is obtained from the products database stored in POS system memory based on the barcode data signal received by the data processing module;

(47) If this product database does not contain the information about the scanned product, then this product database automatically generates a new entry about a new product.

(48) (1003) Once the data processing module receives the barcode data signal, this module receives the image of the placed item from the image capture device configured to receive the video data from the barcode scanning area.

(49) (1004) The data processing module stores the received image of the placed item with the data about the scanned product in the product database.

(50) The received image is stored even when the scanned product data already contains at least one image. Also, a user can set a threshold N for the number of images for each product.

(51) For example, a user sets N to 10. Once the barcode data signal is received, the data processing module receives the data about the scanned product and analyzes the received data. If the number of the images in the image set for the scanned product equals ten, then the data processing module deletes the oldest image and stores the new just received image of the product into the set. Thus, the information about the products is updated all the time. This is necessary, because the appearance of the products changes with time, and there is no need to store old images of the products.

(52) The user may also set a product price threshold. In this case, once the barcode data signal is received, the data processing module receives the data about the scanned product and analyzes the received data. If the price of the product is lower than the preset threshold (for example, less than $100), then the image received at the moment of scanning the item from the image capture device is not added to the product images.

(53) (1005) The stages 1001-1004 are repeated for each item placed against the barcode reader, thereby the product image set is automatically generated or updated to teach the artificial neural network used for verification to detect cashier fraud.

(54) In the disclosed solutions, the artificial neural network may learn from the updated product database.

(55) The product image selection generation or update and artificial neural network learning are continuous processes, because the set of relevant products and their appearance change with time. A user of the POS system may set a specific time when the artificial neural network is taught. For example, once a day or once a week. And the learning may be done by, for example, the data processing module of the POS system, or by the POS system cloud service, or by any other computational device.

(56) The data processing module may generate or update the product image set, teach the artificial neural network, and detect cashier fraud in parallel.

(57) In some embodiment, where all products in the product database are divided into product classes, each product class has its own artificial neural network. Large stores may have an artificial neural network even for each product. As has been described above, the artificial neural networks are used for product verification, thus, the greater the number of artificial neural networks, the more accurate the verification result is, and the more exact and error-free cashier fraud detection is, because each artificial neural network is specifically designed for one particular product or a class of products.

(58) The embodiments of the present group of inventions may be implemented by software, hardware, programmable logic devices, or their combination. In the example embodiments, the programmable logic devices, software, or a set of instructions are stored on one or more different traditional computer-readable data media.

(59) In this description, a computer-readable data medium may be any medium or device which can store, contain, transfer, distribute, or transport instructions (commands) for their use (execution) by a computation device, for example, a computer, wherein the data medium could be a transitory or non-transitory computer-read data medium.

(60) At least some of different operations described in this solution may be performed in a different order and/or in parallel.

(61) Although this technical solution has been described in detail to illustrate the most relevant and preferable embodiments, this invention is not limited to the disclosed embodiments, and it is designed for further modification and various combinations of features different from the described embodiments. For example, the present invention permits that one or more features of any embodiments could be combined, if possible, with other one or more features of other embodiments.