DEVICE AND METHOD FOR FORMING AT LEAST ONE GROUND TRUTH DATABASE FOR AN OBJECT RECOGNITION SYSTEM

Abstract

Described herein are a device and a method for forming at least one ground truth database for an object recognition system and for keeping the at least one ground truth database current. The device comprises includes at least the following components: a data storage unit configured to store color space positions and/or reflectance spectra and/or luminescence spectra of different objects; and a processor programmed for communication with the data storage unit and with the object recognition system.

Claims

1. A device for forming at least one ground truth database for an object recognition system and for keeping the at least one ground truth database current, the device comprising at least the following components: a) a data storage unit configured to store color space positions and/or reflectance spectra and/or luminescence spectra of different objects; and b) a processor programmed for communication with the data storage unit and with the object recognition system, the processor programmed for: receiving, via a communication interface, measured color space positions and/or reflectance spectra and/or luminescence spectra of different objects, assigning each color space position and/or reflectance spectrum and/or luminescence spectrum to one of the different objects as a tag, storing the color space positions and/or reflectance spectra and/or luminescence spectra together with the respective different objects the color space positions and/or reflectance spectra and/or luminescence spectra are assigned to, respectively, in the data storage unit, thus forming the at least one ground truth database, monitoring, by using at least one sensor and/or artificial intelligence tools, a scene including at least some of the different objects for the occurrence of a triggering and/or recognition event, updating and/or supplementing dynamically in at least one of the at least one database the color space positions and/or the reflectance spectra and/or luminescence spectra stored in the respective at least one database in the case the triggering and/or recognition event occurs, and providing immediate access to the up-to-date color space positions and/or reflectance spectra and/or luminescence spectra.

2. The device according to claim 1, further comprising the processor programmed for providing as the at least one ground truth database a master database and a local database, the local database being in conjunction with the master database and the color space positions and/or the reflectance spectra and/or luminescence spectra stored in the local database are updated and/or supplemented over time by receiving from the object recognition system re-measured respective color space positions and/or reflectance spectra and/or luminescence spectra for at least some of the different objects in the scene and, thus, small and continuous changes of the respective objects are at least tracked in the local database.

3. The device according to claim 2, wherein the local database is stored locally in the scene or on a cloud server, the local database being only accessible for the object recognition system which is locally used in the scene.

4. The device according to claim 1, further comprising the processor programmed for tracking small and continuous changes of the different objects by monitoring changes in fluorescence emission magnitude and/or fluorescence emission spectral shapes of the respective objects.

5. The device according to claim 2, further comprising the processor programmed for supplementing the local database by a color space position and/or a reflectance spectrum and/or luminescence spectrum of an object by using the master database when the object is new in the scene and the new object's color space position and/or reflective and luminescence spectrum measured by the locally used object recognition system can be matched to a color space position and/or a reflectance spectrum and/or luminescence spectrum of an object stored in the master database.

6. The device according to claim 2, further comprising the processor programmed for synchronizing the master database and the local database regarding the different objects in the scene.

7. The device according to claim 2, wherein the master database comprises for each of the different objects color space position and/or reflectance spectrum and/or luminescence spectrum of the original object and color space position and/or reflectance spectrum and/or luminescence spectrum of at least one degraded/aged object descending from the original object.

8. A computer-implemented method for forming at least one ground truth database for an object recognition system and for keeping the at least one ground truth database current, the method comprising at least the following steps: providing, via a communication interface, color space positions and/or reflectance spectra and/or luminescence spectra of different objects, assigning, by a processor, each color space position and/or reflectance spectrum and/or luminescence spectrum to one of the different objects as a tag, storing the color space positions and/or reflectance spectra and/or luminescence spectra together with the respective different objects the color space positions and/or reflectance spectra and/or luminescence spectra are assigned to, respectively, in a data storage, thus forming the at least one ground truth database, monitoring, by using at least one sensor and/or artificial intelligence tools, a scene including at least some of the different objects for the occurrence of a triggering and/or recognition event, updating and/or supplementing dynamically in at least one of the at least one database the color space positions and/or the reflectance spectra and/or luminescence spectra stored in the at least one database in the case the triggering and/or recognition event occurs, and providing immediate access to the up-to-date color positions and/or reflectance spectra and/or luminescence spectra.

9. The method according to claim 8, further comprising providing a master database and a local database, the local database being in conjunction with the master database and the color space positions and/or the reflectance spectra and/or luminescence spectra stored in the local database are updated and/or supplemented over time by re-measuring by the object recognition system the respective color space positions and/or reflectance spectra and/or luminescence spectra for the different objects and, thus, small and continuous changes of the respective objects are at least tracked in the local database.

10. The method according to claim 9, wherein the local database is stored locally in the scene or on a cloud server, the local database being only accessible for the object recognition system which is locally used in the scene.

11. The method according to claim 8, wherein small and continuous changes of the different objects are tracked by monitoring changes in fluorescence emission magnitude and/or fluorescence emission spectral shapes of the respective objects.

12. The method according to claim 9, wherein the local database is supplemented by a color space position and/or a reflectance spectrum and/or luminescence spectrum of an object by using the master database when the object is new in the scene and the new object's color space position and/or reflectance spectrum and/or luminescence spectrum measured by the locally used object recognition system can be matched to a color space position and/or a reflectance spectrum and/or luminescence spectrum of an object stored in the master database.

13. The method according to claim 9, wherein the master database and the local database are synchronized regarding the different objects in the scene when at least one of a number of predefined events occurs.

14. The method according to claim 13, wherein the master database comprises for each of the different objects a color space position and/or a reflectance spectrum and/or luminescence spectrum of the original object and a color space position and/or a reflectance spectrum and/or luminescence spectrum of at least one degraded/aged object descending from the original object.

15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a machine to: receive, via a communication interface, color space positions and/or reflectance spectra and/or luminescence spectra of different objects, assign each color space position and/or reflectance spectrum and/or luminescence spectrum to one of the different objects as a tag, store the color space positions and/or reflectance spectra and/or luminescence spectra together with the respective different objects the color space positions and/or reflectance spectra and/or luminescence spectra are assigned to, respectively, in a data storage, thus forming at least one ground truth database, monitor, using at least one sensor and/or artificial intelligence tools, a scene which includes at least some of the different objects for the occurrence of a triggering and/or recognition event, update and/or supplement dynamically in at least one of the at least one database the color space positions and/or the reflectance spectra and/or luminescence spectra stored in the at least one database in the case the triggering and/or recognition event occurs, and provide immediate access to the up-to-date color positions and/or reflectance spectra and/or luminescence spectra.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0087] FIG. 1 shows schematically a flowchart of method for object recognition using at least one ground truth database formed and updated using one embodiment of the proposed device and/or of the proposed method.

[0088] FIG. 2 shows schematically a flowchart of instructions of an embodiment of the proposed computer-readable medium.

DETAILED DESCRIPTION OF THE DRAWINGS

[0089] FIG. 1 shows schematically a flow chart of a method for recognizing via an object recognition system, an object in a scene using a ground truth database which is formed and kept current using an embodiment of the method proposed by the present disclosure.

[0090] In the example described here, an object recognition system is provided which is used to recognize objects in a scene by sensing/measuring via a sensor, e. g. a spectrophotometer, reflectance spectra and/or luminescence spectra of the objects present in the scene and identifying by means of a measured fluorescence spectrum a specific object whose specific fluorescence spectrum is stored as a tag in a respective ground truth database which can be accessed by the object recognition system.

[0091] The object recognition system which is used to recognize objects in the scene has access at least to a local database stored in a data storage unit, the local database storing fluorescence spectra of objects which are or have been located locally in the respective scene. Besides such a local database, the data storage unit can also host a master database which is communicatively connected with the local database but which stores the fluorescence spectra of more than only the locally measured objects. Therefore, the master database is accessible for more than only the object recognition system which is locally used to recognize objects locally in the scene. The master database can also be stored in a further data storage unit which is in a communicative connection with the data storage unit storing the local database.

[0092] The data storage unit storing the local database as well as the data storage unit storing the master database can be realized by single standing-alone servers and/or by a cloud server. Both, the local database as well as the master database can be stored on a cloud.

[0093] The proposed device for forming the local database and also the master database for the object recognition system and for keeping the local database and the master database current, comprises besides the already mentioned at least one data storage unit, a processor which is programmed for a communication with the data storage unit and with the object recognition system. The processor is programmed for: [0094] receiving, via a communication interface, color space positions/coordinates and/or reflectance spectra and/or luminescence spectra of different objects, [0095] assigning each color space position and/or reflectance spectrum and/or luminescence spectrum to one of the different objects as a tag, [0096] storing the color space positions and/or reflectance spectra and/or luminescence spectra together with the respective different objects the color space positions and/or reflectance spectra and/or luminescence spectra are assigned to, respectively, in the data storage unit, thus forming at least one ground truth database, namely the local database and/or the master database, [0097] monitoring, by using at least one sensor and/or artificial intelligence tools, a scene which includes at least some of the different objects for the occurrence of a triggering and/or recognition event, [0098] updating and/or supplementing dynamically in at least one of the local database and the master database the color space positions and/or the reflectance spectra and/or luminescence spectra by continuously monitoring the scene for the occurrence of a triggering and/or recognition event and, thus, tracking small and continuous changes in the scene in the respective database.

[0099] Such method steps can be executed by the processor when an embodiment of the proposed non-transitory computer-readable medium is used/loaded which comprises the instructions as shown in FIG. 2.

[0100] A triggering and/or recognition event can be a new object entering the scene and, thus, provoking/initiating the measuring of a new reflectance spectrum and/or luminescence spectrum within the scene. A further triggering and/or recognition event can be given by receiving newly measured color space positions and/or reflectance spectra and/or luminescence spectra of the objects which have already been present in the scene but which have degraded over time.

[0101] In a step 101 a reflectance spectrum and a fluorescence spectrum are sensed/measured by an object recognition system used locally for recognizing objects in a scene. The object recognition system provides, for example, a specific fluorescence spectrum for an object which is to be recognized/identified. Therefore, the local database storing the fluorescence spectra of all objects which have up to now been identified in the scene, is searched for a matching fluorescence spectrum. In the case a match is found in a method step 102, it is further examined whether the spectrum found in the local database needs to be updated because the identified fluorescence spectrum deviates from the stored fluorescence spectrum, but still meets a confidence threshold to enable an identification on the basis of the measured fluorescence spectrum. Generally, to implement the local database, confidence thresholds and error thresholds are required. For example, a match between a fluorescence spectrum observed in the scene and a fluorescence spectrum in the local database must meet the confidence threshold to enable an identification. However, there may still be some error between the observed and assigned fluorescence spectrum. If this error is greater than the error threshold as indicated by arrow 103, then the stored fluorescence spectrum in the local database is updated in step 104. If it is stated in step 105 that the observed fluorescence spectrum and the fluorescence spectrum stored in the local database meet the error threshold, the object is identified in a step 106 without updating the local database. If there is no matching result found in the local database for the measured fluorescence spectrum, in a step 107, the master database is searched in step 108 for a fluorescence spectrum matching the sensed/measured fluorescence spectrum. If a match is found in the master database in step 109, the object can be identified in a step 110 and the matching fluorescence spectrum of the identified object is added together with its assigned object to the local database, indicating that the respective object is currently located in the scene and, thus, the local database which can be assigned to the respective scene is updated accordingly. If no match can be found in a step 111 in the master database, it is to be stated in step 112 that no match can be detected and no object can be recognized.

[0102] It is further possible to output via an output unit, such as a display, a selection of possible objects and to ask a user to select via a user interface, such as a touch screen, from such selection of possible object identifications, either in the local database or in the master database, and to use the user feedback to improve an accuracy of future identifications within the local database. That means that the objection recognition system can also be trained dynamically by the user feedback, thus improving the prediction dynamically. It is also possible to ask the user via a communication interface if an identification is correct and to use the feedback to improve future identifications within the local database. Additionally, if no match can be found neither in the local database nor in the master database, the object has to be identified manually by a user and its newly measured fluorescence spectrum can then be stored together with the respective object in both, the local database and the master database. Not only a user but also another automated system can “initiate” such an object by adding it to the local database when it is first acquired. Similarly, an object may be “retired” by removing it from the local database (and also from the master database if needed) when it is disposed of at the end of its useful life.

[0103] The object recognition procedure has been described using the example of a fluorescence spectrum of a specific object, the same procedure can be performed using a reflectance spectrum and/or color coordinates of the object to be recognized providing that the respective ground truth databases comprise reflectance spectra and/or color coordinates of objects.

[0104] Generally, an object recognition system can operate by using distinctive fluorescence emission and reflective spectrums as a method of object identification. This necessitates having a database of known or measured fluorescence spectra and/or reflectance spectra that the unknown object is compared to and selecting a best match from the respective database. The present disclosure considers that many fluorescent and/or reflective materials used for object recognition degrade over time with exposure to light or oxygen. Most of these materials have their fluorescence emission reduced in magnitude, but some may undergo changes in their fluorescence emission spectral shapes, i.e. in their fluorescence spectra. The present disclosure proposes now to include a local database in conjunction with a master database. A new object entering a scene would initially be classified with the master database on the assumption that the object has a non-degraded reflectance spectrum and/or luminescence spectrum. Once detected, the object can be included in the local database for quicker identification in the future. The local database is only accessible by the object recognition system locally used in the respective scene. Additionally, the fluorescence spectra and the reflectance spectra of the object measured by the object recognition system can be updated over time, so that small and continuous changes of the object are tracked in the local database. At the end of an object's useful life, it may be identified correctly by the local database despite its current emission spectra better matching another object's original emission spectra in the master database. Confidence thresholds and error thresholds are defined. The match between a spectrum observed in the scene and the spectrum in the local database must meet the confidence threshold to enable an identification. However, due to the possible degradation of the underlying fluorescent and reflective material over time, there may still be some error between the observed and assigned reflectance spectrum and/or fluorescence spectrum. If this error is greater than the error threshold, then the respective spectrum of the object in the local database may need to be updated, thus checking continuously small changes of the object in the local database. This makes it possible to identify an object although it's fluorescent and/or reflective material has changed over time. If no match can be found, it is possible to provide a user via a communication interface with a selection of possible object identifications either in the local database or master database whose spectra are beyond the confidence threshold but still within a possible identification area and to ask the user to select from such provided selection and to use such user feedback to improve the accuracy of future identifications within the local database. Alternatively, the user can also be asked if an identification is correct and to use such feedback also to improve future identifications within the local database. For initiating such a user interaction the proposed device provides a user interface, i. e. a communication interface for that the user can make some inputs. Such user interface is directly connected with the processor and via the processor also with the respective databases. The user interface can also be realized by a standing-alone computing device providing the input device for a user. All suitable known technologies are possible.