A METHOD FOR IDENTIFYING A STATE OF A FOOD PACKAGE BY USING A CAMERA, AND A DEVICE THEREOF

Abstract

A method for identifying a state of a food package by using a camera can include capturing image data depicting a section of the food package by using the camera. The section includes at least one package feature, The method can further include identifying a package feature sub-set of the image data depicting the at least one package feature provided in the section, and determining the state of the food package based on the package feature sub-set of the image data. The state is selected from a food holding state and an emptied state.

Claims

1. A method for identifying a state of a food package by using a camera, said method comprising: capturing image data depicting a section of the food package by using the camera, wherein the section comprises at least one package feature, identifying a package feature sub-set of the image data depicting the at least one package feature provided in the section, and determining the state of the food package based on the package feature sub-set of the image data, wherein the state is selected from a food holding state and an emptied state.

2. The method according to claim 1, wherein the section further comprises a printed code, said method further comprising: identifying an identification data sub-set of the image data depicting the printed code, and determining identification data based on the identification data sub-set of the image data.

3. The method according to claim 2, further comprising: determining position data by using a food package state identification device, transmitting the position data to a server, and storing the identification data, the state of the package and the position data on the server.

4. The method according to claim 3, further comprising: determining if the position data is within a recycling area.

5. The method according to claim 1, wherein the at least one package feature comprises an opening device, and the emptied state comprises a state that the opening device is in an opened state.

6. The method according to claim 1, wherein the at least one package feature comprises at least one weakening line, and the emptied state comprises a state that the package is in a flat-folded state.

7. The method according to claim 2, wherein the printed code is a unique two-dimensional code.

8. The method according to claim 4, wherein the food package state identification device is associated with a user identity, and the method further comprising: transmitting the user identity to the server, and storing the identification data, the state of the package and the user identity on the server.

9. The method according to claim 2, wherein capturing the image data depicting the section of the food package, identifying the identification data sub-set of the image data, determining the identification data based on the identification data sub-set, identifying the package feature sub-set of the image data depicting the opening device the at least one package feature, and determining the state of the food package based on the package feature sub-set of the image data are performed on a server.

10. The method according to claim 2, further comprising: transmitting the identification data to a server, and receiving template data related to the identification data from the server, wherein the template data is taken into account in determining the state of the food package.

11. A device configured to identify a state of a food package, said device comprising: a camera configured to capture image data depicting a section of the food package, wherein the section comprises at least one package feature, a control circuit configured to: identify a package feature sub-set of the image data depicting the at least one package feature, and determine the state of the food package based on the package feature sub-set of the image data, wherein the state is selected from a group consisting of a food holding state and an emptied state.

12. The device according to claim 11, wherein the section further comprises a printed code, wherein the control circuit is further configured to: identify an identification data sub-set of the image data depicting the printed code, and determine identification data based on the identification data sub-set of the image data.

13. The device according to claim 11, wherein the device comprises a camera equipped mobile device or a camera equipped bin.

14. A server configured to identify a state of a food package, said server comprising: a transceiver configured to receive image data depicting a section of the food package from a device, wherein the section comprises at least one package feature, and a control circuit configured to: identify a package feature sub-set of the image data depicting the at least one package feature, and determine the state of the food package based on the package feature sub-set of the image data, wherein the state is selected from a group consisting of a food holding state and an emptied state.

15. The server according to claim 14, wherein the section further comprises a printed code, and the control circuit is further configured to: identify an identification data sub-set of the image data depicting the printed code, and determine identification data based on the identification data sub-set of the image data.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Embodiments of the invention will now be described, by way of example, with reference to the accompanying schematic drawings, in which

[0029] FIG. 1A is an example of image data depicting a top section of a package with a cap in place.

[0030] FIG. 1B is an example of image data depicting the top section also illustrated in FIG. 1A, but with the cap removed.

[0031] FIG. 2 is another example of image data depicting a top section of a package, but unlike the package illustrated in FIGS. 1A and 1B, the package is provided with a straw hole instead of the cap.

[0032] FIG. 3A is still another example of image data depicting a top section of a package with a straw hole, but unlike the package illustrated in FIG. 2, a straw is placed in the straw hole.

[0033] FIG. 3B illustrates the package illustrated in FIG. 3A, but with the straw removed from the straw hole, thereby leaving a punctured straw hole coverage.

[0034] FIG. 4 illustrates a mobile device used for capturing image data depicting a package with a cap removed.

[0035] FIG. 5A illustrates another package in a flat-folded state.

[0036] FIG. 5B illustrates the package also illustrated in FIG. 5A, but with image data covering a different section than in FIG. 5A.

[0037] FIG. 6A schematically illustrates data exchange between the device and a server via a communications network according to a first set up example.

[0038] FIG. 6B schematically illustrates data exchange between the device and the server via the communications network according to a second set up example.

[0039] FIG. 7 is a flow chart illustrating steps of a method for identifying a state of a food package by using a camera.

[0040] FIG. 8 schematically illustrates the device configured to identify the state of the food package.

[0041] FIG. 9 schematically illustrates the server configured to identify the state of the food package.

[0042] FIG. 10 illustrates the device embodied as a bin equipped with the camera.

[0043] FIG. 11 schematically illustrates how the two states, food holding state and emptied state, can be related to different sub-states.

DETAILED DESCRIPTION

[0044] FIG. 1A illustrates by way of example a package 100 comprising a section 102, herein a top section. The section 102 can, as illustrated, be provided with a printed code 104, herein in the form of a two-dimensional code, e.g. a QR code. Further, an opening device 106, herein a spout with a cap, can be provided. In image data depicting the package 100, the printed code 104 may be part of an identification data sub-set 108 of the image data, and the opening device 106 may be part of a package feature sub-set 110 of the image data.

[0045] The printed code 104 may be used for identifying the package 100, that is, the printed code may comprise a unique code that provides for that the package can be identified, which in turn provides for that the package can be traced throughout the life cycle of the package 100. The printed code 104 can be printed in various ways, for instance by using digital printing technology or by using an inkjet printer. Further the printed code may be achieved by embossing or in any other way effecting the material such that a code can be provided. To identify the package 100, the first sub-set of the image data 108 may be input to an image data processing software configured to read and interpret the printed code 104. Further, as an additional alternative, the printed code 104 may be produced such that this is not visible to the human eye, but to the camera.

[0046] The opening device 106 can be opened or closed. By using the package feature sub-set 110 of the image data, the image processing software can determine whether or not the package 100 is in an opened state, as illustrated by way of example in FIG. 1A, or in a closed state as illustrated by way of example in FIG. 1B. As illustrated, when having the opening device 106 embodied as the spout and cap, the opened state may be defined as that the cap is placed on the spout, and the closed state may be defined as that the cap is removed from the spout.

[0047] By having the first sub-set 108 and the second sub-set 110 being part of the image data, that is, capturing information about the printed code 104 and the opening device 106 at the same time, it is possible to conclude the state of the package 100 and also, in case the identification data sub-set 108 is available, an identity of the package.

[0048] FIG. 2 illustrates another example of a package 200 comprising a section 202, more particularly a top section as in FIGS. 1A and 1B. Also in line with FIGS. 1A and 1B, the package 200 can comprise a printed code 204 and an opening device 206. In the image data depicting the package 200, an identification data sub-set 208 can comprise image data related to the printed code 204 and a package feature sub-set 210 can comprise image data related to the opening device 210 in line with FIGS. 1A and 1B.

[0049] However, unlike the package 100 illustrated in FIGS. 1A and 1B, the opening device 206 of the package 200 illustrated in FIG. 2 is a straw hole and not the spout and cap arrangement illustrated in FIGS. 1A and 1B. Even though the opening device 206 in this example is the straw hole 206, it is still possible for the image processing software to determine whether this is in the opened state or in the closed state, which in turn may be used for determining the state of the package 200. If correctly assessed by the image processing software, the package 200 illustrated in FIG. 2 is in the closed state, which implies that the straw hole comprises an unbroken cover, that is, no straw is identified to have been pushed through the straw hole.

[0050] FIGS. 3A and 3B illustrate yet another example of a package 300 with a section 302. In line with the packages 100, 200 illustrated in FIGS. 1A, 1B and 2, the package 300 can comprise a printed code 304 and an opening device 306. An identification subset 308 of image data can comprise the printed code 304 and a package features subset 310 of the image data can comprise the opening device 306.

[0051] As in FIG. 2, the opening device 306 comprises the straw hole. However, unlike the package 200 illustrated in FIG. 2, the opening device 306 is in the opened state. FIG. 3A illustrates a situation where the straw has been pushed through the cover of the straw hole and thereafter has been left in the straw hole. FIG. 3B illustrates a different situation, which is also an example of the opened state, in which the straw has been pushed through the cover, but not left in the straw hole, thereby leaving a broken cover of the straw hole.

[0052] FIG. 4 illustrates by way of example a device 400 embodied as a mobile device 400 that can be used for capturing image data depicting a package with a cap removed. The mobile device 400, which may be a mobile phone, such as a smart phone, a tablet or any other electronic device fit for the purpose, can comprise a camera 402. In the example illustrated in FIG. 4, the camera 402 is a rear camera, that is a camera placed on the rear side of the mobile device. As illustrated, the mobile device 400 may, on a front side, comprise a display 404 presenting image data captured by the camera 402.

[0053] FIG. 5A illustrates by way of example a flat-folded package 500. More particularly, the package 500 is an emptied carton package that is, when being in a food holding state, shaped as a parallelepiped. In this example a section 502 is covering a part of the flat folded package 500, more particularly part of a top section of the package 500 and part of side sections of the package. Further, the package comprises a printed code 504 and an opening device 506 in line with the examples provided in FIGS. 1A, 1B, 2, 3A and 3B.

[0054] An identification subset 508 of the image data depicting the package may comprise the printed code 304 in line with the examples presented above. A package feature subset 510 may however comprise not only the opening device 506, but also additional parts of the package 500 to cover additional package features. The opening device 506 may be one package feature that can be used for determining whether the package 500 is in an emptied state or in a food holding state. A top end line 512, a bottom end line 514, and a longitudinal sealing line 516 are examples of other package features that can be used for the same purpose. For instance, these lines can be compared with one another for determining, or identifying, the state of the package. Even though not illustrated a distance reference mark may be printed on the package 500 to enable the image processing software to reliably estimate lengths of the different lines.

[0055] In addition, a top transversal sealing area 518 can be used as one of the package features. In case this is placed in an uppermost part of the section 502, this may be an indication that the package 500 is flat folded. Similarly, a longitudinal sealing area 520 and a bottom transversal sealing area 522 can be used.

[0056] Weakening lines 524, 526, 528, 530, 532, 534, 536, 538, 540, 542, 544 (hereafter referred to as 524-544) can also form part of the package features. The weakening lines 524-544, also referred to as folding lines, may be areas of the package 500 in which packaging material has been compressed during production of the packaging material in order to facilitate folding of the package 500 in a filling machine. In an end of a life cycle of the package 500 these weakening lines 524-544 may also be used for identifying the state of the package, more particularly whether the package is in the food holding state or in the emptied state.

[0057] Although presented as different groups, the features can be used in combination to identify the state of the package. Further, for other types of packages, different features may be added or used.

[0058] FIG. 5B illustrates by way of example the package 500 as illustrated in FIG. 5A. However, instead of having the section 502 covering a part of the package 500 as illustrated in FIG. 5A, the section 502 covers the entire package 500. An advantage with having the package covering the entire package is that additional package features can be extracted and hence a more reliable identification of the state can be made.

[0059] FIG. 6A schematically illustrates a system 600 for data exchange between the device, herein exemplified as the mobile device 400, and a server 602 via a communications network 604 according to a first set up example.

[0060] In this example, image data depicting the package 100, 200, 300, 500 illustrated in any of the FIG. 1A, 1B, 2, 3A, 3B, 5A or 5B can be captured by the device 400, and based on the image data of the identification data section 108, 208, 308, 508, which is related to the printed code, identification data 608 can be determined, that is, an identity of the package 100, 200, 300, 500 can be established. The identification data 508 can be transmitted from the mobile device 400 to the communications network 604 and from there to the server 602.

[0061] To determine whether the package 100, 200, 300, 500 is in the food holding state, that is, not yet consumed, or in the emptied state, that is, consumed and ready to be recycled, template data 610 may be transmitted from the server 602 via the communications network 604 to the mobile device 400. Since the identity of the package is established it is namely possible to retrieve information about the package 100, 200, 300, 500 from the server 602 or a database connected to the server 602 (not illustrated) about which kind of opening device the package is equipped with, how the opening device is placed in relation to the printed code, how the weakening lines are provided, width and length of the transversal sealing section as well as the longitudinal sealing section, etc. Hence, by having the template data 610 and feeding this into the image processing software, a more reliable and a faster assessment of the state of the package can be made.

[0062] In addition, position data 612 may be transmitted from the mobile device 400 to the server 602, that is, a position where the image data was captured. The position data 612 may be of interest to increase understanding of how the food product held in the package is consumed, which in turn may result in more efficient logistics and distribution.

[0063] The position data 612 may also be used to detect counterfeit products. For instance, if the position data 612 is far away from a retailer to which the package was shipped, this may be an indication that the package, and hence also the product, is not legit, and therefore, as a consequence, it may have been made to unduly collect recycling remuneration.

[0064] Time data 614 associated with the capture of the image data may also be transmitted from the device 400 to the server 602 via the communications network 604. In a similar manner as the position data 612, the time data 614 may be used to increase the understanding of consumer behaviors and/or to indicate a risk that the package is not legit.

[0065] Optionally, as an effect of that the package 100, 200, 300, 500 is identified, package information 616 may be transmitted from the server 602, or a database connected to the server 602, via the communications network 604 to the mobile device 400. The package information 616 may comprise information on material of the package such that this can be placed in a correct bin.

[0066] Further, since the mobile device 400 and/or the software application used for capturing the image data may be linked to a user identity, the user identity 618 can be transmitted from the mobile device 400 to the server 602 via the communications network 604. Having access to the user identity 618 provides additional possibilities to increase the understanding of the consumer behavior (even though this in some jurisdictions may require an acceptance by the user). Further, by having access to the user identity 618, it is possible to provide user-specific package information 616, e.g. a total number of packages recycled in a certain period of time.

[0067] Even though presented in a certain order, it should be understood that the data exchange can be made in different orders as well and also, if not related to one another, exchanged simultaneously.

[0068] FIG. 6B illustrates data exchange between the mobile device 400 and the server via the communications network according to a second set up example.

[0069] Similar to the first set up example illustrated in FIG. 6A, the server 602 is communicatively connected to the mobile device 400 via the communications network 604. However, unlike the first set up example, in the second set up example image data 606 can be transmitted from the mobile device 400 to the server 602 via the communications network 604. In the server 602, based on the image data 606, identification data can determined. That is, instead of using the mobile device 400 for determining the identification data as in the first set up example illustrated in FIG. 6A, the mobile device 400 is solely used for capturing the image data 606 and transmitting this to the server 502 in which the identification data can be determined. Still an option, since determining the identification data may comprise a number of sub-steps is that part of a process for determining the identification data is made in the mobile device 400 and part of the process is made in the server 602.

[0070] In line with the first set up example illustrated in FIG. 6A, the position data 612 may be transmitted from the mobile device 400, which may be GPS equipped or NFC equipped, to the server 602. Further, also in line with the first set up example, the time data 614 may be transmitted from the mobile device 400 to the server 602. Still further and also in line with the first set up example, package information 616 may be transmitted from the server 602 via the communications network 604 to the mobile device 400. In addition, also in line with the first set up example, the user identity 618 may be transmitted from the mobile device 400 to the server 602 via the communications network 614.

[0071] FIG. 7 is a flow chart illustrating steps of a method for identifying consumption of the food product held in the food package 100, 200, 300, 500 by using the mobile device 400.

[0072] In a first step 702, image data 606 depicting a section 102, 202, 302, 502 of the food package can be captured by using a camera 402 of the mobile device 400. As presented above, the section 102, 202, 302 can comprise a printed code 104, 204, 304, 504 and an opening device 106, 206, 306, 506.

[0073] Optionally, in a second step 704, an identification data sub-set 108, 208, 308, 508 of the image data 606 depicting the printed code 104, 204, 304, 504 can be identified.

[0074] Optionally, in a third step 706, identification data 608 can be determined based on the identification data sub-set 108, 208, 308, 508 of the image data 606.

[0075] In a fourth step 708, a package feature sub-set 110, 210, 310, 510 of the image data 606 depicting the package features 106, 206, 306, 506, can be identified.

[0076] In a fifth step 710, a state of the package 100, 200, 300, 500 based on the package feature sub-set 110, 210, 310 of the image data 606 can be determined. The state may be selected from a group consisting of a food holding state and an emptied state.

[0077] Optionally, according to a sixth step 712, position data 612 can be determined by using the mobile device 400, and, according to a seventh step 714, the position data 612 may be transmitted to the server 602, and, according to an eighth step 716, the identification data 608, the state of the package and the position data 612 may be stored on the server 602.

[0078] Optionally, according to a ninth step 718, it can be determined if the position data 612 is within the recycling area or not. In case the mobile device 400 is not within the recycling area, that is, not close to the recycling bin, it may be decided that no recycling is taking place. As described above, the position data 612 may be determined by having the mobile device 400 GPS-enabled, but also in that the mobile device is NFS-enabled and that the recycling bin is provided with a NFC transponder.

[0079] Optionally, according to a tenth step 720, the user identity 618 can be transmitted to the server 602, and according to an eleventh step 722 the identification data 508, the state of the package and the user identity 618 can be stored on the server 602.

[0080] Optionally, in an twelfth step 724, the package information 616 related to the identification data 508 from the server 502 can be received.

[0081] The first step 702, the second step 704, the third step 706, the fourth step 708 and the fifth step 710 may be performed in the server 602.

[0082] Alternatively, in a thirteenth step 726, the identification data 508 can be transmitted to the server 502, and, according to a fourteenth step 728, the template data 510 related to the identification data 508 can be received from the server 502, wherein the template data 510 is taken into account in the step of determining 710 the state of the package, e.g. as explained above.

[0083] Even though described in a certain order, the different steps may also be performed in other orders.

[0084] FIG. 8 illustrates the mobile device 400 configured to identify the state of the food package 100, 200, 300, 500 according to a first example. As described and illustrated in FIG. 4, the mobile device 400 may be equipped with the camera 402.

[0085] For handling data, the mobile device 400 may comprise a control unit 804, such as a processor, a transceiver 806 for transmitting and receiving data, and a memory 808 for storing the data.

[0086] Optionally, the control circuit 804 can be configured to execute an identification data identification function 810 configured to identify an identification data sub-set 108, 208, 308, 508 of the image data 606 depicting the printed code 104, 204, 304, 504. Further, it may be configured to execute an identification data determination function 812 configured to determine identification data 608 based on the identification data sub-set 108, 208, 308, 508 of the image data 606.

[0087] The control circuit 804 may be configured to execute a package feature identification function 814 configured to identify a package feature sub-set 110, 210, 310, 510 of the image data depicting the package 100, 200, 300, 500. It may be configured to execute a state determination function 816 configured to determine a state of the package based on the package feature sub-set of the image data. The state may be selected from a group consisting of the food holding state and the emptied state.

[0088] The control circuit 804 may also be configured to execute a position determination function 818 configured to determine the position data 612 by using the mobile device 400, which may be GPS equipped or NFC-enabled, and the transceiver 806 may be configured to transmit the position data 612 to the server 602.

[0089] The control circuit 804 may further be configured to execute a user identity determination function 820 configured to determine the user identity 618, wherein the transceiver 806 can further be configured to transmit the user identity 618 to the server 602.

[0090] FIG. 9 illustrates the server 602 configured to identify consumption of the food product held in the food package 100, 200, 300 according to a second example.

[0091] The server 602 may comprise a transceiver 906 configured to receive the image data 606 depicting the section 102, 202, 302, 502 of the food package from the mobile device 400, wherein the section 102, 202, 302 can comprises at least one package feature 106, 206, 306, 506, 512-544, and optionally also the opening device 106, 206, 306, 506 and/or the printed code 104, 204, 304, 504.

[0092] Further, the server 602 may comprise a control circuit 904 configured to execute an identification data identification function 910 configured to identify the identification data sub-set 108, 208, 308, 508 of the image data 606 depicting the printed code 104, 204, 304, 504, an identification data determination function 912 configured to determine the identification data 608 based on the identification data sub-set 108, 208, 308, 508 of the image data 606, a package feature identification function 914 configured to identify the package feature sub-set 110, 210, 310, 510 of the image data 606 depicting the opening device 106, 206, 306, 506 and a state determination function 916 configured to determine the state of the opening device 106, 206, 306, 506 based on the second sub-set 110, 210, 310, 510 of the image data 606, wherein the state may be selected from the group consisting of the food holding state or the emptied state.

[0093] Further, the transceiver 906 may further be configured to receive the position data 612, and the control circuit 904 may further be configured to execute a position data area mapping function 918 configured to determine if the position data 612 is within the recycling area or not.

[0094] The control unit 904 may also be configured to execute a package information fetcher function 920 configured to fetch the package information related to the identification data 608. The transceiver 906 may further be configured to transmit the package information 616 to the mobile device 400.

[0095] The first example illustrated in FIG. 8 is linked to the mobile device 400 illustrated in FIG. 6A, that is, in this example the package 100, 200, 300, 500 is identified by the mobile device 400 and the identification data 508 is transmitted to the server 504. The second example illustrated in FIG. 9 is linked to the server 602 illustrated in FIG. 6B. In the example illustrated in FIG. 6B, the image data 606 can be captured by the mobile device 400, but the identification data can be extracted by the server 602.

[0096] As stated above, it may be determined whether the package is in the food holding state or the emptied state. The food holding state may be defined as a state in which the opening device of the package is closed and the package is folded such that a closed space is formed inside the package. The emptied state may be defined as a state in which the package is folded flat and optionally the opening device is opened.

[0097] Even though the device above is exemplified as the mobile device 400 above, the device may be embodied in different ways. For instance, the device may be a bin 1000 provided with a camera 1002 as illustrated in FIG. 10. As illustrated in the example presented in FIG. 10, the bin 1000 may be provided with an opening 1004 for receiving the package 100, 200, 300, 500. Above the opening 1004, the camera 1002 may be provided such the image data 606 can be captured in connection to placing the package in the bin 1000. Even though not illustrated, a display may be provided on the bin 1000 such that the package information and also information on remuneration linked to placing the package in the bin can be displayed. As an alternative, this information may be provided on the mobile phone of the user. The mobile phone and the bin may be linked to each other using NFC communication and/or position data.

[0098] FIG. 11 illustrates schematically the state 1100 of the package 100, 200, 300, 500. More particularly, it is illustrated how the food holding state 1102 and the emptied state 1104 can be linked to different sub-states. For instance, the food holding state 1102 may be linked to a closed state 1106 of the opening device 106, 206, 306, 506, that is, if the opening device is closed this may indicate that the package is in the food holding state 1102. The emptied state 1104 may be linked to the opened state 1108, that is, in case the opening device is opened, this may indicate that the package is in the emptied state 1104. Further, by using e.g. the weakening lines described above, a non-flat folded state 1110 may be identified. In case the package is determined to be in the non-flat folded state 1110, this may indicate that the package is in the food holding state 1102. However, in case the package is determined to a flat folded state 1112, this may be an indication that the package is in the emptied state 1104. The more indications pointing towards a certain state, the higher likelihood that that the package is in this certain state. The sub-states exemplified above may be complemented or replaced by other sub-states for identifying whether the package is in the food holding state or in the emptied state. Further, different types of packages may use different types of sub-states. The sub-states presented above are examples of sub-states that may be used for carton packages.

[0099] From the description above follows that, although various embodiments of the invention have been described and shown, the invention is not restricted thereto, but may also be embodied in other ways within the scope of the subject-matter defined in the following claims.