A METHOD AND A SERVER FOR FACILITATING PROVISION OF FOOD PRODUCT INFORMATION

Abstract

A method for facilitating provision of food product information is provided. The method comprises providing a food package, wherein the food package is provided with a visual marker, capturing, by a camera of a user device, authentic image data depicting the visual marker, extracting marker data from the authentic image data, retrieving artificial image data linked to the marker data from an artificial image database, wherein the artificial image data depicts at least one food product component of a food product held in the food package, determining an artificial image spatial reference point based on the visual marker, generating augmented image data by combining the authentic image data, the artificial image data and the artificial image spatial reference point, providing the augmented image data to a screen of the user device.

Claims

1. A method for facilitating provision of food product information, comprising: providing a food package, wherein the food package is provided with a visual marker, capturing by a camera of a user device, authentic image data depicting the visual marker, extracting marker data from the authentic image data, retrieving artificial image data linked to the marker data from an artificial image database, wherein the artificial image data depicts at least one food product component of a food product held in the food package, determining an artificial image spatial reference point based on the visual marker, generating augmented image data by combining the authentic image data, the artificial image data and the artificial image spatial reference point, and providing the augmented image data to a screen of the user device.

2. The method according to claim 1, further comprising: extracting spatial measurement data from the authentic image data, wherein the spatial measurement data is linked to measurements of the food package and/or measurements of the visual marker, and generating calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the food package and/or linked to the visual marker, provided by a spatial reference database, wherein the augmented image data is generated by combining the authentic image data, the calibrated artificial image data and the artificial image spatial reference point.

3. The method according to claim 2, wherein providing a food package further comprises providing an additional food package, wherein the additional food package is provided with an additional visual marker, wherein the authentic image data depicts the visual marker and the additional visual marker, wherein the marker data is based on the visual marker and the additional visual marker, wherein the artificial image data depicts the at least one food product component of the food product held in the food package and at least one additional food product component of an additional food product held in the additional food package, and wherein the artificial image spatial reference point is based on the visual marker and the additional visual marker.

4. The method according to claim 3, wherein the spatial measurement data is linked to measurements of the food package, measurements of the additional food package, measurements of the visual marker and/or measurements of the additional visual marker, and wherein the spatial reference data is linked to the food package, the additional food package, the visual marker and/or the additional visual marker.

5. The method according to claim 2, further comprising: providing a physical container, such as a bowl, wherein the authentic image data further depicts the physical container, determining geometric data for the physical container in the authentic image data, generating adjusted artificial image data by adjusting the artificial image data based on the geometric data for the physical container, and generating the augmented image data by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

6. The method according to claim 5, wherein the geometric data is determined by comparing the spatial reference data linked to the food package and/or linked to the visual marker, with the physical container.

7. The method according to claim 1, wherein the visual marker is provided on a side of the food package, together with a two-dimensional code, such as a QR-code, wherein the two-dimensional code is configured to launch an application on the user device.

8. The method according to claim 7, wherein the two-dimensional code comprises a unique identifier of the food package.

9. The method according to claim 1, wherein the visual marker is comprised in a décor of the food package.

10. The method according to claim 1, wherein the artificial image data depicts the food product in a first form and the food product is held in the food package in a second form, wherein a first volume of the food product in the first form is different from a second volume of the food product in the second form.

11. The method according to claim 1, wherein the artificial image data depends on a geographical position of the user device.

12. The method according to claim 1, wherein the artificial image data depends on a personal profile of a user, wherein the personal profile comprises user set data for different food product components.

13. A server configured for facilitating provision of food product information, the server comprising: a transceiver, a controller, and a memory, wherein the transceiver is configured to: receive marker data, and transmit augmented image data to a screen of a user device, and wherein the controller is configured to execute instructions stored in the memory, the instructions comprising: a retrieving function configured to retrieve artificial image data linked to the marker data from an artificial image database, wherein the artificial image data depicts at least one food product component of a food product held in a food package provided with a visual marker, a determining function configured to determine an artificial image spatial reference point based on the visual marker, and a first generating function configured to generate augmented image data by combining authentic image data depicting the visual marker, the artificial image data and the artificial image spatial reference point.

14. The server according to claim 13, wherein the controller is configured to execute further instructions stored in the memory and comprising: an extracting function configured to extract spatial measurement data from the authentic image data, wherein the spatial measurement data is linked to measurements of the visual marker, and a second generating function configured to generate calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the visual marker, provided by a spatial reference database.

15. The server according to claim 13, wherein the authentic image data further depicts a physical container, and wherein the controller is configured to execute further instructions stored in the memory and comprising: a second determining function configured to determine geometric data for the physical container in the authentic image data, a third generating function configured to generate adjusted artificial image data by adjusting the artificial image data based on the geometric data for the physical container, and a fourth generating function configured to generate the augmented image data by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Embodiments of the invention will now be described, by way of example, with reference to the accompanying schematic drawings, in which

[0038] FIG. 1 is a schematic view illustrating a system for facilitating provision of food product information.

[0039] FIG. 2A illustrates an example of a scene with authentic image data.

[0040] FIGS. 2B and 2C illustrates two examples of scenes with augmented image data.

[0041] FIG. 3 is a flowchart illustrating the steps of a method for facilitating provision of food product information.

[0042] FIG. 4A to 4C illustrates different ways a visual marker can be provided on a food package.

[0043] FIG. 5 illustrates a server configured for facilitating provision of food product information.

DETAILED DESCRIPTION

[0044] FIG. 1 is a schematic view illustrating a system 100 for facilitating provision of food product information by way of example. A food package 102 can be provided. The food package 102 can hold a food product. The food product can comprise one or more food product components. In this example also an additional food package 122 is provided with an additional visual marker 124. It may also be multiple additional food packages. In the case of multiple food packages, two or more food packages may hold the same food product. They may also hold different food products.

[0045] A physical container 126 can also be provided, as illustrated in this example. It can also be multiple physical containers provided. A physical container 126 can be a bowl, a pot, a glass, or any other physical container able to hold a food product. The physical container 126 may be different from the food package 102.

[0046] The food package 102 can be provided with a visual marker 104. It may also have a two-dimensional code 402, such as a QR-code. The visual marker 104 and two-dimensional code 402 is further described in regard to FIG. 4A to FIG. 4C.

[0047] The system 100 can further comprise a user device 108. The user device can be a mobile device such as a smart phone, or a tablet or any other electronic device fit for the purpose. It can also be smart glasses, wherein a screen is incorporated in a lens of the glasses.

[0048] The user device 108 can comprise a camera 106. The camera 106 can be provided as an external camera, connected to the user device 108 by suitable means.

[0049] The user device 108 can comprise a screen 110. Even though the screen 110 is illustrated as a part of the user device 108, it may be provided as an external screen and connected to the user device 108 by suitable means.

[0050] The camera 106 of the user device 108 can be used to capture authentic image data. The authentic image data can depict one or more visual markers 104 that are provided on the food package 102. The authentic image data can further depict one or more food packages. It can also depict one or more physical containers.

[0051] Marker data 112 can be extracted from the authentic image data by the user device 108. The marker data 112 can be used to retrieve artificial image data 114 linked to the marker data 112, from an artificial image database 116. The artificial image database 116 can be provided locally on the user device 108. It can also be provided externally to the user device 108. The artificial image data 114 can be illustrations of food product components that a food product held in the food package is comprised of. In other words, it can be illustrations of the contents of the food package 102. It can also be illustrations or videos related to the food package 102, such as a guide on how to open the food package. It can also be illustrations related to the producer, such as a mascot or logotype of the company.

[0052] An artificial image spatial reference point can be determined. It can be determined based on a location of the visual marker 104.

[0053] Augmented image data can be generated by combining the authentic image data, the artificial image data and the artificial image spatial reference point. The augmented image data can be interpreted as an image showing both real life objects, such as food packages or physical containers, and animated objects from the artificial image database. The artificial image spatial reference point can be used to align the authentic image data and the artificial image data. In a simpler version of the system 100, the step of determining an artificial image spatial reference point can be skipped. Instead, a pre-determined reference point, such as the center of the screen, can be used to know where to place the artificial image data.

[0054] The augmented image data may be provided to the screen 110 of the user device 108.

[0055] Optionally, a spatial reference database 120 can be provided. The spatial reference database 120 can provide spatial reference data 118 linked to a food package 102 and/or a visual marker 104. The spatial reference data can be geometrical dimensions of the food package or visual marker. To know which package type that is depicted, this information may be linked to the visual marker or other code provided on the package. By knowing which type of package, e.g. Tetra Brik Aseptic 1000 ml Slim, spatial reference data linked to the package being depicted can be retrieved. An advantage of using the package for spatial alignment instead of the visual marker is that the package is larger and as an effect different spatial reference points can be placed further apart, which may provide for improved spatial alignment of the artificial image data and the authentic image data.

[0056] Spatial measurement data can be extracted from the authentic image data. The spatial measurement data can be linked to measurements of a food package 102, as described above, and/or measurements of a visual marker 104. Calibrated artificial image data can be generated by comparing the spatial measurement data with the spatial reference data.

[0057] The augmented image data may be generated using the authentic image data, the calibrated artificial image data and the artificial image spatial reference point. In this way, the size of an artificial object may be calibrated to match the size of the food package 102 and/or the visual marker 104.

[0058] In the case of the physical container 126 being provided in the authentic image data, geometric data of the physical container 126 can be determined, for instance by comparing the physical container 126 to spatial reference data of the food package 102 and/or the visual marker 104.

[0059] FIG. 2A illustrates, by way of example, a scene based on authentic image data. In this example the food package 102 with a visual marker 104, and a physical container 126 is visualized.

[0060] FIG. 2B illustrates. By way of example, a scene based on augmented image data. It illustrates the food product held in the food package 102 but in the physical container 126 in a first form 202. It could also be illustrated in a second form 205 as in FIG. 2C, which shows the same food package 102 and the same food product but in a second form 204 with a different volume. The first and second form can be interpreted as a processed and un-processed form of the food product.

[0061] As an example, the food package 102 could be a package of cream. In FIG. 2B the cream is visualized as un-whipped cream. In FIG. 2C the cream is visualized as whipped cream. In this case, the first form 202 of the product has a smaller volume than the second form 204.

[0062] As another example, the food package 102 could be a package of orange juice. The first form can be un-processed oranges. The second form can be the processed form of orange juice. In this case, the first form of the product has a larger volume than the second form.

[0063] Even though not illustrated in any of FIG. 2A to FIG. 2C the augmented image data can depict food products or food product components of multiple food packages in the same scene. This could be used to illustrate a mix of two food products.

[0064] As another example, a food product can be illustrated by the food product components it's comprised of separately.

[0065] FIG. 3 is a flowchart illustrating the steps of a method 300 for facilitating provision of food product information.

[0066] In a first step 302, a food package can be provided, wherein the food package can be provided with a visual marker.

[0067] In a second step 304, authentic image data depicting the visual marker can be captured by a camera of a user device.

[0068] In a third step 308, marker data can be extracted from the authentic image data.

[0069] In a fourth step 310, artificial image data linked to the marker data can be retrieved from an artificial image database, wherein the artificial image data can depict at least one food product component of a food product held in the food package.

[0070] Optionally, the third and fourth step can be skipped by checking 306 if marker data already has been retrieved. In this case the method 300 can be repeated continuously, to get a real-time feed of augmented image data on the screen of the user device. In other words, the third step may need to be performed only once. In addition, the fourth step may need to be performed only once.

[0071] In a fifth step 312, an artificial image spatial reference point can be determined, based on the visual marker.

[0072] In a sixth step 314, augmented image data can be generated by combining the authentic image data, the artificial image data and the artificial image spatial reference point.

[0073] In a seventh step 316, the augmented image data can be provided to a screen of the user device.

[0074] Optionally, in an eighth step 318, spatial measurement data from the authentic image data can be extracted, wherein the spatial measurement data can be linked to measurements of the food package and/or measurements of the visual marker.

[0075] Optionally, in a ninth step 320, calibrated artificial image data can be generated by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the food package and/or linked to the visual marker, provided by a spatial reference database.

[0076] Optionally, in the sixth step 314, the augmented image data can be generated by combining the authentic image data, the calibrated artificial image data and the artificial image spatial reference point.

[0077] Optionally, in the first step 302, an additional food package can be provided 322, wherein the additional food package can be provided with an additional visual marker, [0078] wherein the authentic image data can depict the visual marker and the additional visual marker, [0079] wherein the marker data can be based on the visual marker and the additional visual marker, [0080] wherein the artificial image data can depict at least one food product component of a food product held in the food package and at least one additional food product component of an additional food product held in the additional food package, [0081] wherein the artificial image spatial reference point can be based on the visual marker and the additional visual marker.

[0082] The spatial measurement data can be linked to measurements of the food package, measurements of the additional food package, measurements of the visual marker and/or measurements of the additional visual marker. Further, the spatial reference data can be linked to the food package, the additional food package, the visual marker and/or the additional visual marker.

[0083] Optionally, in the first step 302, a physical container can be provided 324, wherein the authentic image data further depicts the physical container. Optionally, in a tenth step, geometric data for the physical container in the authentic image data can be determined. Optionally, in an eleventh step, adjusted artificial image data can be generated by adjusting the artificial image data based on the geometric data for the physical container. Optionally, in the sixth step 314, the augmented image data can be generated by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

[0084] Optionally, in the tenth step, the geometric data is determined by comparing the spatial reference data linked to the food package and/or linked to the visual marker, with the physical container.

[0085] FIG. 4A to FIG. 4C illustrates examples of ways the visual marker 104 can be provided on the food package 102. The visual marker 104 can be provided together with the two-dimensional code 402, such as a QR code, or separately.

[0086] FIG. 4A illustrates an example where the visual marker 104 is provided next to the two dimensional code 402 on one side of the food package 102. They can be placed on either side of the food package 102.

[0087] FIG. 4B illustrates an example where the visual marker 104 is provided within the two-dimensional code 402.

[0088] FIG. 4C illustrates an example where the visual marker 104 is comprised in a décor 404 of the food package 102. In other words, the visual marker 104 can be, for instance, a logo on the package that is already printed on the food package 104.

[0089] FIG. 5 illustrates a schematic view of a server 500. The server 500 is configured for facilitating provision of food product information. The server 500 comprises a transceiver 502, a control unit 504 and a memory 508.

[0090] The transceiver 502 is configured to communicate with other devices. The transceiver 502 can be configured to receive marker data, and transmit augmented image data to a screen of a user device.

[0091] The control circuit 504 can be configured to perform the control of functions and operations of the server 500. The control circuit 504 can include a processor 506, such as a central processing unit (CPU). The processor 506 can be configured to execute program code stored in the memory 508, in order to perform functions and operations of the server 500.

[0092] The control circuit 504 can execute a retrieving function 510 configured to retrieve artificial image data linked to the marker data from an artificial image database, wherein the artificial image data can depict at least one food product component of a food product held in the food package.

[0093] The control circuit 504 can execute a determining function 512 configured to determine an artificial image spatial reference point based on the visual marker.

[0094] The control circuit 504 can execute a first generating function 514 configured to generate augmented image data by combining the authentic image data, the artificial image data and the artificial image spatial reference point.

[0095] Optionally, the control circuit 504 can execute an extracting function 516 configured to extract spatial measurement data from the authentic image data, wherein the spatial measurement data is linked to measurements of the visual marker and/or the package depicted in the authentic image data.

[0096] Optionally, the control circuit 504 can execute a second generating function 518 configured to generate calibrated artificial image data by performing a spatial calibration of the artificial image data by comparing the spatial measurement data with spatial reference data linked to the visual marker and/or the package, provided by a spatial reference database.

[0097] Optionally, the control circuit 504 can execute a second determining function 520 configured to determine geometric data for a physical container in the authentic image data.

[0098] Optionally, the control circuit 504 can execute a third generating function 522 configured to generate adjusted artificial image data by adjusting the artificial image data based on the geometric data for the physical container.

[0099] Optionally, the control circuit 504 can execute a fourth generating function 524 configured to generate the augmented image data by combining the authentic image data, the adjusted artificial image data and the artificial image spatial reference point.

[0100] Optionally, the control circuit 504 can be configured to extract marker data from the authentic image data, by means of an second extraction function.

[0101] The server 500 may be provided within the user device.

[0102] From the description above follows that, although various embodiments of the invention have been described and shown, the invention is not restricted thereto, but may also be embodied in other ways within the scope of the subject-matter defined in the following claims.