Food Waste Detection Method and System

Abstract

A system (1) for detecting food related products (2) before thrown away, the system comprising: one or more cameras (11); a display unit (12); a computing device (13) that is communicatively connected to the cameras and the display; and a scale (3) that is communicatively connected to the computing device, the scale holding a trash bin (31), wherein the cameras obtain an image or a video of the products when the products are within a field of view of the cameras and before the products are in the trash bin, the scale configured to weigh the products in the trash bin, and wherein the computing device obtains information about the products from the obtained image or video by applying an image recognition algorithm, receives the weight from the scale and generates and outputs data on the display unit, the data being based on the information about products and the weight.

Claims

1. A system for detecting food related products before being thrown away, the system comprising: one or more cameras; a display unit; a computing device that is communicatively connected to the one or more cameras and the display unit; and a scale that is communicatively connected to the computing device, wherein the scale is configured to hold a trash bin, wherein the one or more cameras are configured to obtain an image or a video of the food related products when the food related products are within a field of view of the one or more cameras and before the food related products are in the trash bin, wherein the scale is configured to obtain weight information of the food related products when the food related products are in the trash bin, and wherein the computing device is configured to: obtain information about the food related products from the obtained image or video by applying an image recognition algorithm; receive the weight information from the scale; and generate and output data on the display unit, wherein the data is based on the information about the food related products and the weight information.

2. The system according to claim 1, wherein the computing device is communicatively connected to a remote server, and wherein the computing device is configured to: transmit the obtained image or video to the remote server for applying the image recognition algorithm; and receive the information about the food related products from the remote server.

3. The system according to claim 2, wherein the computing device is further configured to store one or more of the information about the food related products, the weight information, the output data, and at time stamp in a data storage of the remote server.

4. The system according to claim 1, wherein the computing device is configured to present one of more questions on the display unit about one or more objects in the obtained image or video in case the image recognition algorithm is unable to identify one or more of the food related products from the image or the video, and wherein the display unit comprises a user interface for receiving user input in response to the one or more questions, the response for use by the image recognition algorithm to improve detection of the one or more objects.

5. The system according to claim 1, wherein the one or more cameras are configured to automatically obtain the image or the video when the food related products are within the field of view at a substantially fixed position for a dynamic minimal amount of time necessary for successful detection.

6. The system according to claim 1, wherein the output data comprises a ratio of different food related products, wherein the different food related products are detected by the image recognition algorithm, and wherein the ratio is based on the weight information and the detected different food related products.

7. The system according to claim 1, wherein the one or more cameras comprises a stereoscopic imaging camera for obtaining 3D information about the food related products from the image or the video.

8. The system according to claim 7, wherein the image recognition algorithm is configured to obtain volumetric information from the 3D information, wherein the computing device is configured to obtain a weight estimation of the food related products based on the volumetric information, wherein the stereoscopic camera replaces the scale, and wherein the weight estimation is used instead of the weight information.

9. The system according to claim 1, wherein the one or more cameras comprises a hyperspectral imaging camera for obtaining substance information about the food related products from the image or the video.

10. The system according to claim 1, further comprising a depth sensor for detecting when the food related products are within the field of view of the one or more cameras.

11. The system according to claim 1, wherein the field of view is located in an area around a line of sight from the one or more cameras in a substantially downwards direction.

12. The system according to claim 11, further comprising a housing, wherein the housing comprises the display unit, wherein the housing accommodates the one or more cameras, wherein the housing comprises an outer surface side that is placed at an angle from a horizontal plane, and wherein the cameras are located within the housing at the outer surface side resulting in the line of sight being vertically angled at the angle, the line of sight being perpendicular to the outer surface side, wherein the angle is in a range of 15 to 45 degrees.

13. The system according to claim 12, wherein the housing further comprises the computing device.

14. The system according to claim 12, wherein the housing further comprises a visual indicator indicating where the food related products are to be presented to the one or more cameras.

15. The system according to claim 14, wherein the visual indicator changed its color when the food related products have been registered by the one or more cameras.

16-19. (canceled)

20. The system according to claim 12, wherein the housing and the scale are connected by a vertically aligned support structure for fixing the housing at a vertical distance from the scale.

21. A housing comprising a display unit, the housing further comprising one or more cameras and a computing device, wherein the housing is configured for use in the system according to claim 12.

22. A method for detecting food related products before being thrown away, the method comprising: obtaining an image or a video of the food related products using one or more cameras when the food related products are within a field of view of the one or more cameras and before the food related products are thrown in a trash bin; obtaining weight information of the food related products using a scale when the food related products are in the trash bin, wherein the scale is configured to hold the trash bin; obtaining information in a computing device about the food related products from the obtained image or video by applying an image recognition algorithm; generating and outputting data by the computing device on the display unit, wherein the data is based on the information about the food related products and the weight information.

23. The method according to claim 22, further comprising: transmitting the obtained image or video from the computing device to the remote server for applying the image recognition algorithm; and receiving the information about the food related products from the remote server in the computing device.

24. The method according to claim 22, further comprising: presenting one of more questions on the display unit about one or more objects in the obtained image or video in case the image recognition algorithm is unable to identify one or more of the food related products from the image or the video; and receiving user input from a user interface of the display unit in response to the one or more questions, the response for use by the image recognition algorithm to improve detection of the one or more objects.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0029] Embodiments will now be described, by way of example only, with reference to the accompanying schematic drawings in which corresponding reference symbols indicate corresponding parts, and in which:

[0030] FIG. 1 shows a system of an exemplary embodiment of the invention;

[0031] FIG. 2 shows a schematic side view of a housing and camera of an exemplary embodiment of the invention;

[0032] FIG. 3 shows an elevated side view of a display unit and camera in a housing of an exemplary embodiment of the invention;

[0033] FIG. 4 shows a block diagram of steps of a method of an exemplary embodiment of the invention;

[0034] FIG. 5 shows another exemplary embodiment of a display unit in a housing;

[0035] FIG. 6A shows an elevated side view of an exemplary scale; and

[0036] FIG. 6B shows a side view of an exemplary scale.

[0037] The figures are meant for illustrative purposes only, and do not serve as restriction of the scope or the protection as laid down by the claims.

DESCRIPTION OF EMBODIMENTS

[0038] FIG. 1 shows an exemplary system 1 for detecting food related products 2 before being thrown away in a trash bin 31. The system 1 preferably includes a housing 100 that includes one or more cameras 11, a display unit 12, a computing unit 13 and a communications module 14. Although less preferred, it is possible to have the one or more cameras 11 separated from the housing 100 and/or have the computing device 13 separated from the housing 100. The system 1 may include a scale 3 that is configured for holding a trash bin 31 and weighing the food related products 2 when placed in the trash bin 31. The scale 3 may be integrated within the trash bin 31. Preferably, the trash bin 31 is a recycle bin allowing the food related products 2 to be recycled after being thrown away. The scale 3 may include a communications module 32 for communicating with the computing device 13, typically via the communications module 14.

[0039] The display unit 12 is typically capable of presenting full color bitmap images representative of food detected amongst the food related products 2. The display unit 12 may be configured to display a graphical user interface, for example in the form of selectable button objects or any other user interface elements selectable through a touch screen interface of the display unit 12. The computing device 13 may be any suitable CPU, GPU and/or NPU based computer, for example in the form of a Raspberry Pi™ computer. Preferably the computing device 13 is a small form factor or single board computer to minimize the size requirements of the computing device 13. The communications module 14 may be integrated with the computing device 13. The communications module 14 may be any suitable wireless or wired communications module. The communications module 14 may include multiple different communication interfaces, for example a Bluetooth™ interface for short range communication with the communications module 32 of the scale 3 and a Wi-Fi or LAN interface for communication with a remote server 4. In the example of a Wi-Fi or LAN interface, the communication typically further involves a router (not shown) for connecting to the Internet 5, a local area network or any other suitable network.

[0040] In an exemplary embodiment the housing 100 and the scale 3 may be connected by a pole or other vertically aligned support structure for fixing the housing 100 at a vertical distance from the scale 3. This allows the housing 100 and the scale 3 to be moved around or placed at a desired location as a single unit. The vertically aligned support structure may be used to guide or accommodate electrical cabling and/or data cables for electrical or data/signal connections between the scale 3 and components in the housing 100.

[0041] The remote server 4 typically includes a data storage 41 and a communications module 42. The remote server 4 may be a stand-alone server, implemented as cloud computing server, implemented as a cloud computing service, or any other computer system. The data network 5 may be a local area network, a wide area network, the Internet, or any other suitable network.

[0042] FIG. 2 shows an exemplary embodiment of a camera setup in a housing 100. In FIG. 2 a side view of the housing 100 including a camera 11 is shown. The housing 100 may be attached to a wall 6. In FIG. 2 the housing 100 has an outer surface side 102 at the underside of the housing 100. The outer surface side 102 may be placed under an angle α from a horizontal plane 103, thus the underside of the housing 100 may be placed under angle α. The camera 11 may be installed in the surface side 102. As the line of sight 101 from the camera is typically in a direction perpendicular to the surface side 102, the line of sight 101 may thus be vertically angled under the angle α. Particularly when the housing 100 is attached to the wall 6, the angled line of sight 101 may improve the field of view by eliminating covering a large part of the wall 6 where no food related products 2 can be presented under the camera 11. Furthermore, the angled line of sight 101 enables a slight view of the products from the side, thereby improving the detectability of the food related products 2.

[0043] FIG. 3 shows an exemplary display unit 12 that may be installed in a housing 100. FIG. 3 further shows a camera 11 that may be installed in the housing 100. In the example of FIG. 3 the underside of the housing where the camera 11 is installed, is placed under an angle to allow the line of sight of camera 11 to be vertically angled, as explained with FIG. 2.

[0044] In an embodiment, food related products—preferably everything—that end up in the trash bin 31 may be first captured by a smart camera 11 above the trash bin 31 and then captured by a digital scale 3 located underneath the bin 31. When something is moved within the field of view of the camera, the camera may automatically take a picture or shoot a video as soon as it detects that an object is fully within view and is kept stable (stopped moving) at a fixed location within the field of view. The object may include a plate or a container onto which the food related products 2 are located. The object may be a human hand holding the food related products 2. The user can be provided audiovisual feedback upon successful ingredient detection. The captured image may be sent to the cloud 4, where an ingredient detection may be performed by an image recognition algorithm. Alternatively, the image recognition algorithm may be performed locally, for example in the computing device 13. The model used by the image recognition algorithm may detect/recognized one or more of the food related products 2 on the image and may send the results back to computing device 13 for local feedback on the display unit 12. When the waste 2 is thrown into the bin 31, the digital scale 3 may capture the weight and send this information to the computing device 13 and/or the cloud 4. The weight and image processing result may be linked to each other in the ratio at which the ingredients were recognized and the thus obtained results may be sent back to the computing device where the results may be displayed on the display unit 12. The results may also be stored in a data storage 41, such as a cloud database. Preferably, the results are stored together with a time stamp or any other indication of a date and/or time. This data may then be used at any point in time to generate dashboards that show the actionable results.

[0045] The possible items that are captured in the eye of view of the camera are limited and therefore relatively easy to detect by waste stream. We currently focus on food waste and have identified four different types of food waste streams in a restaurant. However, in practice we see that not all restaurants split their waste, resulting in that the tool also detects other types of waste, for example: plastics, paper, cardboard, cutlery. This gives our detection model an opportunity that goes beyond solely food waste.

[0046] Different types of food waste streams may be captured by the system 1. Examples hereof are: (i) expired products from an inventory; (ii) processed kitchen waste, which may be detectably presented to the camera 11 on metal containers, in bins or pans or big plastic containers; (iii) cutting waste, which may be detectably presented to the camera 11 on cutting plates or bigger plastic or metal bins; and (iv) waste in a restaurant, which may be detectably presented to the camera 11 on smaller plates or bowls (e.g. porcelain).

[0047] The state or condition of the food related products 2 may be detected in the image as well and may be used in the analysis of the waste stream. For example, in a restaurant environment, the following detection criteria may be used: (i) for expired products from an inventory, product may be untouched and therefore more easy to detect; (ii) processed food from a kitchen may be finely chopped, mashed or may include liquid side dishes; (iii) cutting waste from the kitchen may include inedible peals and bones; and (iv) plate waste from the restaurant may include left-overs from plate or parts of products that may be more difficult to detect because it is partly eaten and mixed. The system 1 may be capable of detecting the state of condition to further improve the detection of the food related products 2 and/or to generate recommendations about minimizing food waste.

[0048] In an exemplary embodiment the camera 11 may be placed approximately 50 cm above the bin 31 or another base platform. In another exemplary embodiment the camera 11 may be placed approximately 70 cm above the bin 31 or another base platform. Registration of the food related products 2 may take place between this platform and the camera 11, for example at 40 cm under the camera 11, which may be detected by a depth sensor. The depth sensor may thus be used to trigger the camera 11 to start the registration of the food related products 2. As shown in FIG. 3, the camera 11 may be placed at a slight angle α to the surface 102 on which the camera 11 may be mounted. For example, with reference to FIG. 2, the angle α may be any angle between 15 degrees to 45 degrees from a horizontal plane 103 to enable the camera 11 to have a better perspective, both in taking an image that is unobstructed by the plane upon which the machine is mounted and to get a slight view of the products from the side.

[0049] In an embodiment, the camera 11 may be completely detached from the display unit 12 so as to make the system more suited when space is a limitation. This may also enable the camera 11 to be placed so as to provide an optimal perspective of the food waste that will be registered.

[0050] For the detection of ingredients in the food related products 2, classification of dishes and waste stream origin (e.g. kitchen, restaurant) computer vision technology may be used. At the core of such computer vision technology are neural networks and deep learning. This is a so called semi-supervised machine learning approach. The terminology “supervised” means that the image recognition algorithm is typically trained to incrementally become better at the tasks it should perform (the detection). The training is typically done by giving the computer a lot of examples in which a human—through a user interface such as a graphical user interface—has manually labeled the ingredients, the type of dish and type of waste stream.

[0051] There are different types of image recognition strategies that may be applied. Most used strategies are: (i) classification to classify and assign a label to an image as a whole; (ii) detection to detect and label possibly multiple objects within an image; and/or (iii) segmentation, which is a fine-grained approach where each individual pixel of an image may be assigned a label.

[0052] For a dish and waste stream classification, the first two types of image recognition strategies are most suitable, i.e. classification and detection. For ingredient detection, the third strategy is most suitable, i.e. the more powerful segmentation strategy. With the segmentation strategy, a per-pixel labeling may be used to compute the ratio as to which ingredients occur within the image. The ratio may be used to improve the weight estimate that may be assigned to each individual ingredient. The input images that may be used to train the ingredient detection may require a lot of detail, meaning that for each pixel or a group of pixels the name of the ingredient may be assigned.

[0053] Once trained, the model may be used to independently recognize ingredients in new images as captured by the camera 11 that are fed into the image recognition algorithm. Hence the term “semi”-supervised is applicable: as soon as the model is trained, it may be used to automatically recognize ingredients in images without any required manual actions.

[0054] Additional domain knowledge may be used—such as the physical position of the camera 11, housing 100 and/or scale 3 within a venue such as a restaurant, and/or the menu of the restaurant in question—to improve accuracy by limiting the scope in which the detection algorithm has to operate. For example, the physical position may be used to determine that only certain waste streams will be recorded by the particular system 1, and the menu may be used to limit the variety in ingredients the system 1 may encounter.

[0055] To improve quality and accuracy of the detection algorithms, the one or more cameras 11 may include a stereoscopic camera. A stereoscopic camera is capable of obtaining a 3D depth image that may provide more information on the volume of food that is thrown away and help improve the weight estimate. Compared to a single top-down camera, which may have a problem of occlusion where certain ingredients may be invisible to the camera when covered by other materials, the stereoscopic camera may use two slightly differently angled cameras to provide a better view. Computer vision techniques such as Siamese Neural Networks can use stereoscopic images as input and may be used to better detect the food related products 2 when presented to a stereoscopic camera.

[0056] A stereoscopic camera may be used to obtain volumetric information of the food related products 2. Together with the identification of the food itself, the volumetric information may provide an indication of the weight of the detected food. The stereoscopic camera may then be used instead of the scale 3, in which case the system 1 does not need to include the scale 3.

[0057] To improve quality and accuracy of the detection algorithms, the one or more cameras 11 may include a hyperspectral camera. A hyperspectral camera is capable of obtaining a spectrum per pixel resulting in a lot more information than a standard RGB camera. This information may be used to detect, for example, levels of fat, protein and sugar, and may simplify and improve quality of detection of ingredients.

[0058] The weight of what ends up in the bin 31 may be registered to the respective food waste picture, possibly to the ratio in which the ingredients are detected in the image. This process may be performed in a short time frame, e.g. within seconds, after the picture is taken and the image and weight data may be sent combinedly to the remote server 4.

[0059] In case the image recognition algorithm cannot determine/detect the food related products 2 from the image, the image may be sent to the remote server 4 or to another remote server or cloud for redetection within the most up-to-date detection model, which may result in a multiple-choice option of images being sent to the display device 12 as a feedback screen. The end user may then select one or multiple images in the feedback screen to identify the product(s). If (parts of) the image is not in the multiple choice, the user may be offered to provide further feedback, for example in the form of a selectable “explain” button to write down what product(s) was not detected. This feedback from the user may be directly added to the detection model of the image recognition algorithm.

[0060] Feedback from the end user may be provided in various manners. For example, the display device 12 may include a touch screen interface for providing the feedback. Alternatively or additionally, a speech recognition interface may be installed in the housing 100 allowing the end user to interact with the system. Alternatively or additionally, one or more buttons on the housing 100 may enable the end user to provide feedback to the system.

[0061] The system 1 may be used in various use cases. Non-limiting examples of use cases are: (i) disposal during cleanup by staff in a kitchen or restaurant; (ii) discard during cleanup by a guest in a kitchen or restaurant; (iii) assembly line detection where the food related products 2 will be presented to the one or more cameras 11 without human involvement, for example in self-service restaurants; (iv) detection of food related products in tray carts where multiple trays are collected before throwing away leftovers and trash and cleaning the trays and cutlery in self-service restaurants and health care institutions.

[0062] Typically, the scale 3 and trash bin 31 will be located underneath the camera 11, but it is possible to place the scale 3 and trash bin 31 at another location. Preferably, the scale 3 and trash bin 31 are located in a vicinity of the camera 11 to ease the handling of the waste from holding underneath the camera to throwing away the waste in the bin.

[0063] The system 1 may be used to stimulate waste reduction by providing performance data, possibly anonymously, between neighboring and/or peer restaurants for comparison.

[0064] The system may alternatively or additionally detect other types of waste, besides food related products 2. Examples hereof are plastics, paper, cardboard and cutlery.

[0065] The system 1 may be integrated with 3rd party vendors providing for example stock management solutions or point of sale solutions.

[0066] FIG. 4 shows an exemplary block diagram of steps that may be performed by parts of a system 1 as shown in FIG. 1. Comparing FIG. 4 with FIG. 1, the smart camera may be similar to the one or more cameras 11, the smart scale may be similar to the scale 3, the touchscreen terminal may be similar to the display unit 12, the cloud storage may be similar to the data storage 41. Starting from the smart camera, in FIG. 4 the camera may automatically take a picture of a container or plate that is in view of the camera or depth sensor. Direct intermediate image feedback may be transmitted to the touch screen terminal, where the obtained image may be presented as a full color bitmap image. The picture that has been taken may be input to a computer vision program, which may be running on a local computer device 13 and/or in a remote server 4 such as a cloud. The computer vision program may apply the image recognition algorithm to the image to obtain detection results. After the picture has been taken, the waste may be thrown in the bin. Starting from the smart scale, the scale may then detect a waste change of waste thrown in the bin. The waste change may be indicative of the weight of the last waste thrown in the bin. Direct intermediate weight feedback may be transmitted to the touch screen terminal, where the obtained weight information may be presented. The detection results and the weight information may be matched and a final registration feedback indicative of the waste that has been thrown away may be presented on the touch screen terminal. The detection results and weight information may additionally or alternatively be stored in the cloud storage, from where statistical information may be generated and actionable insights may be presented for example on a dashboard output on a website or in a dashboard application. The touchscreen terminal may be used to obtain re-enforcement learning feedback provided by the user, which may be fed into the computer vision program to improve the detectability of the waste.

[0067] FIG. 5 shows an exemplary display unit 12 in a housing 100, which is a non-limiting alternative to the display unit and housing shown in FIG. 3. In this example the camera 11 may be located at a higher location than the display 12. The housing 12 may further include one or more light sources 15 for illuminating the food related products 2 enabling better detection by the camera 11, The other two circles shown at the top of the housing indicate the possibility of having further sensors, cameras and/or further light sources. These further sensors, cameras and/or light sources may be different in functionality from the camera 11 and light source 15, e.g. a depth sensor or infrared light source.

[0068] The housing 100 may include a visual indicator 16 indicating the height at which the food related products are to be presented to the camera before throwing in the trash bin 31. This helps the system in letting the user place the products at an optimal position with respect to the camera to detect what will be thrown in the bin. The visual indicator 16 may be implemented in various manners, for example as an illuminated LED strip. When using a light emitting visual indicator, the color of the light may be changed to indicate a status. For example, the color of the visual indicator 16 may turn green when the products have been registered by the camera thus indicating that the products may be thrown into the bin.

[0069] The housing 100 may include an audible indicator 17 for providing feedback to the user, e.g. to indicate that the products have been registered by the camera thus indicating that the products may be thrown into the bin.

[0070] FIGS. 6A and 6B and show an exemplary scale 3, which may be used in the system of FIG. 1. The weighing scale 3 of FIGS. 6A and 6B is designed to be as low as physically possible to allow more space for the bin 31 and more space between the bin 31 and the camera 11. Moreover, the scale 3 of FIGS. 6A and 6B includes at least one—e.g. two —ramps 33, i.e. sloped side walls, to allow a trash bin to be rolling on and off the scale 3. The ramps 33 may form a single piece of material with the top part of the scale, i.e. be part of the moving part of the scale 3 when measuring weight.

[0071] One or more embodiments may be implemented as a computer program product for use with a computer system. The program(s) of the program product may define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. The computer-readable storage media may be non-transitory storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory devices within a computer such as CD-ROM disks readable by a CD-ROM drive, ROM chips or any type of solid-state non-volatile semiconductor memory) on which information may be permanently stored; and (ii) writable storage media, e.g., hard disk drive or any type of solid-state random-access semiconductor memory, flash memory, on which alterable information may be stored.