METHOD FOR CONTROLLING A HOME APPLIANCE

Abstract

A method for controlling a home appliance depending on a load includes taking a load image with an appliance camera, sending the image from the appliance to the server, processing the image on the server, and generating a processing result. Control data is determined on the server for controlling the appliance based on the result and is then sent to the appliance and/or the processing result is sent to the appliance enabling the appliance to determine data for controlling the appliance based on the processing result. The control data and/or the processing result is received from the server on the appliance and based thereon appliance working programs are controlled. Image processing generating the result includes upscaling the image using a trained generative adversarial network and analyzing the load using a trained neural network. A server, home appliance and system including both are also provided.

Claims

1-14. (canceled)

15. A method for controlling a home appliance in dependence on a load to be handled by the home appliance, the method comprising: taking a load image of the load with a camera disposed at the home appliance; sending the load image from the home appliance to a server; receiving the load image on the server; processing the load image on the server to generate a processing result by upscaling the load image using a trained generative adversarial network and analyzing the load using a trained neural network; based on the processing result, determining control data on the server to control the home appliance and sending the control data to the home appliance, or sending the processing result to the home appliance to enable the home appliance to determine control data to control the home appliance based on the processing result; receiving control data on the home appliance, the control data derived from the load image from the server, or receiving a processing result on the home appliance, the processing result derived from the load image from the server, and based on the processing result, determining control data on the home appliance; and controlling at least one working program of the home appliance based on the control data.

16. The method according to claim 15, which further comprises providing the trained neural network as a deep convolutional neural network or a deep convolutional semantic segmentation neural network.

17. The method according to claim 15, which further comprises providing the trained generative adversarial network as a super resolution generative adversarial network and using the upscaling to increase resolution of the load image by a factor of between two and eight.

18. The method according to claim 17, which further comprises increasing the resolution of the load image by a factor of between four and eight.

19. The method according to claim 17, which further comprises increasing the resolution of the load image by a factor of between four and six.

20. The method according to claim 15, which further comprises using the processing result to indicate at least one of: an amount of the load, types of the load in dependence on a location of the load within the home appliance, and a segmentation of the load according to the types of the load.

21. The method according to claim 15, which further comprises providing the processing result as a labeled segmented image of the load, and using labels to refer to types of the load.

22. The method according to claim 15, which further comprises providing a dishwasher as the home appliance, and providing at least one of glass, plastic, metal, ceramic and empty space as types of the load.

23. The method according to claim 22, which further comprises providing the control data as at least one of: a control command relating to at least one of a drying time, a rinse time or a spray pressure of a spray arm, when an amount of plastic within the load surpasses a plastic threshold, or a control command relating to at least one of a temperature, a spray pressure of a spray arm or a quantity of rinse-aid, when an amount of glass within the load surpasses a glass threshold, or a control command relating to instructing a spray arm to skip the empty space, when an amount of empty space within the load surpasses a gap threshold.

24. A server for controlling a home appliance in dependence on a load to be handled by the home appliance, the server comprising: a receiving module configured to receive a load image of the load from the home appliance; a processing module configured to process the load image to generate a processing result, said processing module including a trained generative adversarial network to upscale the load image and a trained neural network to analyze the load; and a determination module configured to determine control data to control the home appliance based on the processing result and a sending module configured to send the control data to the home appliance, or a sending module configure to send the processing result to the home appliance to enable the home appliance to determine control data to control the home appliance based on the processing result.

25. The server according to claim 24, wherein said processing module includes at least one of: a deep convolutional neural network or a deep convolutional semantic segmentation neural network as said trained neural network, or a super resolution generative adversarial network as said trained generative adversarial network, to upscale the load image by a factor between two and eight.

26. The server according to claim 25, wherein the factor is between four and eight.

27. The server according to claim 25, wherein the factor is between four and six.

28. A home appliance or dishwasher, comprising: a camera for taking a load image of a load, said camera being adapted to provide the load image with a resolution of less than 20,000 pixels, and the load image being configured to be scaled up on a server using a trained generative adversarial network; a sending module for sending the load image to the server; a receiving module for receiving control data derived from the load image from the server, or a receiving module for receiving a processing result derived from the load image from the server and a determination module for determining control data based on the processing result; and a control module for controlling at least one working program of the home appliance based on the control data.

29. The home appliance according to claim 28, wherein the resolution of the load image is less than 10,000 pixels.

30. The home appliance according to claim 24, which further comprises: at least one rack for holding the load; a front of the home appliance; said camera being disposed at said front of the home appliance; and said camera being configured to capture the load image of the load when said at least one rack is in a pulled-out position.

31. The home appliance according to claim 30, which further comprises a sensor for detecting the pulled-out position of said at least one rack.

32. A system for controlling a home appliance in dependence on a load to be handled by the home appliance, the system comprising: a server and a home appliance; said server including: a receiving module configured to receive a load image of the load from said home appliance; a processing module configured to process the load image to generate a processing result, said processing module including a trained generative adversarial network to upscale the load image and a trained neural network to analyze the load; a determination module configured to determine control data to control said home appliance based on the processing result and a sending module configured to send the control data to said home appliance, or a sending module configured to send the processing result to said home appliance to enable said home appliance to determine control data to control said home appliance based on the processing result; said home appliance including: a camera for taking the load image of the load; a sending module for sending the load image to said server; a receiving module for receiving control data derived from the load image from said server, or a receiving module for receiving a processing result derived from the load image from said server and a determination module for determining control data based on the processing result; and a control module for controlling at least one working program of said home appliance based on the control data.

33. A non-transitory computer program product, comprising instructions which, when executing the program on a computer, cause the computer to carry out the method according to claims 15.

Description

[0020] In the following, further details and relating advantages of embodiments of the invention will be explained with reference to the figures, which show

[0021] FIG. 1 an embodiment of a system according to the invention;

[0022] FIG. 2 a possible arrangement of a camera at an embodiment of a home appliance according to the invention;

[0023] FIG. 3 two exemplary load images taken by the camera according to FIG. 2;

[0024] FIG. 4 an example of a load image and a resulting labeled segmented image of the load;

[0025] FIG. 5 an example how the neural network may be trained;

[0026] FIG. 6 a schematic illustration of an embodiment of a GAN;

[0027] FIG. 7 an example how a GAN may be trained;

[0028] FIG. 8 an exemplary combination of a GAN with a neural network;

[0029] FIG. 9 a first embodiment of a method according to the invention;

[0030] FIG. 10 a second embodiment of a method according to the invention;

[0031] FIG. 11 a first embodiment of a home appliance and a server according to the invention;

[0032] FIG. 12 a second embodiment of a home appliance and a server according to the invention; and

[0033] FIG. 13 an exemplary recognition accuracy of an embodiment of a server according to the invention.

[0034] In the following, similar features and features having similar functions will be referred to with the same reference sign, if not stated otherwise.

[0035] FIG. 1 illustrates an embodiment of a system according to the invention. On the left side an embodiment of a home appliance 1, in the form of a dishwasher, is shown with a lower rack 2 and an upper rack 3. A camera 4 is able to take load images of each of the racks. It is connected to a CPU 5, which is coupled to a communication module 6. The load images are sent to a Wifi hotspot 7 in the home, which forwards them via the Internet 8 to a server 9, which is implemented as a cloud service. The server 9 analyzes the load images and sends the results back to the home appliance 1, which controls its behavior based on the results.

[0036] FIG. 2 illustrates a possible arrangement of the camera 4 at the home appliance 1. While FIG. 2a shows a side view, a front view is depicted in FIG. 2b. The camera 4 is arranged at the front of the home appliance 1 and may have a field of view 10, for example, between 80° and 130°. In the shown embodiment, the angle of the field of view in the vertical dimension is 89° and in a horizontal dimension is 122° . While a first sensor 11 is able to detect a position of the lower rack, a second sensor 12 monitors the position of the upper rack 3. In this way, the camera 4 can be triggered in the right situation to take a picture of the load.

[0037] FIG. 3a shows an example of a load image of the upper rack 3 taken by the camera 4. An example of a load image of the lower rack 2 is illustrated in FIG. 3b.

[0038] FIG. 4 illustrates an example of the semantic segmentation. In FIG. 4a an exemplary load image 13 is shown and in FIG. 4b an exemplary labeled segmented image 14 of the load, which corresponds to the load image 13. Possible types of load are empty 15, plastic 16, metal 17, glass 18 and ceramic 19. As can be seen in FIG. 4b, the load image 13 comprises 3 regions, a first region 20 loaded with metal, a second region 21 loaded with ceramic and a third region 22, which is empty.

[0039] FIG. 5 shows an example of how a deep convolutional semantic segmentation neural network 23 can be trained with high-resolution images 24, which have been correctly segmented by humans (correct segmentation 25). The high-resolution images 24 are fed into the neural network 23, which delivers a result in the form of an estimated segmentation 26. The estimated segmentation 26 is compared to the correct segmentation 25 and deviations are evaluated according to a cost function. This means that a loss 27 is calculated. As is well known in the prior art, the loss is used to adapt the weights and biases of the neural network 23 using backpropagation.

[0040] FIG. 6 shows a basic principle of how a GAN may work, which is well known in the prior art. A discriminator 28 (which is a neural network) is trained with high-resolution images 29 first. When the discriminator is sufficiently trained, a GAN 30 takes low-resolution images 31 as an input and tries to generate super-resolution images 32 (i.e. upscaled images) which are indistinguishable for the discriminator 28 from high-resolution images 29. A loss 33 is calculated, which is used to adapt the weights and biases of the GAN 30.

[0041] In the embodiment shown in FIG. 7, the low-resolution images 31 are generated for training purposes from the high-resolution images 24 by downscaling 34. The low-resolution images 31 are afterwards upscaled by the GAN 30 and a loss between the original high-resolution images 24 and the images upscaled by the GAN 30 is calculated to train the GAN 30. In this way, the GAN 30 learns to generate realistic high-resolution images of the load from low-resolution load images.

[0042] FIG. 8 illustrates how an embodiment of the processing module according to the invention may be trained. Once the GAN 30 has been sufficiently trained, the weights and biases of the GAN 30 are fixed, i.e. the training of the GAN is completed. Low-resolution images 31 are fed into the GAN 30, which upscales the images and feeds them into the deep convolutional semantic segmentation neural network 23, which outputs estimated segmentations 26. The estimated segmentations 26 are compared with the correct segmentations 25 to calculate the loss 27, which is used to fine-tune the weights and biases of the neural network 23.

[0043] FIG. 9 illustrates a first embodiment of the method according to the invention. The steps carried out on the home appliance 1 are shown on the left side and the steps carried out on the server 9 are located on the right side. In a first step S1, a load image 13 of the load is taken with a camera 4. The load image 13 is then sent to the server 9 in step S2. In step S3, the load image 13 is received on the server 9, where it is upscaled using a trained GAN 30 in step S4. The upscaled load image is fed into a neural network 23, which analyzes the load image in step S5. Based on the processing result of the neural network, e.g. a labeled segmented image of the load 14, control data to control the home appliance is determined in step S6. The control data is sent to the home appliance in step S7, where it is received in step S8. In step S12, the home appliance controls at least one working program of the home appliance based on the control data.

[0044] FIG. 10 shows a second embodiment of the method according to the invention. Again, the steps carried out on the home appliance 1 are located on the left side and the steps carried out on the server 9 are depicted on the right side. The steps S1 to S5 are the same as in the first embodiment. However, the step S6 is missing in the second embodiment. Instead, in step S9, the processing result is send to the home appliance, where it is received in step S10. In step S11, the home appliance determines control data based on the processing result by itself, which is used afterwards in step S12 to control at least one working program of the home appliance. In comparison to the second embodiment, the first embodiment has the disadvantage that the server may need knowledge about the type and configuration of the home appliance to be able to determine reasonable control data. However, it has the advantage that the determination of the control data may be carried out by a CPU on the server with a higher performance in comparison to the CPU of the home appliance.

[0045] FIG. 11 illustrates a first embodiment of a home appliance 1 according to the invention and a first embodiment of a server 9 according to the invention. The first embodiment of the home appliance 1 comprises a camera 4 for taking a load image of the load to be handled by the home appliance 1. Furthermore, it has a sending module 40 for sending the load image to a server and a receiving module 41, which is adapted to receive control data derived from the load image from the server in the first embodiment. In addition, the home appliance 1 comprises a control module 42 for controlling at least one working program of the home appliance based on the control data. The first embodiment of the server 9 comprises a receiving module 50, which is configured to receive a load image of the load from the home appliance 1, and a processing module 51 configured to process the load image to generate a processing result. The processing module 51 is equipped with a GAN 30 and a neural network 23. In addition, the server 9 comprises a determination module 52 configured to determine control data to control the home appliance based on the processing result and a sending module 53 configure to send the control data to the home appliance 1.

[0046] FIG. 12 shows a second embodiment of a home appliance 1 according to the invention and a second embodiment of a server 9 according to the invention. The main difference between the first and the second embodiment of the home appliance is that the second embodiment of the home appliance 1 comprises in addition a determination module 43 for determining control data based on the processing result. In comparison to the first embodiment of the server 9, the second embodiment of the server 9 does not comprise a determination module 52, instead the sending module 53 of the second embodiment of the server 9 is configured to send the processing result to the home appliance 1 to enable the home appliance to determine control data to control the home appliance based on the processing result on its own.

[0047] The following table illustrates an important advantage which may be achieved by the present invention:

TABLE-US-00001 Down sample 1x 2x 4x 5x 6x 7x 8x 16x scale (640 × 480) (320 × 240) (160 × 120) (128 × 96) (106 × 80) (91 × 68) (80 × 60) (40 × 30) Pixels 307200 76800 19200 12288 8480 6188 4800 1200 Recognition ~83% ~82% ~79% ~77% ~75% ~70% ~65% ~49% accuracy with GAN Recognition ~83% ~78% ~52% ~50% ~48% ~43% ~40% ~40% accuracy without GAN

[0048] In the first row the resolution of the image, which is processed by the processing module, is given. The highest resolution is 640×480 pixels, which leads to an overall number of pixels of 307.200 (see the second row). If this resolution is downscaled by a factor of 2, both dimensions are divided by two and a resolution of 320×240 results (see third column), which leads to an overall number of pixels of 76.800 (which is one fourth of 307.200). In the third row, the recognition accuracy of the semantic image segmentation performed by the processing module is shown, which results if the image having the resolution mentioned in the first row is fed into a GAN, where the image is upsampled to 640×480, before it is fed into the neural network. The recognition accuracy of the semantic image segmentation which results if the image having the resolution mentioned in the first row is fed directly into the neural network, without using a GAN before, is shown in the fourth row.

[0049] As can be seen in the third row, when going from a downscaling factor of 1 to a downscaling factor of 6, the recognition accuracy of the semantic image segmentation degrades only slightly from 83% to 75% if a GAN is used to upsample the images. If no GAN is used, the degradation is much severer and decreases from 83% to 48%. For a dishwasher, which is a non-life threating application, a recognition accuracy between 75% and 80% can be accepted. This allows to use a camera with a lower resolution at the dishwasher, which is of course much cheaper. Thus, the usage of a GAN on the server side allows to save hardware costs on the home appliance side, without significantly impairing the recognition accuracy.

[0050] FIG. 13 tries to visualize this correlation graphically. While the recognition accuracy is plotted on the vertical axis, the downscaling factor is outlined on the horizontal axis. The dashed line 60 relates to the recognition accuracy if no GAN is used and the solid line 61 shows the recognition accuracy if a GAN is used. A skilled person immediately understands that at a downscaling factor of 16 both approaches lead to insufficient results, but especially inbetween a downscaling factor of 4 and 8, the approach with GAN yields significantly better results than the approach without GAN.

[0051] The invention described above may among other things have the following advantages: The sending of the load image to the server allows to use the high computational power of the server, which especially allows to use neural networks for object recognition and image segmentation. In this way, the load to be handled by the home appliance may be recognized especially well, such that the home appliance can adapt its behavior very accurately in dependence on the load. Due to the usage of a GAN the overall cost of the camera hardware at the home appliance may be kept at a minimum. High-speed image acquisition may be possible because with a decrease of the resolution the camera may be able to take pictures at a higher speed. This may decrease the blurriness of pictures which are taken in motion, especially when the racks of the dishwasher are pushed in. The performance under low light conditions may be improved (when it is dusky in the kitchen). Moreover, the signal-to-noise-ratio may be improved. Less data processing may be needed at the dishwasher and the amount of data which needs to be sent to the server may be lower, which further reduces the costs.

[0052] The description with regard to the figures is to be interpreted in an illustrative, rather than in a restrictive sense. Many modifications may be made to the described embodiments without departing from the scope of the invention as set forth in the appended claims.

LIST OF REFERENCE SIGNS

[0053] 1 home appliance

[0054] 2 lower rack

[0055] 3 upper rack

[0056] 4 camera

[0057] 5 CPU

[0058] 6 communication module

[0059] 7 Wifi hotspot

[0060] 8 Internet

[0061] 9 server

[0062] 10 field of view

[0063] 11 sensor

[0064] 12 sensor

[0065] 13 load image

[0066] 14 labeled segmented image of the load

[0067] 15 empty

[0068] 16 plastic

[0069] 17 metal

[0070] 18 glass

[0071] 19 ceramic

[0072] 20 first region with metal

[0073] 21 second region with ceramic

[0074] 22 third region, which is empty

[0075] 23 neural network

[0076] 24 high-resolution images

[0077] 25 correct segmentation

[0078] 26 estimated segmentation

[0079] 27 loss

[0080] 28 discriminator

[0081] 29 high-resolution images

[0082] 30 GAN

[0083] 31 low-resolution images

[0084] 32 super-resolution images (upscaled images)

[0085] 33 loss

[0086] 34 downscaling

[0087] 35 loss

[0088] 40 sending module

[0089] 41 receiving module

[0090] 42 control module

[0091] 43 determination module

[0092] 50 receiving module

[0093] 51 processing module

[0094] 52 determination module

[0095] 53 sending module

[0096] 60 recognition accuracy without GAN

[0097] 61 recognition accuracy with GAN

[0098] S1 taking a load image

[0099] S2 sending the load image

[0100] S3 receiving the load image

[0101] S4 upscaling the load image

[0102] S5 analyzing the load using a neural network

[0103] S6 determining control data

[0104] S7 sending control data

[0105] S8 receiving control data

[0106] S9 sending the processing result

[0107] S10 receiving the processing result

[0108] S11 determining control data

[0109] S12 controlling at least one working program of the home appliance