Computing Device Programmed to Automatically Detect and Generate Digitized Information Relating to a Process Involving a User-Operated Device

Abstract

A computing device programmed for automatically generating and storing, in computer storage, a digitalization of information relating to a process involving a user-operated device. The computing device detects and records process details as a user uses a user-operated device. Specifically, a user interacts with a kitchen appliance via a user interface of the appliance. An action sequence of manual interactions is captured, each manual interaction triggering a function for processing ingredients in the kitchen appliance. Operating parameters of the kitchen appliance, detected via at least one sensor device of the kitchen appliance during the action sequence at the kitchen appliance, are automatically correlated with the manual interactions. Using the operating parameter data and the sequence of manual user interactions, partial steps of a user-known recipe are automatically identified by the computing device based on comparing the recorded information with reference data sets within library data.

Claims

1. A method for automatically generating and storing, in computer storage, a digitalization of a user-known recipe, the method comprising: generating a recording of manual user interactions with a kitchen appliance during a cooking process; wherein generating the recording includes capturing an action sequence of the manual user interactions, via a user interface of the kitchen appliance, during the cooking process; wherein each manual user interaction of the action sequence triggers a function for processing one or more ingredients stored in the kitchen appliance; detecting multiple operating parameters of the kitchen appliance by at least one sensor device of the kitchen appliance during the action sequence of manual user interactions; recording, in a set of operating parameter data, each of the multiple operating parameters during or as a result of one or more of the manual user interactions, correlating the multiple operating parameters with the recording of the manual user interactions; analyzing one or more reference data sets of a reference database to identify a set of partial steps of the user-known recipe and one or more ingredients involved in the set of partial steps; wherein each of the one or more reference data sets comprises one or more of: reference operating parameters, reference user interactions, or associations of known ingredients with reference action sequences and/or reference operating parameters; and wherein each partial step of the identified set of partial steps is associated with reference data comprising one or both of: reference operating parameters that are identical or similar to operating parameter data of the set of operating parameter data, or reference user interactions that are identical or similar to at least a portion of the action sequence of the recording of manual user interactions; based, at least in part, on the identified set of partial steps of the user-known recipe and the one or more ingredients involved in the set of partial steps, storing, in the computer storage, a plurality of action sequences to create the digitalization of the user-known recipe.

2. The method of claim 1, further comprising recording, in the set of operating parameter data, at least one curve of values of at least one of the multiple operating parameters over time.

3. The method of claim 2, wherein said analyzing the one or more reference data sets of the reference database further comprises comparing the at least one curve of values to the one or more reference data sets.

4. The method of claim 1, wherein: the multiple operating parameters comprises a particular operating parameter; and said detecting multiple operating parameters comprises recording a time course of the particular operating parameter.

5. The method of claim 1, further comprising: receiving, via the user interface of the kitchen appliance, manual edit information from a user; adjusting the digitalization of the user-known recipe based, at least in part, on the manual edit information.

6. The method of claim 1, further comprising: sending the plurality of action sequences to a remote device that is remote from the kitchen appliance; wherein storing, in the computer storage, the plurality of action sequences to create the digitalization of the user-known recipe is performed by the remote device.

7. The method of claim 1, further comprising identifying a particular ingredient, of the one or more ingredients, based on comparing information from the set of operating parameter data and the recording of manual user interactions with the one or more reference data sets.

8. The method of claim 7, further comprising: displaying, on a display unit of the kitchen appliance, digitalization information comprising one or both of: the particular ingredient, or the action sequence of the manual user interactions; wherein the kitchen appliance is configured to receive confirmation information, for the digitalization information, via the user interface.

9. The method of claim 1, further comprising detecting user confirmation of addition of an ingredient to the kitchen appliance, wherein said detecting the multiple operating parameters of the kitchen appliance is performed responsive to said detecting user confirmation.

10. The method of claim 1, further comprising identifying a critical action sequence, of the action sequence of manual user interactions, based, at least in part, on validation information for one or more of: an ingredient of the one or more ingredients, the action sequence of manual user interactions, or the digitalization of the user-known recipe.

11. The method of claim 10, further comprising automatically generating the validation information comprising: locating a particular ingredient, of the one or more ingredients, in a validation database that comprises a validation data set associated with the particular ingredient; and generating the validation information based, at least in part, on comparing (a) at least one operating parameter, of the multiple operating parameters, and/or at least one manual user interaction, of the action sequence of manual user interactions, with (b) the validation data set.

12. The method of claim 1, further comprising outputting a suggestion for improvement of the digitalization of the user-known recipe based on data from the reference database and/or a validation database that comprises a validation data set associated with an ingredient of the one or more ingredients.

13. A computerized kitchen appliance comprising: one or more ingredient processing devices; a user interface; computer storage; at least one sensor device; and a computing unit that is configured to perform automatically generating and storing, in the computer storage, a digitalization of a user-known recipe, comprising: generating a recording of manual user interactions with the computerized kitchen appliance during a cooking process; wherein generating the recording includes capturing an action sequence of the manual user interactions, via the user interface of the computerized kitchen appliance, during the cooking process; wherein each manual user interaction of the action sequence triggers a function for processing one or more ingredients stored in the computerized kitchen appliance; detecting multiple operating parameters of the computerized kitchen appliance by the at least one sensor device of the computerized kitchen appliance during the action sequence of manual user interactions; recording, in a set of operating parameter data, each of the multiple operating parameters during or as a result of one or more of the manual user interactions, correlating the multiple operating parameters with the recording of the manual user interactions; analyzing one or more reference data sets of a reference database to identify a set of partial steps of the user-known recipe and one or more ingredients involved in the set of partial steps; wherein each of the one or more reference data sets comprises one or more of: reference operating parameters, reference user interactions, or associations of known ingredients with reference action sequences and/or reference operating parameters; and wherein each partial step of the identified set of partial steps is associated with reference data comprising one or both of: reference operating parameters that are identical or similar to operating parameter data of the set of operating parameter data, or reference user interactions that are identical or similar to at least a portion of the action sequence of the recording of manual user interactions; based, at least in part, on the identified set of partial steps of the user-known recipe and the one or more ingredients involved in the set of partial steps, storing, in the computer storage, a plurality of action sequences to create the digitalization of the user-known recipe.

14. A computing system comprising: a kitchen appliance comprising: one or more ingredient processing devices, a user interface, and at least one sensor device; computer storage; and a computing unit, communicatively coupled to the kitchen appliance and the computer storage, the computing unit being configured to perform automatically generating and storing, in the computer storage, a digitalization of a user-known recipe, comprising: generating a recording of manual user interactions with the kitchen appliance during a cooking process; wherein generating the recording includes capturing an action sequence of the manual user interactions, via the user interface of the kitchen appliance, during the cooking process; wherein each manual user interaction of the action sequence triggers a function for processing one or more ingredients stored in the kitchen appliance; detecting multiple operating parameters of the kitchen appliance by the at least one sensor device of the kitchen appliance during the action sequence of manual user interactions; recording, in a set of operating parameter data, each of the multiple operating parameters during or as a result of one or more of the manual user interactions, correlating the multiple operating parameters with the recording of the manual user interactions; analyzing one or more reference data sets of a reference database to identify a set of partial steps of the user-known recipe and one or more ingredients involved in the set of partial steps; wherein each of the one or more reference data sets comprises one or more of: reference operating parameters, reference user interactions, or associations of known ingredients with reference action sequences and/or reference operating parameters; and wherein each partial step of the identified set of partial steps is associated with reference data comprising one or both of: reference operating parameters that are identical or similar to operating parameter data of the set of operating parameter data, or reference user interactions that are identical or similar to at least a portion of the action sequence of the recording of manual user interactions; based, at least in part, on the identified set of partial steps of the user-known recipe and the one or more ingredients involved in the set of partial steps, storing, in the computer storage, a plurality of action sequences to create the digitalization of the user-known recipe.

15. The computing system of claim 14, wherein the at least one sensor device comprises one or more of: a scale, a current sensor, a temperature sensor, a timepiece, or a camera.

16. The computing system of claim 14, the computing unit being further configured to perform identifying a particular ingredient, of the one or more ingredients, based on comparing information from the set of operating parameter data and the recording of manual user interactions with the one or more reference data sets.

17. The computing system of claim 16, further comprising: a display unit configured to display digitalization information comprising one or both of: the particular ingredient, or the action sequence of the manual user interactions; wherein the user interface is configured to receive confirmation information for the digitalization information.

18. The computing system of claim 14, wherein the computing unit is further configured to perform causing output of a suggestion for improvement of the digitalization of the user-known recipe based on data from the reference database and/or a validation database that comprises a validation data set associated with an ingredient of the one or more ingredients.

19. The computing system of claim 14, wherein the computing unit is further configured to perform recording, in the set of operating parameter data, at least one curve of values of at least one of the multiple operating parameters over time.

20. The computing system of claim 19, wherein said analyzing the one or more reference data sets of the reference database further comprises comparing the at least one curve of values to the one or more reference data sets.

Description

[0061] Further advantages, features and details of the invention result from the following description, in which examples of the execution of the invention are described in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention either individually or in any combination. It is shown:

[0062] FIG. 1 shows a system according to the invention with a kitchen appliance according to the invention in a schematic representation in a first embodiment,

[0063] FIG. 2 a method according to the invention in a schematic representation of the procedural steps in a second embodiment,

[0064] FIG. 3a-b a more detailed description of the procedural steps of the method according to the invention of the second embodiment,

[0065] FIG. 4 a method according to the invention in a schematic representation in a third embodiment,

[0066] FIG. 5 a recipe according to the invention of the third embodiment.

[0067] FIGS. 6a-c Data comparisons for an inventive method in further embodiments,

[0068] In the following figures, the identical reference signs are used for the same technical characteristics, even for different embodiments.

[0069] FIG. 1 shows an invention-compliant system 1 for the digitalization of a cooking process 200. The invention-compliant kitchen appliance 10 is provided, which is suitable for performed the cooking process 200 with a cooking vessel 11. The cooking vessel 11 is configured to hold ingredients 2, so that they can be added to the cooking vessel 11 via a lid, for example. Furthermore, the kitchen appliance 10 has a processing device 12 for processing the ingredients 2. The processing device 12 comprises an agitator 12.1 for mixing and/or crushing the ingredients 2. In addition, the processing device 12 has a heating element 12.2 which is configured to heat the ingredients 2. Preferably the heating element 12.2 is an electric heating element and the agitator 12.1 is an electrically driven agitator. Furthermore, the kitchen appliance 10 includes a user interface 13 for starting a manual user interaction 201. The user interface 13 includes a display unit 13.1 and a rotary knob 13.2. The user interface 13 allows the user to conveniently operate the kitchen appliance 10. In addition, the kitchen appliance 10 has a sensor device 20 for detecting 103 at least one operating parameter 210 during the cooking process 200. In the example shown, the sensor device 20 is part of the kitchen appliance 10. However, it is also conceivable that the sensor device 20 is attached to the kitchen appliance 10 as an additional module or is provided as an independent unit beside the kitchen appliance 10. Furthermore, the kitchen appliance 10 has an integrated control unit 14, which is connected to an internal memory unit 32 of the kitchen appliance 10. In the internal memory unit 32, for example, the operating parameters and the manual user interaction can be stored. Furthermore, the control unit 14 can be connected to a computing unit 31 via a data interface 15. This enables to start processes on the computing unit 31 in which the user operates the user interface 13 of the kitchen appliance 10, whose signal is processed accordingly by the integrated control unit 14 and, if necessary, triggers communication with the computing unit 31. The computing unit 31 is configured to identify an ingredient 2 by interpretation 110 of at least one operating parameter 210 and manual user input 201. The operating parameter 210 is preferably a measured value which is recorded in the kitchen appliance 10, especially during the cooking process 200. For this purpose, the sensor device 20 includes a scale 21 to record the weight of the added ingredient 2. In particular, the scale 21 comprises three load cells 21.1, which are configured as a stand for the kitchen appliance. This means that a weight distribution within the cooking vessel 11 can also be recorded by the scale 21 in order to determine ingredient 2. For example, it is conceivable that a certain flow and/or pouring behaviour of the ingredient 2 at a certain speed of the agitator 12.1 is characteristic of the ingredient 2 to be identified. In particular, the agitator 12.1 can be equipped with a tachometer for this purpose. Furthermore, it is conceivable that the sensor device 20 has a current sensor 22 through which the motor current of the agitator 12.1 can be detected. A higher motor current at a given speed can therefore indicate a higher resistance due to ingredient 2. In addition, a temperature sensor 23 is provided to measure the temperature of ingredient 2 and/or in cooking vessel 11. Temperature sensor 23 can therefore also be used to determine a further characteristic, namely, for example, the heating behaviour, in particular the specific heat capacity, ingredient 2. Furthermore, a camera 24 is provided in order to visually record ingredient 2 and, for example, to be able to infer ingredient 2 from a change in the colour of ingredient 2 during a heating process. The sensor device 20 further comprises a timepiece 25, by means of which a progression of time of the at least one operating parameter 210 can be recorded. Thus, the timepiece 25, which can include in particular a watch, can give a temporal dimension to the operating parameter 210, thus allowing a higher reliability in the identification 104 of the ingredient 2. In this example, the identification 104 or interpretation 110 of the operating parameter 210 and the manual user interaction 201 is performed by a server 30 that has the computing unit 31. In order to be able to perform a comparison of the data with known reference and/or validation data, the server 30 also has a storage unit 32 with a reference database 33 and a validation database 34. In addition, the server 30 is part of or connected to a network 35, in particular a neural network. Over network 35, which has a plurality of network nodes 35.1, it is possible to perform a big data analysis, the network nodes 35.1 comprising databases with known characteristics of ingredients in large numbers, which can be analyzed and made available to the server 30. This enables the computing unit 31, in particular by accessing the network 35, to perform the interpretation 110 of at least one operating parameter 210 and thus to conclude on the ingredient 2. In order to display the data to the user in intermediate steps and/or as a result, the data can also be sent to a mobile terminal 3, i.e., in particular a smartphone, tablet or the like, which can communicate with the kitchen appliance 10 and/or the server 30.

[0070] FIG. 2 shows a method 100 for the digitalization of a cooking process 200 for a kitchen appliance 10. This preferably requires the user to start 101 cooking process 200 first. The user thereby confirms that a cooking process 200 is being performed, which is at least partially performed manually. The kitchen appliance 10 also recognizes that certain sections of the following process or actions are to be recorded. Thereupon, an intake 102 of an ingredient 2 is provided by a cooking vessel 11 of the kitchen appliance 10. If the user has completely filled in ingredient 2, a confirmation 121 is required by which the user informs the kitchen appliance 10 that ingredient 2 is completely filled in. With the confirmation 121 or after the confirmation 121 a manual user interaction 201 takes place, which triggers a certain behaviour of the kitchen appliance 10. For example, the user starts an agitator 12.1 of the kitchen appliance 10 at a certain speed or starts the heating process of a heating element 12.2. In particular, after detecting 103 at least one operating parameter 210 as a result of the manual user interaction 201, the ingredient 2 is identified 104 by interpreting 110 the at least one operating parameter 210 and the at least one manual user interaction 201.

[0071] FIG. 3a shows the interpretation of 110 in a schematic, exemplary representation. In this case, there is a storage 112 of at least one operating parameter 210 during a cooking step of the cooking process 200, which was triggered by a manual user interaction 201. Preferably, several operating parameters 210 are recorded and stored simultaneously, so that the most comprehensive possible picture of the characteristic behaviour of ingredient 2 is available. In connection with the manual user interaction 201 performed, a comparison 111 is finally performed, for which reference data records 211 of a reference database 33, for example, are used. Reference records 211 may include library data of known ingredients that reflect their characteristic behaviour in the user interaction performed. For example, it is conceivable that onions generate a characteristic motor current curve when they are comminuted at a specified speed in the agitator 12.1 of the kitchen appliance 10. At the same time, it may be provided that the onions are to be heated and that their heating curve is also recorded, thus indicating the specific heat capacity of ingredient 2. If a similar material behaviour is now found in the reference data sets 211, it is highly probable that onions can be concluded. In particular, the reference database 33 may be supplemented with or based on data from a big data analysis.

[0072] After identifying 104, an assignment 105 of the ingredient 2 and/or the manual user interaction 201 to an action sequence 202 preferably takes place. The action sequence 202 can preferably comprise a part of a recipe 203, so that an assignment 107 of the action sequence 202 to a recipe 203 subsequently takes place. This means that recipe 203 can preferably be digitalized step by step. Furthermore, after or during identification 104 the ingredient 2, the assignment 105 to an action sequence 202 and the assignment 107 to recipe 203 can be displayed 106 to the user, so that the user is informed at any time during the cooking process about the current status of the data acquisition. In addition, depending on the display 106, a confirmation 121 may be provided by the user before the next procedural step takes place. This ensures that errors are detected early in the process and can be corrected manually or automatically. After assigning 107 to the recipe 203, a sending 108 of the recipe 203 and/or the action sequence 202 to a mobile terminal 3 and/or a server 30 is also provided. The mobile terminal device 3 can, for example, be a smartphone or tablet of the user, so that he has the recipe 203 and/or action sequence 202 digitally available. The recipe 203 and/or the action sequence 202 can, for example, be made available to an online community via a server 30. Preferably before or after sending 108 the corresponding data to server 30 or mobile device 3, validation 120 is provided.

[0073] FIG. 3b shows a schematic representation of validation 120. A comparison 123 of the recorded operating parameter 210 with a validation data set 212 is provided. Preferably, validation record 212 is generated by a big data analysis and/or retrieved from a validation database 34. For this purpose, a finding 122 of the ingredient 2 in a validation database 34 is provided, whereby a text element of the ingredient 2 is generated and a finding of the text element can take place in the validation database 34. It is also conceivable that finding 122 the ingredient 2 in the validation database 34 on the basis of validation record 212 itself represents a validation result 120.1. In addition or alternatively, validation 120 may include a confirmation 121 from a user. Validating 120 ensures that the recorded cooking process meets 200 specific quality criteria. In particular, this can ensure that only recipes 203 with harmless ingredients 2 are made available, for example, in an online community. Furthermore, on the basis of validation 120, an output of an improvement suggestion can be made, in which certain action sequences 202 are improved. This can be based, for example, on an average perception of food quality. Preferably, confirmation 121 by the user, finding of 122 of ingredient 2 in validation database 34 and/or comparison 123 may lead to validation result 120.1, which classifies ingredient 2, action sequence 202 and/or recipe 203 as critical or non-critical.

[0074] In particular, based on the validation result 120.1, it is also possible to output 109 of an improvement suggestion before or after sending 108 of the data to server 30. The improvement suggestion can include, for example, adaptations of recipe 304 to a quality specification or the like.

[0075] FIG. 4 also shows a method 100 in another example. The starting 101 of a digitalization of a cooking process 200 is planned. First an action sequence 202 comprising several ingredients 2.1, 2.2, 2.3 and several manual user interactions 201 is recorded and finally interpreted. For example, it is planned to first record 102 of a first ingredient 2.1 in a cooking vessel 11 of a kitchen appliance 10. The first ingredient 2.1 is followed by the addition 102 of a second ingredient 2.2 Preferably, confirmation 121 is required after the addition of each of the ingredients 2.1, 2.1 by the user, so that it is clear for the interpretation of the data that two different ingredients 2.1, 2.2 have been added. Finally, each ingredient 2.1, 2.2 can be assigned an operating parameter of 210, i.e., in particular a weight, via a scale 21 of the kitchen appliance 10. This is followed by a manual user interaction 201, which starts or carries out, for example, comminution and mixing of the existing ingredients 2.1, 2.2. In particular, a manual user interaction 201 can include several manual, preferably different, partial interactions 201.1, 201.2, which in turn represent partial sequences. During comminution, at least one operating parameter 210 is recorded, which can be assigned to user interaction 201. Afterwards, 102 is added to a third ingredient 2.3, which is followed by another manual user interaction 201. The manual user interaction 201 can include a renewed comminution of the total mass now present in the cooking vessel 11 and/or heating. For example, the first ingredient may include 2.1 onions and the second ingredient 2.2 cloves of garlic. These are first crushed and steamed before the third ingredient 2.3 tomatoes are added, which are crushed and heated again, while onions and garlic cloves are still present in the cooking vessel 11. This also results in the recording of at least one further operating parameter 210, which can be assigned to manual user interaction 201. Thus, the entire action sequence 202 with different inputs and system reactions is initially available. The sequence of actions, i.e., in particular the sequence of actions in combination with the recorded operating parameters 210, can thus lead to an adjustment 111, via which such standardized sequences of actions can be found via a reference database 33. In this way, known partial steps of a recipe 203 can also be identified as a whole and, accordingly, the ingredients involved 2.1, 2.2, 2.3.

[0076] According to FIG. 5, several action sequences 202 finally result in a recipe 203, which was at least partially performed manually by the user during the cooking process 200 and is then available in digital form.

[0077] FIG. 6a schematically shows a course of time of an operating parameter 210, which is compared with a reference data set 211, which is also present as a temporal course, in order to compare 111 of the data. If the operating parameters 210 and the reference data set 211 deviate from each other within a certain tolerance, the presence of a certain ingredient 2 whose characteristic is mapped by the reference data set 211 over time can be concluded with high probability.

[0078] FIG. 6b schematically shows the temporal course of the operating parameter 210, which for validating 120 is compared with a validation data set 212, which comprises a limit value, in a comparison 123. If the operating parameter 210 does not exceed the limit value of the validation data set 212 within the recorded time course, the section of the cooking process 200 assigned to the operating parameter 210 can be classified as non-critical.

[0079] FIG. 6c shows another possibility of validating 120 in another example. An ingredient 2 itself is validated, whereby the ingredient 2 is searched in a validation database 34 with entries for different ingredients 2. In validation database 34, each ingredient 2 is also assigned a validation data record 212, which classifies the respective ingredient 2 as critical or non-critical. If a finding 122 of the ingredient 2 to be validated in the validation database 34 is successful, the associated validation data record 212 is output, in particular as validation result 120.1, so that a classification can be displayed to the user, for example.

[0080] The preceding explanation of the embodiments describes the present invention exclusively in the context of examples. Of course, individual features of the embodiments can be freely combined with each other, if technically reasonable, without leaving the scope of the present invention.

REFERENCE CHARACTER LIST

[0081] 1 System [0082] 2 Ingredients [0083] 2.1 First ingredient [0084] 2.2 Second ingredient [0085] 2.3 Third ingredient [0086] 3 mobile device [0087] 10 Kitchen appliance [0088] 11 Cooking vessel [0089] 12 Processing device [0090] 12.1 Agitator [0091] 12.2 Heating element [0092] 13 User interface [0093] 13.1 Display unit [0094] 13.2 Rotary knob [0095] 14 Control unit [0096] 15 Data Interface [0097] 20 Sensor device [0098] 21 Scale [0099] 21.1 Load cell [0100] 22 Current sensor [0101] 23 Temperature sensor [0102] 24 Camera [0103] 25 Timer [0104] 30 Server [0105] 31 Computing unit [0106] 32 Memory unit [0107] 33 Reference database [0108] 34 Validation database [0109] 35 Network [0110] 35.1 Network node [0111] 100 Methods [0112] 101 Start [0113] 102 Receiving of 2 [0114] 103 Detection of 210 [0115] 104 Identifying of 2 [0116] 105 Assigned to 202 [0117] 106 Display [0118] 107 Assigned to 203 [0119] 108 Send to 203 [0120] 109 Output of a suggestion for improvement [0121] 110 Interpreting of 201 and 210 [0122] 111 Adjustment [0123] 112 Save [0124] 120 Validate [0125] 120.1 Result of validation [0126] 121 Confirmation [0127] 122 Location of 2 [0128] 123 Compare [0129] 200 Cooking process [0130] 201 Manual user interaction [0131] 201.1 Partial interaction [0132] 201.2 Partial interaction [0133] 202 Action sequence [0134] 203 Recipe [0135] 210 Operating parameters [0136] 211 Reference dataset [0137] 212 Validation data record