Methods and systems for food preparation in a robotic cooking kitchen
11117253 · 2021-09-14
Assignee
Inventors
Cpc classification
G05B2219/40395
PHYSICS
A47J36/321
HUMAN NECESSITIES
G05B19/42
PHYSICS
International classification
G05B19/42
PHYSICS
Abstract
The present disclosure is directed to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the human chef's movements and actions. Monitoring a human chef is carried out in an instrumented application-specific setting, a standardized robotic kitchen in this instance, and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef.
Claims
1. A smart kitchen, comprising: a kitchen module, comprising: a kitchen furniture, and one or more smart appliances; one or more sensors; a display for displaying one or more controlling functions in the kitchen module; and one or more processors for controlling the one or more smart appliances; wherein during a first mode, the one or more sensors for recording a first set of data in a recipe file for controlling one or more smart appliances, and during a second mode, the one or more sensors for receiving a second set of data for subsequent corrective or feedback during execution of the recipe file by the one or more smart appliances; and wherein the one or more processors executes the recipe file for controlling the one or more smart appliances, each smart appliance executing one or more food preparation operations based on one or more graphical curves from the recipe file, the one or more processors generating a command for controlling the one or more smart appliances from the recipe file or from the user, the one or more graphical curves comprising one or more data over a time period.
2. The smart kitchen of claim 1, wherein the one or more sensors is integrated in the kitchen module.
3. The smart kitchen of claim 1, wherein the one or more sensors is located outside of the kitchen module.
4. The smart kitchen of claim 1, wherein the one or more sensors is integrated in the one or more smart appliances.
5. The smart kitchen of claim 1, wherein the information comprises data and instructions.
6. The smart kitchen of claim 1, further comprising one or more smart containers for storing ingredients.
7. The smart kitchen of claim 1, further comprising one or more dispensing devices for dosing ingredients.
8. The smart kitchen of claim 1, further comprising a recipe Library database for storing a plurality of recipe files for execution by the one or more smart appliances within the kitchen module, each of the recipe files includes one or more cooking operations with a start time and an end time.
9. The smart kitchen of claim 1, wherein the one or more processors receives data from a chef and record the data in a recipe file.
10. The smart kitchen of claim 1, where the one or more processors in the kitchen module coupled to a network for controlling the one or more smart appliances and for communicating information between the kitchen module and the graphical or voice user interface, the interface receiving a voice activation command from a user, the interface generates generating a voice-based information to the user.
11. The smart kitchen of claim 1, where the kitchen module comprises one or more robotic arms and one or more robotic end effectors, the processor executing the recipe file in which the one or more robotic arms and the one or more robotic end effectors operating the one or more smart appliances.
12. The smart kitchen of claim 1, wherein the processor generates a visual graphical representation of the smart kitchen.
13. A method for operating a kitchen module in a smart kitchen, comprising: providing a kitchen module, the kitchen module including a kitchen furniture, one or more smart appliances, and one or more sensors; during a first mode, recording by one or more sensors a first set of data in a recipe file for controlling the one or more smart appliances, and during a second mode, the one or more sensors receiving a second set of data for subsequent corrective or supportive feedback during the recipe execution, or to define position and/or orientation of a standard object in the kitchen module, or to define a shape, a size, a type, a position, or an orientation of a non-standard object in the kitchen module, the one or more sensors sensing the non-standard object with respect to a shape, a dimension, an orientation or a position information for making a real-time adjustment; displaying one or more real-time smart appliances statuses upon receiving a chef or user command via a graphical or voice user interface for selecting the recipe file for recording or execution, displaying recipe execution feedback, visualizing object position and orientation in the kitchen, or visualize controlling functions of the one or more smart appliances, or any combinations thereof, in the kitchen module during recipe recording and execution; detecting by one or more processors operational states of the one or more smart appliances and operating the one or smart appliances; and controlling the one or more smart appliances by e one or more processors in the kitchen module coupled to a network, the one or more processors communicating information between the kitchen module and the graphical or voice user interface; wherein the one or more processors executes the recipe file for controlling the one or more smart appliances, each smart appliance executing one or more food preparation operations based on one or more graphical curves from the recipe file, the one or more processors generating a command for controlling the one or more smart appliances from the recipe file or from the user, the one or more graphical curves comprising one or more data over a time period.
14. The method of claim 13, during an execution mode, further comprising executing the recipe file for controlling the one or more smart appliances and checking on the real-time status of the one or more smart appliances.
15. The method of claim 14, wherein the one or more processors informs a user via a visual interface or a voice interface on the real-time status of the one or more smart appliances.
16. The method of claim 14, wherein the one more processors assists a user with food preparation via a visual interface or a voice interface based on the real-time status data collected from the one or more smart appliances.
17. The method of claim 14, during the recording mode, further comprising recording the recipe file from real-time sensor data, the real-time sensor data including sensor data and timing data.
18. The method of claim 17, wherein the real-time sensor data comprises one or more video sensors, one or more microphone sensors, one or more temperature sensors, and/or sensor data from the one or more smart appliances.
19. The method of claim 13, further comprising retrieving the file from a cloud database.
20. The method of claim 13, wherein the displaying step comprises displaying a real-time virtual kitchen model via a visual interface.
21. The smart kitchen of claim 1, wherein during the second mode, the one or more sensors receiving the second set of data to define a position and an orientation of a standard object in the kitchen module, or the one or more sensors receiving the second set of data to define a shape, a size, a type, a position, or an orientation of a non-standard object in the kitchen module, the one or more sensors sensing the non-standard object with respect to its shape, dimension, orientation or position information for making a real-time adjustment.
22. The smart kitchen of claim 1, wherein the step of the graphical or voice user interface for operating the kitchen module or obtaining real-time information from the one or more objects in the kitchen module comprises (1) receiving a chef or user command to select the recipe for recording or execution (2) displaying one or more real-time smart appliances statuses, (3) displaying recipe execution feedback, (4) visualizing one or more objects, a respective objection position and/or a respective object orientation in the kitchen, or (5) controlling the one or more smart appliances, or any combinations thereof.
23. The smart kitchen of claim 1, wherein the kitchen module comprises one or more actuators to reallocate one or more objects in the kitchen module, including one or more shelfs, one or more cookwares, or one or more small appliances.
24. The smart kitchen of claim 1, wherein the one or more smart appliances comprises a pan having a plurality of cooking zones, each cooking zone associated with a respective control and a respective sensor, the respective control measuring an adjusting the temperature of a respective cooking with the respective sensor, the pan replicating the chef creating recipe process by controlling and adjusting one or more desired temperature curves over time in the plurality of cooking zones.
25. The smart kitchen of claim 1, further comprising one or more ingredient storages having one or more with sensors for receiving real-time sensor data to determine ingredient properties in the one or more ingredient storages.
26. A smart kitchen, comprising: a kitchen module, comprising: a kitchen furniture, a worktop area, one or more smart appliances, and one or more sensors; one or more sensors; a display for displaying one or more controlling functions in the kitchen module; and one or more processors in the kitchen module coupled to a network for controlling the one or more smart appliances and for communicating information between the kitchen module and the graphical or voice user interface; wherein during a first mode, the one or more sensors for recording a first set of data in a recipe file for controlling one or more smart appliances, and during a second mode, the one or more sensors for receiving a second set of data for subsequent corrective or supportive feedback during execution of the recipe file by the one or more smart appliances; and wherein the one or more smart appliances comprises a smart oven communicatively coupled to an operating control unit, the smart having one or more temperature sensors and/or one or more humidity sensors, the smart oven replicating the chef creating recipe process with the sensor curves of a particular food recipe in which the operating control unit adjusts the temperature and humidity of the smart oven by real-time data received from the one more temperature sensors and/or one or more humidity sensors.
27. A smart kitchen, comprising: a kitchen module, comprising: a kitchen furniture, a worktop area, one or more smart appliances, and one or more sensors; one or more sensors; a display for displaying one or more controlling functions in the kitchen module; and one or more processors in the kitchen module coupled to a network for controlling the one or more smart appliances and for communicating information between the kitchen module and the graphical or voice user interface; wherein during a first mode, the one or more sensors for recording a first set of data in a recipe file for controlling one or more smart appliances, and during a second mode, the one or more sensors for receiving a second set of data for subsequent corrective or supportive feedback during execution of the recipe file by the one or more smart appliances; and wherein the one or more smart appliances comprises a charcoal grill communicatively coupled to a control unit and an electric ignition system, the charcoal grill including one or more temperature sensors and one or more humidity sensors, charcoal grill replicating the chef creating recipe process with the sensor curves of a particular food recipe in which the electric ignition system modulates electric power to the charcoal grill with real-time data received from the one more temperature sensors and/or one or more humidity sensors.
28. The method of claim 13, wherein the non-standard object comprises an ingredient that may have a different size, a different dimension, a different shape, a different color, a different texture, a different weight, and/or any combination thereof.
29. The method of claim 13, wherein the standard object including the name of the standard object, a predetermined size, a predetermined dimension, a predetermined color, a predetermined shape, a predetermined texture, a predetermined weight, a standard position, a standard orientation, a virtual three-dimensional model of the standard object, and/or any combination.
30. The method of claim 13, wherein the kitchen modules comprises a non-standard kitchen module, the standard object placed in a different position and/or in a different orientation.
31. The smart kitchen of claim 1, wherein the one or more processors are located in the one or more smart appliances.
32. The smart kitchen of claim 1, wherein the one or more processors are located in the kitchen module.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The invention will be described with respect to specific embodiments thereof, and reference will be made to the drawings, in which:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)
(18)
(19)
(20)
(21)
(22)
(23)
(24)
(25)
(26)
(27)
(28)
(29)
(30)
(31)
(32)
(33)
(34)
(35)
(36)
(37)
(38)
(39)
(40)
(41)
(42)
(43)
(44)
(45)
(46)
(47)
(48)
(49)
(50)
(51)
(52)
(53)
(54)
(55)
(56)
(57)
(58)
(59)
(60)
(61)
(62)
(63)
(64)
(65)
(66)
(67)
(68)
(69)
(70)
(71)
(72)
(73)
(74)
(75)
(76)
(77)
(78)
(79)
(80)
(81)
(82)
(83)
(84)
(85)
(86)
(87)
(88)
(89)
(90)
(91)
(92)
(93)
(94)
(95)
(96)
(97)
(98)
(99)
(100)
(101)
DETAILED DESCRIPTION
(102) A description of structural embodiments and methods of the present invention is provided with reference to
(103) The following definitions apply to the elements and steps described herein. These terms may likewise be expanded upon.
(104) Abstracted Data—refers to the abstracted recipe of utility for machine-execution which has many other data-elements that a machine needs to know for proper execution and replication. This so-called meta-data, or additional data corresponding to a particular step in the cooking process, whether it be direct sensor-data (clock-time, water-temperature, camera-image, utensil or ingredient used, etc.) or data generated through interpretation or abstraction of larger data-sets (such as a 3-dimensional range cloud from a laser used to extract the location and types of objects in the image, overlaid with texture and color maps from a camera-picture, etc.), is time-stamped and used by the robotic kitchen to set, control and monitor all processes and associated methods and equipment needed at every point in time as it steps through the sequence of steps in the recipe.
(105) Abstracted Recipe—refers to a representation of a chef's recipe, which a human knows as represented by the use of certain ingredients, in certain sequences, prepared and combined through a sequence of processes and methods as well as skills of the human chef. An abstracted recipe used by a machine for execution in an automated way requires different types of classifications and sequences. While the overall steps carried out are identical to those of the human chef, the abstracted recipe of utility to the robotic kitchen requires that additional meta-data be a part of every step in the recipe. Such meta-data includes the cooking time, variables such as temperature (and its variations over time), oven-setting, tool/equipment used, etc. Basically a machine-executable recipe-script needs to have all possible measured variables of import to the cooking process (all measured and stored while the human chef was preparing the recipe in the chef studio) correlated to time, both overall and that within each process-step of the cooking-sequence. Hence the abstracted recipe is a representation of the cooking steps mapped into a machine-readable representation or domain, which takes the required process from the human-domain to that of the machine-understandable and machine-executable domain through a set of logical abstraction steps.
(106) Acceleration—refers to the maximum rate of speed-change at which a robotic arm can accelerate around an axis or along a space-trajectory over a short distance.
(107) Accuracy—refers to how closely a robot can reach a commanded position. Accuracy is determined by the difference between the absolute position of the robot compared to the commanded position. Accuracy can be improved, adjusted, or calibrated with external sensing such as sensors on a robotic hand or a real-time three-dimensional model using multiple (multi-mode) sensors.
(108) Action Primitive—In one embodiment, the term refers to an indivisible robotic action, such as moving the robotic apparatus from location X1 to location X2, or sensing the distance from an object for food preparation without necessarily obtaining a functional outcome. In another embodiment, the term refers to an indivisible robotic action in a sequence of one or more such units for accomplishing a mini-manipulation. These are two aspects of the same definition.
(109) Automated Dosage System—refers to dosage containers in a standardized kitchen module where a particular size of food chemical compounds (such as salt, sugar, pepper, spice, any kind of liquids, such as water, oil, essences, ketchup, etc.) that is released upon application.
(110) Automated Storage and Delivery System—refers to storage containers in a standardized kitchen module that maintain a specific temperature and humidity for storing food; each storage container is assigned a code (e.g., a bar code) for the robotic kitchen to identify and retrieval where a particular storage container delivers the food contents stored therein.
(111) Data Cloud—refers to a collection of sensor or data-based numerical measurement values from a particular space (three-dimensional laser/acoustic range measurement, RGB-values from a camera image, etc.) collected at certain intervals and aggregated based on a multitude of relationships, such as time, location, etc.
(112) Degree of Freedom (“DOF”)—refers to a defined mode and/or direction in which a mechanical device or system can move. The number of degrees of freedom is equal to the total number of independent displacements or aspects of motion. The total number of degrees of freedom is doubled for two robotic arms.
(113) Edge Detection—refers to a software-based computer program(s) capable of identifying the edges of multiple objects that may be overlapping in a two-dimensional-image of a camera yet successfully identifying their boundaries to aid in object identification and planning for grasping and handling.
(114) Equilibrium Value—refers to the target position of a robotic appendage, such as a robotic arm where the forces acting upon it are in equilibrium, i.e. there is no net force and thus no net movement.
(115) Execution Sequence Planner—refers to a software-based computer program(s) capable of creating a sequence of execution scripts or commands for one or more elements or systems capable of being computer controlled, such as arm(s), dispensers, appliances, etc.
(116) Food Execution Fidelity—refers to a robotic kitchen which is intended to replicate the recipe-script generated in the chef studio by watching and measuring and understanding the steps and variables and methods and processes of the human chef, thereby trying to emulate his/her techniques and skills. The fidelity of how close the execution of the dish-preparation comes to that of the human-chef is measured by how close the robotically-prepared dish resembles the human-prepared dish as measured by a variety of subjective elements, such as consistency, color, taste, etc. The notion is that, the more closely the dish prepared by the robotic kitchen is to that prepared by the human chef, the higher the fidelity of the replication process.
(117) Food Preparation Stage (also referred to as “Cooking stage”)—refers to a combination, either sequential or in parallel, of one or more mini-manipulations including action primitives, and computer instructions for controlling the various kitchen equipment and appliances in the standardized kitchen module; one or more food preparation stages collectively represent the entire food preparation process for a particular recipe.
(118) Geometric Reasoning—refers to a software-based computer program(s) capable of using two-dimensional (2D)/three-dimensional (3D) surface- and/or volumetric data to reason as to the actual shape and size of a particular volume; the ability to determine or utilize boundary information also allows for inferences as to the start end of a particular geometric element and the number present (in an image or model).
(119) Grasp Reasoning—refers to a software-based computer program(s) capable of relying on geometric and physical reasoning to plan a multi-contact (point/area/volume) contact-interaction between a robotic end-effector (gripper, link, etc.), or even tools/utensils held by the end-effector, so as to successfully and stably contact, grasp and hold the object in order to manipulate it in three-dimensional space.
(120) Hardware Automation Device—Fixed process device capable of executing pre-programmed steps in succession without the ability to modify any of them; such devices are used for repetitive motions that are not in need of any modulation.
(121) Ingredient management and manipulation—refers to defining each ingredient in detail (including size, shape, weight, dimensions, characteristics and properties), one or more real-time adjustments in the variables associated with the particular ingredient that may differ from the previous stored ingredient details (such as the size of a fish fillet, the dimensions of an egg, etc.), and the process in executing the different stages for the manipulation movements to an ingredient.
(122) Kitchen Module (or Kitchen Volume)—a standardized full kitchen module with standardized sets of kitchen equipment, standardized sets of kitchen tools, standardized sets of kitchen handles, and standardized sets of kitchen containers, with predefined space and dimensions for storing, accessing, and operating each kitchen element in the standardized full kitchen module. One objective of a kitchen module is to predefine as much of the kitchen equipment, tools, handles, containers, etc. as possible so as to provide a relatively fixed kitchen platform for the movements of robotic arms and hands. Both a chef in the chef kitchen studio and a person at home with a robotic kitchen (or a person at a restaurant) uses the standardized kitchen module so as to maximize the predictability of the kitchen hardware, while minimizing the risks of differentiations, variations and deviations between the chef kitchen studio and a home robotic kitchen. Different embodiments of the kitchen module are possible, including a standalone kitchen module and an integrated kitchen module. The integrated kitchen module is fitted into a conventional kitchen area of a typical house. The kitchen module operates in at least two modes, a robotic mode and a normal (manual) mode.
(123) Machine Learning—refers to the technology wherein a software component or program improves its performance based on experience and feedback. One kind of machine learning is reinforcement learning, often used in robotics, where desirable actions are rewarded and undesirable ones are penalized. Another kind is case-based learning, where previous solutions, e.g. sequences of actions by a human teacher or by the robot itself are remembered, together with any constraints or reasons for the solutions, and then are applied or reused in new settings. There are also additional kinds of machine learning, such as inductive and transductive methods.
(124) Mini-Manipulation—refers to a combination (or a sequence) of one or more steps that accomplish a basic functional outcome with a threshold value of the highest level of probability (examples of threshold value as within 0.1, 0.001, or 0.001 of the optimal value). Each step can be an action primitive or another (smaller) mini-manipulation, similar to a computer program comprised of basic coding steps and other computer programs that may stand alone or serve as sub-routines. For instance, a mini-manipulation can be grasping an egg, comprised of the motor actions required for reaching out a robotic arm moving the robotic fingers into the right configuration, and applying the correct delicate amount of force for grasping—all primitive actions. Another mini-manipulation can be breaking-an-egg-with-a-knife, including the grasping mini-manipulation, followed with one robotic hand, followed by grasping-a-knife mini-manipulation with the other hand, followed by the primitive action of striking the egg with the knife using a predetermined force.
(125) Model Elements and Classification—refers to one or more software-based computer program(s) capable of understanding elements in a scene as being items that are used or needed in different parts of a task; such as a bowl for mixing and the need for a spoon to stir, etc. Multiple elements in a scene or a world-model may be classified into groupings allowing for faster planning and task-execution.
(126) Motion Primitives—refers to motion actions that define different levels/domains of detailed action steps, e.g. a high level motion primitive would be to grab a cup, and a low level motion primitive would be to rotate a wrist by five degrees.
(127) Multimodal Sensing Unit—refers to a sensing unit comprised of multiple sensors capable of sensing and detection in multiple modes or electromagnetic bands or spectra, particularly capable of capturing three-dimensional position and/or motion information; the electromagnetic spectrum can range from low to high frequencies and need not be limited to that perceivable by a human being. Additional modes might include, but are not limited to, other physical senses such as touch, smell, etc.
(128) Number of Axes—three axes are required to reach any point in space. To fully control the orientation of the end of the arm (i.e. the wrist), three additional rotational axes (yaw, pitch, and roll) are required.
(129) Parameters—refers to variables that can take numerical values or ranges of numerical values. Three kinds of parameters are particularly relevant: parameters in the instructions to a robotic device (e.g. the force or distance in an arm movement), user settable parameters (e.g. prefers meat well done vs. medium), and chef-defined parameters (e.g. set oven temperature to 350 F).
(130) Parameter adjustment—refers to the process of changing the values of parameters based on inputs. For instance changes in the parameters of instructions to the robotic device can be based on the properties (e.g. size, shape, orientation) of, but not limited to, the ingredients, position/orientation of kitchen tools, equipment, appliances, speed, and time duration of a mini-manipulation.
(131) Payload or carrying capacity—refers to how much weight a robotic arm can carry and hold (or even accelerate) against the force of gravity, as a function of its endpoint location.
(132) Physical Reasoning—refers to a software-based computer program(s) capable of relying on geometrically-reasoned data and using physical information (density, texture, typical geometry and shape) to assist an inference-engine (program) to better model the object and also predict its behavior in the real world, particularly when grasped and/or manipulated/handled.
(133) Raw Data—refers to all measured and inferred sensory-data and representation information that is collected as part of the chef-studio recipe-generation process while watching/monitoring a human chef preparing a dish. Raw data can range from a simple data-point such as clock-time, to oven temperature (over time), camera-imagery, three-dimensional laser-generated scene representation data, to appliances/equipment used, tools employed, ingredients (type and amount) dispensed and when, etc. All the information the studio-kitchen collects from its built-in sensors and stores in raw, time-stamped form is considered raw data. Raw data is then used by other software processes to generate an even higher level of understanding and recipe-process understanding, turning raw data into additional time-stamped processed/interpreted data.
(134) Robotic Apparatus—refers the set of robotic sensors and effectors. The effectors comprise one or more robotic arms, and one or more robotic hands for operation in the standardized robotic kitchen. The sensors comprise cameras, range sensors, force sensors (haptic sensors) that transmit their information to the processor or set of processors that control the effectors.
(135) Recipe Cooking Process—refers to a robotic script containing abstract and detailed levels of instructions to a collection of programmable and hard automation devices, so as to allow computer-controllable devices to execute a sequenced operation within its environment (e.g. a kitchen replete with ingredients, tools, utensils and appliances).
(136) Recipe Script—refers to a recipe script as a sequence in time containing a structure and a list of commands and execution primitives (simple to complex command software) that, when executed by the robotic kitchen elements (robot-arm, automated equipment, appliances, tools, etc.) in a given sequence, should result in the proper replication and creation of the same dish as prepared by the human chef in the studio-kitchen. Such a script is sequential in time and equivalent to the sequence employed by the human chef to create the dish, albeit in a representation that is suitable and understandable by the computer-controlled elements in the robotic kitchen.
(137) Recipe Speed Execution—refers to managing a timeline in the execution of recipe steps in preparing a food dish by replicating a chef's movements, where the recipe steps include standardized food preparation operations (e.g., standardized cookware, standardized equipment, kitchen processors, etc.), mini-manipulations, and cooking of non-standardized objects.
(138) Repeatability—refers to an acceptable preset margin in how accurately the robotic arms/hands can repeatedly return to a programmed position. If the technical specification in a control memory requires the robotic hand to move to a certain X-Y-Z position and within +/−0.1 mm of that position, then the repeatability is measured for the robotic hands to return to within +/−0.1 mm of the taught and desired/commanded position.
(139) Robotic Recipe Script—refers to a computer-generated sequence of machine-understandable instructions related to the proper sequence of robotically/hard-automation execution of steps to mirror the required cooking steps in a recipe to arrive at the same end-product as if cooked by a chef.
(140) Robotic Costume—External instrumented device(s) or clothing, such as gloves, clothing with camera-trackable markers, jointed exoskeleton, etc., used in the chef studio to monitor and track the movements and activities of the chef during all aspects of the recipe cooking process(es).
(141) Scene Modeling—refers to a software-based computer program(s) capable of viewing a scene in one or more cameras' fields of view, and being capable of detecting and identifying objects of importance to a particular task. These objects may be pre-taught and/or be part of a computer library with known physical attributes and usage-intent.
(142) Smart Kitchen Cookware/Equipment—refers to an item of kitchen cookware (e.g., a pot or a pan) or an item of kitchen equipment (e.g., an oven, a grill, or a faucet) with one or more sensors that prepares a food dish based on one or more graphical curves (e.g., a temperature curve, a humidity curve, etc.).
(143) Software Abstraction Food Engine—refers to a software engine that is defined as a collection of software loops or programs, acting in concert to process input data and create a certain desirable set of output data to be used by other software engines or an end-user through some form of textual or graphical output interface. An abstraction software engine is a software program(s) focused on taking a large and vast amount of input data from a known source in a particular domain (such as three-dimensional range measurements that form a data-cloud of three-dimensional measurements as seen by one or more sensors), and then processing the data to arrive at interpretations of the data in a different domain (such as detecting and recognizing a table-surface in a data-cloud based on data having the same vertical data value, etc.), in order to identify, detect and classify data-readings as pertaining to an object in three-dimensional space (such as a table-top, cooking pot, etc.). The process of abstraction is basically defined as taking a large data set from one domain and inferring structure (such as geometry) in a higher level of space (abstracting data points), and then abstracting the inferences even further and identifying objects (pots, etc.) out of the abstracted data-sets to identify real-world elements in an image, which can then be used by other software engines to make additional decisions (handling/manipulation decisions for key objects, etc.). A synonym for “software abstraction engine” in this application could be also “software interpretation engine” or even “computer-software processing and interpretation algorithm”.
(144) Task Reasoning—refers to a software-based computer program(s) capable of analyzing a task-description and breaking it down into a sequence of multiple machine-executable (robot or hard-automation systems) steps so as to achieve a particular end result defined in the task description.
(145) Three-dimensional World Object Modeling and Understanding—refers to a software-based computer program(s) capable of using sensory data to create a time-varying three-dimensional model of all surfaces and volumes so as to enable it to detect, identify and classify objects within the same and understand their usage and intent.
(146) Torque vector—refers to the torsion force upon a robotic appendage including its direction and magnitude.
(147) Volumetric Object Inference (Engine)—refers to a software-based computer program(s) capable of using geometric data and edge-information as well as other sensory data (color, shape, texture, etc.) to allow for identification of three-dimensionality of one or more objects to aid in the object identification and classification process.
(148)
(149) The robotic food preparation software 14 includes the multimodal three-dimensional sensors 20, a capturing module 28, a calibration module 30, a conversion algorithm module 32, a replication module 34, a quality check module 36 with a three-dimensional vision system, a same result module 38, and a learning module 40. The capturing module 28 captures the movements of the chef as the chef prepares a food dish. The calibration module 30 calibrates the robotic arms 22 and robotic hands 24 before, during and after the cooking process. The conversion algorithm module 32 is configured to convert the recorded data from a chef's movements collected in the chef studio into recipe modified data (or transformed data) for use in a robotic kitchen where robotic hands replicate the food preparation of the chef's dish. The replication module 34 is configured to replicate the chef's movements in a robotic kitchen. The quality check module 36 is configured to perform quality check functions of a food dish prepared by the robotic kitchen during, prior to, or after the food preparation process. The same result module 38 is configured to determine whether the food dish prepared by a pair of robotic arms and hands in the robotic kitchen would taste the same or substantially the same as if prepared by the chef. The learning module 40 is configured to provide learning capabilities to the computer 16 that operates the robotic arms and hands.
(150)
(151) The standardized robotic kitchen 50 is designed for detecting, recording and emulating a chef's cooking movements, controlling significant parameters such as temperature over time, and process execution at robotic kitchen stations with designated appliances, equipment and tools. The chef kitchen 44 provides a computing kitchen environment 16 with gloves with sensors or a costume with sensors for recording and capturing a chef's 50 movements in the food preparation for a specific recipe. Upon recording the movements and recipe process of the chef 49 for a particular dish into a software recipe file in memory 52, the software recipe file is transferred from the chef kitchen 44 to the robotic kitchen 48 via a communication network 46, including a wireless network and/or a wired network connected to the Internet, so that the user (optional) 60 can purchase one or more software recipe files or the user can be subscribed to the chef kitchen 44 as a member that receives new software recipe files or periodic updates of existing software recipe files. The household robotic kitchen system 48 serves as a robotic computing kitchen environment at residential homes, restaurants, and other places in which the kitchen is built for the user 60 to prepare food. The household robotic kitchen system 48 includes the robotic cooking engine 56 with one or more robotic arms and hard-automation devices for replicating the chef's cooking actions, processes and movements based on a received software recipe file from the chef studio system 44.
(152) The chef studio 44 and the robotic kitchen 48 represent an intricately linked teach-playback system, which has multiple levels of fidelity of execution. While the chef studio 44 generates a high-fidelity process model of how to prepare a professionally cooked dish, the robotic kitchen 48 is the execution/replication engine/process for the recipe-script created through the chef working in the chef studio. Standardization of a robotic kitchen module is a means to increase performance fidelity and success/guarantee.
(153) The varying levels of fidelity for recipe-execution depend on the correlation of sensors and equipment (besides of course the ingredients) between those in the chef studio 44 and that in the robotic kitchen 48. Fidelity can be defined as a dish tasting identical to that prepared by a human chef (indistinguishably so) at one of the (perfect replication/execution) ends of the spectrum, while at the opposite end the dish could have one or more substantial or fatal flaws with implications to quality (overcooked meat or pasta), taste (burnt elements), edibility (incorrect consistency) or even health-implications (undercooked meat such as chicken/pork with salmonella exposure, etc.).
(154) A robotic kitchen that has identical hardware and sensors and actuation systems that can replicate the movements and processes akin to those by the chef that were recorded during the chef-studio cooking process is more likely to result in a higher fidelity outcome. The implication here is that the setups need to be identical, which has a cost and volume implication. The robotic kitchen 48 can however still be implemented using more standardized non-computer-controlled or computer-monitored elements (pots with sensors, networked appliances such as ovens, etc.), requiring more sensor-based understanding to allow for more complex execution monitoring. Since uncertainty has now increased as to key elements (correct amount of ingredients, cooking temperatures, etc.) and processes (use of stirrer/masher in case a blender is not available in a robotic home kitchen), the guarantees of having an identical outcome to that from the chef will undoubtedly be lower.
(155) An emphasis in the present disclosure is that the notion of a chef studio 44 coupled with a robotic kitchen is a generic concept. The level of the robotic kitchen 48 is variable all the way from a home-kitchen outfitted with a set of arms and environmental sensors, all the way to an identical replica of the studio-kitchen, where a set of arms and articulated motions, tools and appliances and ingredient-supply can replicate the chef's recipe in an almost identical fashion. The only variable to contend with will be the quality-degree of the end-result or dish in terms of quality, looks, taste, edibility and health.
(156) A potential method to mathematically describe this correlation between the recipe-outcome and the input variables in the robotic kitchen can best be described by the function below:
F.sub.recipe-outcome=F.sub.studio(I,E,P,M,V)+F.sub.RobKit(E.sub.f,I,R.sub.e,P.sub.mf) where F.sub.studio=Recipe Script Fidelity of Chef-Studio F.sub.RobKit=Recipe Script Execution by Robotic Kitchen I=Ingredients E=Equipment P=Processes M=Methods V=Variables (Temperature, Time, Pressure, etc.) E.sub.f=Equipment Fidelity R.sub.e=Replication Fidelity P.sub.mf=Process Monitoring Fidelity
(157) The above equation relates the degree to which the outcome of a robotically-prepared recipe matches that a human chef would prepare and serve (F.sub.recipe-outcome) to the level that the recipe was properly captured and represented by the chef studio 44 (F.sub.studio) based on the ingredients (I) used, the equipment (E) available to execute the chef's processes (P) and methods (M) by properly capturing all the key variables (V) during the cooking process; and how the robotic kitchen is able to represent the replication/execution process of the robotic recipe script by a function (F.sub.RobKit) that is primarily driven by the use of the proper ingredients (I), the level of equipment fidelity (E.sub.f) in the robotic kitchen compared to that in the chef studio, the level to which the recipe-script can be replicated (R.sub.e) in the robotic kitchen, and to what extent there is an ability and need to monitor and execute corrective actions to achieve the highest process monitoring fidelity (P.sub.mf) possible.
(158) The functions (F.sub.studio) and (F.sub.RobKit) can be any combination of linear or non-linear functional formulas with constants, variables and any form of algorithmic relationships. An example for such algebraic representations for both functions could be.
F.sub.studio=I(fct. sin(Temp))+E(fct. Cooptop1*5)+P(fct. Circle(spoon)+V(fct. 0.5*time)
(159) Delineating that the fidelity of the preparation process is related to the temperature of the ingredient which varies over time in the refrigerator as a sinusoidal function, the speed with which an ingredient can be heated on the cooktop on specific station at a particular multiplicative rate, and related to how well a spoon can be moved in a circular path of a certain amplitude and period, and that the process needs to be carried out at no less than ½ the speed of the human chef for the fidelity of the preparation process to be maintained.
F.sub.RobKit=E.sub.f,(Cooktop2,Size)+I(1.25*Size+Linear(Temp))+R.sub.e(Motion-Profile)+P.sub.mf(Sensor-Suite Correspondence)
(160) Delineating that the fidelity of the replication process in the robotic kitchen is related to the appliance type and layout for a particular cooking-area and the size of the heating-element, the size and temperature profile of the ingredient being seared and cooked (thicker steak requiring more cooking time), while also preserving the motion-profile of any stirring and bathing motions of a particular step like searing or mousse-beating, and whether the correspondence between sensors in the robotic kitchen and the chef-studio is sufficiently high to trust the monitored sensor data to be accurate and detailed enough to provide a proper monitoring fidelity of the cooking process in the robotic kitchen during all steps in a recipe.
(161) The outcome of a recipe is not only a function of what fidelity the human chef's cooking steps/methods/process/skills were captured with by the chef studio, but also with what fidelity these can be executed by the robotic kitchen, where each of them has key elements that impact their respective subsystem performance.
(162)
(163) The standardized hard automation dispenser(s) 82 is a device or a series of devices that is/are programmable and/or controllable via the cooking computer 16 to feed or provide pre-packaged (known) amounts or dedicated feeds of key materials for the cooking process, such as spices (salt, pepper, etc.), liquids (water, oil, etc.) or other dry materials (flour, sugar, etc.). The standardized hard automation dispensers 82 may be located at a specific station or be able to be robotically accessed and triggered to dispense according to the recipe sequence. In other embodiments, a robotic hard automation module may be combined or sequenced in series or parallel with other such modules or robotic arms or cooking utensils. In this embodiment, the standardized robotic kitchen 50 includes robotic arms 70 and robotic hands 72 and robotic hands as controlled by the robotic food preparation engine 56 in accordance with a software recipe file stored in the memory 52 for replicating a chef's precise movements in preparing a dish to produce the same tasting dish as if the chef had prepared it himself or herself. The three-dimensional vision sensors 66 provide capability to enable three-dimensional modeling of objects, providing a visual three-dimensional model of the kitchen activities, and scanning the kitchen volume to assess the dimensions and objects within the standardized robotic kitchen 50. The retractable safety glass 68 comprises a transparent material on the robotic kitchen 50, which when in an ON state extends the safety glass around the robotic kitchen to protect surrounding human beings from the movements of robotic arms 70 and hands 72, hot water and other liquids, steam, fire and other dangers influents. The robotic food preparation engine 56 is communicatively coupled to an electronic memory 52 for retrieving a software recipe file previously sent from the chef studio system 44 for which the robotic food preparation engine 56 is configured to execute processes in preparing and replicating the cooking method and processes of a chef as indicated in the software recipe file. The combination of robotic arms 70 and robotic hands 72 serves to replicate the precise movements of the chef in preparing a dish so that the resulting food dish will taste identical (or substantially identical) to the same food dish prepared by the chef. The standardized cooking equipment 74 includes an assortment of cooking appliances 46 that are incorporated as part of the robotic kitchen 50, including, but not limited to, a stove/induction/cooktop (electric cooktop, gas cooktop, induction cooktop), an oven, a grill, a cooking steamer, and a microwave oven. The standardized cookware and sensors 76 are used as embodiments for the recording of food preparation steps based on the sensors on the cookware and cooking a food dish based on the cookware with sensors, which include a pot with sensors, a pan with sensors, an oven with sensors, and a charcoal grill with sensors. The standardized cookware 78 includes frying pans, sauté pans, grill pans, multi-pots, roasters, woks, and braisers. The robotic arms 70 and the robotic hands 72 operate the standardized handles and utensils 80 in the cooking process. In one embodiment, one of the robotic hands 72 is fitted with a standardized handle, which is attached to a fork head, a knife head, and a spoon head for selection as required. The standardized hard automation dispensers 82 are incorporated into the robotic kitchen 50 to provide for expedient (via both robot arms 70 and human use) key and common/repetitive ingredients that are easily measured/dosed out or pre-packaged. The standardized containers 86 are storage locations that store food at room temperature. The standardized refrigerator containers 88 refer to, but are not limited to, a refrigerator with identified containers for storing fish, meat, vegetables, fruit, milk, and other perishable items. The containers in the standardized containers 86 or standardized storages 88 can be coded with container identifiers from which the robotic food preparation engine 56 is able to ascertain the type of food in a container based on the container identifier. The standardized containers 86 provide storage space for non-perishable food items such as salt, pepper, sugar, oil, and other spices. Standardized cookware with sensors 76 and the cookware 78 may be stored on a shelf or a cabinet for use by the robotic arms 70 for selecting a cooking tool to prepare a dish. Typically, the raw fish, the raw meat, and vegetables are pre-cut and stored in the identified standardized storages 88. The kitchen countertop 90 provides a platform for the robotic arms 70 to handle the meat or vegetables as needed, which may or may not include cutting or chopping actions. The kitchen faucet 92 provides a kitchen sink space for washing or cleaning food in preparation for a dish. When the robotic arms 70 have completed the recipe process to prepare a dish and the dish is ready for serving, the dish is placed on a serving counter 90, which further allows for the dining environment to be enhanced by adjusting the ambient setting with the robotic arms 70, such as placement of utensils, wine glasses, and a chosen wine compatible with the meal. One embodiment of the equipment in the standardized robotic kitchen module 50 is a professional series as to increase the universal appeal to prepare various types of dishes.
(164) The standardized robotic kitchen module 50 has as one objective the standardization of the kitchen module 50 and various components with the kitchen module itself, to ensure consistency in both the chef kitchen 44 and the robotic kitchen 48 to maximize the preciseness of recipe replication while minimizing the risks of deviations from precise replication of a recipe dish between the chef kitchen 44 and the robotic kitchen 48. One main purpose of having the standardization of the kitchen module 50 is to obtain the same result of the cooking process (or the same dish) between a first food dish prepared by the chef and a subsequent replication of the same recipe process via the robotic kitchen. Conceiving a standardized platform in the standardized robotic kitchen module 50 between the chef kitchen 44 and the robotic kitchen 48 has several key considerations: same timeline, same program or mode, and quality check. The same timeline in the standardized robotic kitchen 50 where the chef prepares a food dish at the chef kitchen 44 and the replication process by the robotic hands in the robotic kitchen 48 refers to the same sequence of manipulations, the same initial and ending time of each manipulation, and the same speed of moving an object between handling operations. The same program or mode in the standardized robotic kitchen 50 refers to the use and operation of standardized equipment during each manipulation recording and execution step. The quality check refers to three-dimensional vision sensors in the standardized robotic kitchen 50 which monitor and adjust in real time each manipulation action during the food preparation process to correct any deviation and avoid a flawed result. The adoption of the standardized robotic kitchen module 50 reduces and minimizes the risks of not obtaining the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen using robotic arms and hands. Without the standardization of a robotic kitchen module and the components within the robotic kitchen module, the increased variations between the chef kitchen 44 and the robotic kitchen 48 increase the risks of not being able to obtain the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen because more elaborate and complex adjustment algorithms will be required with different kitchen modules, different kitchen equipment, different kitchenware, different kitchen tools, and different ingredients between the chef kitchen 44 and the robotic kitchen 48.
(165) The standardized robotic kitchen module 50 includes standardization of many aspects. First, the standardized robotic kitchen module 50 includes standardized positions and orientations (in the XYZ coordinate plane) of any type of kitchenware, kitchen containers, kitchen tools and kitchen equipment (with standardized fixed holes in the kitchen module and device positions). Secondly, the standardized robotic kitchen module 50 includes a standardized cooking volume dimension and architecture. Thirdly, the standardized robotic kitchen module 50 includes standardized equipment sets, such as an oven, a stove, a dish washer, a faucet, etc. Fourth, the standardized robotic kitchen module 50 includes standardized kitchenware, standardized cooking tools, standardized cooking devices, standardized containers, and standardized food storage in a refrigerator, in terms of shape, dimension, structure, material, capabilities, etc. Fifth, in one embodiment, the standardized robotic kitchen module 50 includes a standardized universal handle for handling any kitchenware, tools, instruments, containers, and equipment, which enable a robotic hand to hold the standardized universal handle in only one correct position, while avoiding any improper grasps or incorrect orientations. Sixth, the standardized robotic kitchen module 50 includes standardized robotic arms and hands with a library of manipulations. Seventh, the standardized robotic kitchen module 50 includes a standardized kitchen processor for standardized ingredient manipulations. Eighth, the standardized robotic kitchen module 50 includes standardized three-dimensional vision devices for creating dynamic three-dimensional vision data, as well as other possible standard sensors, for recipe recording, execution tracking, and quality check functions. Ninth, the standardized robotic kitchen module 50 includes standardized types, standardized volumes, standardized sizes, and standardized weights for each ingredient during a particular recipe execution.
(166)
(167) The input module 50 is configured to receive any type of input information such as software recipe files sent from another computing device. The calibration module 94 is configured to calibrate itself with the robotic arms 70, the robotic hands 72, and other kitchenware and equipment components within the standardized robotic kitchen module 50. The quality check module 96 is configured to determine the quality and freshness of raw meat, raw vegetables, milk-associated ingredients and other raw foods at the time that the raw food is retrieved for cooking, as well as checking the quality of raw foods when receiving the food into the standardized food storage 88. The quality check module 96 can also be configured to conduct quality testing of an object based on senses, such as the smell of the food, the color of the food, the taste of the food, and the image or appearance of the food. The chef movements recording module 98 is configured to record the sequence and the precise movements of the chef when the chef prepares a food dish. The cookware sensor data recording module 100 is configured to record sensory data from cookware equipped with sensors (such as a pan with sensors, a grill with sensors, or an oven with sensors) placed in different zones within the cookware, thereby producing one or more sensory curves. The result is the generation of a sensory curve, such as temperature curve (and/or humidity), that reflects the temperature fluctuation of cooking appliances over time for a particular dish. The memory module 102 is configured as a storage location for storing software recipe files, for either replication of chef recipe movements or other types of software recipe files including sensory data curves. The recipe abstraction module 104 is configured to use recorded sensor data to generate machine-module specific sequenced operation profiles. The chef movements replication module 106 is configured to replicate the chef's precise movements in preparing a dish based on the stored software recipe file in the memory 52. The cookware sensory replication module 108 is configured to replicate the preparation of a food dish by following the characteristics of one or more previously recorded sensory curves which was generated when the chef 49 prepared a dish by using the standardized cookware with sensors 76. The robotic cooking module 110 is configured to control and operate standardized kitchen operations, mini-manipulations, non-standardized objects, and the various kitchen tools and equipment in the standardized robotic kitchen 50. The real time adjustment module 112 is configured to provide real-time adjustments to the variables associated with a particular kitchen operation or a mini operation so as to produce a resulting process that is a precise replication of the chef movement or a precise replication of the sensory curve. The learning module 114 is configured to provide learning capabilities to the robotic cooking engine 56 to optimize the precise replication in preparing a food dish by robotic arms 70 and the robotic hands 72, as if the food dish was prepared by a chef, using a method such as case-based (robotic) learning. The mini-manipulation library database module 116 is configured to store
(168) a first database library of mini-manipulations. The standardized kitchen operation library database module 117 is configured to store a second database library of standardized kitchenware and how to operate this standardized kitchenware. The output module 118 is configured to send output computer files or control signals external to the robotic cooking engine.
(169)
(170) These individual software modules generate such information (but are not thereby limited to only these modules) as (i) chef-location and cooking-station ID via a location and configuration module 152, (ii) configuration of arms (via torso), (iii) tools handled and when and how, (iv) utensils used and locations on the station through the hardware and variable abstraction module 154, (v) processes executed with them and (vi) variables (temperature, lid y/n, stirring, etc.) in need of monitoring through the process module B156, (vii) temporal (start/finish, type) distribution and (viii) types of processes (stir, fold, etc.) being applied, and (ix) ingredients added (type, amount, state of prep, etc.), through the cooking sequence and process abstraction module 158.
(171) All this information is then used to create a machine-specific (not just for the robotic-arms, but also ingredient dispensers, tools and utensils, etc.) set of recipe instructions through the stand-alone module 160, which are organized as a script of sequential/parallel overlapping tasks to be executed and monitored. This recipe-script is stored (162) alongside the entire raw data set (164) in the data storage module 166 and is made accessible to either a remote robotic cooking station through the robotic kitchen interface module 168 or a human user 170 via a graphical user interface (GUI) 172.
(172)
(173) The robotic kitchen 48 engages in a recipe replication process 106, whose profile depends on whether the kitchen is of a standardized or non-standardized type, which is checked by a process 186.
(174) The robotic kitchen execution is dependent on the type of kitchen available to the user. If the robotic kitchen uses the same/identical (at least functionally) equipment as used in the in the chef studio, the recipe replication process is primarily one of using the raw data and playing it back as part of the recipe-script execution process. Should the kitchen however differ from the (ideal) standardized kitchen, the execution engine(s) will have to rely on the abstracted data to generate kitchen-specific execution sequences to try to achieve a similar step-by-step result.
(175) Since the cooking process is continually monitored by all sensor units in the robotic kitchen via a monitoring process 194, regardless of whether the known studio equipment 196 or the mixed/atypical non-chef studio equipment 198 is being used, the system is able to make modifications as needed depending on a recipe progress check 200. In one embodiment of the standardized kitchen, raw data is typically played back through an execution module 188 using chef-studio type equipment, and the only adjustments that are expected are adaptations 202 in the execution of the script (repeat a certain step, go back to a certain step, slow down the execution, etc.) as there is a one-to-one correspondence between taught and played-back data-sets. However, in the case of the non-standardized kitchen, the chances are very high that the system will have to modify and adapt the actual recipe itself and its execution via a recipe script modification module 204, to suit the available tools/appliances 192 which differ from those in the chef studio 44 or the measured deviations from the recipe script (meat cooking too slowly, hot-spots in pot burning the roux, etc.). Overall recipe-script progress is monitored using a similar process 206, which differs depending on whether chef-studio equipment 208 or mixed/atypical kitchen equipment 210 is being used.
(176) A non-standardized kitchen is less likely to result in a close-to-human chef cooked dish, as compared to using a standardized robotic kitchen that has equipment and capabilities reflective of those used in the studio-kitchen. The ultimate subjective decision is of course that of the human (or chef) tasting, which is a quality evaluation 212, yielding to a (subjective) quality decision 214.
(177)
(178) A data process-mapping algorithm 220 uses the simpler (typically single-unit) variables to determine where the process action is taking place (cooktop and/or oven, fridge, etc.) and assigns a usage tag to any item/appliance/equipment being used whether intermittently or continuously. It associates a cooking step (baking, grilling, ingredient-addition, etc.) to a specific time-period and tracks when, where and which and how much of what ingredient was added. This (time-stamped) information dataset is then made available for the data-melding process during the recipe-script generation process 222.
(179) The data extraction and mapping process 224 is primarily focused on taking two-dimensional information (such as from monocular/single-lensed cameras) and extracting key information from the same. In order to extract the important and more abstracted descriptive information from each successive image, several algorithmic processes have to be applied to this dataset. Such processing steps can include (but are not limited to) edge-detection, color and texture-mapping, and then using the domain-knowledge in the image, coupled with object-matching information (type and size) extracted from the data reduction and abstraction process 226, to allow for the identification and location of the object (whether an item of equipment or ingredient, etc.), again extracted from the data reduction and abstraction process 226, allowing one to associate the state (and all associated variables describing the same) and items in an image with a particular process-step (frying, boiling, cutting, etc.). Once this data has been extracted and associated with a particular image at a particular point in time, it can be passed to the recipe-script generation process 222 to formulate the sequence and steps within a recipe.
(180) The data-reduction and abstraction engine (set of software routines) 226 is intended to reduce the larger three-dimensional data sets and extract from them key geometric and associative information. A first step is to extract from the large three-dimensional data point-cloud only the specific workspace area of importance to the recipe at that particular point in time. Once the data-set has been trimmed, key geometric features will be identified by a process known as template matching; this allows for the identification of such items as horizontal table-tops, cylindrical pots and pans, arm and hand locations, etc. Once typical known (template) geometric entities are determined in a data-set a process of object identification and matching proceeds to differentiate all items (pot vs. pan, etc.) and associates the proper dimensionality (size of pot or pan, etc.) and orientation of the same, and places them within the three-dimensional world model being assembled by the computer. All this abstracted/extracted information is then also shared with the data-extraction and mapping engine 224, prior to all being fed to the recipe-script generation engine 222.
(181) The recipe-script generation engine process 222 is responsible for melding (blending/combining) all the available data and sets into a structured and sequential cooking script with clear process-identifiers (prepping, blanching, frying, washing, plating, etc.) and process-specific steps within each, which can then be translated into robotic-kitchen machine-executable command-scripts that are synchronized based on process-completion and overall cooking time and cooking progress. Data melding will at least involve, but will not solely be limited to, the ability to take each (cooking) process step and populating the sequence of steps to be executed with the properly associated elements (ingredients, equipment, etc.), methods and processes to be used during the process steps, and the associated key control—(set oven/cooktop temperatures/settings) and monitoring-variables (water or meat temperature, etc.) to be maintained and checked to verify proper progress and execution. The melded data is then combined into a structured sequential cooking script that will resemble a set of minimally descriptive steps (akin to a recipe in a magazine) but with a much larger set of variables associated with each element (equipment, ingredient, process, method, variable, etc.) of the cooking process at any one point in the procedure. The final step is to take this sequential cooking script and transform it into an identically structured sequential script that is translatable by a set of machines/robot/equipment within a robotic kitchen 48. It is this script the robotic kitchen 48 uses to execute the automated recipe execution and monitoring steps.
(182) All raw (unprocessed) and processed data as well as the associated scripts (both structure sequential cooking-sequence script and the machine-executable cooking-sequence script) are stored in the data and profile storage unit/process 228 and time-stamped. It is from this database that the user, by way of a GUI, can select and cause the robotic kitchen to execute a desired recipe through the automated execution and monitoring engine 230, which is continually monitored by its own internal automated cooking process, with necessary adaptations and modifications to the script generated by the same and implemented by the robotic-kitchen elements, in order to arrive at a completely plated and served dish.
(183)
(184) The mini-manipulation library is a command-software repository, where motion behaviors and processes are stored based on an off-line learning process, where the arm/wrist/finger motions and sequences to successfully complete a particular abstract task (grab the knife and then slice; grab the spoon and then stir; grab the pot with one hand and then use other hand to grab spatula and get under meat and flip it inside the pan; etc.). This repository has been built up to contain the learned sequences of successful sensor-driven motion-profiles and sequenced behaviors for the hand/wrist (and sometimes also arm-position corrections), to ensure successful completions of object (appliance, equipment, tools) and ingredient manipulation tasks that are described in a more abstract language, such as “grab the knife and slice the vegetable”, “crack the egg into the bowl”, “flip the meat over in the pan”, etc. The learning process is iterative and is based on multiple trials of a chef-taught motion-profile from the chef studio, which is then executed and iteratively modified by the offline learning algorithm module, until an acceptable execution-sequence can be shown to have been achieved. The mini-manipulation library (command software repository) is intended to have been populated (a-priori and offline) with all the necessary elements to allow the robotic-kitchen system to successfully interact with all equipment (appliances, tools, etc.) and main ingredients that require processing (steps beyond just dispensing) during the cooking process. While the human chef wore gloves with embedded haptic sensors (proximity, touch, contact-location/-force) for the fingers and palm, the robotic hands are outfitted with similar sensor-types in locations to allow their data to be used to create, modify and adapt motion-profiles to successfully execute desired motion-profiles and handling-commands.
(185) The object-manipulation portion of the robotic-kitchen cooking process (robotic recipe-script execution software module for the interactive manipulation and handling of objects in the kitchen environment) 252 is further elaborated below. Using the robotic recipe-script database 254 (which contains data in raw, abstracted cooking-sequence and machine-executable script forms), the recipe script executor module 256 steps through a specific recipe execution-step. The configuration playback module 258 selects and passes configuration commands through to the robot arm system (torso, arm, wrist and hands) controller 270, which then controls the physical system to emulate the required configuration (joint-positions/-velocities/-torques, etc.) values.
(186) The notion of being able to faithfully carry out proper environment interaction manipulation and handling tasks is made possible through a real-time process-verification by way of (i) 3D world modeling as well as (ii) mini-manipulation. Both the verification and manipulation steps are carried out through the addition of the robot wrist and hand configuration modifier 260. This software module uses data from the 3D world configuration modeler 262, which creates a new 3D world model at every sampling step from sensory data supplied by the multimodal sensor(s) unit(s), in order to ascertain that the configuration of the robotic kitchen systems and process matches that required by the recipe script (database); if not, it enacts modifications to the commanded system-configuration values to ensure the task is completed successfully. Furthermore, the robot wrist and hand configuration modifier 260 also uses configuration-modifying input commands from the mini-manipulation motion profile executor 264. The hand/wrist (and potentially also arm) configuration modification data fed to the configuration modifier 260 are based on the mini-manipulation motion profile executor 264 knowing what the desired configuration playback should be from 258, but then modifying it based on its 3D object model library 266 and the a-priori learned (and stored) data from the configuration and sequencing library 268 (which was built based on multiple iterative learning steps for all main object handling and processing steps).
(187) While the configuration modifier 260 continually feeds modified commanded configuration data to the robot arm system controller 270, it relies on the handling/manipulation verification software module 272 to verify not only that the operation is proceeding properly but also whether continued manipulation/handling is necessary. In the case of the latter (answer ‘N’ to the decision), the configuration modifier 260 re-requests configuration-modification (for the wrist, hands/fingers and potentially the arm and possibly even torso) updates from both the world modeler 262 and the mini-manipulation profile executor 264. The goal is simply to verify that a successful manipulation/handling step or sequence has been successfully completed. The handling/manipulation verification software module 272 carries out this check by using the knowledge of the recipe script database F2 and the 3D world configuration modeler 262 to verify the appropriate progress in the cooking step currently being commanded by the recipe script executor 256. Once progress has been deemed successful, the recipe script index increment process 274 notifies the recipe script executor 256 to proceed to the next step in the recipe-script execution.
(188)
(189) The multimodal sensor-unit(s) 302, comprising, but not limited to, video cameras 304, IR cameras and rangefinders 306, stereo (or even trinocular) camera(s) 308 and multi-dimensional scanning lasers 310, provide multi-spectral sensory data to the main software abstraction engines 312 (after being acquired & filtered in the data acquisition and filtering module 314). The data is used in a scene understanding module 316 to carry out multiple steps such as (but not limited to) building high- and lower-resolution (laser: high-resolution; stereo-camera: lower-resolution) three-dimensional surface volumes of the scene, with superimposed visual and IR-spectrum color and texture video information, allowing edge-detection and volumetric object-detection algorithms to infer what elements are in a scene, allowing the use of shape-/color-/texture- and consistency-mapping algorithms to run on the processed data to feed processed information to the Kitchen Cooking Process Equipment Handling Module 318. In the module 318, software-based engines are used for the purpose of identifying and three-dimensionally locating the position and orientation of kitchen tools and utensils and identifying and tagging recognizable food elements (meat, carrots, sauce, liquids, etc.) so as to generate data to let the computer build and understand the complete scene at a particular point in time so as to be used for next-step planning and process monitoring. Engines required to achieve such data and information abstraction include, but are not limited to, grasp reasoning engines, geometry reasoning engines, physical reasoning engines and task reasoning engines. Output data from both engines 316 and 318 are then used to feed the scene modeler and content classifier 320, where the 3D world model is created with all the key content required for executing the robotic cooking script executor. Once the fully-populated model of the world is understood, it can be used to feed the motion and handling planner 322 (if robotic-arm grasping and handling are necessary, the same data can be used to differentiate and plan for grasping and manipulating food and kitchen items depending on the required grip and placement) to allow for planning motions and trajectories for the arm(s) and attached end-effector(s) (grippers, multi-fingered hands). A follow-on Execution Sequence planner 324 creates the proper sequencing of task-based commands for all individual robotic/automated kitchen elements, which are then used by the robotic kitchen actuation systems 326. The entire sequence above is repeated in a continuous closed loop during the robotic recipe-script execution and monitoring phase.
(190)
(191)
(192)
(193)
(194)
(195)
(196) In some embodiments a chef performs the same food preparation operation multiple times, yielding values of the sensor reading, and parameters in the corresponding robotic instructions that vary somewhat from one time to the next. The set of sensor readings for each sensor across multiple repetitions of the preparation of the same food dish provides a distribution with a mean, standard deviation and minimum and maximum values. The corresponding variations on the robotic instructions (also called the effector parameters) across multiple executions of the same food dish by the chef also define distributions with mean, standard deviation, minimum and maximum values. These distributions may be used to determine the fidelity (or accuracy) of subsequent robotic food preparations.
(197) In one embodiment the estimated average accuracy of a robotic food preparation operation is given by:
(198)
(199) Where C represents the set of Chef parameters (1.sup.st through n.sup.th) and R represents the set of Robotic Apparatus parameters (correspondingly (1.sup.st through n.sup.th). The numerator in the sum represents the difference between robotic and chef parameters (i.e. the error) and the denominator normalizes for the maximal difference). The sum gives the total normalized cumulative error
(200)
and multiplying by 1/n gives the average error. The complement of the average error corresponds to the average accuracy.
(201) Another version of the accuracy calculation weighs the parameters for importance, where each coefficient (each α.sub.i) represents the importance of the i.sup.th parameter, the normalized cumulative error is
(202)
and the estimated average accuracy is given by:
(203)
(204)
(205)
(206) In order to operate a mechanical robotic mechanism such as the ones described in the embodiments of this invention, a skilled artisan realizes that many mechanical and control problems need to be addressed, and the literature in robotics describes methods to do just that. The establishment of static and/or dynamic stability in a robotics system is an important consideration. Especially for robotic manipulation, dynamic stability is a strongly desired property, in order to prevent accidental breakage or movements beyond those desired or programmed. Dynamic stability is illustrated in
(207) The cited literature addresses conditions for dynamic stability that are imported by reference into the present invention to enable proper functioning of the robotic arms. These conditions include the fundamental principle for calculating torque to the joints of a robotic arm:
(208)
(209) where T is the torque vector (T has n components, each corresponding to a degree of freedom of the robotic arm), M is the inertial matrix of the system (M is a positive semi-definite n-by-n matrix), C is a combination of centripetal and centrifugal forces, also an n-by-n matrix, G(q) is the gravity vector, and q is the position vector. And they include finding stable points and minima, e.g. via the LaGrange equation if the robotic positions (x's) can be described by twice-differentiable functions (y's).
(210) In order for the system comprised of the robotic arms and hands/grippers to be stable, it is important that the system be properly designed and built and have an appropriate sensing and control system which operates within the boundary of acceptable performance. The reason that this is important is that one wants to achieve the best (highest speed with highest position/velocity and force/torque tracking and all under stable conditions) performance possible given the physical system and what its controller is asking it to do.
(211) When one speaks of proper design, the notion is one of achieving proper observability and controllability of the system. Observability implies that the key variables of the system (joint/finger positions and velocities, forces and torques) are measurable by the system, which implies one needs to have the ability to sense these variables, which in turn implies the presence and use of the proper sensing devices (internal or external). Controllability implies that one (computer in this case) have the ability to shape or control the key axes of the system based on observed parameters from internal/external sensors; this usually implies an actuator or direct/indirect control over a certain parameter by way of a motor or other computer-controlled actuation system. The ability to make the system as linear in its response as possible, thereby negating the detrimental effects of nonlinearities (stiction, backlash, hysteresis, etc.), allows for control schemes like PID gain-scheduling and nonlinear controllers like sliding-mode control to guarantee system stability and performance even in the light of system-modeling uncertainties (errors in mass/inertia estimates, dimensional geometry discretization, sensor/torque discretization anomalies, etc.) which are always present in any higher-performance control system.
(212) Furthermore, the use of a proper computing and sampling system is significant, as the system's ability to follow rapid motions with a certain maximum frequency content is clearly related to what control bandwidth (closed-loop sampling rate of the computer control system) the entire system is able to achieve and thus the frequency-response (ability to track motions of certain speeds and motion-frequency content) the system is able to exhibit.
(213) All the above characteristics are significant when it comes to ensuring that a highly redundant system can actually carry out the complex and dexterous tasks a human chef requires for a successful recipe-script execution, in both a dynamic and a stable fashion.
(214) Machine learning in the context of robotic manipulation of relevance to the invention can involve well known methods for parameter adjustment, such as reinforcement learning. An alternate and preferred embodiment for this invention is a different and more appropriate learning technique for repetitive complex actions such as preparing and cooking a meal with multiple steps over time, namely case-based learning. Case-based reasoning, also known as analogical reasoning, has been developed over time.
(215) As a general overview, case-based reasoning comprises the following steps:
(216) A. Constructing and remembering cases. A case is a sequence of actions with parameters that are successfully carried out to achieve an objective. The parameters include distances, forces, directions, positions, and other physical or electronic measures whose values are required to successfully carry out the task (e.g. a cooking operation). First, 1. storing aspects of the problem that was just solved together with: 2. the method(s) and optionally intermediate steps to solve the problem and its parameter values, and 3. (typically) storing the final outcome.
B. Applying cases (at a later point of time) 4. Retrieving one or more stored cases whose problems bear strong similarity to the new problem, 5. Optionally adjusting the parameters from the retrieved case(s) to apply to the current case (e.g. an item may weigh somewhat more, and hence a somewhat stronger force is needed to lift it), 6. Using the same methods and steps from the case(s) with the adjusted parameters (if needed) at least in part to solve the new problem.
Hence, case-based reasoning consists of remembering solutions to past problems and applying them with possible parametric modification to new very similar problems. However, in order to apply case-based reasoning to the robotic manipulation challenge, something more is needed. Variation in one parameter of the solution plan will cause variation in one or more coupled parameters. This requires transformation of the problem solution, not just application. We call the new process case-based robotic learning since it generalizes the solution to a family of close solutions (those corresponding to small variations in the input parameters—such as exact weight, shape and location of the input ingredients). Case-based robotic learning operates as follows:
C. Constructing, remembering and transforming robotic manipulation cases 1. Storing aspects of the problem that was just solved together with: 2. The value of the parameters (e.g. the inertial matrix, forces, etc. from equation 1), 3. Perform perturbation analysis by varying the parameter(s) pertinent to the domain (e.g. in cooking, vary the weight of the materials or their exact starting position), to see how much parameter values can vary and still obtain the desired results, 4. Via perturbation analysis on the model, record which other parameter values will change (e.g. forces) and by how much they should change, and 5. If the changes are within operating specification of the robotic apparatus, store the transformed solution plan (with the dependencies among parameters and projected change calculations for their values).
D. Applying cases (at a later point of time) 6. Retrieve one or more stored cases with the transformed exact values (now ranges, or calculations for new values depending on values of the input parameters), but still whose initial problems bear strong similarity to the new problem, including parameter values and value ranges, and 7. Use the transformed methods and steps from the case(s) at least in part to solve the new problem.
As the chef teaches the robot (the two arms and the sensing devices, such as haptic feedback from fingers, force-feedback from joints, and one or more observation cameras), the robot learns not only the specific sequence of movements, and time correlations, but also the family of small variations around the chef's movements to be able to prepare the same dish regardless of minor variations in the observable input parameters—and thus it learns a generalized transformed plan, giving it far greater utility than rote memorization. For additional information on case-based reasoning and learning, see materials by Leake, 1996 Book, Case-Based Reasoning: Experiences, Lessons and Future Directions, http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=4068324&fileld=S0269888 900006585dl.acm.org/citation.cfm?id=524680; Carbonell, 1983, Learning by Analogy: Formulating and Generalizing Plans from Past Experience, http://link.springer.com/chapter/10.1007/978-3-662-12405-5_5, which these references are incorporated by reference herein in their entireties.
(217) As depicted in
(218)
(219) A person of skill in the art will appreciate that the probability of overall success can be low even if the probability of success of individual stages is relatively high. For instance, given 10 stages and a probability of success of each stage being 90%, the probability of overall success is (0.9).sup.10=0.28 or 28%.
(220) A stage in preparing a food dish comprises one or more mini-manipulations, where each mini-manipulation comprises one or more robotic actions leading to a well-defined intermediate result. For instance, slicing a vegetable can be a mini-manipulation consisting of grasping the vegetable with one hand, grasping a knife with the other, and applying repeated knife movements until the vegetable is sliced. A stage in preparing a dish can comprise one or multiple slicing mini-manipulations.
(221) The probability of success formula applies equally well at the level of stages and at the level of mini-manipulations, so long as each mini-manipulation is relatively independent of other mini-manipulations.
(222) In one embodiment, in order to mitigate the problem of reduced certainty of success due to potential compounding errors, standardized methods for most or all of the mini-manipulations in all of the stages are recommended. Standardized operations are ones that can be pre-programmed, pre-tested, and if necessary pre-adjusted to select the sequence of operations with the highest probability of success. Hence, if the probability of standardized methods via the mini-manipulations within stages is very high, so will be the overall probability of success of preparing the food dish, due to the prior work, until all of the steps have been perfected and tested. For instance, to return to the above example, if each stage utilizes reliable standardized methods, and its success probability is 99% (instead of 90% as in the earlier example), then the overall probability of success will be (0.99).sup.10=90.4%, assuming there are 10 stages as before. This is clearly better than 28% probability of an overall correct outcome.
(223) In another embodiment, more than one alternative method is provided for each stage, wherein, if one alternative fails, another alternative is tried. This requires dynamic monitoring to determine the success or failure of each stage, and the ability to have an alternate plan. The probability of success for that stage is the complement of the probability of failure for all of the alternatives, which mathematically is written as:
(224)
(225) In the above expression s.sub.i is the stage and A(s.sub.i) is the set of alternatives for accomplishing S.sub.i. The probability of failure for a given alternative is the complement of the probability of success for that alternative, namely 1−P(s.sub.i|a.sub.j), and the probability of all the alternatives failing is the product in the above formula. Hence, the probability that not all will fail is the complement of the product. Using the method of alternatives, the overall probability of success can be estimated as the product of each stage with alternatives, namely:
(226)
(227) With this method of alternatives, if each of the 10 stages had 4 alternatives, and the expected success of each alternative for each stage was 90%, then the overall probability of success would be (1−(1−(0.9)).sup.4).sup.10=0.99 or 99% versus just 28% without the alternatives. The method of alternatives transforms the original problem from a chain of stages with multiple single points of failure (if any stage fails) to one without single points of failure, since all the alternatives would need to fail in order for any given stage to fail, providing more robust outcomes.
(228) In another embodiment, both standardized stages comprising standardized mini-manipulations, and alternate means of the food dish preparation stages are combined, yielding even more robust behavior. In such a case, the corresponding probability of success can be very high, even if alternatives are only present for some of the stages or mini-manipulations.
(229) In another embodiment only the stages with lower probability of success are provided alternatives, in case of failure, for instance stages for which there is no very reliable standardized method, or for which there is potential variability, e.g. depending on odd-shaped materials. This embodiment reduces the burden of providing alternatives to all stages.
(230)
(231)
(232) A predefined mini-manipulation is available to achieve each functional result (e.g., the egg is cracked). Each mini-manipulation comprises of a collection of action primitives which act together to accomplish the functional result. For example, the robot may begin by moving its hand towards the egg, touching the egg to localize its position and verify its size, and executing the movements and sensing actions necessary to grasp and lift the egg into the known and predetermined configuration.
(233) Multiple mini-manipulations may be collected into stages such as making a sauce for convenience in understanding and organizing the recipe. The end result of executing all of the mini-manipulations to complete all of the stages is that a food dish has been replicated with a consistent result each time.
(234)
(235) The robotic hand 72 has the RGB-D sensor 500 placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The RGB-D sensor 500 provides guidance to the robotic hand 72 in moving the robotic hand 72 toward the direction of the object and to make necessary adjustments to grab an object. Second, a sonar sensor 502f and/or a tactile pressure sensor are placed near the palm of the robotic hand 72, for detecting the distance and shape, and subsequent contact, of the object. The sonar sensor 502f can also guide the robotic hand 72 to move toward the object. Additional types of sensors in the hand may include ultrasonic sensors, lasers, radio frequency identification (RFID) sensors, and other suitable sensors. In addition, the tactile pressure sensor serves as a feedback mechanism so as to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to safely lift the object. In addition, the sonar sensor 502f in the palm of the robotic hand 72 provides a tactile sensing function to grab and handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand exerts on the knife and applies to the beef can be detected by the tactile sensor when the knife finishes slicing the beef, i.e. when the knife has no resistance, or when holding an object. The pressure distributed is not only to secure the object, but also not to break it (e.g. an egg).
(236) Furthermore, each finger on the robotic hand 72 has haptic vibration sensors 502a-e and sonar sensors 504a-e on the respective fingertips, as shown by a first haptic vibration sensor 502a and a first sonar sensor 504a on the fingertip of the thumb, a second haptic vibration sensor 502b and a second sonar sensor 504b on the fingertip of the index finger, a third haptic vibration sensor 502c and a third sonar sensor 504c on the fingertip of the middle finger, a fourth haptic vibration sensor 502d and a fourth sonar sensor 504d on the fingertip of the ring finger, and a fifth haptic vibration sensor 502e and a fifth sonar sensor 504e on the fingertip of the pinky. Each of the haptic vibration sensors 502a, 502b, 502c, 502d and 502e can simulate different surfaces and effects by varying the shape, frequency, amplitude, duration and direction of a vibration. Each of the sonar sensors 504a, 504b, 504c, 504d and 504e provides sensing capability on the distance and shape of the object, sensing capability for the temperature or moisture, as well as feedback capability. Additional sonar sensors 504g and 504h are placed on the wrist of the robotic hand 72.
(237)
(238)
(239)
(240)
(241) Together the thenar eminence 532 and hypothenar eminence 534 support application of large forces from the robot arm to an object in the working space such that application of these forces puts minimal stress on the robot hand joints (e.g., picture of the rolling pin). Extra joints within the palm 520 themselves are available to deform the palm. The palm 520 should deform in such a way as to enable the formation of an oblique palmar gutter for tool grasping in a way similar to a chef (typical handle grasp). The palm 520 should deform in such a way as to enable cupping, for conformable grasping of convex objects such as dishes and food materials in a manner similar to the chef, as shown by a cupping posture 542 in
(242) Joints within the palm 520 that may support these motions include the thumb carpometacarpal joint (CMC), located on the radial side of the palm near the wrist, which may have two distinct directions of motion (flexion/extension and abduction/adduction). Additional joints required to support these motions may include joints on the ulnar side of the palm near the wrist (the fourth finger F4 528 and the fifth finger F5 530 CMC joints), which allow flexion at an oblique angle to support cupping motion at the hypothenar eminence 534 and formation of the palmar gutter.
(243) The robotic palm 520 may include additional/different joints as needed to replicate the palm shape observed in human cooking motions, e.g., a series of coupled flexure joints to support formation of an arch 540 between the thenar and hypothenar eminences 532 and 534 to deform the palm 520, such as when the thumb F1 522 touches the pinky finger F5 530, as illustrated in
(244) When the palm is cupped, the thenar eminence 532, the hypothenar eminence 534, and the MCP pads 536 form ridges around a palmar valley that enable the palm to close around a small spherical object (e.g., 2 cm).
(245) The shape of the deformable palm will be described using locations of feature points relative to a fixed reference frame, as shown in
(246) Feature points are measured by calibrated cameras mounted in the workspace as the chef performs cooking tasks. Trajectories of feature points in time are used to match the chef motion with the robot motion, including matching the shape of the deformable palm. Trajectories of feature points from the chef's motion may also be used to inform robot deformable palm design, including shape of the deformable palm surface and placement and range of motion of the joints of the robot hand.
(247) In the embodiment as depicted in
(248)
(249) The visual pattern consists of surface markings 552 on the robot hand or on a glove worn by the chef. These surface markings may be covered by a food safe transparent glove 554, but the surface markings 552 remain visible through the glove.
(250) When the surface markings 552 are visible in a camera image, two-dimensional feature points may be identified within that camera image by locating convex or concave corners within the visual pattern. Each such corner in a single camera image is a two-dimensional feature point.
(251) When the same feature point is identified in multiple camera images, the three-dimensional location of this point can be determined in a coordinate frame which is fixed with respect to the standardized robotic kitchen 50. This calculation is performed based on the two-dimensional location of the point in each image and the known camera parameters (position, orientation, field of view, etc. . . . ).
(252) A reference frame 556 fixed to the robotic hand 72 can be obtained using a reference frame visual pattern. In one embodiment, the reference frame 556 fixed to the robotic hand 72 comprises of an origin and three orthogonal coordinate axes. It is identified by locating features of the reference frame's visual pattern in multiple cameras, and using known parameters of the reference frame visual pattern and known parameters of the cameras to extract the origin and coordinate axes.
(253) Three-dimensional shape feature points expressed in the coordinate frame of the food preparation station can be converted into the reference frame of the robot hand once the reference frame of the robot hand is observed.
(254) The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame fixed to the hand of the robot or the chef.
(255) As illustrated in
(256)
(257) Shape feature point locations are determined based on sensor signals. The sensors provide an output which allows calculation of distance in a reference frame which is attached to the magnet, which furthermore is attached to the hand of the robot or the chef.
(258) The three-dimensional location of each shape feature point is calculated based on the sensor measurements and known parameters obtained from sensor calibration. The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame, which is fixed to the hand of the robot or the chef. For additional information on common contact regions on the human hand and function in grasping, see the material from Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura. “Patterns of static prehension in normal hands.” American Journal of Occupational Therapy 34, no. 7 (1980): 437-445, which this reference is incorporated by reference herein in its entirety.
(259)
(260)
(261)
(262)
(263)
(264)
(265)
(266)
(267)
(268) The robotic hand 72 includes a camera sensor 684, such as an RGB-D sensor, an imaging sensor or a visual sensing device, placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The imaging sensor 682f provides guidance to the robotic hand 72 in moving the robotic hand 72 towards the direction of the object and to make necessary adjustments to grab an object. In addition, a sonar sensor, such as a tactile pressure sensor, may be placed near the palm of the robotic hand 72, for detecting the distance and shape of the object. The sonar sensor 682f can also guide the robotic hand 72 to move toward the object. Each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g includes ultrasonic sensors, laser, radio frequency identification (RFID), and other suitable sensors. In addition, each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g serves as a feedback mechanism to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to grab and lift the object. In addition, the sonar sensor 682f in the palm of the robotic hand 72 provides tactile sensing function to handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand 72 exerts on the knife and applies to the beef, allows the tactile sensor to detect when the knife finishes slicing the beef, i.e., when the knife has no resistance. The distributed pressure is not only to secure the object, but also so as not to exert too much pressure so as to, for example, not to break an egg). Furthermore, each finger on the robotic hand 72 has a sensor on the finger tip, as shown by the first sensor 682a on the finger tip of the thumb, the second sensor 682b on the finger tip of the index finger, the third sensor 682c on the finger tip of the middle finger, the fourth sensor 682d on the finger tip of the ring finger, and the fifth sensor 682f on the finger tip of the pinky. Each of the sensors 682a, 682b, 682c, 682d, 682e provide sensing capability on the distance and shape of the object, sensing capability for temperature or moisture, as well as tactile feedback capability.
(269) The RGB-D sensor 684 and the sonar sensor 682f in the palm, plus the sonar sensors 682a, 682b, 682c, 682d, 682e in the finger tip of each finger, provide a feedback mechanism to the robotic hand 72 as a means to grab a non-standardized object, or a non-standardized kitchen tool. The robotic hands 72 may adjust the pressure to a sufficient degree to grab ahold of the non-standardized object. A program library 690 that stores sample grabbing functions 692, 694, 696 according to a specific time interval for which the robotic hand 72 can draw from in performing a specific grabbing function, is illustrated in
(270)
(271)
(272)
(273)
(274)
(275)
(276)
(277) To create the mini-manipulation that results in cracking an egg with a knife, multiple parameter combinations must be tested to identify a set of parameters that ensure the desired functional result—that the egg is cracked—is achieved. In this example, parameters are identified to determine how to grasp and hold an egg in such a way so as not to crush it. An appropriate knife is selected through testing, and suitable placements are found for the fingers and palm so that it may be held for striking. A striking motion is identified that will successfully crack an egg. An opening motion and/or force are identified that allows a cracked egg to be opened successfully.
(278) The teaching/learning process for the robotic apparatus involves multiple and repetitive tests to identify the necessary parameters to achieve the desired final functional result.
(279) These tests may be performed over varying scenarios. For example, the size of the egg can vary. The location at which it is to be cracked can vary. The knife may be at different locations. The mini-manipulation must be successful in all of these variable circumstances.
(280) Once the learning process has been completed, results are stored as a collection of action primitives that together are known to accomplish the desired functional result.
(281)
(282) As an example of the operative relationship between the creation of a mini-manipulation in
(283)
(284)
(285)
(286) At step 892, the computer 16 tests and validates the specific successful parameter combination for X number of times (such as one hundred times). At step 894, the computer 16 computes the number of failed results during the repeated test of the specific successful parameter combination. At step 896, the computer 16 selects the next one-time successful parameter combination from the temporary library, and returns the process back to step 892 for testing the next one-time successful parameter combination X number of times. If no further one-time successful parameter combination remains, the computer 16 stores the test results of one or more sets of parameter combinations that produce a reliable (or guaranteed) result at step 898. If there are more than one reliable sets of parameter combinations, at step 899, the computer 16 determines the best or optimal set of parameter combinations and stores the optimal set of parameter combination which is associated with the specific mini-manipulation for use in the mini-manipulation library database by the robotic apparatus in the standardized robotic kitchen 50 during the food preparation stages of a recipe.
(287)
(288)
(289)
(290)
(291)
(292)
(293)
(294)
(295)
(296)
(297)
(298)
(299)
(300)
(301) One embodiment of the flow charts in functioning as a recipe filter, an ingredient filter, an equipment filter, an account and social network access, a personal partner page, a shopping cart page, and the information on the purchased recipe, registration setting, create a recipe are illustrated in
(302) In
(303) An example of partnership between users of the platform is demonstrated in
(304)
(305)
(306)
(307) The standardized robotic kitchen 50 in
(308) Based on the proper placement of the augmented sensor system 1854 placed somewhere in the robotic kitchen, such as on a computer-controllable railing, or on the torso of a robot with arms and hands, allows for 3D-tracking and raw data generation, both during chef-monitoring for machine-specific recipe-script generation, and monitoring the progress and successful completion of the robotically-executed steps in the stages of the dish replication in the standardized robotic kitchen 50.
(309) The standardized robotic kitchen 50 in
(310)
(311) The standardized robotic kitchen 50 depicts another possible configuration for the use of one or more augmented sensor systems 20. The standardized robotic kitchen 50 shows a multitude of augmented sensor systems 20 placed in the corners above the kitchen work-surface along the length of the kitchen axis with the intent to effectively cover the complete visible three-dimensional workspace of the standardized robotic kitchen 50.
(312) The proper placement of the augmented sensor system 20 in the standardized robotic kitchen 50, allows for three-dimensional sensing, using video-cameras, lasers, sonars and other two- and three-dimensional sensor systems to enable the collection of raw data to assist in the creation of processed data for real-time dynamic models of shape, location, orientation and activity for robotic arms, hands, tools, equipment and appliances, as they relate to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50.
(313) Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and mini-manipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.). The process of recipe-script execution based on key measurable variables is an essential part of the use of the augmented (also termed multi-modal) sensor system 20 during the execution of the replicating steps for a particular dish in a standardized robotic kitchen 50.
(314)
(315)
(316)
(317)
(318) To access the ingredients storage-and-supply unit, part of the countertop with sliding doors can be opened, where the recipe software controls the doors and moves designated containers and ingredients to the access location where the robotic arm(s) may pick up the containers, open the lid, remove the ingredients out of the containers to a designated place, reseal the lid and move the containers back into storage. The container is moved from the access location back to its default location in the storage unit, and a new/next container item is then uploaded to the access location to be picked up.
(319) An alternative embodiment for an ingredient storage-and-supply unit 1210 is depicted in
(320) In
(321)
(322)
(323)
(324)
(325)
(326)
(327)
(328) The top level 1292-1 contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc.
(329) The counter level 1292-2 not only houses the robotic arms 70, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher.
(330) The lower level 1292-3 houses the combination convection oven and microwave 1316, the dish-washer 1318 and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware and packing materials and cutlery.
(331)
(332) The perspective view of the robotic kitchen 50 clearly identifies one of the many possible layouts and locations for equipment at all three levels, including the top level 1292-1 (storage pantry 1304, standardized cooking tools and ware 1320, storage ripening zone 1298, chilled storage zone 1300, and frozen storage zone 1302, the counter level 1292-2 (robotic arms 70, sink 1308, chopping/cutting area 1310, charcoal grill 1312, cooking appliances 1314 and serving counter 1306) and the lower level (dish-washer 1318 and oven and microwave 1316).
(333)
(334)
(335)
(336)
(337)
(338)
(339)
(340)
(341)
(342)
(343)
(344)
(345)
(346)
(347)
(348)
(349)
(350)
(351)
(352)
(353)
(354)
(355)
(356)
(357)
(358)
(359)
(360)
(361)
(362)
(363)
(364)
(365)
(366)
(367)
(368)
(369)
(370)
(371)
(372)
(373)
(374)
(375)
(376) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 82 is included, a cabinet volume 1320 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(377) The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(378) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(379)
(380) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage pantry volume 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(381) The counter level not only houses monitoring sensors 1892 and control units 1894, but also includes a counter area with a sink and electronically controllable faucet 1308, another counter area 1310 with removable working surfaces for cutting/chopping on a board, etc., a charcoal-based slatted grill 1312, and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(382) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1892 providing data to one or more control units 1896, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(383)
(384)
(385)
(386)
(387)
(388) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(389) The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(390) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 3=378 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(391)
(392) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment.
(393) At the simplest level this includes a cabinet volume 1294 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(394) The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(395) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1315, the dish-washer 1318, the hard automation controlled ingredient dispensers 82 (not shown), and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(396)
(397) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(398) The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 2020 while also including a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices 2022 are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
(399) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not shown), and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(400)
(401) The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(402) The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 1316 while also including a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a (smart) charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1184 providing data to one or more control units 1186, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices (not shown) are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
(403) The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not showed)s, and a larger cabinet volume 1309 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1307 providing data to one or more control units 376, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
(404)
(405)
(406)
(407)
(408)
(409)
(410)
(411)
(412)
(413)
(414) In one embodiment the emotional profile can be detected via machine learning methods based on statistical classifiers where the inputs are any measured levels of pheromones, hormones, or other features such as visual or auditory cues. If the set of features is {x.sub.1, x.sub.2, x.sub.3, . . . , x.sub.n} represented as a vector and y represents the emotional state, then the general form of an emotion-detection statistical classifier is:
(415)
Where the function f is a decision tree, a neural network, a logistic regressor, or other statistical classifier described in the machine learning literature. The first term minimizes the empirical error (the error detected while training the classifier) and the second term minimizes the complexity—e.g. Occam's razor, finding the simplest function and set of parameters p for that function that yield the desired result.
(416) Additionally, in order to determine which pheromones or other features make the most difference (add the most value) to predicting emotional state, an active-learning criterion can be added, generally expressed as:
(417)
Where L is a “loss function”, f is the same statistical classifier as in the previous equation, and y-hat is the known outcome. We measure whether the statistical classifier performs better (smaller loss function) by addition new features, and if so keep them, otherwise not.
(418) Parameters, values and quantities that evolve over time can be assessed to create a human emotional profile by detecting the change or transformation from one moment to the next. There are identifiable qualities to an emotional expression. A robot with emotions in response to its environment could make quicker and more effective decisions, e.g. when a robot is motivated by fear or joy or desire it might make better decisions and attain the goals more effectively and efficiently.
(419) The robotic emotion engine replicates the human hormone emotions and pheromone emotions, either individually or in combination. Hormone emotions refer to how hormones change inside of a person's body and how that affects a person's emotions. Pheromone emotions refer to pheromones that are outside a person's body, such as smell, that affect a person's emotions. A person's emotional profile can be constructed by understanding and analyzing the hormone and pheromone emotions. The robotic emotion engine attempts to understand a person's emotions such as anger and fear by using sensors to detect a person's hormone and pheromone profile.
(420) There are nine key physiological sign parameters to be measured in order to build a person's emotional profile: (1) sets of hormones 2221, which are secreted internally and trigger various biochemical pathways that cause certain effects, e.g. adrenalin and insulin are hormones, (2) sets of pheromones 2222, which are secreted externally, and have an effect on another person in a similar way, e.g. androstenol, androstenone and androstadienone, (3) micro expression 2223, which is a brief, involuntary facial expression shown by humans according to emotions experienced, (4) the heart rate 2224 or heart beat, e.g., when a person's heart rate increases, (5) sweat 2225 (e.g., goose bumps) e.g. face blushes and palms get sweaty and in the state of being excited or nervous, (6) pupil dilation 2226 (and iris sphincter, biliary muscle), e.g. pupil dilation for a short time in response to feelings of fear, (7) reflex movement v7, which is the movement/action primarily controlled by the spinal arc, as a response to an external stimulus, e.g. jaw jerk reflex, (8) body temperature 2228 (9) pressure 2229. The analysis 2230 on how these parameters change over a certain time 2231 may reveal a person's emotional state and profile.
(421)
(422)
(423) When a user experiences an emotion or mood swing, physiological parameters such as hormone, heart rate, sweat, pheromones can be detected and recorded with a port connecting to a person's body, above the skin and directly to the vein. The start time and end time of the mood change can be determined by the person himself or herself as the person's emotional state changes. For example, a person initiates four manual emotion cycles and creates four timelines within a week, and as determined by the person, the first one lasts 2.8 hour from the time he tags the start till the time he tags the end. The second cycle last for 2 hours, the third one last for 0.8 hours, and the fourth one last for 1.6 hours.
(424)
(425)
(426)
(427)
(428) The computer module 1420 includes modules that include, but are not limited to, a robotic painting engine 1352 interfaced to a painting movement emulator 1422, a painting control module 1421 that acts based on visual feedback of the painting execution processes, a memory module 1380 to store painting execution program files, algorithms 1423 for learning the selection and usage of the appropriate drawing tools, as well as an extended simulation validation and calibration module 1378.
(429)
(430) One embodiment of the art platform standardization is defined as follows. First, standardized position and orientation (xyz) of any kind of art tools (brushes, paints, canvas, etc.) in the art platform. Second, standardized operation volume dimensions and architecture in each art platform. Third, standardized art tools set in each art platform. Fourth, standardized robotic arms and hands with a library of manipulations in each art platform. Fifth, standardized three-dimensional vision devices for creating dynamic three-dimensional vision data for painting recording and execution tracking and quality check function in each art platform. Sixth, standardized type/producer/mark/of all using paints during particular painting execution. Seventh, standardized type/producer/mark/size of canvas during particular painting execution.
(431) One main purpose to have Standardized Art Platform is to achieve the same result of the painting process (i.e., the same painting) executing by the original painter and afterward duplicated by robotic Art Platform. Several main points to emphasize in using the standardized Art Platform: (1) have the same timeline (same sequence of manipulations, same initial and ending time of each manipulation, same speed of moving object between manipulations) of Painter and automatic robotic execution; and (2) there are quality checks (3D vision, sensors) to avoid any fail result after each manipulation during the painting process. Therefore the risk of not having the same result is reduced if the painting was done at the standardized art platform. If a non-standardized art platform is used, this will increase the risk of not having the same result (i.e. not the same painting) because adjustment algorithms may be required when the painting is not executed at not the same volume, with the same art tools, with the same paint or with the same canvas in the painter studio as in the robotic art platform.
(432)
(433)
(434)
(435)
(436) In the case the user selects the robot engine to select the title/composer in step 1482, the engine uses its own interpretation of creativity in step 1492, to offer the human user to provide input to the selection process in step 1493. Should the human decline providing input, the robotic musician engine uses settings such as manual inputs to tonality, pitch and instrumentation as well as melodic variation in step 1499, to gather the necessary input in step 1130 to generate and upload selected instrument playing execution program files in step 1501, allowing the user to select the preferred one in step 1503, after the robotic musician engine has confirmed the selection in step 1502. The choice made by the human is then stored as a personal choice in the personal profile database in step 1504. Should the human decide to provide input to the query in step 1493, the user will be able in step 1493 to provide additional emotional input to the selection process (facial expressions, photo, news article, etc.). The input from step 194 is received by the robotic musician engine in step 1495, allowing it to proceed to step 1496, where the engine carries out a sentiment analysis related to all available input data and uploads a music selection based on the mood and style appropriate to the emotional input data from the human. Upon confirmation of selection for the uploaded music selection in step 1497 by the robotic musician engine, the user may select the ‘start’ button to play the program file for the selection in step 1498.
(437) In the case where the human wants to be intimately involved in the selection of the title/composer, the system provides a list of performers for the selected title to the human on a display in step 1483. In step 1484 the user selects the desired performer, a choice input that the system receives in step 1485. In step 1486 the robotic musician engine generates and uploads the instrument playing execution program files, and proceeds in step 1487 to compare potential limitations between a human and a robotic musician's playing performance on a particular instrument, thereby allowing it to calculate a potential performance gap. A checking step 1488 decides whether there exists a gap. Should there be a gap, the system will suggest other selections based on the user's preference profile in step 1489. Should there be no performance gap, the robotic musician engine will confirm the selection in step 1490 and allow the user to proceed to step 1491, where the user may select the ‘start’ button to play the program file for the selection.
(438)
(439)
(440)
(441)
(442) The disk drive unit 2240 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 246) embodying any one or more of the methodologies or functions described herein. The software 246 may also reside, completely or at least partially, within the main memory 244 and/or within the processor 226 during execution thereof the computer system 224, the main memory 228, and the instruction-storing portions of processor 226 constituting machine-readable media. The software 246 may further be transmitted or received over a network 18 via the network interface device 248.
(443) While the machine-readable medium 244 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term “machine-readable medium” shall also be taken to include any tangible medium that is capable of storing a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term “machine-readable medium” shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.
(444) In general terms, there may be considered a method of motion capture and analysis for a robotics system, comprising sensing a sequence of observations of a person's movements by a plurality of robotic sensors as the person prepares a product using working equipment; detecting in the sequence of observations mini-manipulations corresponding to a sequence of movements carried out in each stage of preparing the product; transforming the sensed sequence of observations into computer readable instructions for controlling a robotic apparatus capable of performing the sequences of mini-manipulations; storing at least the sequence of instructions for mini-manipulations to electronic media for the product. This may be repeated for multiple products. The sequence of mini-manipulations for the product is preferably stored as an electronic record. The mini-manipulations may be abstracted parts of a multi-stage process, such as cutting an object, heating an object (in an oven or on a stove with oil or water), or similar. Then, the method may further comprise: transmitting the electronic record for the product to a robotic apparatus capable of replicating the sequence of stored mini-manipulations, corresponding to the original actions of the person. Moreover, the method may further comprise executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
(445) In another general aspect, there may be considered a method of operating a robotics apparatus, comprising providing a sequence of pre-programmed instructions for standard mini-manipulations, wherein each mini-manipulation produces at least one identifiable result in a stage of preparing a product; sensing a sequence of observations corresponding to a person's movements by a plurality of robotic sensors as the person prepares the product using equipment; detecting standard mini-manipulations in the sequence of observations, wherein a mini-manipulation corresponds to one or more observations, and the sequence of mini-manipulations corresponds to the preparation of the product; transforming the sequence of observations into robotic instructions based on software implemented methods for recognizing sequences of pre-programmed standard mini-manipulations based on the sensed sequence of person motions, the mini-manipulations each comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; storing the sequence of mini-manipulations and their corresponding robotic instructions in electronic media. Preferably, the sequence of instructions and corresponding mini-manipulations for the product are stored as an electronic record for preparing the product. This may be repeated for multiple products. The method may further include transmitting the sequence of instructions (preferably in the form of the electronic record) to a robotics apparatus capable of replicating and executing the sequence of robotic instructions. The method may further comprise executing the robotic instructions for the product by the robotics apparatus, thereby obtaining substantially the same result as the original product prepared by the human. Where the method is repeated for multiple products, the method may additionally comprise providing a library of electronic descriptions of one or more products, including the name of the product, ingredients of the product and the method (such as a recipe) for making the product from ingredients.
(446) Another generalized aspect provides a method of operating a robotics apparatus comprising receiving an instruction set for a making a product comprising of a series of indications of mini-manipulations corresponding to original actions of a person, each indication comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; providing the instruction set to a robotic apparatus capable of replicating the sequence of mini-manipulations; executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
(447) A further generalized method of operating a robotic apparatus may be considered in a different aspect, comprising executing a robotic instructions script for duplicating a recipe having a plurality of product preparation movements; determining if each preparation movement is identified as a standard grabbing action of a standard tool or a standard object, a standard hand-manipulation action or object, or a non-standard object; and for each preparation movement, one or more of: instructing the robotic cooking device to access a first database library if the preparation movement involves a standard grabbing action of a standard object; instructing the robotic cooking device to access a second database library if the food preparation movement involves a standard hand-manipulation action or object; and instructing the robotic cooking device to create a three-dimensional model of the non-standard object if the food preparation movement involves a non-standard object. The determining and/or instructing steps may be particularly implemented at or by a computer system. The computing system may have a processor and memory.
(448) Another aspect may be found in a method for product preparation by robotic apparatus, comprising replicating a recipe by preparing a product (such as a food dish) via the robotic apparatus, the recipe decomposed into one or more preparation stages, each preparation stage decomposed into a sequence of mini-manipulations and active primitives, each mini-manipulation decomposed into a sequence of action primitives. Preferably, each mini manipulation has been (successfully) tested to produce an optimal result for that mini manipulation in view of any variations in positions, orientations, shapes of an applicable object, and one or more applicable ingredients.
(449) A further method aspect may be considered in a method for recipe script generation, comprising receiving filtered raw data from sensors in the surroundings of a standardized working environment module, such as a kitchen environment; generating a sequence of script data from the filtered raw data; and transforming the sequence of script data into machine-readable and machine-executable commands for preparing a product, the machine-readable and machine-executable commands including commands for controlling a pair of robotic arms and hands to perform a function. The function may be from the group consisting of one or more cooking stages, one or more mini-manipulations, and one or more action primitives. A recipe script generation system comprising hardware and/or software features configured to operate in accordance with this method may also be considered.
(450) In any of these aspects, the following may be considered. The preparation of the product normally uses ingredients. Executing the instructions typically includes sensing properties of the ingredients used in preparing the product. The product may be a food dish in accordance with a (food) recipe (which may be held in an electronic description) and the person may be a chef. The working equipment may comprise kitchen equipment. These methods may be used in combination with any one or more of the other features described herein. One, more than one or all of the features of the aspects may be combined, so a feature from one aspect may be combined with another aspect for example. Each aspect may be computer-implemented and there may be provided a computer program configured to perform each method when operated by a computer or processor. Each computer program may be stored on a computer-readable medium. Additionally or alternatively, the programs may be partially or fully hardware-implemented. The aspects may be combined. There may also be provided a robotics system configured to operate in accordance with the method described in respect of any of these aspects.
(451) In another aspect, there may be provided a robotics system, comprising: a multi-modal sensing system capable of observing human motions and generating human motions data in a first instrumented environment; and a processor (which may be a computer), communicatively coupled to the multi-modal sensing system, for recording the human motions data received from the multi-modal sensing system and processing the human motions data to extract motion primitives, preferably such that the motion primitives define operations of a robotics system. The motion primitives may be mini-manipulations, as described herein (for example in the immediately preceding paragraphs) and may have a standard format. The motion primitive may define specific types of action and parameters of the type of action, for example a pulling action with a defined starting point, end point, force and grip type. Optionally, there may be further provided a robotics apparatus, communicatively coupled to the processor and/or multi-modal sensing system. The robotics apparatus may be capable of using the motion primitives and/or the human motions data to replicate the observed human motions in a second instrumented environment.
(452) In a further aspect, there may provided a robotics system, comprising: a processor (which may be a computer), for receiving motion primitives defining operations of a robotics system, the motion primitives being based on human motions data captured from human motions; and a robotics system, communicatively coupled to the processor, capable of using the motion primitives to replicate human motions in an instrumented environment. It will be understood that these aspects may be further combined.
(453) A further aspect may be found in a robotics system comprising: first and second robotic arms; first and second robotic hands, each hand having a wrist coupled to a respective arm, each hand having a palm and multiple articulated fingers, each articulated finger on the respective hand having at least one sensor; and first and second gloves, each glove covering the respective hand having a plurality of embedded sensors. Preferably the robotics system is a robotic kitchen system.
(454) There may further be provided, in a different but related aspect, a motion capture system, comprising: a standardized working environment module, preferably a kitchen; plurality of multi-modal sensors having a first type of sensors configured to be physically coupled to a human and a second type of sensors configured to be spaced away from the human. One or more of the following may be the case: the first type of sensors may be for measuring the posture of human appendages and sensing motion data of the human appendages; the second type of sensors may be for determining a spatial registration of the three-dimensional configurations of one or more of the environment, objects, movements, and locations of human appendages; the second type of sensors may be configured to sense activity data; the standardized working environment may have connectors to interface with the second type of sensors; the first type of sensors and the second type of sensors measure motion data and activity data, and send both the motion data and the activity data to a computer for storage and processing for product (such as food) preparation.
(455) An aspect may additionally or alternatively be considered in a robotic hand coated with a sensing gloves, comprising: five fingers; and a palm connected to the five fingers, the palm having internal joints and a deformable surface material in three regions; a first deformable region disposed on a radial side of the palm and near the base of the thumb; a second deformable region disposed on a ulnar side of the palm, and spaced apart from the radial side; and a third deformable region disposed on the palm and extend across the base of the fingers. Preferably, the combination of the first deformable region, the second deformable region, the third deformable region, and the internal joints collectively operate to perform a mini manipulation, particularly for food preparation.
(456) In respect of any of the above system, device or apparatus aspects, there may further be provided method aspects comprising steps to carry out the functionality of the system. Additionally or alternatively, optional features may be found based on any one or more of the features described herein with respect to other aspects
(457) TABLE-US-00001 TABLE A Types of Equipment Types of Equipment KITCHEN ACCESSORIES 1 Funnels 1.1. stainless steel funnel 1.2. plastic funnel 1.3 silicone funnel 1.4 convertible funnel 2 Colanders 2.1 quadratic colanders 2.2 oval ladle-vases 2.3 colanders with folding handles 2.4 flat colander 2.5 plastic colanders 2.6 small round colanders 2.7 suspended colanders 2.8 cover-colander 2.9 stainless steel and aluminum colanders 2.1 cone colanders 3 Kitchen Appliances 3.1. Whisk 3.2. scoop, spatula 3.2.1 cook spatula 3.2.2. spatula with slots 3.2.3. confectionery spatula 3.5 Spoons 3.5.1 serving spoon 3.5.2 spoon-tongs 3.5.3 spoon with slots 3.5.4 spoon for rice 3.5.5 ladle spoon 3.5.6 ice cream spoon 3.5.7 honey spoon 3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionery syringe for cookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9 skimmer 3.10 Meat fork 3.11 Brush 3.12 coffee filter 3.5.7 honey spoon 3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionery syringe for cookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9 skimmer 3.10 Meat fork 3.11 Brush 3.23 ties for rolls 3.2 dough mini-scraper 3.25 grill tongs 3.26 spaghetti tongs 3.27 ice tongs 3.28 sugar tongs 3.29 package clip 3.30 package clip 3.31 citrus spray 3.32 Dough press 3.33 scoop for bulk 3.34 salad serving tongs (tweezers) 3.35 accessories for tubes 3.36 Pestle 3.37 Mortar 3.38 roller for cutting of the rings 3.39 opener for caps 3.40 meat tenderizer; meat softener 3.41 egg yolk separator 3.42 Apron 3.33 scoop for bulk 3.43 tools for decoration 3.44 jar for oil and vinegar 3.45 mug for milk boiling 3.46 napkins 3.47 tablecloth 3.48 marker for glasses 3.49 potato masher 3.50 Basket 3.51 meat tenderizer 3.52 cocotte 3.53 brush for washing of the vegetables 3.54 lids for cups 3.55 rope for baking 3.56 jar for herbs storage 3.57 Mortar 3.58 scraper for glass ceramic plates 3.59 Teapot for tea 3.60 clothespin for notes on the fridge 3.61 railing systems 3.62 hanger for kitchen tools 3.63 plunger with not adhering surface 3.64 silicone plunger 3.65 rolling pin with adjustable thickness 3.66 vacuum bags with pump 3.67 gas lighter 3.68 bone forceps 4 kitchen timers, thermometers 4.1 timer for meat roasting 4.2 digital thermometer 4.3 holder for thermometer 4.4 meat thermometer 4.5 digital timer 4.6 elector. digital timer 4.7 _aramel thermometer 5 Mills for spices 5.1 mill for black pepper 5.2 electric mill 5.3 combined mill for pepper and salt (2 in 1) 5.4 mill for spices 5.5 mill for greens 6 Measuring utensils 6.1. Measuring container (plastic bottle) 6.2. measuring jar 6.3. measuring jug 6.4. measuring bowl 6.5. mechanical dispenser for ice cream 7 mechanical mixers 8 Bowl 8.1. metal bowl 8.2. stainless steel bowl 8.3. plastic bowl 8.4. plastic bowl 8.5. bowls for food 9 Sets 9.2 wine set 9.3 sets for spices 9.6. cupcakes baking set 9.7 accessory kit for baking 9.8 set of bar tools 9.9 set of kitchen tools 9.10 Set for eggs and pancakes baking 11 Slicing and cutting of products 11.1 Cutter 11.2 holder for onions cutting 11.2 cutting boards 11.3 universal professional knives 11.4 kitchen shears 11.5 hatchet 11.6 meat hatchet 11.7 Hammer for meat with hatchet 11.8 Hoe 11.9 Hammer for meat 11.10 Knives 11.11 knife for greens 11.12 knife for oranges 11.13 knife for kiwi 11.14 knife for pineapple 11.15 Spiral knife for carrots 11.16 multifunctional knife 11.17 vegetable knife 11.18 Pizza Cutter 11.19 universal knife 11.20 knife for slicing 11.21 cook knife 11.22 gastronomic knife 11.23 opener 11.24 Cheese knife 11.25 boning knife 11.26 lettuce knife 11.27 knife for steaks 11.28 butcher knife 11.29 shredding knife 11.30 bread knife 11.31 fish knife 11.32 knife for sandwiches 11.33 Santoku knife 11.34 knife for fruit coring 11.35 Butter knife 12 openers 12.1 tin-opener 12.2 corkscrew 12.3 corkscrew on a stand 12.4 lever corkscrew 12.5 folding corkscrew 12.6 opener for waiter 12.7 openers 13 stand and holders 13.1 stands for hot 13.2 stand for kitchen utensils storing 13.3 toothpick holder 13.4 Bottle holder 13.5 Holder for capsules 13.6 stand for spoon 13.7 stand for coffee capsules 13.8 Coasters 13.9 Napkin holder 13.10 stand for eggs 13.11 stand for openers 13.12 stand for scoops 13.13 stand for cooking and serving of eggs 13.14 stand for ladle 13.15 Holder for paper towels 13.16 Transforming stand for kitchen appliances 13.17 stand for mug 13.18 stand mugs and saucers 13.19 stand for kitchen knives 13.20 stand for chicken 13.21 napkin-stand 13.22 heated stand 13.23 stands for cake 14 Appliances for peeling and cutting 14.1 grater for vegetables 14.2 grater 14.3 garlic masher 14.4 egg cutter 14.5 Manual vegetable cutter 14.6 Peeler for vegetables 14.7 Nutcracker 14.8 The device for separating the yolks from the whites 14.9 grasping for carrots cleaning 14.10 scraper fish scales 14.11 cutter for fruits 14.12 roller for holes 14.13 tongs for fish bones 14.14 spiral vegetable cutter 15 Bottle Caps 15.1 champagne cork (stopper) 15.2 stoppers for wine 15.3 The opener to remove the crown corks from bottles 16 sieves 16.1 sieve for tea 16.2 sieve-tongs for tea 16.3 Strainer for spices 16.4 Strainer for tea 16.5 Universal sieve 16.6 flour sieve 16.7 sieve to form the “Bird's Nest” 16.8 The Chinese sieve with a mesh insert 16.9 sieve with support 16.10 Mug-sieve for flour 16.11 sieve on the handle 17 Salt and pepper shakers 17.1 container for seasoning 17.2 salt cellar 17.3 containers for oil and vinegar 18 Dish dryers 18.1 salad dryer dryer-placemat dryer for crockery and cutlery 19 Cutlery Accessories 19.1 cutlery tray 19.2 cutlery holder 19.3 cutlery container 19.4 strainer for cutlery 19.5 wall hanger for kitchen tools 19.6 cutlery organizer 19.7 mat for cutlery 19.8 sliding tray for cutlery 19.9 dryer for cutlery 19.10 glass for cutlery 19.11 napkin for the cutlery 19.12 case for cutlery 19.13 tray for cutlery 19.14 mitten-potholder 19.15 box for cutlery 19.16 full-size rack (cassette) for cutlery 19.17 Stand without containers for cutlery 19.18 cassette for cutlery 19.19 container for cutlery 19.20 station for cutlery 19.21 Shelf for cutlery 20 Decorations for cocktails 21.1 Ducts 22.2. Sticks 23 Mold 23.1 molds for ice 23.2 molds for children 23.3 Molds for shaping products 23.4 Molds for dumplings 24 Measuring container 24.1 Measuring container 24.2 A mixing container with the dispenser 24.3 Measuring container with the funnel 24.4 Beaker 24.5 Scoop 26 kitchen scissors 26.1 Scissors for BBQ 26.2 Kitchen scissors with bottle opener 26.3 Scissors for greens 26.4 Kitchen multipurpose scissors 26.5 Kitchen scissors for poultry 27 utensil for storage 27.1 container for storage 27.2 Bottles for liquid spices, oils 27.3 jars for storage 27.4 lunchbox 27.5 foldable lunchbox 27.6 jar for hermetic storage of bulk products 27.7 Sprayer for oil/vinegar 27.8 jar for bulk products 27.9 containers for spices 27.10 container for seasoning 27.11 Container for tea 28 potholders 28.1 oven-glove 28.2 silicone potholders 28.3 dishcloth railing with hooks 29 silicone mats 29.1 baking mat 29.2 mat for baking cakes 29.3 mat for drying of the glasses 29.4 cooking mat 29.5 Mat for drying of the dishes 30 graters, presses, rubbing machines 30.1 grater with a handle 30.2 grater 30.3 multifunction grater 30.4 grater shredder 30.5 grind for the green 30.6 grind for the garlic 30.7 Slicer for tomatoes 30.8 grater with rotating drums 30.9 universal device for grinding 30.10 mechanical grater 30.11 garlic peeling tube 30.12 rubbing machine 30.13 press for vegetables 30.14 press for garlic 30.2 press for hamburgers 31 knife sharpener 31.1 electric sharpener 31.2 sharpening stone 31.3 ceramic sharpener 32 breadbox 33 lattice with legs Kitchen dishes 1 for alcohol 1.2 Brandy set with dispenser 1.3 souvenir cups 1.4 stemware 1.5 pail of ice 1.6 stemware 1.7 champagne bucket 1.8 stemware 1.9 carafe 1.10 server 1.11 bottle holder 2 tableware 2.1 first course dish 2.2 dish for bouillon 2.3 bouillon bowl 2.4 oiler 2.5 round dish 2.6 duck pan 2.7 Set for making chocolate fondue 2.8 Set for making cheese fondue 2.9 salad bowl 2.10 dish for cake 2.11 compartmental dish 2.12 set of cutlery 2.13 serving spoon and fork 2.14 dish with lid 2.15 steam table 2.16 ice-cream bowl 2.17 Flatware 2.18 saucer 2.19 saucer for jam 2.20 mustard-pot 2.21 pepper-pot 2.22 ash-pot 2.23 deep table plate 2.24 dinner plate 2.25 snack plate 2.26 deep dessert plate 2.27 dessert plate 2.28 plate for pies 2.29 horseradish-pot 3 Utensils for table 3.1 Pad for tableware 3.2 serving mat 3.3. serving tray 3.4 glass burner 4 Dishes for tea, coffee, dessert 4.1 sugar-bowl 4.2 mug 4.3 mug with teapot 4.4 mug with stand 4.5 mug with lid 4.6 tea set 4.7 dish 4.8 french-press 4.9 teapot 4.10 teapot with strainer 4.11 glass teapot 4.12 ice-cream bowl 4.13 multifunctional vase 4.14 glasses 4.15 soup bowl 4.16 wicker basket 4.17 vase 3-tier 4.18 tea set 4.19 napkin rings 4.20 pannier for fruits 4.21 table trash basket 4.22 biscuit dish 4.23 candy dish 4.24 coffee sets 5 CUTLERY 5.1 Table fork 5.2 fork for sprat 5.3 fork for crayfish 5.4 fork for oysters 5.6 fork for lemons 5.7 big spoon 5.8 dessert spoon 5.9 tea spoon 5.10 coffee spoon 5.11 lemonade spoon 5.12 ladle-spoon 5.13 spoon for hot snacks 5.14 ice cream spoon 5.15 mustard spoon 5.16 salt spoon 5.17 spatula for cakes 5.18 spatula for caviar 5.19 spatula for fish 5.20 table knife 5.21 knife and fork for the fish 5.22 knife and fork snack 5.23 knife and fork dessert 5.24 Butterknife 5.25 tool kits for lobster, crayfish 5.26 devices for spices 5.27 grille and asparagus tongs 5.28 salad unit (salad fork and spoon) 5.29 sugar-tongs 5.30 tongs for cakes and sugar 5.31 ice tongs 5.32 can-opener 5.33 fork oyster 5.34 plug for crayfish 5.36 cocotte fork 5.37 fork for canned fish in oil (sprat, sardines) 5.38 spinner for champagne 5.39 spoon to mix whiskey with soda water Kitchen appliances 1 aerogrill 2 blenders, grinder 3 coffee Maker 4 coffee grinder (coffee mill) 5 Food Processor 6 mixer 7 mini oven 8 multicooker 9 meat grinders 10 steamers 11 Raclette grill 12 Juicers 13 toasters 14 egg cooker 15 electric range 15.1 electric induction stove 16 electric kettle 16.1 thermopots 17 bread makers 18 microwaves 19 weights for kitchen 20 electric driers 21 weights for kitchen Children's dishes 1 Children Sets for baking 2 Children cutlery 3 Children thermoses 4 Children Sets of dishes List of ingredient data 1 Ingredient name 2 Ingredient Photo 3 Manufacturer 4 Country 5 Type of Ingredient 6 Type of cuisine 7 Relation to Vegetarianism 8 Spice 9 Energy value 10 Description of the Ingredient 11 Status 12 Price List of equipment data 1 Equipment name 2 Equipment photo 3 Manufacturer 4 Brand name 5 Dimensions 6 Weight 7 Connectivity 8 Type of cuisine 9 Type of equipment 10 Description of equipment 11 Year 12 Status 13 Price List of recipe data 1 Name of the recipe 2 Recipe author 3 Recipe Photo 4 Preparation time 5 Basis of the dish 6 Type of cuisine 7 Type of the dish 8 Relation to Vegetarianism 9 Spice 10 Energy value 11 Number of persons 12 Description of the recipe 13 Description of the stages of cooking 14 Ingredients 15 Type of equipment 16 Video of recipe cooking 17 User Rating 18 Expert Rating 19 Amount of sales 21 Automatic cooking 22 Price
(458) TABLE-US-00002 TABLE B Types of Ingredients 1 MEAT and MEAT PRODUCTS 1 Basturma 2 Fat 3 brisket cooked and smoked 4 Hare 5 leather duck 6 Sausage 7 Sausages 8 Sausages “Hunting party” 9 Horsemeat 10 Bones with bone marrow 11 Roe 12 Rabbit 13 Meat 14 Moosemeat 15 Venison 16 Liver 17 Kidney 18 Smoked ribs 19 Salami 20 Sausages 21 Cervelat 22 Sausages 23 Hungarian smoked bacon 24 bacon fat-tailed 25 Steak 26 ribeye steak 27 Farce 28 crocodile fillet 29 Jamon 30 Choriso (spanish sausage) 31 Skewers 32 Sowbelly 33 Deer tongue 34 Frog legs LAMB, VEAL 1 breast of lamb 2 loin of lamb 3 blade lamb 4 veal brains 5 mutton ham 6 veal ham 7 leg of lamb 8 Heel muscle mutton 9 lamb offal 10 veal kidneys 11 lamb chops 12 gras cow 13 veal heart 14 lamb testicles 15 veal fillet 16 veal cheeks 17 minced lamb 18 minced veal 19 veal tail 20 veal tongue 21 eggs bullish BEEF 1 beef brisket 2 beef fillet 3 beef (sirloin) 4 beef on the bone 5 beef eye muscle 6 legs of beef 7 ham beef 8 gras beef 9 beef ribs 10 beef heart 11 minced beef 12 tail beef 13 beef tongue PORK 1 bacon 2 smoked bacon 3 Pork 4 Ham 5 pork brisket 6 smoked pork belly 7 Pork Intestine 8 legs of pork 9 boar ham 10 pork ham 11 pork ribs 12 knuckle of pork 13 Fat 14 pork (pork neck or loin) 15 pork ears 16 minced pork 17 pig tail 18 pork tongue 2 BIRDS 1 garshnep 2 turkey breast 3 chicken breast 4 chicken breast, smoked 5 duck breast 6 Goose 7 chicken ventricles 8 turkey 9 turkey wings 10 chicken wings 11 chicken 12 smoked chicken 13 grouse 14 Coot 15 duck leg 16 crow's feet 17 chicken legs 18 chicken ham 19 Quail 20 gras chicken 21 chicken giblets 22 grouse 23 chicken hearts 24 Duck 25 smoked duck 26 Pheasant 27 minced chicken 28 chicken fillet 29 foie gras 30 chicken 31 chicken gutted 32 neck duck 3 FISH and SEAFOOD 1 anchovies 2 arctic char 3 mullet 4 Black Sea goby 5 shrimp head 6 Butterfish 7 scallops 8 dorado 9 Ruff 10 caviar 11 red caviar 12 Tobiko caviar 13 Squid 14 flounder 15 cuttlefish 16 Carp 17 Sprat 18 Smelt 19 crab sticks 20 Shrimps 21 King shrimps 22 Salad shrimps 23 Tiger prawn 24 Bream 25 salmon 26 Smoked salmon 27 Mussels 28 Mussels with shells 29 Pollock 30 Molluscs 31 Sea food 32 Sea fish 33 sole (fish) 34 Crab meat 35 Krill meat 36 Burbot 37 Perch 38 Lobster 39 Cisco 40 sturgeon 41 octopus 42 baby octopus 43 shrimp broth 44 halibut 45 Pangasius 46 cod liver oil 47 Haddock 48 Crayfish 49 dried crustaceans 50 Hot smoked fish 51 red fish salted 52 Swordfish 53 Saury 54 Sardines 55 Herring 56 Salmon 57 smoked salmon 58 salted salmon 59 Seabass 60 Whitefish 61 Ramp 62 Mackerel 63 smoked mackerel 64 Sheatfish 65 Starlet 66 Walleye 67 Dried seaweed 68 Tilapia 69 Carp 70 Cod 71 Hot smoked cod 72 black cod 73 Tuna 74 Turbot 75 Eel 76 smoked eel 77 Snails 78 Oysters 79 white fish fillets 80 catfish fillets 81 fillet of carp 82 fish fillet 83 salmon fillet 84 salted herring fillets 85 perch fillet 86 Trout 87 smoked trout 88 Squid Ink 89 cervical shrimp 90 cervical cancers 91 Sprats 92 Pike 4 VEGETABLES 2 Artichokes 3 Eggplant 4 Yam 5 broccoli tops 6 beet tops 7 Broccoli 8 Rutabaga 9 Galangal 10 Peas 11 pea sprouts 12 pea pods 13 green peas 14 Daikon 15 Melon 16 Ginseng 17 Ginger 18 Zucchini 19 Feces 20 Cabbage 21 Brussels sprouts 22 Sauerkraut 23 Chinese cabbage 24 Cabbage 25 Romanesco cabbage 26 savoy cabbage 27 Cauliflower 28 Potatoes 29 young potatoes 30 Kohlrabi 31 root anise 32 salsify root 33 parsley root 34 celery root 35 fresh corn 36 white onion 37 pearl bow 38 onion 39 red onion 40 dry onion 41 small onion 42 Shallots 43 cassava 44 mini corn 45 mini peppers 46 mini-tomatoes 47 carrots 48 cucumber 49 parsnips 50 squash 51 bell peppers 52 cayenne pepper 53 fresh chili pepper 54 jalapeno peppers 55 tomato 56 pickled tomatoes 57 cherry tomatoes 58 sunflower sprouts 59 wheat germ 60 soybean seedlings 61 germinated soybeans 62 rhubarb 63 Radish 64 wild radish 65 Turnip 66 beansprouts 67 Beet 68 Asparagus 69 chopped tomatoes 70 Sweet 71 Pumpkin 72 green beans 73 Fennel 74 physalis 75 horseradish 76 zucchini 77 Garlic 78 endive 5 FRUITS 1 Apricot 2 Avocado 3 quince 4 fresh pineapple 5 Orange 6 banana 7 Hawthorn 8 cranberries 9 grapes 10 Cherry 11 Dried cherries 12 blueberries 13 Garnet 14 Grapefruit 15 Pear 16 Blackberry 17 strawberries 18 pomegranate seeds 19 carambola 20 Kiwi 21 Strawberry 22 Cranberry 23 coconut 24 gooseberry 25 kumquat 26 Lime 27 lemon 28 Litchi 29 raspberries 30 mango 31 Mandarin 32 Passionfruit 33 mini pineapple 34 Nectarine 35 buckthorn 36 papaya 37 Peach 38 Pomelo 39 Rowan 40 Drain 41 red currants 42 black currant 43 tamarind 44 Feijoa 45 fruit to taste 46 persimmon 47 cherries 48 Cherry 49 blueberries 50 Apple 51 frozen berries 52 juniper berries 53 fresh berries 6 GROCERY 1 Agar 2 Adjika 3 rice paper 4 vanilla extract 5 vermicelli rice 6 egg noodles 7 Algae 8 Glucose 9 Jam 10 raspberry jam 11 fresh yeast 12 Gelatin 13 liquid Smokehouse 14 Sweetener 15 corn muffins 16 Ketchup 17 citric acid 18 Candy 19 Confiture 20 strawberry jam 21 food dye 22 Starch 23 potato starch 24 corn starch 25 bread crumbs 26 Noodles 27 buckwheat noodles 28 Pad Thai noodles 29 rice noodles 30 glass noodles 31 noodles harusame 32 egg noodles 33 Mayonnaise 34 poppy sweet 35 Pasta 36 cannelloni pasta 37 pasta lumakoni 38 pasta feathers 39 fusilli pasta 40 pumpkin marmalade 41 jujube fruit 42 Marzipan 43 Mirin 44 coconut milk 45 almond milk 46 soy milk 47 Muesli 48 Pasta 49 peanut paste 50 red curry paste 51 tamarind paste 52 Tom Yam Paste 53 chili paste 54 Molasses 55 Pectin 56 Penne 57 Jam 58 elderberry syrup 59 vanilla syrup 60 syrup vishnevny 61 ginger syrup 62 caramel syrup 63 maple syrup 64 strawberry syrup 65 coffee syrup 66 corn syrup 67 raspberry syrup 68 mango syrup 69 honey syrup 70 almond syrup 71 walnut syrup 72 blackcurrant syrup 73 chocolate syrup 74 cranberry sauce 75 worcestershire sauce 76 pomegranate sauce 77 kimchi sauce 78 Pesto 79 fish sauce 80 fish sauce nampla 81 Tabasco sauce 82 teriyaki sauce 83 sauce tkemali 84 oyster sauce 85 sweet chili sauce 86 Japanese walnut sauce 87 spaghetti 88 crumbs of white bread 89 breadcrumbs 90 pastry decorations 91 candied 7 MILK PRODUCTS and EGGS 1 yogurt 2 natural yoghurt 3 Kefir 4 margarine 5 butter 6 melted butter 7 Milk 8 baked milk 9 buttermilk 10 curdled 11 cream 12 sour cream 13 Whey 14 Thane 15 Curd 16 curd beaded 17 quail eggs 18 Egg 8 MUSHROOMS 2 mushrooms 3 Ceps 4 Enoki mushrooms 5 Chinese dried mushrooms 6 portobello mushrooms 7 dried mushrooms 8 shiitake mushrooms 9 milkmushrooms 10 chanterelles 11 boletus 12 honey fungus 13 saffron milk cap 14 morels 15 truffles 16 meadow mushrooms 9 CHEESE 1 cheese 2 cheese Adyghe 3 brie cheese 4 feta cheese 5 Burrata cheese 6 Gouda cheese 7 Dutch cheese 8 blue cheese 9 Gorgonzola 10 grana padano cheese 11 Gruyere cheese 12 Dor Blue cheese 13 Camembert 14 goat cheese 15 cheese sausage 16 mascarpone cheese 17 Monterey Jack cheese 18 mozzarella cheese 19 soft cheese 20 goat cheese 21 parmesan cheese 22 pecorino cheese 23 processed cheese 24 cheese Poshehonsky 25 ricotta cheese 26 Roquefort cheese 27 blue cheese 28 cream cheese 29 suluguni 30 cheese curd 31 feta cheese 32 philadelphia cheese 33 cheddar cheese 34 edam cheese 35 Emmentaler cheese 10 NUTS and DRIED FRUITS 1 peanuts 2 barberry 3 walnuts (peeled) 4 raisins 5 Figs 6 Chestnut 7 Dried cranberries 8 coconut 9 dried apricots 10 Filbert (hazelnut) 11 almonds 12 Nuts 13 pine nuts 14 cashew nuts 15 Dried peaches 16 sunflower seeds 17 pumpkin seeds 18 Dried Fruits 19 Dates 20 Pistachios 21 Hazelnuts 22 Prunes 11 BEVERAGES 1 Water 2 water orange 3 mineral water 4 water pink 5 GABA-tea 6 Hibiscus 7 Kvass 8 bread kvass 9 Coke 10 Kuding 11 Lemonade 12 Mate 13 Juice 14 carbonated drink 15 Bitter Brandy 16 Rooibos 17 pineapple juice 18 orange juice 19 birch juice 20 grape juice 21 cherry juice 22 pomegranate juice 23 strawberry juice 24 cranberry juice 25 gooseberry juice 26 lime juice 27 mango juice 28 tangerine juice 29 peach juice 30 currant juice 31 tomato juice 32 apple juice 33 Sprite 34 Tonic 35 tea white 36 tea yellow 37 green tea 38 red tea 39 Puer tea 40 Puer tea in Mandarin 41 oolong tea 42 black tea 43 Espresso 12 ALCOHOL 1 Balm 2 Bitter 3 Brandy 4 Bourbon 5 Vermouth 6 Wine 7 white wine 8 sparkling wine 9 red wine 10 dry red wine 11 wine sangria 12 Whiskey 13 Vodka 14 anise vodka 15 Grappa 16 Gin 17 Irish cream liqueur 18 Calvados 19 Cachaca 20 Brandy 21 Liqueur 22 orange liqueur 23 coffee liqueur 24 chocolate liqueur 25 Madeira 26 Marsala 27 Martini 28 Beer 29 cherry beer 30 Port 31 Rum 32 white rum 33 black rum 34 Sake 35 sambuca 36 Cider 37 tequila 38 sherry 39 Champagne (Brut) 40 schnapps 13 GREENS AND HERBS 1 Basil 2 basil red 3 bouquet garni 4 oregano 5 greens 6 dried herbs 7 cabbage pak choi 8 chervil 9 cilantro 10 oxalis 11 oat root 12 fresh coriander 13 nettle 14 Watercress 15 watercress 16 rose petals 17 lemongrass 18 bamboo leaves 19 banana leaves 20 grape leaves 21 Grape leaves (salty) 22 kaffir lime leaves 23 lime leaves 24 dandelion leaves 25 green onion 26 Leek 27 marjoram 28 Chard 29 melissa 30 lemon balm 31 Mint 32 oregano 33 parsley 34 dried parsley 35 plantain 36 wormwood 37 chopped camomile 38 arugula 39 iceberg lettuce 40 green salad 41 corn salad 42 lettuce 43 leaf lettuce 44 salad Mizuno 45 Oakleaf lettuce 46 radicchio salad 47 romaine lettuce 48 salad Friess 49 salad mix 50 celery 51 Lemon grass (lemon grass) 52 Italian herbs 53 spicy herbs 54 Dill 55 dandelion flowers 56 flowers 57 lavender flowers 58 chicory 59 thyme 60 Ramson 61 saffron 62 rosehips 63 chives 64 spinach 65 sorrel 66 tarragon 14 Cereals, legumes and flours 1 beans 2 mung beans 3 bulgur 4 puffed rice 5 buckwheat green 6 Quinoa 7 buckwheat 8 corn grits 9 semolina 10 Oats 11 pearl barley 12 cereal wheat 13 couscous 14 Flour 15 buckwheat flour 16 chestnut flour 17 corn flour 18 almond flour 19 Chickpea flour 20 oat flour 21 wheat flour 22 rye flour 23 rice flour 24 Chickpeas 25 Bran 26 Millet 27 Figure 28 Figure baya 29 basmati rice 30 brown rice 31 wild rice 32 Round grain rice 33 semola (flour made from durum wheat) 34 Beans 35 white beans 36 red beans 37 buckwheat flakes 38 cereal grains 39 oat flakes 40 Lentils 41 Barley 15 Spices and Seasonings 1 star anise 2 white pepper 3 Vanillin 4 Vanilla 5 vanilla essence 6 vanilla powder 7 Wasabi 8 Caltrop 9 garam masala 10 Carnation 11 cloves minced 12 Mustard 13 sweet mustard 14 allspice peas 15 grain mustard 16 Cumin 17 ground ginger 18 Capers 19 Cardamom 20 Curry 21 Coriander 22 ground coriander 23 Cinnamon 24 coffee essence 25 balsamic cream 26 Sesame 27 Turmeric 28 bay leaf 29 lemon pepper 30 poppy seed 31 Olives 32 olives dry 33 avocado oil 34 anchovy butter 35 peanut oil 36 mustard oil 37 oil for frying 38 scented oil 39 grapeseed oil 40 canola oil 41 corn oil 42 sesame oil 43 linseed oil 44 olive oil 45 Peanut butter 46 sunflower oil 47 lean oil 48 vegetable oil 49 oil, refined 50 oil seed-bearing 51 soybean oil 52 truffle oil 53 oil pumpkin 54 almonds hammers 55 miso paste 56 sea salt 57 Nutmeg 58 Olives 59 Ligurian olives 60 hot red pepper 61 hot peppers 62 Fenugreek 63 Paprika 64 lemongrass paste 65 peperoncini 66 pepper pink polka dots 67 Chili 68 Dried chili peppers 69 mustard powder 70 seasoning fish 71 baking powder 72 rosemary 73 pink ground pepper 74 Sugar 75 vanilla sugar 76 brown sugar 77 sugar muskovado 78 sugar cane 79 powdered sugar 80 nasturtium seeds 81 Nigella seeds 82 fennel seeds 83 spice mix “taco” 84 Soda 85 ginger juice squeezed 86 lemon juice 87 Salt 88 citrate 89 grape sauce 90 sauce narsharab 91 ponzu sauce 92 soy sauce 93 tomato sauce 94 chili sauce 95 Spices 96 sumac 97 thyme 98 cumin 99 Mediterranean herbs 100 French herbs 101 vinegar 102 balsamic vinegar 103 wine vinegar 104 white wine vinegar 105 red wine vinegar 106 cherry vinegar 107 raspberry vinegar 108 rice vinegar 109 apple cider vinegar 110 hops suneli 111 Savory 112 chutney 113 black pepper 114 black pepper peas 115 dry garlic 116 Sage 16 PREPARED PRODUCTS 1 canned pineapple 2 canned artichokes 3 Marinated artichokes 4 baguette 5 Loaf 6 Bars of chocolate 7 meringue 8 biscuit 9 beans, canned 10 Bun 11 buns for hamburgers 12 Broth 13 beef broth 14 chicken broth 15 fish broth 16 Jam 17 Apricot jam 18 lingonberry jam 19 cherry jam 20 black currant jam 21 raspberry jam 22 blueberry jam 23 Wafer 24 canned cherry 25 Glaze 26 Dijon mustard 27 croutons 28 marinated mushrooms 29 Demiglas apple 30 Yeast 31 Jelly 32 leaven 33 marshmallows 34 crushed tomatoes in juice 35 pickled ginger 36 Cocoa 37 marinated cactus 38 Pickled capers 39 sour cabbage 40 sea kale 41 Kimchi 42 wafer cakes 43 gherkins 44 natural coffee 45 instant coffee 46 Crackers 47 Chocolate Crumb 48 Croissant 49 bouillon cubes 50 canned corn 51 marinated corn 52 Pita 53 Lanspik 54 Ice 55 Letcho 56 lasagna sheets 57 canned salmon 58 pickled onions 59 canned mandarins 60 marshmallow 61 hazelnut oil 62 sweet curd 63 Yoghurt 64 Honey 65 honey in the comb 66 Mix ginger 67 condensed milk 68 condensed milk boiled 69 milk powder 70 pickled carrots 71 ice cream 72 vanilla ice cream 73 chocolate ice cream 74 salted cucumber 75 pickled cucumbers 76 pickled cucumbers 77 Pecans 78 beet broth 79 corn sticks 80 bread sticks 81 tomato paste 82 Pasta Chocolate 83 Pate 84 frozen dumplings 85 hot pepper pickled 86 canned peaches 87 Cookies 88 Biscuit 89 Cookies Savoiardi 90 chocolate cookies 91 Pita 92 Supplements 93 tomatoes in juice 94 canned tomatoes 95 Popcorn 96 Prosciutto 97 Gingerbread 98 mango puree 99 mashed potatoes 100 tomato puree 101 apple puree 102 pickle cucumber 103 Roll 104 Pickled beets 105 pork jerky 106 sugar syrup 107 whipped cream 108 cream of coconut 109 Malt 110 Sorbet 111 barbecue sauce 112 sauce bearnez 113 Béchamel 114 Worcestershire sauce 115 sauce Demiglas 116 sauce for soups “Bright udon” 117 sweet and sour sauce 118 Salsa 119 sweet sauce 120 chocolate sauce 121 berry sauce 122 asparagus, soya 123 caramel chips 124 crushed crackers 125 Tartlets 126 Tahini 127 pasta for lasagna 128 dough for ravioli 129 pizza dough 130 yeast dough 131 dough kataifi 132 shortbread dough 133 pastry dough 134 puff pastry 135 dough dry 136 filo pastry 137 dried tomatoes 138 Tortilla 139 Toast 140 Tofu 141 tuna fish oil 142 tuna canned in its own juice 143 Tahini 144 Rice Stuffing 145 Canned beans 146 white bread 147 toast bread 148 rye bread 149 sweet bread 150 black bread 151 rye bread 152 corn flakes 153 ciabatta 154 tea Away 155 potato chips 156 corn chips 157 Marinated mushrooms 158 chocolate corn balls 159 Chocolate 160 white chocolate 161 bitter chocolate 162 milk chocolate 163 dark chocolate
(459) TABLE-US-00003 TABLE C Lists of Food Preparation Methods and Equipment, Cuisine and Bases A list of food preparation methods 1; “0”; “The fried” 2; “0”; The boiled” 3; “0”; The stewed” 4; “0”; “The baked” 5; “0”; “The cut” A list of Equipment 1; “0”; “ KITCHEN ACCESSORIES” 2; “1”; “funnels” 3; “2”; “stainless steel funnel” 4; “2”; “plastic funnel” 5; “2”; “silicone funnel” 6; “2”; “convertible funnel” 7; “1”; “colanders” 8; “7”; “quadratic colanders” 9; “7”; “oval ladle-vases” 10; “7”; “colanders with folding handles” 11; “7”; “flat colander” 12; “7”; “plastic colanders” 13; “7”; “small round colanders” 14; “7”; “suspended colanders” 15; “7”; “cover-colander” 16; “7”; “stainless steel and aluminum colanders” 17; “7”; “cone colanders” 18; “1”; “Kitchen Appliances” 19; “18”; “whisk” 20; “18”; “scoop spatula” 21; “20”; “cook spatula” 22; “20”; “spatula with slots” 23; “20”; “confectionery spatula” 24; “18”; “spoons” 25; “24”serving spoon” 26; “24”; “spoon-tongs” 27; “24”; “spoon with slots” 28; “24”; “spoon for rice” 29; “24”; “ladle spoon” 30; “24”; “ice cream spoon” 31; “24”; “honey spoon” 32; “24”; “spaghetti spoon” 33; “24”; “serving spoon” 34; “18”; “confectionery syringe for cookies and cream” 35; “18”; “soup ladle” 36; “18”; “potato masher” 37; “18”; “skimmer” 38; “18”; “Meat fork” 39; “18”; “brush” 40; “18”; “coffee filter” 41; “18”; “whisk” 42; “18”; “silicone brush” 43; “18”; “silicone juicer” 44; “18”; “earthen saucer” 45; “18”; “tea filter” 46; “18”; “pump dispenser for oil and vinegar” 47; “18”; “clip for silicone spoon for the edge of the pan” 48; “18”; “transformed spoons for salad” 49; “18”; “device for cherry seeds removing” 50; “18”; “sink mat” 51; “18”; “ties for rolls” 52; “18”; “dough mini-scraper” 53; “18”; “grill tongs” 54; “18”; “spaghetti tongs” 55; “18”; “ice tongs” 56; “18”; “sugar tongs” 57; “18”; “package clip” 58; “18”; “package clip” 59; “18”; “citrus spray” 60; “18”; “Dough press” 61; “18”; “scoop for bulk” 62; “18”; “salad serving tongs (tweezers)” 63; “18”; “accessories for tubes” 64; “18”; “pestle” 65; “18”; “mortar” 66; “18”; “roller for cutting of the rings” 67; “18”; “opener for caps” 68; “18”; “meat tenderizer; meat softener” 69; “18”; “egg yolk separator” 70; “18”; “apron” 71; “18”; “tools for decoration” 72; “18”; “jar for oil and vinegar” 73; “18”; “mug for milk boiling” 74; “18”; “napkins” 75; “18”; “tablecloth” 76; “18”; “marker for glasses” 78; “18”; “basket” 79; “18”; “meat tenderizer” 80; “18”; “cocotte” 81; “18”; “brush for washing of the vegetables” 82; “18”; “lids for cups” 83; “18”; “rope for baking” 84; “18”; “jar for herbs storage” 86; “18”; “scraper for glass ceramic plates” 87; “18”; “Teapot for tea” 88; “18”; “clothespin for notes on the fridge” 89; “18”; “railing systems” 90; “18”; “hanger for kitchen tools” 91; “18”; plunger with not adhering surface” 92; “18”; “silicone plunger” 93; “18”; “rolling pin with adjustable thickness” 94; “18”; “vacuum bags with pump” 95; “18”; “gas lighter” 96; “18”; “bone forceps” 97; “1”; “kitchen timers thermometers” 98; “97”; “timer for meat roasting” 99; “97”; “digital thermometer” 100; “97”; “holder for thermometer” 101; “97”; “meat thermometer” 102; “97”; “digital timer” 103; “97”; “electr. digital timer” 104; “97”; “aramel thermometer” 105; “1”; “Mills for spices” 106; “105”; “mill for black pepper” 107; “105”; “electric mill” 108; “105”; “combined mill for pepper and salt (2 in 1)” 109; “105”; “mill for spices” 110; “105”; “mill for greens” 111; “1”; “Measuring utensils” 112; “111”; “Measuring container (plastic bottle)” 113; “111”; “measuring jar” 114; “111”; “measuring jug” 115; “111”; “measuring bowl” 116; “111”; “mechanical dispenser for ice cream” 117; “1”; “mechanical mixers” 118; “1”; “bowl” 119; “118”; “metal bowl” 120; “118”; “stainless steel bowl” 121; “118”; “plastic bowl” 122; “118”; “plastic bowl” 123; “118”; “bowls for food” 124; “1”; “sets” 125; “124”; “wine set” 126; “124”; “sets for spices” 127; “124”; “cupcakes baking set” 128; “124”; “accessory kit for baking” 129; “124”; “set of bar tools” 130; “124”; “set of kitchen tools” 131; “124”; “Set for eggs and pancakes baking” 132; “1”; “Slicing and cutting of products” 133; “132”; “cutter” 134; “132”; “holder for onions cutting” 135; “132”; “cutting boards” 136; “132”; “universal professional knives” 137; “132”; “kitchen shears” 138; “132”; “hatchet” 139; “132”; “meat hatchet” 140; “132”; “Hammer for meat with hatchet” 141; “132”; “hoe” 142; “132”; “Hammer for meat” 143; “132”; “knives” 144; “143”; “knife for greens” 145; “143”; “knife for oranges” 146; “143”; “knife for kiwi” 147; “143”; “knife for pineapple” 148; “143”; “Spiral knife for carrots” 149; “143”; “multifunctional knife” 150; “143”; “vegetable knife” 151; “143”; “Pizza Cutter” 152; “143”; “universal knife” 153; “143”; “knife for slicing” 154; “143”; “cook knife” 155; “143”; “gastronomic knife” 156; “143”; “opener” 157; “143”; “Cheese knife” 158; “143”; “boning knife” 159; “143”; “lettuce knife” 160; “143”; “knife for steaks” 161; “143”; “butcher knife” 162; “143”; “shredding knife” 163; “143”; “bread knife” 164; “143”; “fish knife” 165; “143”; “knife for sandwiches” 166; “143”; “Santoku knife” 167; “143”; “knife for fruit coring” 168; “143”; “Butter knife” 169; “169”; “openers” 170; “169”; “tin-opener” 171; “169”; “corkscrew” 172; “169”; “corkscrew on a stand” 173; “169”; “lever corkscrew” 174; “169”; “folding corkscrew” 175; “169”; “opener for waiter” 178; “494”; “stands for hot” 179; “494”; “stand for kitchen utensils storing” 180; “494”; “toothpick holder” 181; “494”; “Bottle holder” 182; “494”; “Holder for capsules” 183; “494”; “stand for spoon” 184; “494”; “stand for coffee capsules” 185; “494”; “Coasters” 186; “494”; “Napkin holder” 187; “494”; “stand for eggs” 188; “494”; “stand for openers” 189; “494”; “stand for scoops” 190; “494”; “stand for cooking and serving of eggs” 191; “494”; “stand for ladle” 192; “494”; “Holder for paper towels” 193; “494”; “Transforming stand for kitchen appliances” 194; “494”; “stand for mug” 195; “494”; “stand mugs and saucers” 196; “494”; “stand for kitchen knives” 197; “494”; “stand for chicken” 198; “494”; “napkin-stand” 199; “494”; “heated stand” 200; “494”; “stands for cake” 201; “1”; “Appliances for peeling and cutting” 202; “201”; “grater for vegetables” 203; “305”; “grater” 204; “201”; “garlic masher” 205; “201”; “egg cutter” 206; “201”; “Manual vegetable cutter” 207; “201”; “Peeler for vegetables” 208; “201”; “Nutcracker” 209; “201”; “The device for separating the yolks from the whites” 210; “201”; “grasping for carrots cleaning” 211; “201”; “scraper fish scales” 212; “201”; “cutter for fruits” 213; “201”; “oller for holes” 214; “201”; “tongs for fish bones” 215; “201”; “spiral vegetable cutter” 216; “1”; “Bottle Caps” 217; “216”; “champagne cork (stopper)” 218; “216”; “stoppers for wine” 219; “216”; “The opener to remove the crown corks from bottles” 220; “1”; “sieves” 221; “220”; “sieve for tea” 222; “220”; “sieve-tongs for tea” 223; “220”; “Strainer for spices” 224; “220”; “Strainer for tea” 225; “220”; “Universal sieve” 226; “220”; “flour sieve” 228; “220”; “The Chinese sieve with a mesh insert” 229; “220”; “sieve with support” 230; “220”; “Mug-sieve for flour” 231; “220”; “sieve on the handle” 232; “1”; “Salt and pepper shakers” 233; “282”; “container for seasoning” 234; “232”; “salt cellar” 235; “232”; “containers for oil and vinegar” 236; “1”; “Dish dryers” 237; “236”; “salad dryer” 238; “236”; “dryer-placemat” 239; “236”; “dryer for crockery and cutlery” 240; “1”; “Cutlery Accessories” 241; “240”; “cutlery tray” 242; “240”; “cutlery holder” 243; “240”; “cutlery container” 244; “240”; “strainer for cutlery” 245; “240”; “wall hanger for kitchen tools” 246; “240”; “cutlery organizer” 247; “240”; “mat for cutlery” 248; “240”; “sliding tray for cutlery” 249; “240”; “dryer for cutlery” 250; “240”; “glass for cutlery” 251; “240”; “napkin for the cutlery” 252; “240”; “case for cutlery” 253; “240”; “tray for cutlery” 254; “240”; “mitten-potholder” 255; “240”; “box for cutlery” 256; “240”; “full-size rack (cassette) for cutlery” 257; “240”; “Stand without containers for cutlery” 258; “240”; “cassette for cutlery” 259; “240”; “container for cutlery” 260; “240”; “station for cutlery” 261; “240”; “Shelf for cutlery” 262; “1”; “Decorations for cocktails” 263; “262”; “ducts” 264; “262”; “sticks” 266; “496”; “molds for ice” 267; “496”; “molds for children” 268; “496”; “Molds for shaping products” 269; “496”; “Molds for dumplings” 271; “497”; “Measuring container” 272; “497”; “A mixing container with the dispenser” 273; “497”; “Measuring container with the funnel” 274; “497”; “beaker” 275; “497”; “scoop” 276; “1”; “kitchen scissors” 277; “276”; “Scissors for BBQ” 278; “276”; “Kitchen scissors with bottle opener” 279; “276”; “Scissors for greens” 280; “276”; “Kitchen multipurpose scissors” 281; “276”; “Kitchen scissors for poultry” 282; “1”; “utensil for storage” 283; “282”; “container for storage” 284; “282”; “ Bottles for liquid spices oils” 285; “282”; “jars for storage” 286; “282”; “lunchbox” 287; “282”; “foldable lunchbox” 288; “282”; “jar for hermetic storage of bulk products' 289; “282”; “Sprayer for oil/vinegar” 290; “282”; “jar for bulk products” 291; “282”; “containers for spices” 293; “282”; “Container for tea” 294; “1”; “potholders” 295; “294”; “oven-glove” 296; “294”; “silicone potholders” 297; “294”; “dishcloth” 298; “1”; “railing with hooks” 299; “1”; “silicone mats” 300; “299”; “baking mat” 301; “299”; “mat for baking cakes” 302; “299”; “mat for drying of the glasses” 303; “299”; “cooking mat” 304; “299”; “Mat for drying of the dishes” 305; “1”; “graters presses rubbing machines” 306; “305”; “grater with a handle” 308; “305”; “multifunction grater” 309; “305”; “grater shredder” 310; “305”; “grind for the green” 311; “305”; “grind for the garlic” 312; “305”; “Slicer for tomatoes” 313; “305”; “grater with rotating drums” 314; “305”; “universal device for grinding” 315; “305”; “mechanical grater” 316; “305”; “garlic peeling tube” 317; “305”; “rubbing machine” 318; “305”; “press for vegetables” 319; “305”; “press for garlic” 320; “305”; “press for hamburgers” 321; “1”; “knife sharpener” 322; “321”; “electric sharpener” 323; “321”; “sharpening stone” 324; “321”; “ceramic sharpener” 325; “1”; “breadbox” 326; “1”; “lattice with legs” 327; “339”; “Flatware” 328; “327”; “for alcohol” 329; “540”; “Cognac set with the batcher” 330; “540”; “Glasses souvenir” 331; “540”; “Glasses” 332; “540”; “Bucket for ice” 333; “540”; “Shot glasses” 334; “540”; “Bucket for champagne” 335; “540”; “Wine glasses” 336; “540”; “decanter” 337; “540”; “tray” 338; “540”; “Support under a bottle” 339; “327”; “tableware” 340; “339”; “first course dish” 341; “339”; “dish for bouillon” 342; “339”; “bouillon bowl” 343; “339”; “oiler” 344; “339”; “round dish” 345; “339”; “duck pan” 346; “339”; “Set for making chocolate fondue 347; “339”; “Set for making cheese fondue” 348; “339”; “salad bowl” 349; “339”; “dish for cake” 350; “339”; “compartmental dish” 351; “339”; “set of cutlery” 352; “339”; “serving spoon and fork” 353; “339”; “dish with lid” 354; “339”; “steam table” 355; “374”; “ice-cream bowl” 357; “339”; “saucer” 358; “339”; “saucer for jam” 359; “339”; “mustard-pot” 360; “339”; “pepper-pot” 361; “339”; “ash-pot” 362; “339”; “deep table plate” 363; “339”; “dinner plate” 364; “339”; “snack plate” 365; “339”; “deep dessert plate” 366; “339”; “dessert plate” 367; “339”; “plate for pies” 368; “339”; “horseradish-pot” 369; “327”; “Utensils for table” 370; “369”; “Pad for tableware” 371; “369”; “serving mat” 372; “369”; “serving tray” 373; “369”; “glass burner” 374; “327”; “Dishes for tea coffee desert” 375; “374”; “sugar-bowl” 376; “374”; “mug” 377; “374”; “mug with teapot” 378; “374”; “mug with stand” 379; “374”; “mug with lid” 380; “374”; “tea set” 381; “374”; “dish” 382; “374”; “french-press” 383; “374”; “teapot” 384; “374”; “teapot with strainer” 385; “374”; “glass teapot” 387; “374”; “multifunctional vase” 388; “540”; “Glasses” 389; “374”; “soup bowl” 390; “374”; “wicker basket” 391; “374”; “vase 3-tier” 393; “374”; “napkin rings” 394; “374”; “pannier for fruits” 395; “374”; “table trash basket” 396; “374”; “biscuit dish” 397; “374”; “candy dish” 398; “374”; “coffee sets” 399; “327”; “CUTLERY” 437; “0”; “Kitchen appliances” 438; “437”; “aerogrill” 439; “437”; “blenders grinder” 440; “437”; “coffee Maker” 441; “437”; “coffee grinder (coffee mill)” 442; “437”; “Food Processor” 443; “437”; “mixer” 444; “437”; “mini oven” 445; “437”; “multicooker” 446; “437”; “meat grinders” 447; “437”; “steamers” 448; “437”; “Raclette grill” 449; “437”; “Juicers” 450; “437”; “toasters” 451; “437”; “egg cooker” 452; “437”; “electric range” 453; “437”; “electric induction stove” 454; “437”; “electric kettle” 455; “437”; “thermopots” 456; “437”; “bread makers” 457; “437”; “microwaves” 458; “437”; “weights for kitchen” 459; “437”; “electric driers” 461; “0”; “Children's dishes” 462; “461”; “Children Sets for baking” 463; “461”; “Children cutlery” 464; “461”; “Children thermoses” 465; “461”; “Children Sets of dishes” 488; “437”; “deep fryer” 491; “339”; “baking sheet” 494; “1”; “stand and holders” 495; “220”; “sieve to form the “Bird's Nest”” 496; “1”; “mold” 497; “1”; “Measuring container” 498; “339”; “pan” 499; “339”; “frying pan” 500; “437”; “Cookware for induction cookers” 501; “437”; “Juice cookers” 502; “437”; “Milk cooker” 503; “437”; “Covers/splash screens” 504; “437”; “Microwave cookware” 505; “437”; “Braziers roasters” 506; “437”; “Turk” 507; “437”; “Dumpling (manti) cookers” 508; “437”; “Sets” 509; “437”; “Samovars” 510; “437”; “Kasans” 511; “437”; “Electric stove” 512; “437”; “Casseroles (pans)” 513; “512”; “casseroles (pans) with non-stick coating” 514; “512”; “aluminum casseroles (pans)” 515; “512”; “Stainless steel casseroles (pans)” 516; “512”; “Enameled casseroles (pans)” 517; “512”; “Teflon coated casseroles (pans)” 518; “512”; “Heat-proof glass casseroles (pans)” 519; “512”; “Ladles” 520; “512”; “Ceramic casseroles (pans)” 521; “512”; “Set of casseroles (pans)” 522; “512”; “Pressure cooker” 523; “512”; “Pan-steamer” 524; “512”; “casseroles for induction cookers” 525; “512”; “Pan-fryer” 526; “512”; “Cast iron casserole (pot)” 527; “512”; “Titanium casserole” 528; “437”; “Frying pans skillet” 529; “528”; “Frying pan with ceramic coating” 530; “528”; “Frying pans with non-stick coating” 531; “528”; “Frying pan with removable handle” 532; “528”; “Stewpots” 533; “528”; “Frying pans for grill” 534; “528”; “Wok” 535; “528”; “Pancake pans” 536; “528”; “Electric frying pans” 537; “528”; “Cast iron skillet” 538; “528”; “Multifunctional frying pan” 539; “528”; “Titanium frying pan” 540; “437”; “Drinkware” 541; “540”; “Wine glasses” 542; “540”; “Water glasses” 543; “540”; “Beer glasses” 544; “540”; “Kegs” 545; “540”; “Carafes” 546; “540”; “Decanters” 547; “540”; “Jugs” 548; “540”; “Shots” 549; “540”; “Wine glasses for champagne” 550; “540”; “Glasses for brandy/cognac” 551; “540”; “Wine glasses for a cocktail/martini” A list of Cuisine 1; “0”; “Abkhaz” 2; “0”; “Australian” 3; “0”; “Austrian” 4; “0”; “Azerbaijan” 5; “0”; “Albanian” 6; “0”; “Algerian” 7; “0”; “American” 8; “0”; “English” 9; “0”; “Arabic” 10; “0”; “Argentine” 11; “0”; “Armenian” 12; “0”; “Bashkir” 13; “0”; “Belarusian” 14; “0”; “Belgian” 15; “0”; “Bulgarian” 16; “0”; “Bosnian” 17; “0”; “Brazilian” 18; “0”; “Hungarian” 19; “0”; “Venezuelan” 20; “0”; “Vietnamese” 21; “0”; “Greek” 22; “0”; “Georgian” 23; “0”; “Danish” 24; “0”; “Jewish” 25; “0”; “Israeli” 26; “0”; “Indian” 27; “0”; “Indonesian” 28; “0”; “Jordanian” 29; “0”; “Iraqi” 30; “0”; “Iranian” 31; “0”; “Irish” 32; “0”; “Icelandic” 33; “0”; “Spanish” 34; “0”; “Italian” 35; “0”; “Cambodian” 36; “0”; “Canadian” 37; “0”; “Cypriot” 38; “0”; “Chinese” 39; “0”; “Colombian” 40; “0”; “Korean” 41; “0”; “Creole” 42; “0”; “Costa Rica” 43; “0”; “Latvian” 44; “0”; “Lebanese” 45; “0”; “Libyan” 46; “0”; “Lithuanian” 47; “0”; “Macedonian” 48; “0”; “Malaysian” 49; “0”; “Moroccan” 50; “0”; “Mexican” 51; “0”; “Moldavian” 52; “0”; “Mongolian” 53; “0”; “German” 54; “0”; “Dutch” 55; “0”; “New Zealand” 56; “0”; “Norwegian” 57; “0”; “Ossetian” 58; “0”; “Pakistani” 59; “0”; “Palestinian” 60; “0”; “Panamanian” 61; “0”; “Peruvian” 62; “0”; “Polish” 63; “0”; “Portuguese” 64; “0”; “Romanian” 65; “0”; “Russian” 66; “0”; “Serbian” 67; “0”; “Singaporean” 68; “0”; “Syrian” 69; “0”; “Slovak” 70; “0”; “Slovenian” 71; “0”; “Thai” 72; “0”; “Tatar” 73; “0”; “Tibetan” 74; “0”; “Tunisian” 75; “0”; “Turkish” 76; “0”; “Turkmen” 77; “0”; “Ukrainian” 78; “0”; “Philippine” 79; “0”; “Finnish” 80; “0”; “French” 81; “0”; “Croatian” 82; “0”; “Montenegrin” 83; “0”; “Czech” 84; “0”; “Chilean” 85; “0”; “Chuvash” 86; “0”; “Chukotka” 87; “0”; “Swedish” 88; “0”; “Swiss” 89; “0”; “Scottish” 90; “0”; “Ecuadorian” 91; “0”; “Estonian” 92; “0”; “Japanese” 93; “0”; “Raw food diet” 94; “0”; “European” 95; “0”; “International” 96; “0”; “Multinational” 97; “0”; “Lean” 98; “0”; “Caucasian” 99; “0”; “Children” A list of bases: 1; “0”; “Meat and meat products” 2; “1”; “Basturma” 3; “1”; “Fat” 4; “1”; “brisket cooked and smoked” 5; “1”; “Hare” 6; “1”; “leather duck” 7; “1”; “Sausage” 8; “1”; “Sausages” 9; “1”; “Sausages “Hunting party”” 10; “1”; “Horsemeat” 11; “1”; “Bones with bone marrow” 12; “1”; “Roe” 13; “1”; “Rabbit” 14; “1”; “Meat” 15; “1”; “Moosemeat” 16; “1”; “Venison” 17; “1”; “Liver” 18; “1”; “Kidney” 19; “1”; “Smoked ribs” 20; “1”; “Salami” 21; “1”; “Sausages” 22; “1”; “Cervelat” 23; “1”; “Sausages” 24; “1”; “Hungarian smoked bacon” 25; “l”; “bacon fat-tailed” 26; “1”; “Steak” 27; “1”; “ribeye steak” 28; “1”; “Farce” 29; “1”; “crocodile fillet” 30; “1”; “Jamon” 31; “1”; “Choriso (spanish sausage)” 32; “1”; “Skewers” 33; “1”; “Sowbelly” 34; “1”; “Deer tongue” 35; “1”; “LAMB” 36; “35”; “breast of lamb” 37; “35”; “loin of lamb” 38; “35”; “blade lamb” 39; “35”; “veal brains” 40; “35”; “mutton ham” 41; “35”; “veal ham” 42; “35”; “leg of lamb” 43; “35”; “Heel muscle mutton” 44; “35”; “lamb offal” 45; “35”; “veal kidneys” 46; “35”; “lamb chops” 47; “35”; “gras cow” 48; “35”; “veal heart” 49; “35”; “lamb testicles” 50; “35”; “VEAL” 51; “35”; “veal cheeks” 52; “35”; “minced lamb” 53; “35”; “minced veal” 54; “35”; “veal tail” 55; “35”; “veal tongue” 56; “35”; “eggs bullish” 57; “1”; “BEEF” 58; “57”; “beef brisket” 59; “57”; “” 60; “57”; “beef (sirloin)” 61; “57”; “beef on the bone” 62; “57”; “beef eye muscle” 63; “57”; “legs of beef” 64; “57”; “ham beef” 65; “57”; “gras beef” 66; “57”; “beef ribs” 67; “57”; “beef heart” 68; “57”; “minced beef” 69; “57”; “tail beef” 70; “57”; “beef tongue” 71; “1”; “PORK” 72; “71”; “bacon” 73; “71”; “smoked bacon” 74; “71”; “pork” 75; “71”; “ham” 76; “71”; “pork brisket” 77; “71”; “smoked pork belly” 78; “71”; “Pork Intestine” 79; “71”; “legs of pork” 80; “71”; “boar ham” 81; “71”; “pork ham” 82; “71”; “pork ribs” 83; “71”; “knuckle of pork” 84; “71”; “fat” 85; “71”; “pork (pork neck or loin)” 86; “71”; “pork ears” 87; “71”; “minced pork” 88; “71”; “pig tail” 89; “71”; “pork tongue” 90; “0”; “Birds” 91; “90”; “garshnep” 92; “90”; “turkey breast” 93; “90”; “chicken breast” 94; “90”; “chicken breast smoked” 95; “90”; “duck breast” 96; “90”; “Goose” 97; “90”; “chicken ventricles” 98; “90”; “turkey” 99; “90”; “turkey wings” 100; “90”; “chicken wings” 101; “90”; “chicken” 102; “90”; “smoked chicken” 103; “90”; “grouse” 104; “90”; “coot” 105; “90”; “duck leg” 106; “90”; “crow's feet” 107; “90”; “chicken legs” 108; “90”; “chicken ham” 109; “90”; “quail” 110; “90”; “gras chicken” 111; “90”; “chicken giblets” 112; “90”; “grouse” 113; “90”; “chicken hearts” 114; “90”; “duck” 115; “90”; “smoked duck” 116; “90”; “Pheasant” 117; “90”; “minced chicken” 118; “90”; “chicken fillet” 119; “90”; “foie gras” 120; “90”; “chicken” 121; “90”; “chicken gutted” 122; “90”; “neck duck” 123; “0”; “FISH and SEAFOOD” 124; “123”; “anchovies” 125; “123”; “arctic char” 126; “123”; “mullet” 127; “123”; “Black Sea goby” 128; “123”; “shrimp head” 129; “123”; “Butterfish” 130; “123”; “scallops” 131; “123”; “dorado” 132; “123”; “ruff” 133; “123”; “caviar” 134; “123”; “red caviar” 135; “123”; “Tobiko caviar” 136; “123”; “squid” 137; “123”; “flounder” 138; “123”; “cuttlefish” 139; “123”; “carp” 140; “123”; “sprat” 141; “123”; “smelt” 142; “123”; “crab sticks” 143; “123”; “Shrimps” 144; “123”; “King shrimps” 145; “123”; “Salad shrimps” 146; “123”; “Tiger prawn” 147; “123”; “Bream” 148; “123”; “salmon” 149; “123”; “Smoked salmon” 150; “123”; “Mussels” 151; “123”; “Mussels with shells' 152; “123”; “Pollock” 153; “123”; “Molluscs” 154; “123”; “Sea food” 155; “123”; “Sea fish” 156; “123”; “sole (fish)” 157; “123”; “Crab meat” 158; “123”; “Krill meat” 159; “123”; “Burbot” 160; “123”; “Frog legs” 161; “123”; “Perch” 162; “123”; “Lobster” 163; “123”; “cisco” 164; “123”; “sturgeon” 165; “123”; “octopus” 166; “123”; “baby octopus” 167; “123”; “shrimp broth” 168; “123”; “halibut” 169; “123”; “Pangasius” 170; “123”; “cod liver oil” 171; “123”; “haddock” 172; “123”; “crayfish” 173; “123”; “dried crustaceans” 174; “123”; “Hot smoked fish” 175; “123”; “red fish salted” 176; “123”; “swordfish” 177; “123”; “saury” 178; “123”; “sardines” 179; “123”; “herring” 180; “123”; “salmon” 181; “123”; “smoked salmon” 182; “123”; “salted salmon” 183; “123”; “seabass” 184; “123”; “whitefish” 185; “123”; “ramp” 186; “123”; “mackerel” 187; “123”; “smoked mackerel” 188; “123”; “sheatfish” 189; “123”; “starlet” 190; “123”; “walleye” 191; “123”; “Dried seaweed” 192; “123”; “tilapia” 193; “123”; “carp” 194; “123”; “cod” 195; “123”; “Hot smoked cod” 196; “123”; “black cod” 197; “123”; “tuna” 198; “123”; “turbot” 199; “123”; “eel” 200; “123”; “smoked eel” 201; “123”; “snails” 202; “123”; “oysters” 203; “123”; “white fish fillets” 204; “123”; “catfish fillets” 205; “123”; “fillet of carp” 206; “123”; “fish fillet” 207; “123”; “salmon fillet” 208; “123”; “salted herring fillets” 209; “123”; “perch fillet” 210; “123”; “trout” 211; “123”; “smoked trout” 212; “123”; “Squid Ink” 213; “123”; “cervical shrimp” 214; “123”; “cervical cancers” 215; “123”; “sprats” 216; “123”; “pike” 217; “0”; “VEGETABLES” 218; “217”; “watermelon” 219; “217”; “Artichokes” 220; “217”; “eggplant” 221; “217”; “yam” 222; “217”; “broccoli tops” 223; “217”; “beet tops” 224; “217”; “broccoli” 225; “217”; “rutabaga” 226; “217”; “galangal” 227; “217”; “peas” 228; “217”; “pea sprouts” 229; “217”; “pea pods” 230; “217”; “green peas” 231; “217”; “daikon” 232; “217”; “melon” 233; “217”; “Ginseng” 234; “217”; “Ginger” 235; “217”; “zucchini” 236; “217”; “feces” 237; “217”; “cabbage” 238; “217”; “Brussels sprouts” 239; “217”; “sauerkraut” 240; “217”; “Chinese cabbage” 241; “217”; “Cabbage” 242; “217”; “Romanesco cabbage” 243; “217”; “savoy cabbage” 244; “217”; “cauliflower” 245; “217”; “potatoes” 246; “217”; “young potatoes' 247; “217”; “kohlrabi” 248; “217”; “root anise” 249; “217”; “salsify root” 250; “217”; “parsley root” 251; “217”; “celery root” 252; “217”; “fresh corn” 253; “217”; “white onion” 254; “217”; “pearl bow” 255; “217”; “onion” 256; “217”; “red onion” 257; “217”; “dry onion” 258; “217”; “small onion” 259; “217”; “Shallots” 260; “217”; “cassava” 261; “217”; “mini corn” 262; “217”; “mini peppers” 263; “217”; “mini-tomatoes” 264; “217”; “carrots” 265; “217”; “cucumber” 266; “217”; “parsnips” 267; “217”; “squash” 268; “217”; “bell peppers” 269; “217”; “cayenne pepper” 270; “217”; “fresh chili pepper” 271; “217”; “jalapeno peppers” 272; “217”; “tomato” 273; “217”; “pickled tomatoes” 274; “217”; “cherry tomatoes” 275; “217”; “sunflower sprouts” 276; “217”; “wheat germ” 277; “217”; “soybean seedlings” 278; “217”; “germinated soybeans” 279; “217”; “rhubarb” 280; “217”; “Radish” 281; “217”; “wild radish” 282; “217”; “Turnip” 283; “217”; “beansprouts” 284; “217”; “beet” 285; “217”; “Asparagus” 286; “217”; “chopped tomatoes” 287; “217”; “Sweet” 288; “217”; “Pumpkin” 289; “217”; “green beans” 290; “217”; “Fennel” 291; “217”; “physalis” 292; “217”; “horseradish” 293; “217”; “zucchini” 294; “217”; “garlic” 295; “217”; “endive” 296; “0”; “FRUITS” 297; “296”; “Apricot” 298; “296”; “Avocado” 299; “296”; “quince” 300; “296”; “fresh pineapple” 301; “296”; “Orange” 302; “296”; “banana” 303; “296”; “Hawthorn” 304; “296”; “cranberries” 305; “296”; “grapes” 306; “296”; “Cherry” 307; “296”; “Dried cherries” 308; “296”; “blueberries” 309; “296”; “Garnet” 310; “296”; “Grapefruit” 311; “296”; “pear” 312; “296”; “Blackberry” 313; “296”; “strawberries” 314; “296”; “pomegranate seeds” 315; “296”; “carambola” 316; “296”; “kiwi” 317; “296”; “Strawberry” 318; “296”; “Cranberry” 319; “296”; “coconut” 320; “296”; “gooseberry” 321; “296”; “kumquat” 322; “296”; “Lime” 323; “296”; “lemon” 324; “296”; “Litchi” 325; “296”; “raspberries” 326; “296”; “mango” 327; “296”; “Mandarin” 328; “296”; “Passionfruit” 329; “296”; “mini pineapple 330; “296”; “Nectarine” 331; “296”; “buckthorn” 332; “296”; “papaya” 333; “296”; “Peach” 334; “296”; “Pomelo” 335; “296”; “Rowan” 336; “296”; “drain” 337; “296”; “red currants” 338; “296”; “black currant” 339; “296”; “tamarind” 340; “296”; “Feijoa” 341; “296”; “fruit to taste” 342; “296”; “persimmon” 343; “296”; “cherries” 344; “296”; “Cherry” 345; “296”; “blueberries” 346; “296”; “apple” 347; “296”; “frozen berries” 348; “296”; “juniper berries” 349; “296”; “fresh berries” 350; “0”; “GROCERY” 351; “350”; “agar” 352; “350”; “adjika” 353; “350”; “rice paper” 354; “350”; “vanilla extract” 355; “350”; “vermicelli rice” 356; “350”; “egg noodles” 357; “350”; “algae” 358; “350”; “glucose” 359; “350”; “jam” 360; “350”; “raspberry jam” 361; “350”; “fresh yeast” 362; “350”; “gelatin” 363; “350”; “liquid Smokehouse” 364; “350”; “sweetener” 365; “350”; “corn muffins” 366; “350”; “ketchup” 367; “350”; “citric acid” 368; “350”; “candy” 369; “350”; “confiture” 370; “350”; “strawberry jam” 371; “350”; “food dye” 372; “350”; “starch” 373; “350”; “potato starch” 374; “350”; “corn starch” 375; “350”; “bread crumbs” 376; “350”; “Noodles” 377; “350”; “buckwheat noodles” 378; “350”; “Pad Thai noodles” 379; “350”; “rice noodles” 380; “350”; “glass noodles” 381; “350”; “noodles harusame” 382; “350”; “egg noodles” 383; “350”; “mayonnaise” 384; “350”; “poppy sweet” 385; “350”; “pasta” 386; “350”; “cannelloni pasta” 387; “350”; “pasta lumakoni” 388; “350”; “pasta feathers” 389; “350”; “fusilli pasta” 390; “350”; “pumpkin marmalade” 391; “350”; “jujube fruit” 392; “350”; “marzipan” 393; “350”; “mirin” 394; “350”; “coconut milk” 395; “350”; “almond milk” 396; “350”; “soy milk” 397; “350”; “muesli” 398; “350”; “Pasta” 399; “350”; “peanut paste” 400; “350”; “red curry paste” 401; “350”; “tamarind paste” 402; “350”; “Tom Yam Paste” 403; “350”; “chili paste” 404; “350”; “molasses” 405; “350”; “pectin” 406; “350”; “Penne” 407; “350”; “jam” 408; “350”; “elderberry syrup” 409; “350”; “vanilla syrup” 410; “350”; “syrup vishnevny” 411; “350”; “ginger syrup” 412; “350”; “caramel syrup” 413; “350”; “maple syrup” 414; “350”; “strawberry syrup” 415; “350”; “coffee syrup” 416; “350”; “corn syrup” 417; “350”; “raspberry syrup” 418; “350”; “mango syrup” 419; “350”; “honey syrup” 420; “350”; “almond syrup” 421; “350”; “walnut syrup” 422; “350”; “blackcurrant syrup” 423; “350”; “chocolate syrup” 424; “350”; “cranberry sauce” 425; “350”; “worcestershire sauce” 426; “350”; “pomegranate sauce” 427; “350”; “kimchi sauce” 428; “350”; “pesto” 429; “350”; “fish sauce” 430; “350”; “fish sauce nam pla” 431; “350”; “Tabasco sauce” 432; “350”; “teriyaki sauce” 433; “350”; “sauce tkemali” 434; “350”; “oyster sauce” 435; “350”; “sweet chili sauce” 436; “350”; “Japanese walnut sauce” 437; “350”; “spaghetti” 438; “350”; “; “crumbs of white bread” 439; “350”; “breadcrumbs” 440; “350”; “pastry decorations” 441; “350”; “candied” 442; “0”; “MILK PRODUCTS and EGGS” 443; “442”; “yogurt” 444; “442”; “natural yoghurt” 445; “442”; “kefir” 446; “442”; “margarine” 447; “442”; “butter” 448; “442”; “melted butter” 449; “442”; “milk” 450; “442”; “baked milk” 451; “442”; “buttermilk” 452; “442”; “curdled” 453; “442”; “cream” 454; “442”; “sour cream” 455; “442”; “whey” 456; “442”; “Thane” 457; “442”; “curd” 458; “442”; “curd beaded” 459; “442”; “quail eggs” 460; “442”; “egg” 461; “0”; “mushrooms” 462; “461”; “oyster mushrooms” 463; “461”; “” 464; “461”; “ceps” 465; “461”; “Enoki mushrooms” 466; “461”; “Chinese dried mushrooms” 467; “461”; “portobello mushrooms” 468; “461”; “dried mushrooms” 469; “461”; “shiitake mushrooms” 470; “461”; “milkmushrooms” 471; “461”; “chanterelles” 472; “461”; “boletus” 473; “461”; “honey fungus” 474; “461”; “saffron milk cap” 475; “461”; “morels” 476; “461”; “truffles” 477; “461”; “meadow mushrooms” 478; “0”; “CHEESE” 479; “478”; “cheese” 480; “478”; “cheese Adyghe” 481; “478”; “brie cheese” 482; “478”; “feta cheese” 483; “478”; “Burrata cheese” 484; “478”; “Gouda cheese” 485; “478”; “Dutch cheese” 486; “478”; “blue cheese” 487; “478”; “Gorgonzola” 488; “478”;” ; “grana padano cheese” 489; “478”; “Gruyere cheese” 490; “478”; “-”; “Dor Blue cheese” 491; “478”; “Camembert” 492; “478”; “goat cheese” 493; “478”; “cheese sausage” 494; “478”; “mascarpone cheese” 495; “478”; “Monterey Jack cheese” 496; “478”; “mozzarella cheese” 497; “478”; “soft cheese” 498; “478”; “goat cheese” 499; “478”; “parmesan cheese” 500; “478”; “pecorino cheese” 501; “478”; “processed cheese” 502; “478”; “cheese Poshehonsky” 503; “478”; “ricotta cheese” 504; “478”; “Roquefort cheese” 505; “478”; “blue cheese” 506; “478”; “cream cheese” 507; “478”; “suluguni” 508; “478”; “cheese curd” 509; “478”; “feta cheese” 510; “478”; “philadelphia cheese” 511; “478”; “cheddar cheese” 512; “478”; “edam cheese” 513; “478”; “Emmentaler cheese” 514; “0”; “NUTS and DRIED FRUITS” 515; “514”; “peanuts” 516; “514”; “barberry” 517; “514”; “walnuts (peeled)” 518; “514”; “raisins” 519; “514”; “figs” 520; “514”; “Chestnut” 521; “514”; “Dried cranberries” 522; “514”; “coconut” 523; “514”; “dried apricots” 524; “514”; “Filbert (hazelnut)” 525; “514”; “almonds” 526; “514”; “nuts” 527; “514”; “pine nuts” 528; “514”; “cashew nuts” 529; “514”; “Dried peaches” 530; “514”; “sunflower seeds” 531; “514”; “pumpkin seeds” 532; “514”; “Dried Fruits” 533; “514”; “Dates” 534; “514”; “pistachios” 535; “514”; “hazelnuts” 536; “514”; “prunes” 537; “0”; “BEVERAGES” 538; “537”; “water” 539; “537”; “water orange” 540; “537”; “mineral water” 541; “537”; “water pink” 542; “537”; “GABA-tea” 543; “537”; “Hibiscus” 544; “537”; “kvass” 545; “537”; “bread kvass” 546; “537”; “-; “Coke” 547; “537”; “Kuding” 548; “537”; “lemonade” 549; “537”; “mate” 550; “537”; “juice” 551; “537”; “carbonated drink” 552; “537”; “Bitter Brandy” 553; “537”; “Rooibos” 554; “537”; “pineapple juice” 555; “537”; “orange juice” 556; “537”; “birch juice” 557; “537”; “grape juice” 558; “537”; “cherry juice” 559; “537”; “pomegranate juice” 560; “537”; “strawberry juice” 561; “537”; “cranberry juice” 562; “537”; “gooseberry juice” 563; “537”; “lime juice” 564; “537”; “mango juice” 565; “537”; “tangerine juice” 566; “537”; “peach juice” 567; “537”; “currant juice” 568; “537”; “tomato juice” 569; “537”; “apple juice” 570; “537”; “sprite” 571; “537”; “tonic” 572; “537”; “tea white” 573; “537”; “tea yellow” 574; “537”; “green tea” 575; “537”; “red tea” 576; “537”; “Puer tea” 577; “537”; “Puer tea in Mandarin” 578; “537”; “oolong tea” 579; “537”; “black tea” 580; “537”; “Espresso” 581; “0”; “ALCOHOL” 582; “581”; “Balm” 583; “581”; “Bitter” 584; “581”; “brandy” 585; “581”; “bourbon” 586; “581”; “vermouth” 587; “581”; “wine” 588; “581”; “white wine” 589; “581”; “sparkling wine” 590; “581”; “red wine” 591; “581”; “dry red wine” 592; “581”; “wine sangria” 593; “581”; “whiskey” 594; “581”; “Vodka” 595; “581”; “anise vodka” 596; “581”; “grappa” 597; “581”; “gin” 598; “581”; “-”; “Irish cream liqueur” 599; “581”; “Calvados” 600; “581”; “Cachaca” 601; “581”; “brandy” 602; “581”; “liqueur” 603; “581”; “orange liqueur” 604; “581”; “coffee liqueur” 605; “581”; “chocolate liqueur” 606; “581”; “Madeira” 607; “581”; “Marsala” 608; “581”; “Martini” 609; “581”; “beer” 610; “581”; “cherry beer” 611; “581”; “port” 612; “581”; “rum” 613; “581”; “white rum” 614; “581”; “black rum” 615; “581”; “Sake” 616; “581”; “sambuca” 617; “581”; “cider” 618; “581”; “tequila” 619; “581”; “sherry” 620; “581”; “( )”; “Champagne (Brut)” 621; “581”; “schnapps” 622; “0”; “GREENS AND HERBS” 623; “622”; “basil” 624; “622”; “basil red” 625; “622”; “bouquet garni” 626; “622”; “oregano” 627; “622”; “greens” 628; “622”; “dried herbs” 629; “622”; “-; “cabbage pak choi” 630; “622”; “chervil” 631; “622”; “cilantro” 632; “622”; “oxalis” 633; “622”; “oat root” 634; “622”; “fresh coriander” 635; “622”; “nettle” 636; “622”; “Watercress” 637; “622”; “watercress” 638; “622”; “rose petals” 639; “622”; “lemongrass” 640; “622”; “bamboo leaves” 641; “622”; “banana leaves” 642; “622”; “grape leaves” 643; “622”; “Grape leaves (salty)” 644; “622”; “kaffir lime leaves” 645; “622”; “lime leaves” 646; “622”; “dandelion leaves” 647; “622”; “green onion” 648; “622”; “-; “Leek” 649; “622”; “marjoram” 650; “622”; “chard” 651; “622”; “melissa” 652; “622”; “lemon balm” 653; “622”; “Mint” 654; “622”; “oregano” 655; “622”; “parsley” 656; “622”; “dried parsley” 657; “622”; “plantain” 658; “622”; “wormwood” 659; “622”; “chopped camomile” 660; “622”; “arugula” 661; “622”; “iceberg lettuce” 662; “622”; “green salad” 663; “622”; “corn salad” 664; “622”; “lettuce” 665; “622”; “leaf lettuce” 666; “622”; “salad Mizuno” 667; “622”; “: “Oakleaf lettuce” 668; “622”; “radicchio salad” 669; “622”; “romaine lettuce” 670; “622”; “salad Friess” 671; “622”; “salad mix” 672; “622”; “celery” 673; “622”; “Lemon grass (lemon grass)” 674; “622”; “Italian herbs” 675; “622”; “spicy herbs” 676; “622”; “dill” 677; “622”; “dandelion flowers” 678; “622”; “flowers” 679; “622”; “lavender flowers” 680; “622”; “chicory” 681; “622”; “thyme” 682; “622”; “Ramson” 683; “622”; “saffron” 684; “622”; “rosehips” 685; “622”; “chives” 686; “622”; “spinach” 687; “622”; “sorrel” 688; “622”; “tarragon” 689; “0”; “Cereals legumes and flours” 690; “689”; “beans” 691; “689”; “mung beans” 692; “689”; “bulgur” 693; “689”; “puffed rice” 694; “689”; “buckwheat green” 695; “689”; “Quinoa” 696; “689”; “buckwheat” 697; “689”; “corn grits” 698; “689”; “semolina” 699; “689”; “oats” 700; “689”; “pearl barley” 701; “689”; “cereal wheat” 702; “689”; “couscous” 703; “689”; “flour” 704; “689”; “buckwheat flour” 705; “689”; “chestnut flour” 706; “689”; “corn flour” 707; “689”; “almond flour” 708; “689”; “Chickpea flour” 709; “689”; “oat flour” 710; “689”; “wheat flour” 711; “689”; “rye flour” 712; “689”; “rice flour” 713; “689”; “chickpeas” 714; “689”; “bran” 715; “689”; “millet” 716; “689”; “Figure” 717; “689”; “Figure baya” 718; “689”; “basmati rice” 719; “689”; “brown rice” 720; “689”; “wild rice” 721; “689”; “Round grain rice” 722; “689”; “semola (flour made from durum wheat)” 723; “689”; “Beans” 724; “689”; “white beans” 725; “689”; “red beans” 726; “689”; “buckwheat flakes” 727; “689”; “cereal grains” 728; “689”; “oat flakes” 729; “689”; “lentils” 730; “689”; “barley” 731; “0”; “Spices and Seasonings” 732; “731”; “star anise” 733; “731”; “white pepper” 734; “731”; “vanillin” 735; “731”; “vanilla” 736; “731”; “vanilla essence” 737; “731”; “vanilla powder” 738; “731”; “wasabi” 739; “731”; “caltrop” 740; “731”; “garam masala” 741; “731”; “Carnation” 742; “731”; “cloves minced” 743; “731”; “mustard” 744; “731”; “sweet mustard” 745; “731”; “allspice peas” 746; “731”; “grain mustard” 747; “731”; “Cumin” 748; “731”; “ground ginger” 749; “731”; “capers” 750; “731”; “cardamom” 751; “731”; “curry” 752; “731”; “coriander” 753; “731”; “ground coriander” 754; “731”; “cinnamon” 755; “731”; “coffee essence” 756; “731”; “balsamic cream” 757; “731”; “sesame” 758; “731”; “turmeric” 759; “731”; “bay leaf” 760; “731”; “lemon pepper” 761; “731”; “poppy seed” 762; “731”; “olives” 763; “731”; “olives dry” 764; “731”; “avocado oil” 765; “731”; “anchovy butter” 766; “731”; “peanut oil” 767; “731”; “mustard oil” 768; “731”; “oil for frying” 769; “731”; “scented oil” 770; “731”; “grapeseed oil” 771; “731”; “canola oil” 772; “731”; “corn oil” 773; “731”; “sesame oil” 774; “731”; “linseed oil” 775; “731”; “olive oil” 776; “731”; “Peanut butter” 777; “731”; “sunflower oil” 778; “731”; “lean oil” 779; “731”; “vegetable oil” 780; “731”; “oil refined” 781; “731”; “oil seed-bearing” 782; “731”; “soybean oil” 783; “731”; “truffle oil” 784; “731”; “oil pumpkin” 785; “731”; “almonds hammers” 786; “731”; “miso paste” 787; “731”; “sea ??salt” 788; “731”; “nutmeg” 789; “731”; “olives” 790; “731”; “Ligurian olives” 791; “731”; “hot red pepper” 792; “731”; “hot peppers” 793; “731”; “fenugreek” 794; “731”; “paprika” 795; “731”; “lemongrass paste” 796; “731”; “peperoncini” 797; “731”; “pepper pink polka dots” 798; “731”; “chili” 799; “731”; “Dried chili peppers” 800; “731”; “mustard powder” 801; “731”; “seasoning fish” 802; “731”; “baking powder” 803; “731”; “rosemary” 804; “731”; “pink ground pepper” 805; “731”; “sugar” 806; “731”; “vanilla sugar” 807; “731”; “brown sugar” 808; “731”; “sugar muskovado” 809; “731”; “sugar cane” 810; “731”; “powdered sugar” 811; “731”; “nasturtium seeds” 812; “731”; “Nigella seeds” 813; “731”; “fennel seeds” 814; “731”; “”; “spice mix “taco”” 815; “731”; “Soda” 816; “731”; “ginger juice squeezed” 817; “731”; “lemon juice” 818; “731”; “salt” 819; “731”; “citrate” 820; “731”; “grape sauce” 821; “731”; “sauce narsharab” 822; “731”; “ponzu sauce” 823; “731”; “soy sauce” 824; “731”; “tomato sauce” 825; “731”; “chili sauce” 826; “731”; “Spices” 827; “731”; “sumac” 828; “731”; “thyme” 829; “731”; “cumin” 830; “731”; “Mediterranean herbs” 831; “731”; “French herbs” 832; “731”; “vinegar” 833; “731”; “balsamic vinegar” 834; “731”; “wine vinegar” 835; “731”; “white wine vinegar” 836; “731”; “red wine vinegar” 837; “731”; “cherry vinegar” 838; “731”; “raspberry vinegar” 839; “731”; “rice vinegar” 840; “731”; “apple cider vinegar” 841; “731”; -”; “hops suneli” 842; “731”; “Savory” 843; “731”; “chutney” 844; “731”; “black pepper” 845; “731”; “black pepper peas” 846; “731”; “dry garlic” 847; “731”; “sage” 848; “0”; “PREPARED PRODUCTS” 849; “848”; “canned pineapple” 850; “848”; “canned artichokes” 851; “848”; “Marinated artichokes” 852; “848”; “baguette” 853; “848”; “loaf” 854; “848”; “Bars of chocolate” 855; “848”; “meringue” 856; “848”; “biscuit” 857; “848”; “beans canned” 858; “848”; “bun” 859; “848”; “buns for hamburgers” 860; “848”; “broth” 861; “848”; “beef broth” 862; “848”; “chicken broth” 863; “848”; “fish broth” 864; “848”; “Jam” 865; “848”; “Apricot jam” 866; “848”; “lingonberry jam” 867; “848”; “cherry jam” 868; “848”; “black currant jam” 869; “848”; “raspberry jam” 870; “848”; “blueberry jam” 871; “848”; “Wafer” 872; “848”; “canned cherry” 873; “848”; “Glaze” 874; “848”; “Dijon mustard” 875; “848”; “croutons” 876; “848”; “marinated mushrooms” 877; “848”; “Demiglas apple” 878; “848”; “yeast” 879; “848”; “Jelly” 880; “848”; “leaven” 881; “848”; “marshmallows” 882; “848”; “crushed tomatoes in juice” 883; “848”; “pickled ginger” 884; “848”; “Cocoa” 885; “848”; “marinated cactus” 886; “848”; “Pickled capers” 887; “848”; “sour cabbage” 888; “848”; “sea ??kale” 889; “848”; “Kimchi” 890; “848”; “wafer cakes” 891; “848”; “gherkins” 892; “848”; “natural coffee” 893; “848”; “instant coffee” 894; “848”; “crackers” 895; “848”; “Chocolate Crumb” 896; “848”; “croissant” 897; “848”; “bouillon cubes” 898; “848”; “canned corn” 899; “848”; “marinated corn” 900; “848”; “pita” 901; “848”; “lanspik” 902; “848”; “ice” 903; “848”; “letcho” 904; “848”; “lasagna sheets” 905; “848”; “canned salmon” 906; “848”; “pickled onions” 907; “848”; “canned mandarins” 908; “848”; “marshmallow” 909; “848”; “hazelnut oil” 910; “848”; “sweet curd” 911; “848”; “yoghurt” 912; “848”; “honey” 913; “848”; “honey in the comb” 914; “848”; “Mix ginger” 915; “848”; “condensed milk” 916; “848”; “condensed milk boiled” 917; “848”; “milk powder” 918; “848”; “pickled carrots” 919; “848”; “ice cream” 920; “848”; “vanilla ice cream” 921; “848”; “chocolate ice cream” 922; “848”; “salted cucumber” 923; “848”; “pickled cucumbers” 924; “848”; “pickled cucumbers” 925; “848”; “pecans” 926; “848”; “beet broth” 927; “848”; “corn sticks” 928; “848”; “bread sticks” 929; “848”; “tomato paste” 930; “848”; “Pasta Chocolate” 931; “848”; “pate” 932; “848”; “frozen dumplings” 933; “848”; “hot pepper pickled” 934; “848”; “canned peaches” 935; “848”; “Cookies” 936; “848”; “Biscuit” 937; “848”; “Cookies Savoiardi” 938; “848”; “chocolate cookies” 939; “848”; “pita” 940; “848”; “supplements” 941; “848”; “tomatoes in juice” 942; “848”; “canned tomatoes” 943; “848”; “popcorn” 944; “848”; “prosciutto” 945; “848”; “gingerbread” 946; “848”; “mango puree” 947; “848”; “mashed potatoes” 948; “848”; “tomato puree” 949; “848”; “apple puree” 950; “848”; “pickle cucumber” 951; “848”; “roll” 952; “848”; “Pickled beets” 953; “848”; “pork jerky” 954; “848”; “sugar syrup” 955; “848”; “whipped cream” 956; “848”; “cream of coconut” 957; “848”; “malt” 958; “848”; “sorbet” 959; “848”; “barbecue sauce” 960; “848”; “sauce bearnez” 961; “848”; “bechamel” 962; “848”; “Worcestershire sauce” 963; “848”; “sauce Demiglas” 964; “848”; “” 965; “848”; “sweet and sour sauce” 966; “848”; “salsa” 967; “848”; “sweet sauce” 968; “848”; “chocolate sauce” 969; “848”; “berry sauce” 970; “848”; “asparagus soya” 971; “848”; “caramel chips” 972; “848”; “crushed crackers” 973; “848”; “tartlets” 974; “848”; “tahini” 975; “848”; “pasta for lasagna” 976; “848”; “dough for ravioli” 977; “848”; “pizza dough” 978; “848”; “yeast dough” 979; “848”; “dough kataifi” 980; “848”; “shortbread dough” 981; “848”; “pastry dough” 982; “848”; “puff pastry” 983; “848”; “dough dry” 984; “848”; “filo pastry” 985; “848”; “dried tomatoes” 986; “848”; “Tortilla” 987; “848”; “toast” 988; “848”; “tofu” 989; “848”; “tuna fish oil” 990; “848”; “tuna canned in its own juice” 991; “848”; “Tahini” 992; “848”; “ Rice Stuffing” 993; “848”; “Canned beans” 994; “848”; “white bread” 995; “848”; “toast bread” 996; “848”; “rye bread” 997; “848”; “sweet bread” 998; “848”; “black bread” 999; “848”; “rye bread” 1000; “848”; “corn flakes” 1001; “848”; “ciabatta” 1002; “848”; “tea Away” 1003; “848”; “potato chips” 1004; “848”; “corn chips” 1005; “848”; “Marinated mushrooms” 1006; “848”; “chocolate corn balls” 1007; “848”; “Chocolate” 1008; “848”; “white chocolate” 1009; “848”; “bitter chocolate” 1010; “848”; “milk chocolate” 1011; “848”; “dark chocolate” 1012; “50”; “veal fillet” 1013; “57”; “beef fillet” 1014; “848”; “sauce for soups “Bright udon”” 1015; “296”; “Lemon” 1016; “217”; “Carrots” 1017; “217”; “Tomato”
(460) TABLE-US-00004 TABLE D Types of Cuisine and Dishes Types of Cuisine 1 Abkhaz cuisine 2 Australian cuisine 3 Austrian cuisine 4 Azerbaijan cuisine 5 Albanian cuisine 6 Algerian cuisine 7 American cuisine 8 English cuisine 9 Arabic cuisine 10 Argentine cuisine 11 Armenian cuisine 12 Bashkir cuisine 13 Belarusian cuisine 14 Belgian cuisine 15 Bulgarian cuisine 16 Bosnian cuisine 17 Brazilian cuisine 18 Hungarian cuisine 19 Venezuelan cuisine 20 Vietnamese cuisine 21 Greek cuisine 22 Georgian cuisine 23 Danish cuisine 24 Jewish cuisine 25 Israeli cuisine 26 Indian cuisine 27 Indonesian cuisine 28 Jordanian cuisine 29 Iraqi cuisine 30 Iranian cuisine 31 Irish cuisine 32 Icelandic cuisine 33 Spanish cuisine 34 Italian cuisine 35 Cambodian cuisine 36 Canadian cuisine 37 Cypriot cuisine 38 Chinese cuisine 39 Colombian cuisine 40 Korean cuisine 41 Creole cuisine 42 Costa Rica cuisine 43 Latvian cuisine 44 Lebanese cuisine 45 Libyan cuisine 46 Lithuanian cuisine 47 Macedonian cuisine 48 Malaysian cuisine 49 Moroccan cuisine 50 Mexican cuisine 51 Moldavian cuisine 52 Mongolian cuisine 53 German cuisine 54 Dutch cuisine 55 Zealand cuisine 56 Norwegian cuisine 57 Ossetian cuisine 58 Pakistani cuisine 59 Palestinian cuisine 60 Panamanian cuisine 61 Peruvian cuisine 62 Polish cuisine 63 Portuguese cuisine 64 Romanian cuisine 65 Russian cuisine 66 Serbian cuisine 67 Singaporean cuisine 68 Syrian cuisine 69 Slovak cuisine 70 Slovenian cuisine 71 Thai cuisine 72 Tatar cuisine 73 Tibetan cuisine 74 Tunisian cuisine 75 Turkish cuisine 76 Turkmen cuisine 77 Ukrainian cuisine 78 Philippine cuisine 79 Finnish cuisine 80 French cuisine 81 Croatian cuisine 82 Montenegrin cuisine 83 Czech cuisine 84 Chilean cuisine 85 Chuvash cuisine 86 Chukotka cuisine 87 Swedish cuisine 88 Swiss cuisine 89 Scottish cuisine 90 Ecuadorian cuisine 91 Estonian cuisine 92 Japanese cuisine 93 Raw food diet 94 Estonian cuisine 95 Japanese cuisine 96 Raw food diet 1 Types of Dishes 1 Snacks 2 Salads 3 Entrees 4 Main Dishes 5 Desserts 6 Drinks 7 Sauces and marinades 8 Baking 9 Semimanufactures and preservatives
(461) TABLE-US-00005 TABLE E List of Robotic Food Preparation System (One Embodiment) Level Sys Responsible Major of No Category Category System Party(s) Challenges Completion Notes 01 Hardware Robot Hands Productionization, Robustness, Cost, Weight 02 Hardware Robot Arms 03 Hardware Robot Armature Rails 04 Hardware Capture/ Dynamic 3D Vision Training System 05 Hardware Capture/ Data Input Training 06 Hardware Capture/ Editing System Training 07 Hardware Kitchen Cabinets Module 08 Hardware Kitchen Fixtures Module 09 Hardware Kitchen Lighting with ability to Module computer-operating mode 10 Hardware Kitchen Protection/Safety Module Screen with ability to computer-operating mode 11 Hardware Kitchen Appliances Module 12 Hardware Kitchen Automatic Storage Module device with ability to computer-operating mode 13 Hardware Kitchen Automatic modular Module dispenser for flowing, liquid ingredients and water with ability to computer-operating mode Hardware Kitchen Freshness ingredients Module analytical device Hardware Kitchen Built-in electronic scales Module (in the tabletop) with ability to computer- operating mode 14 Hardware Kitchen Cleaning Module 15 Hardware Kitchen Waste Disposal Module Hardware Kitchen Multi-functional Module professional steam- oven with ability to computer-operating mode Hardware Kitchen Multi-functional Module professional kitchen processor with ability to computer-operating mode Hardware Kitchen Top-loaded dishwasher Module with ability to computer-operating mode Hardware Kitchen Professional Stove with Module turning control regulators/buttons operated with ability to computer-operating mode Hardware Kitchen Standard dimension Module layout Hardware Kitchen Anti-wieting, smoke, Module steam ventilation system autonomous or be connected to the duct with ability to computer-operating mode Hardware Kitchen Kitchen sink with tap Module with ability to computer-operating mode 16 Hardware Control/ CPU Power Modules 17 Hardware Control/ I/O Touchscreen Power Modules 18 Hardware Control/ Power Supply Power Modules 19 Hardware Accessories Utensils 20 Hardware Accessories Food Containers/Cartridges 21 Software Robot OS Module Hardware Kitchen Professional Stove with Module turning control regulators/buttons operated with ability to computer-operating mode 21 Software Robot OS Module 22 Software Robot Apps Module 23 Software Robot hand firmware Module 24 Software Robot Arm firmware Module 25 Software Robot Rail Control Module 26 Software Capture/ OS Training 27 Software Capture/ apps Training 28 Software Capture/ Vision Training 29 Software Capture/ Data Input Training 30 Software Capture/ Editing System Training 31 Software Kitchen OS Module 32 Software Kitchen App Module 33 Software Kitchen Controller Module Protection/Safety 34 Software Kitchen Controller, Appliances Module 35 Software Kitchen Controller, Storage Module 36 Software Kitchen Controller, Cleaning Module 36 Software Kitchen Controller, Steam-oven Module 36 Software Kitchen Controller, Kitchen Module Processor 36 Software Kitchen Controller, Dishwasher Module 36 Software Kitchen Controller, Stove Module 36 Software Kitchen Controller, Ventilation Module system 36 Software Kitchen Controller, Lighting Module 37 Software Kitchen Controller, Waste Module 37 Software Kitchen Controller, Tap Module 37 Software Kitchen Controller, Dispensing Module device 37 Software Kitchen Controller, Scales Module 37 Software Kitchen Controller, Freshness Module Indicator 38 Software Control/ OS Power Modules 39 Software Control/ I/O Touchscreen Power Modules 40 Software Control/ Control Apps Power Modules 41 Other Food Food Recipe Development 42 Other Food Food Container Prep 43 Other Food Food Order/Delivery 44 Other Logistics Safety/Regulatory 45 Other Logistics Sales/Distribution 46 Other Logistics Installation/Maintenance 47 Other Logistics Packaging/Shipping Container 48 Other Logistics Return Management 49 Other Logistics Technical Training 50 Other Logistics Manuals 51 Other Logistics Warranty 52 Production Robot 53 Production Kitchen Module 54 Production Integration/ Shipping 55 Production 56 Production 57 Production 58 Production 59 Production 60 Production
(462) The present invention has been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the invention may be practiced in other embodiments. The particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. The system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements, or entirely in software elements. The particular division of functionality between the various systems components described herein is merely example and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.
(463) In various embodiments, the present invention can be implemented as a system or a method for performing the above-described techniques, either singly or in any combination. The combination of any specific features described herein is also provided, even if that combination is not explicitly described. In another embodiment, the present invention can be implemented as a computer program product comprising a computer-readable storage medium and computer program code, encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-described techniques.
(464) As used herein, any reference to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase “in one embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
(465) Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally perceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, transformed, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
(466) It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that, throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “displaying” or “determining” or the like refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
(467) Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware, and/or hardware, and, when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
(468) The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers and/or other electronic devices referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
(469) The algorithms and displays presented herein are not inherently related to any particular computer, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references above to specific languages are provided for disclosure of enablement and best mode of the present invention.
(470) In various embodiments, the present invention can be implemented as software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device can include, for example, a processor, an input device (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination thereof), an output device (such as a screen, speaker, and/or the like), memory, long-term storage (such as magnetic storage, optical storage, and/or the like), and/or network connectivity, according to techniques that are well known in the art. Such an electronic device may be portable or non-portable. Examples of electronic devices that may be used for implementing the invention include a mobile phone, personal digital assistant, smartphone, kiosk, desktop computer, laptop computer, consumer electronic device, television, set-top box, or the like. An electronic device for implementing the present invention may use an operating system such as, for example, iOS available from Apple Inc. of Cupertino, Calif., Android available from Google Inc. of Mountain View, Calif., Microsoft Windows 7 available from Microsoft Corporation of Redmond, Wash., webOS available from Palm, Inc. of Sunnyvale, Calif., or any other operating system that is adapted for use on the device. In some embodiments, the electronic device for implementing the present invention includes functionality for communication over one or more networks, including for example a cellular telephone network, wireless network, and/or computer network such as the Internet.
(471) Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term “connected” to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.
(472) As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
(473) The terms “a” or “an,” as used herein, are defined as one or more than one. The term “plurality,” as used herein, is defined as two or more than two. The term “another,” as used herein, is defined as at least a second or more.
(474) An ordinary artisan should require no additional explanation in developing the methods and systems described herein but may find some possibly helpful guidance in the preparation of these methods and systems by examining standardized reference works in the relevant art.
(475) While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised which do not depart from the scope of the present invention as described herein. It should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. The terms used should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims but should be construed to include all methods and systems that operate under the claims set forth herein below. Accordingly, the invention is not limited by the disclosure, but instead its scope is to be determined entirely by the following claims.