METHODS AND SYSTEMS FOR FOOD PREPARATION IN A ROBOTIC COOKING KITCHEN
20190381654 ยท 2019-12-19
Inventors
Cpc classification
G05B2219/40395
PHYSICS
A47J36/321
HUMAN NECESSITIES
G05B19/42
PHYSICS
International classification
B25J9/00
PERFORMING OPERATIONS; TRANSPORTING
Abstract
The present disclosure is directed to methods, computer program products, and computer systems for instructing a robot to prepare a food dish by replacing the human chef's movements and actions. Monitoring a human chef is carried out in an instrumented application-specific setting, a standardized robotic kitchen in this instance, and involves using sensors and computers to watch, monitor, record and interpret the motions and actions of the human chef, in order to develop a robot-executable set of commands robust to variations and changes in the environment, capable of allowing a robotic or automated system in a robotic kitchen to prepare the same dish to the standards and quality as the dish prepared by the human chef.
Claims
1. A robotic kitchen system, comprising: one or more robotic arms; one or more robotic end effectors coupled to the one or more robotic arms, each end effector coupled to a respective robotic arm; at least one processor communicatively coupled to the one or more robotic arms, the at least one processor being operable to: receive an electronic recipe file including a machine-executable sequential commands script from a source; and control the one or more robotic arms and the one or more robotic end effectors to replicate one or more cooking operations by executing the machine-executable sequential commands script of the electronic recipe file, each corresponding arm and end effector performing a cooking operation upon receiving a command.
2. The system of claim 1, further comprising one or more sensors configured to collect sensor data, each corresponding arm and end effector performing a cooking operation upon receiving a command based at least in part on the collected sensor data.
3. The system of claim 2, wherein the sensor data for operating in an instrumented environment, comprising (a) detecting the presence or absence of one or more objects in a standard position in an instrumented environment, (b) identifying an object type, position and orientation for a standard object placed in a non-standard position, and (c) identifying an object type, shape, dimensions, positions and orientations for a non-standard object placed in a non-standard position.
4. The system of claim 1, wherein the processor is computed to collect sensor data as feedback to confirm or make real-time adjustment to a current operation.
5. The system of claim 1, wherein the processor is computed to collect sensor data as feedback on one or smart appliances, one or more tools, or one or more ingredients for command execution.
6. The system of claim 5, wherein sensor data comprises object status data, appliance status data, temperature data, humidity data, object color data, and object changing-shape data.
7. The system of claim 1, wherein each command in the machine-executable sequential commands script comprises at least one preprogrammed cooking operation.
8. The system of claim 1, wherein each command in the machine-executable sequential commands script comprises at least one pretested cooking operation.
9. The system of claim 1, wherein the machine-executable sequential commands script, executed by the processor, comprises at least one command for controlling for controlling one or more smart appliances.
10. The system of claim 1, wherein each corresponding arm and end effector performs a cooking operation upon receiving a cooking command by a user.
11. The system of claim 1, wherein each corresponding arm and end effector performs a cooking operation upon receiving a command from the electronic recipe file.
12. The system of claim 1, wherein the electronic recipe file is prerecorded, or generated based on one or more recipe creator real-time commands, via a computer interface.
13. The system of claim 1, wherein the command to perform a cooking operation comprises a specific type of cooking operation, at least one parameter of the specific cooking operation, and timing data associated with the specific cooking operation.
14. The system of claim 1, wherein the robotic kitchen system operates in a first mode and a second mode: during the first mode, the processor recording the machine-executable sequential commands script; and during the second mode, the processor executing the machine-executable sequential commands script.
15. A robotic kitchen system, comprising: a kitchen module defining an instrumented environment for which the robotic kitchen module operates within; one or more robotic arms; one or more robotic end effectors coupled to the one or more robotic arms, each end effector coupled to a respective robotic arm; one or more sensors are configured to collect sensor data; at least one processor communicatively coupled to the one or more robotic arms, the at least one processor being operable to: receive an electronic recipe file including a machine-executable sequential command script from a source; and control the one or more robotic arms and the one or more robotic end effectors within the kitchen module to replicate one or more food preparation operations by executing the machine-executable sequential command script of the electronic recipe file, each corresponding arm and end effector performing a cooking operation in response to receiving a command using at least in part on the collected sensor data.
16. The system of claim 15, wherein the machine-executable sequential commands script, executed by the processor, comprises at least one command for controlling for controlling one or more smart appliances.
17. The system of claim 15, wherein the robotic kitchen system is located outside the kitchen module, the kitchen module defining an instrumented environment, the robotic kitchen system operating within the instrumented environment.
18. The kitchen system of claim 15, wherein the kitchen module comprises one or more actuators for repositioning the one or more robotic arms and the one or more robotic end effectors to perform a specific operation within a requested part of the kitchen module.
19. The system of claim 15, wherein the robotic kitchen system operates in a first mode and a second mode: during the first mode, the processor recording the machine-executable sequential commands script; and during the second mode, the processor executing the machine-executable sequential commands script.
20. The system of claim 15, wherein the processor adjusting parameters for the robotic instructions machine-executable sequential command script based on at least in part of the sensor data.
21. A method for operating a robotic system, performed by at least one processor, comprising: executing an electronic recipe file that contains a food preparation recipe, the electronic recipe file including machine-executable sequential command script and timing data originated from a recipe creator; and controlling at least one robotic apparatus in an instrumented environment for replicating the food preparation recipe by executing the machine-executable sequential command script of the electronic recipe file; wherein machine-executable sequential command script comprises one or more commands, each command in the command script activating at a predetermined time with a specified time duration.
22. The method of claim 21, wherein the controlling step comprises receiving sensor data in the instrumented environment as generated by one or more sensors, comprising (a) detecting the presence or absence of one or more objects in a standard position in an instrumented environment, (b) identifying an object type, position and orientation for a standard object placed in a non-standard position, and (c) identifying an object type, shape, dimensions, positions and orientations for a non-standard object placed in a non-standard position.
23. The method of claim 21, wherein the instrumented environment comprises one or more predefined objects, including one or more ingredients, one or more appliances, one or more utensils, one or more containers, one or more equipment, or in any combination; and predefined operation space dimensions on x-axis, y-axis and z-axis.
24. The method of claim 21, wherein the instrumented environment comprises a standardized instrumented environment that includes one or more standardized objects, each standardized object in a respective standardized position in a respective standardized orientation.
25. The method of claim 21, wherein the machine-executable sequential command script comprises a sequence of food preparation commands; and wherein the electronic recipe file is prerecorded, or generated based on one or more recipe creator real-time commands, via a computer interface.
26. The method of claim 22, wherein the instrumented environment is used to create one or more single commands and execute the one or more single commands.
27. The method of claim 22, wherein the instrumented environment is used to create machine-executable sequential command script and execute the machine-executable sequential command script.
28. The method of claim 21, wherein the timing data comprises at least one predetermined start time of execution associated with at least one particular cooking operation.
29. The method of claim 21, wherein the timing data comprises a predetermined start time of execution and duration for a specific cooking operation.
30. A robotic kitchen system, comprising: one or more robotic arms; one or more robotic end effectors coupled to the one or more robotic arms, each end effector coupled to a respective robotic arm; one or more sensors are configured to collect sensor data; at least one processor communicatively coupled to the one or more robotic arms, the at least one processor being operable to: receive an electronic recipe file including a mini-manipulation machine-executable sequential command script from a source; control the one or more robotic arms and the one or more robotic end effectors to replicate one or more cooking operations by executing the mini-manipulation machine-executable sequential commands script of the electronic recipe file, each corresponding arm and end effector performing a cooking operation upon receiving a command based at least in part on the collected sensor data.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The invention will be described with respect to specific embodiments thereof, and reference will be made to the drawings, in which:
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
[0036]
[0037]
[0038]
[0039]
[0040]
[0041]
[0042]
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
[0053]
[0054]
[0055]
[0056]
[0057]
[0058]
[0059]
[0060]
[0061]
[0062]
[0063]
[0064]
[0065]
[0066]
[0067]
[0068]
[0069]
[0070]
[0071]
[0072]
[0073]
[0074]
[0075]
[0076]
[0077]
[0078]
[0079]
[0080]
[0081]
[0082]
[0083]
[0084]
[0085]
[0086]
[0087]
[0088]
[0089]
[0090]
[0091]
[0092]
[0093]
[0094]
[0095]
[0096]
[0097]
[0098]
[0099]
[0100]
[0101]
[0102]
[0103]
[0104]
[0105]
[0106]
[0107]
[0108]
[0109]
[0110]
[0111]
[0112]
[0113]
[0114]
[0115]
[0116]
[0117]
DETAILED DESCRIPTION
[0118] A description of structural embodiments and methods of the present invention is provided with reference to
[0119] The following definitions apply to the elements and steps described herein. These terms may likewise be expanded upon.
[0120] Abstracted Datarefers to the abstracted recipe of utility for machine-execution which has many other data-elements that a machine needs to know for proper execution and replication. This so-called meta-data, or additional data corresponding to a particular step in the cooking process, whether it be direct sensor-data (clock-time, water-temperature, camera-image, utensil or ingredient used, etc.) or data generated through interpretation or abstraction of larger data-sets (such as a 3-dimensional range cloud from a laser used to extract the location and types of objects in the image, overlaid with texture and color maps from a camera-picture, etc.), is time-stamped and used by the robotic kitchen to set, control and monitor all processes and associated methods and equipment needed at every point in time as it steps through the sequence of steps in the recipe.
[0121] Abstracted Reciperefers to a representation of a chef's recipe, which a human knows as represented by the use of certain ingredients, in certain sequences, prepared and combined through a sequence of processes and methods as well as skills of the human chef. An abstracted recipe used by a machine for execution in an automated way requires different types of classifications and sequences. While the overall steps carried out are identical to those of the human chef, the abstracted recipe of utility to the robotic kitchen requires that additional meta-data be a part of every step in the recipe. Such meta-data includes the cooking time, variables such as temperature (and its variations over time), oven-setting, tool/equipment used, etc. Basically a machine-executable recipe-script needs to have all possible measured variables of import to the cooking process (all measured and stored while the human chef was preparing the recipe in the chef studio) correlated to time, both overall and that within each process-step of the cooking-sequence. Hence the abstracted recipe is a representation of the cooking steps mapped into a machine-readable representation or domain, which takes the required process from the human-domain to that of the machine-understandable and machine-executable domain through a set of logical abstraction steps.
[0122] Accelerationrefers to the maximum rate of speed-change at which a robotic arm can accelerate around an axis or along a space-trajectory over a short distance.
[0123] Accuracyrefers to how closely a robot can reach a commanded position. Accuracy is determined by the difference between the absolute position of the robot compared to the commanded position. Accuracy can be improved, adjusted, or calibrated with external sensing such as sensors on a robotic hand or a real-time three-dimensional model using multiple (multi-mode) sensors.
[0124] Action PrimitiveIn one embodiment, the term refers to an indivisible robotic action, such as moving the robotic apparatus from location X1 to location X2, or sensing the distance from an object for food preparation without necessarily obtaining a functional outcome. In another embodiment, the term refers to an indivisible robotic action in a sequence of one or more such units for accomplishing a mini-manipulation. These are two aspects of the same definition.
[0125] Automated Dosage Systemrefers to dosage containers in a standardized kitchen module where a particular size of food chemical compounds (such as salt, sugar, pepper, spice, any kind of liquids, such as water, oil, essences, ketchup, etc.) that is released upon application.
[0126] Automated Storage and Delivery Systemrefers to storage containers in a standardized kitchen module that maintain a specific temperature and humidity for storing food; each storage container is assigned a code (e.g., a bar code) for the robotic kitchen to identify and retrieval where a particular storage container delivers the food contents stored therein.
[0127] Data Cloudrefers to a collection of sensor or data-based numerical measurement values from a particular space (three-dimensional laser/acoustic range measurement, RGB-values from a camera image, etc.) collected at certain intervals and aggregated based on a multitude of relationships, such as time, location, etc.
[0128] Degree of Freedom (DOF)refers to a defined mode and/or direction in which a mechanical device or system can move. The number of degrees of freedom is equal to the total number of independent displacements or aspects of motion. The total number of degrees of freedom is doubled for two robotic arms.
[0129] Edge Detectionrefers to a software-based computer program(s) capable of identifying the edges of multiple objects that may be overlapping in a two-dimensional-image of a camera yet successfully identifying their boundaries to aid in object identification and planning for grasping and handling.
[0130] Equilibrium Valuerefers to the target position of a robotic appendage, such as a robotic arm where the forces acting upon it are in equilibrium, i.e. there is no net force and thus no net movement.
[0131] Execution Sequence Plannerrefers to a software-based computer program(s) capable of creating a sequence of execution scripts or commands for one or more elements or systems capable of being computer controlled, such as arm(s), dispensers, appliances, etc.
[0132] Food Execution Fidelityrefers to a robotic kitchen which is intended to replicate the recipe-script generated in the chef studio by watching and measuring and understanding the steps and variables and methods and processes of the human chef, thereby trying to emulate his/her techniques and skills. The fidelity of how close the execution of the dish-preparation comes to that of the human-chef is measured by how close the robotically-prepared dish resembles the human-prepared dish as measured by a variety of subjective elements, such as consistency, color, taste, etc. The notion is that, the more closely the dish prepared by the robotic kitchen is to that prepared by the human chef, the higher the fidelity of the replication process.
[0133] Food Preparation Stage (also referred to as Cooking stage)refers to a combination, either sequential or in parallel, of one or more mini-manipulations including action primitives, and computer instructions for controlling the various kitchen equipment and appliances in the standardized kitchen module; one or more food preparation stages collectively represent the entire food preparation process for a particular recipe.
[0134] Geometric Reasoningrefers to a software-based computer program(s) capable of using two-dimensional (2D)/three-dimensional (3D) surface- and/or volumetric data to reason as to the actual shape and size of a particular volume; the ability to determine or utilize boundary information also allows for inferences as to the start end of a particular geometric element and the number present (in an image or model).
[0135] Grasp Reasoningrefers to a software-based computer program(s) capable of relying on geometric and physical reasoning to plan a multi-contact (point/area/volume) contact-interaction between a robotic end-effector (gripper, link, etc.), or even tools/utensils held by the end-effector, so as to successfully and stably contact, grasp and hold the object in order to manipulate it in three-dimensional space.
[0136] Hardware Automation DeviceFixed process device capable of executing pre-programmed steps in succession without the ability to modify any of them; such devices are used for repetitive motions that are not in need of any modulation.
[0137] Ingredient management and manipulationrefers to defining each ingredient in detail (including size, shape, weight, dimensions, characteristics and properties), one or more real-time adjustments in the variables associated with the particular ingredient that may differ from the previous stored ingredient details (such as the size of a fish fillet, the dimensions of an egg, etc.), and the process in executing the different stages for the manipulation movements to an ingredient.
[0138] Kitchen Module (or Kitchen Volume)a standardized full kitchen module with standardized sets of kitchen equipment, standardized sets of kitchen tools, standardized sets of kitchen handles, and standardized sets of kitchen containers, with predefined space and dimensions for storing, accessing, and operating each kitchen element in the standardized full kitchen module. One objective of a kitchen module is to predefine as much of the kitchen equipment, tools, handles, containers, etc. as possible so as to provide a relatively fixed kitchen platform for the movements of robotic arms and hands. Both a chef in the chef kitchen studio and a person at home with a robotic kitchen (or a person at a restaurant) uses the standardized kitchen module so as to maximize the predictability of the kitchen hardware, while minimizing the risks of differentiations, variations and deviations between the chef kitchen studio and a home robotic kitchen. Different embodiments of the kitchen module are possible, including a standalone kitchen module and an integrated kitchen module. The integrated kitchen module is fitted into a conventional kitchen area of a typical house. The kitchen module operates in at least two modes, a robotic mode and a normal (manual) mode.
[0139] Machine Learningrefers to the technology wherein a software component or program improves its performance based on experience and feedback. One kind of machine learning is reinforcement learning, often used in robotics, where desirable actions are rewarded and undesirable ones are penalized. Another kind is case-based learning, where previous solutions, e.g. sequences of actions by a human teacher or by the robot itself are remembered, together with any constraints or reasons for the solutions, and then are applied or reused in new settings. There are also additional kinds of machine learning, such as inductive and transductive methods.
[0140] Mini-Manipulationrefers to a combination (or a sequence) of one or more steps that accomplish a basic functional outcome with a threshold value of the highest level of probability (examples of threshold value as within 0.1, 0.001, or 0.001 of the optimal value). Each step can be an action primitive or another (smaller) mini-manipulation, similar to a computer program comprised of basic coding steps and other computer programs that may stand alone or serve as sub-routines. For instance, a mini-manipulation can be grasping an egg, comprised of the motor actions required for reaching out a robotic arm moving the robotic fingers into the right configuration, and applying the correct delicate amount of force for graspingall primitive actions. Another mini-manipulation can be breaking-an-egg-with-a-knife, including the grasping mini-manipulation, followed with one robotic hand, followed by grasping-a-knife mini-manipulation with the other hand, followed by the primitive action of striking the egg with the knife using a predetermined force.
[0141] Model Elements and Classificationrefers to one or more software-based computer program(s) capable of understanding elements in a scene as being items that are used or needed in different parts of a task; such as a bowl for mixing and the need for a spoon to stir, etc. Multiple elements in a scene or a world-model may be classified into groupings allowing for faster planning and task-execution.
[0142] Motion Primitivesrefers to motion actions that define different levels/domains of detailed action steps, e.g. a high level motion primitive would be to grab a cup, and a low level motion primitive would be to rotate a wrist by five degrees.
[0143] Multimodal Sensing Unitrefers to a sensing unit comprised of multiple sensors capable of sensing and detection in multiple modes or electromagnetic bands or spectra, particularly capable of capturing three-dimensional position and/or motion information; the electromagnetic spectrum can range from low to high frequencies and need not be limited to that perceivable by a human being. Additional modes might include, but are not limited to, other physical senses such as touch, smell, etc.
[0144] Number of Axesthree axes are required to reach any point in space. To fully control the orientation of the end of the arm (i.e. the wrist), three additional rotational axes (yaw, pitch, and roll) are required.
[0145] Parametersrefers to variables that can take numerical values or ranges of numerical values. Three kinds of parameters are particularly relevant: parameters in the instructions to a robotic device (e.g. the force or distance in an arm movement), user settable parameters (e.g. prefers meat well done vs. medium), and chef-defined parameters (e.g. set oven temperature to 350 F).
[0146] Parameter adjustmentrefers to the process of changing the values of parameters based on inputs. For instance changes in the parameters of instructions to the robotic device can be based on the properties (e.g. size, shape, orientation) of, but not limited to, the ingredients, position/orientation of kitchen tools, equipment, appliances, speed, and time duration of a mini-manipulation.
[0147] Payload or carrying capacityrefers to how much weight a robotic arm can carry and hold (or even accelerate) against the force of gravity, as a function of its endpoint location.
[0148] Physical Reasoningrefers to a software-based computer program(s) capable of relying on geometrically-reasoned data and using physical information (density, texture, typical geometry and shape) to assist an inference-engine (program) to better model the object and also predict its behavior in the real world, particularly when grasped and/or manipulated/handled.
[0149] Raw Datarefers to all measured and inferred sensory-data and representation information that is collected as part of the chef-studio recipe-generation process while watching/monitoring a human chef preparing a dish. Raw data can range from a simple data-point such as clock-time, to oven temperature (over time), camera-imagery, three-dimensional laser-generated scene representation data, to appliances/equipment used, tools employed, ingredients (type and amount) dispensed and when, etc. All the information the studio-kitchen collects from its built-in sensors and stores in raw, time-stamped form is considered raw data. Raw data is then used by other software processes to generate an even higher level of understanding and recipe-process understanding, turning raw data into additional time-stamped processed/interpreted data.
[0150] Robotic Apparatusrefers the set of robotic sensors and effectors. The effectors comprise one or more robotic arms, and one or more robotic hands for operation in the standardized robotic kitchen. The sensors comprise cameras, range sensors, force sensors (haptic sensors) that transmit their information to the processor or set of processors that control the effectors.
[0151] Recipe Cooking Processrefers to a robotic script containing abstract and detailed levels of instructions to a collection of programmable and hard automation devices, so as to allow computer-controllable devices to execute a sequenced operation within its environment (e.g. a kitchen replete with ingredients, tools, utensils and appliances).
[0152] Recipe Scriptrefers to a recipe script as a sequence in time containing a structure and a list of commands and execution primitives (simple to complex command software) that, when executed by the robotic kitchen elements (robot-arm, automated equipment, appliances, tools, etc.) in a given sequence, should result in the proper replication and creation of the same dish as prepared by the human chef in the studio-kitchen. Such a script is sequential in time and equivalent to the sequence employed by the human chef to create the dish, albeit in a representation that is suitable and understandable by the computer-controlled elements in the robotic kitchen.
[0153] Recipe Speed Executionrefers to managing a timeline in the execution of recipe steps in preparing a food dish by replicating a chef's movements, where the recipe steps include standardized food preparation operations (e.g., standardized cookware, standardized equipment, kitchen processors, etc.), mini-manipulations, and cooking of non-standardized objects.
[0154] Repeatabilityrefers to an acceptable preset margin in how accurately the robotic arms/hands can repeatedly return to a programmed position. If the technical specification in a control memory requires the robotic hand to move to a certain X-Y-Z position and within +/0.1 mm of that position, then the repeatability is measured for the robotic hands to return to within +/0.1 mm of the taught and desired/commanded position.
[0155] Robotic Recipe Scriptrefers to a computer-generated sequence of machine-understandable instructions related to the proper sequence of robotically/hard-automation execution of steps to mirror the required cooking steps in a recipe to arrive at the same end-product as if cooked by a chef.
[0156] Robotic CostumeExternal instrumented device(s) or clothing, such as gloves, clothing with camera-trackable markers, jointed exoskeleton, etc., used in the chef studio to monitor and track the movements and activities of the chef during all aspects of the recipe cooking process(es).
[0157] Scene Modelingrefers to a software-based computer program(s) capable of viewing a scene in one or more cameras' fields of view, and being capable of detecting and identifying objects of importance to a particular task. These objects may be pre-taught and/or be part of a computer library with known physical attributes and usage-intent.
[0158] Smart Kitchen Cookware/Equipmentrefers to an item of kitchen cookware (e.g., a pot or a pan) or an item of kitchen equipment (e.g., an oven, a grill, or a faucet) with one or more sensors that prepares a food dish based on one or more graphical curves (e.g., a temperature curve, a humidity curve, etc.).
[0159] Software Abstraction Food Enginerefers to a software engine that is defined as a collection of software loops or programs, acting in concert to process input data and create a certain desirable set of output data to be used by other software engines or an end-user through some form of textual or graphical output interface. An abstraction software engine is a software program(s) focused on taking a large and vast amount of input data from a known source in a particular domain (such as three-dimensional range measurements that form a data-cloud of three-dimensional measurements as seen by one or more sensors), and then processing the data to arrive at interpretations of the data in a different domain (such as detecting and recognizing a table-surface in a data-cloud based on data having the same vertical data value, etc.), in order to identify, detect and classify data-readings as pertaining to an object in three-dimensional space (such as a table-top, cooking pot, etc.). The process of abstraction is basically defined as taking a large data set from one domain and inferring structure (such as geometry) in a higher level of space (abstracting data points), and then abstracting the inferences even further and identifying objects (pots, etc.) out of the abstracted data-sets to identify real-world elements in an image, which can then be used by other software engines to make additional decisions (handling/manipulation decisions for key objects, etc.). A synonym for software abstraction engine in this application could be also software interpretation engine or even computer-software processing and interpretation algorithm.
[0160] Task Reasoningrefers to a software-based computer program(s) capable of analyzing a task-description and breaking it down into a sequence of multiple machine-executable (robot or hard-automation systems) steps so as to achieve a particular end result defined in the task description.
[0161] Three-dimensional World Object Modeling and Understandingrefers to a software-based computer program(s) capable of using sensory data to create a time-varying three-dimensional model of all surfaces and volumes so as to enable it to detect, identify and classify objects within the same and understand their usage and intent.
[0162] Torque vectorrefers to the torsion force upon a robotic appendage including its direction and magnitude.
[0163] Volumetric Object Inference (Engine)refers to a software-based computer program(s) capable of using geometric data and edge-information as well as other sensory data (color, shape, texture, etc.) to allow for identification of three-dimensionality of one or more objects to aid in the object identification and classification process.
[0164]
[0165] The robotic food preparation software 14 includes the multimodal three-dimensional sensors 20, a capturing module 28, a calibration module 30, a conversion algorithm module 32, a replication module 34, a quality check module 36 with a three-dimensional vision system, a same result module 38, and a learning module 40. The capturing module 28 captures the movements of the chef as the chef prepares a food dish. The calibration module 30 calibrates the robotic arms 22 and robotic hands 24 before, during and after the cooking process. The conversion algorithm module 32 is configured to convert the recorded data from a chef's movements collected in the chef studio into recipe modified data (or transformed data) for use in a robotic kitchen where robotic hands replicate the food preparation of the chef's dish. The replication module 34 is configured to replicate the chef's movements in a robotic kitchen. The quality check module 36 is configured to perform quality check functions of a food dish prepared by the robotic kitchen during, prior to, or after the food preparation process. The same result module 38 is configured to determine whether the food dish prepared by a pair of robotic arms and hands in the robotic kitchen would taste the same or substantially the same as if prepared by the chef. The learning module 40 is configured to provide learning capabilities to the computer 16 that operates the robotic arms and hands.
[0166]
[0167] The standardized robotic kitchen 50 is designed for detecting, recording and emulating a chef's cooking movements, controlling significant parameters such as temperature over time, and process execution at robotic kitchen stations with designated appliances, equipment and tools. The chef kitchen 44 provides a computing kitchen environment 16 with gloves with sensors or a costume with sensors for recording and capturing a chef's 50 movements in the food preparation for a specific recipe. Upon recording the movements and recipe process of the chef 49 for a particular dish into a software recipe file in memory 52, the software recipe file is transferred from the chef kitchen 44 to the robotic kitchen 48 via a communication network 46, including a wireless network and/or a wired network connected to the Internet, so that the user (optional) 60 can purchase one or more software recipe files or the user can be subscribed to the chef kitchen 44 as a member that receives new software recipe files or periodic updates of existing software recipe files. The household robotic kitchen system 48 serves as a robotic computing kitchen environment at residential homes, restaurants, and other places in which the kitchen is built for the user 60 to prepare food. The household robotic kitchen system 48 includes the robotic cooking engine 56 with one or more robotic arms and hard-automation devices for replicating the chef's cooking actions, processes and movements based on a received software recipe file from the chef studio system 44.
[0168] The chef studio 44 and the robotic kitchen 48 represent an intricately linked teach-playback system, which has multiple levels of fidelity of execution. While the chef studio 44 generates a high-fidelity process model of how to prepare a professionally cooked dish, the robotic kitchen 48 is the execution/replication engine/process for the recipe-script created through the chef working in the chef studio. Standardization of a robotic kitchen module is a means to increase performance fidelity and success/guarantee.
[0169] The varying levels of fidelity for recipe-execution depend on the correlation of sensors and equipment (besides of course the ingredients) between those in the chef studio 44 and that in the robotic kitchen 48. Fidelity can be defined as a dish tasting identical to that prepared by a human chef (indistinguishably so) at one of the (perfect replication/execution) ends of the spectrum, while at the opposite end the dish could have one or more substantial or fatal flaws with implications to quality (overcooked meat or pasta), taste (burnt elements), edibility (incorrect consistency) or even health-implications (undercooked meat such as chicken/pork with salmonella exposure, etc.).
[0170] A robotic kitchen that has identical hardware and sensors and actuation systems that can replicate the movements and processes akin to those by the chef that were recorded during the chef-studio cooking process is more likely to result in a higher fidelity outcome. The implication here is that the setups need to be identical, which has a cost and volume implication. The robotic kitchen 48 can however still be implemented using more standardized non-computer-controlled or computer-monitored elements (pots with sensors, networked appliances such as ovens, etc.), requiring more sensor-based understanding to allow for more complex execution monitoring. Since uncertainty has now increased as to key elements (correct amount of ingredients, cooking temperatures, etc.) and processes (use of stirrer/masher in case a blender is not available in a robotic home kitchen), the guarantees of having an identical outcome to that from the chef will undoubtedly be lower.
[0171] An emphasis in the present disclosure is that the notion of a chef studio 44 coupled with a robotic kitchen is a generic concept. The level of the robotic kitchen 48 is variable all the way from a home-kitchen outfitted with a set of arms and environmental sensors, all the way to an identical replica of the studio-kitchen, where a set of arms and articulated motions, tools and appliances and ingredient-supply can replicate the chef's recipe in an almost identical fashion. The only variable to contend with will be the quality-degree of the end-result or dish in terms of quality, looks, taste, edibility and health.
[0172] A potential method to mathematically describe this correlation between the recipe-outcome and the input variables in the robotic kitchen can best be described by the function below:
F.sub.recipe-outcome=F.sub.studio(I,E,P,M,V)+F.sub.RobKit(E.sub.f,I,R.sub.e,P.sub.mf) [0173] where F.sub.studio=Recipe Script Fidelity of Chef-Studio [0174] F.sub.RobKit=Recipe Script Execution by Robotic Kitchen [0175] I=Ingredients [0176] E=Equipment [0177] P=Processes [0178] M=Methods [0179] V=Variables (Temperature, Time, Pressure, etc.) [0180] E.sub.f=Equipment Fidelity [0181] R.sub.e=Replication Fidelity [0182] P.sub.mf=Process Monitoring Fidelity
[0183] The above equation relates the degree to which the outcome of a robotically-prepared recipe matches that a human chef would prepare and serve (F.sub.recipe-outcome) to the level that the recipe was properly captured and represented by the chef studio 44 (F.sub.studio) based on the ingredients (I) used, the equipment (E) available to execute the chef's processes (P) and methods (M) by properly capturing all the key variables (V) during the cooking process; and how the robotic kitchen is able to represent the replication/execution process of the robotic recipe script by a function (F.sub.RobKit) that is primarily driven by the use of the proper ingredients (I), the level of equipment fidelity (E.sub.f) in the robotic kitchen compared to that in the chef studio, the level to which the recipe-script can be replicated (R.sub.e) in the robotic kitchen, and to what extent there is an ability and need to monitor and execute corrective actions to achieve the highest process monitoring fidelity (P.sub.mf) possible.
[0184] The functions (F.sub.studio) and (F.sub.RobKit) can be any combination of linear or non-linear functional formulas with constants, variables and any form of algorithmic relationships. An example for such algebraic representations for both functions could be.
F.sub.studio=I(fct. sin(Temp))+E(fct. Cooptop1*5)+P(fct. Circle(spoon)+V(fct. 0.5*time)
[0185] Delineating that the fidelity of the preparation process is related to the temperature of the ingredient which varies over time in the refrigerator as a sinusoidal function, the speed with which an ingredient can be heated on the cooktop on specific station at a particular multiplicative rate, and related to how well a spoon can be moved in a circular path of a certain amplitude and period, and that the process needs to be carried out at no less than the speed of the human chef for the fidelity of the preparation process to be maintained.
F.sub.RobKit=E.sub.f,(Cooktop2,Size)+I(1.25*Size+Linear(Temp))+R.sub.e(Motion-Profile)+P.sub.mf(Sensor-Suite Correspondence)
[0186] Delineating that the fidelity of the replication process in the robotic kitchen is related to the appliance type and layout for a particular cooking-area and the size of the heating-element, the size and temperature profile of the ingredient being seared and cooked (thicker steak requiring more cooking time), while also preserving the motion-profile of any stirring and bathing motions of a particular step like searing or mousse-beating, and whether the correspondence between sensors in the robotic kitchen and the chef-studio is sufficiently high to trust the monitored sensor data to be accurate and detailed enough to provide a proper monitoring fidelity of the cooking process in the robotic kitchen during all steps in a recipe.
[0187] The outcome of a recipe is not only a function of what fidelity the human chef's cooking steps/methods/process/skills were captured with by the chef studio, but also with what fidelity these can be executed by the robotic kitchen, where each of them has key elements that impact their respective subsystem performance.
[0188]
[0189] The standardized hard automation dispenser(s) 82 is a device or a series of devices that is/are programmable and/or controllable via the cooking computer 16 to feed or provide pre-packaged (known) amounts or dedicated feeds of key materials for the cooking process, such as spices (salt, pepper, etc.), liquids (water, oil, etc.) or other dry materials (flour, sugar, etc.). The standardized hard automation dispensers 82 may be located at a specific station or be able to be robotically accessed and triggered to dispense according to the recipe sequence. In other embodiments, a robotic hard automation module may be combined or sequenced in series or parallel with other such modules or robotic arms or cooking utensils. In this embodiment, the standardized robotic kitchen 50 includes robotic arms 70 and robotic hands 72 and robotic hands as controlled by the robotic food preparation engine 56 in accordance with a software recipe file stored in the memory 52 for replicating a chef's precise movements in preparing a dish to produce the same tasting dish as if the chef had prepared it himself or herself. The three-dimensional vision sensors 66 provide capability to enable three-dimensional modeling of objects, providing a visual three-dimensional model of the kitchen activities, and scanning the kitchen volume to assess the dimensions and objects within the standardized robotic kitchen 50. The retractable safety glass 68 comprises a transparent material on the robotic kitchen 50, which when in an ON state extends the safety glass around the robotic kitchen to protect surrounding human beings from the movements of robotic arms 70 and hands 72, hot water and other liquids, steam, fire and other dangers influents. The robotic food preparation engine 56 is communicatively coupled to an electronic memory 52 for retrieving a software recipe file previously sent from the chef studio system 44 for which the robotic food preparation engine 56 is configured to execute processes in preparing and replicating the cooking method and processes of a chef as indicated in the software recipe file. The combination of robotic arms 70 and robotic hands 72 serves to replicate the precise movements of the chef in preparing a dish so that the resulting food dish will taste identical (or substantially identical) to the same food dish prepared by the chef. The standardized cooking equipment 74 includes an assortment of cooking appliances 46 that are incorporated as part of the robotic kitchen 50, including, but not limited to, a stove/induction/cooktop (electric cooktop, gas cooktop, induction cooktop), an oven, a grill, a cooking steamer, and a microwave oven. The standardized cookware and sensors 76 are used as embodiments for the recording of food preparation steps based on the sensors on the cookware and cooking a food dish based on the cookware with sensors, which include a pot with sensors, a pan with sensors, an oven with sensors, and a charcoal grill with sensors. The standardized cookware 78 includes frying pans, saut pans, grill pans, multi-pots, roasters, woks, and braisers. The robotic arms 70 and the robotic hands 72 operate the standardized handles and utensils 80 in the cooking process. In one embodiment, one of the robotic hands 72 is fitted with a standardized handle, which is attached to a fork head, a knife head, and a spoon head for selection as required. The standardized hard automation dispensers 82 are incorporated into the robotic kitchen 50 to provide for expedient (via both robot arms 70 and human use) key and common/repetitive ingredients that are easily measured/dosed out or pre-packaged. The standardized containers 86 are storage locations that store food at room temperature. The standardized refrigerator containers 88 refer to, but are not limited to, a refrigerator with identified containers for storing fish, meat, vegetables, fruit, milk, and other perishable items. The containers in the standardized containers 86 or standardized storages 88 can be coded with container identifiers from which the robotic food preparation engine 56 is able to ascertain the type of food in a container based on the container identifier. The standardized containers 86 provide storage space for non-perishable food items such as salt, pepper, sugar, oil, and other spices. Standardized cookware with sensors 76 and the cookware 78 may be stored on a shelf or a cabinet for use by the robotic arms 70 for selecting a cooking tool to prepare a dish. Typically, the raw fish, the raw meat, and vegetables are pre-cut and stored in the identified standardized storages 88. The kitchen countertop 90 provides a platform for the robotic arms 70 to handle the meat or vegetables as needed, which may or may not include cutting or chopping actions. The kitchen faucet 92 provides a kitchen sink space for washing or cleaning food in preparation for a dish. When the robotic arms 70 have completed the recipe process to prepare a dish and the dish is ready for serving, the dish is placed on a serving counter 90, which further allows for the dining environment to be enhanced by adjusting the ambient setting with the robotic arms 70, such as placement of utensils, wine glasses, and a chosen wine compatible with the meal. One embodiment of the equipment in the standardized robotic kitchen module 50 is a professional series as to increase the universal appeal to prepare various types of dishes.
[0190] The standardized robotic kitchen module 50 has as one objective the standardization of the kitchen module 50 and various components with the kitchen module itself, to ensure consistency in both the chef kitchen 44 and the robotic kitchen 48 to maximize the preciseness of recipe replication while minimizing the risks of deviations from precise replication of a recipe dish between the chef kitchen 44 and the robotic kitchen 48. One main purpose of having the standardization of the kitchen module 50 is to obtain the same result of the cooking process (or the same dish) between a first food dish prepared by the chef and a subsequent replication of the same recipe process via the robotic kitchen. Conceiving a standardized platform in the standardized robotic kitchen module 50 between the chef kitchen 44 and the robotic kitchen 48 has several key considerations: same timeline, same program or mode, and quality check. The same timeline in the standardized robotic kitchen 50 where the chef prepares a food dish at the chef kitchen 44 and the replication process by the robotic hands in the robotic kitchen 48 refers to the same sequence of manipulations, the same initial and ending time of each manipulation, and the same speed of moving an object between handling operations. The same program or mode in the standardized robotic kitchen 50 refers to the use and operation of standardized equipment during each manipulation recording and execution step. The quality check refers to three-dimensional vision sensors in the standardized robotic kitchen 50 which monitor and adjust in real time each manipulation action during the food preparation process to correct any deviation and avoid a flawed result. The adoption of the standardized robotic kitchen module 50 reduces and minimizes the risks of not obtaining the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen using robotic arms and hands. Without the standardization of a robotic kitchen module and the components within the robotic kitchen module, the increased variations between the chef kitchen 44 and the robotic kitchen 48 increase the risks of not being able to obtain the same result between the chef's prepared food dish and the food dish prepared by the robotic kitchen because more elaborate and complex adjustment algorithms will be required with different kitchen modules, different kitchen equipment, different kitchenware, different kitchen tools, and different ingredients between the chef kitchen 44 and the robotic kitchen 48.
[0191] The standardized robotic kitchen module 50 includes standardization of many aspects. First, the standardized robotic kitchen module 50 includes standardized positions and orientations (in the XYZ coordinate plane) of any type of kitchenware, kitchen containers, kitchen tools and kitchen equipment (with standardized fixed holes in the kitchen module and device positions). Secondly, the standardized robotic kitchen module 50 includes a standardized cooking volume dimension and architecture. Thirdly, the standardized robotic kitchen module 50 includes standardized equipment sets, such as an oven, a stove, a dish washer, a faucet, etc. Fourth, the standardized robotic kitchen module 50 includes standardized kitchenware, standardized cooking tools, standardized cooking devices, standardized containers, and standardized food storage in a refrigerator, in terms of shape, dimension, structure, material, capabilities, etc. Fifth, in one embodiment, the standardized robotic kitchen module 50 includes a standardized universal handle for handling any kitchenware, tools, instruments, containers, and equipment, which enable a robotic hand to hold the standardized universal handle in only one correct position, while avoiding any improper grasps or incorrect orientations. Sixth, the standardized robotic kitchen module 50 includes standardized robotic arms and hands with a library of manipulations. Seventh, the standardized robotic kitchen module 50 includes a standardized kitchen processor for standardized ingredient manipulations. Eighth, the standardized robotic kitchen module 50 includes standardized three-dimensional vision devices for creating dynamic three-dimensional vision data, as well as other possible standard sensors, for recipe recording, execution tracking, and quality check functions. Ninth, the standardized robotic kitchen module 50 includes standardized types, standardized volumes, standardized sizes, and standardized weights for each ingredient during a particular recipe execution.
[0192]
[0193] The input module 50 is configured to receive any type of input information such as software recipe files sent from another computing device. The calibration module 94 is configured to calibrate itself with the robotic arms 70, the robotic hands 72, and other kitchenware and equipment components within the standardized robotic kitchen module 50. The quality check module 96 is configured to determine the quality and freshness of raw meat, raw vegetables, milk-associated ingredients and other raw foods at the time that the raw food is retrieved for cooking, as well as checking the quality of raw foods when receiving the food into the standardized food storage 88. The quality check module 96 can also be configured to conduct quality testing of an object based on senses, such as the smell of the food, the color of the food, the taste of the food, and the image or appearance of the food. The chef movements recording module 98 is configured to record the sequence and the precise movements of the chef when the chef prepares a food dish. The cookware sensor data recording module 100 is configured to record sensory data from cookware equipped with sensors (such as a pan with sensors, a grill with sensors, or an oven with sensors) placed in different zones within the cookware, thereby producing one or more sensory curves. The result is the generation of a sensory curve, such as temperature curve (and/or humidity), that reflects the temperature fluctuation of cooking appliances over time for a particular dish. The memory module 102 is configured as a storage location for storing software recipe files, for either replication of chef recipe movements or other types of software recipe files including sensory data curves. The recipe abstraction module 104 is configured to use recorded sensor data to generate machine-module specific sequenced operation profiles. The chef movements replication module 106 is configured to replicate the chef's precise movements in preparing a dish based on the stored software recipe file in the memory 52. The cookware sensory replication module 108 is configured to replicate the preparation of a food dish by following the characteristics of one or more previously recorded sensory curves which was generated when the chef 49 prepared a dish by using the standardized cookware with sensors 76. The robotic cooking module 110 is configured to control and operate standardized kitchen operations, mini-manipulations, non-standardized objects, and the various kitchen tools and equipment in the standardized robotic kitchen 50. The real time adjustment module 112 is configured to provide real-time adjustments to the variables associated with a particular kitchen operation or a mini operation so as to produce a resulting process that is a precise replication of the chef movement or a precise replication of the sensory curve. The learning module 114 is configured to provide learning capabilities to the robotic cooking engine 56 to optimize the precise replication in preparing a food dish by robotic arms 70 and the robotic hands 72, as if the food dish was prepared by a chef, using a method such as case-based (robotic) learning. The mini-manipulation library database module 116 is configured to store a first database library of mini-manipulations. The standardized kitchen operation library database module 117 is configured to store a second database library of standardized kitchenware and how to operate this standardized kitchenware. The output module 118 is configured to send output computer files or control signals external to the robotic cooking engine.
[0194]
[0195] These individual software modules generate such information (but are not thereby limited to only these modules) as (i) chef-location and cooking-station ID via a location and configuration module 152, (ii) configuration of arms (via torso), (iii) tools handled and when and how, (iv) utensils used and locations on the station through the hardware and variable abstraction module 154, (v) processes executed with them and (vi) variables (temperature, lid y/n, stirring, etc.) in need of monitoring through the process module B156, (vii) temporal (start/finish, type) distribution and (viii) types of processes (stir, fold, etc.) being applied, and (ix) ingredients added (type, amount, state of prep, etc.), through the cooking sequence and process abstraction module 158.
[0196] All this information is then used to create a machine-specific (not just for the robotic-arms, but also ingredient dispensers, tools and utensils, etc.) set of recipe instructions through the stand-alone module 160, which are organized as a script of sequential/parallel overlapping tasks to be executed and monitored. This recipe-script is stored (162) alongside the entire raw data set (164) in the data storage module 166 and is made accessible to either a remote robotic cooking station through the robotic kitchen interface module 168 or a human user 170 via a graphical user interface (GUI) 172.
[0197]
[0198] The robotic kitchen 48 engages in a recipe replication process 106, whose profile depends on whether the kitchen is of a standardized or non-standardized type, which is checked by a process 186.
[0199] The robotic kitchen execution is dependent on the type of kitchen available to the user. If the robotic kitchen uses the same/identical (at least functionally) equipment as used in the in the chef studio, the recipe replication process is primarily one of using the raw data and playing it back as part of the recipe-script execution process. Should the kitchen however differ from the (ideal) standardized kitchen, the execution engine(s) will have to rely on the abstracted data to generate kitchen-specific execution sequences to try to achieve a similar step-by-step result.
[0200] Since the cooking process is continually monitored by all sensor units in the robotic kitchen via a monitoring process 194, regardless of whether the known studio equipment 196 or the mixed/atypical non-chef studio equipment 198 is being used, the system is able to make modifications as needed depending on a recipe progress check 200. In one embodiment of the standardized kitchen, raw data is typically played back through an execution module 188 using chef-studio type equipment, and the only adjustments that are expected are adaptations 202 in the execution of the script (repeat a certain step, go back to a certain step, slow down the execution, etc.) as there is a one-to-one correspondence between taught and played-back data-sets. However, in the case of the non-standardized kitchen, the chances are very high that the system will have to modify and adapt the actual recipe itself and its execution via a recipe script modification module 204, to suit the available tools/appliances 192 which differ from those in the chef studio 44 or the measured deviations from the recipe script (meat cooking too slowly, hot-spots in pot burning the roux, etc.). Overall recipe-script progress is monitored using a similar process 206, which differs depending on whether chef-studio equipment 208 or mixed/atypical kitchen equipment 210 is being used.
[0201] A non-standardized kitchen is less likely to result in a close-to-human chef cooked dish, as compared to using a standardized robotic kitchen that has equipment and capabilities reflective of those used in the studio-kitchen. The ultimate subjective decision is of course that of the human (or chef) tasting, which is a quality evaluation 212, yielding to a (subjective) quality decision 214.
[0202]
[0203] A data process-mapping algorithm 220 uses the simpler (typically single-unit) variables to determine where the process action is taking place (cooktop and/or oven, fridge, etc.) and assigns a usage tag to any item/appliance/equipment being used whether intermittently or continuously. It associates a cooking step (baking, grilling, ingredient-addition, etc.) to a specific time-period and tracks when, where and which and how much of what ingredient was added. This (time-stamped) information dataset is then made available for the data-melding process during the recipe-script generation process 222.
[0204] The data extraction and mapping process 224 is primarily focused on taking two-dimensional information (such as from monocular/single-lensed cameras) and extracting key information from the same. In order to extract the important and more abstracted descriptive information from each successive image, several algorithmic processes have to be applied to this dataset. Such processing steps can include (but are not limited to) edge-detection, color and texture-mapping, and then using the domain-knowledge in the image, coupled with object-matching information (type and size) extracted from the data reduction and abstraction process 226, to allow for the identification and location of the object (whether an item of equipment or ingredient, etc.), again extracted from the data reduction and abstraction process 226, allowing one to associate the state (and all associated variables describing the same) and items in an image with a particular process-step (frying, boiling, cutting, etc.). Once this data has been extracted and associated with a particular image at a particular point in time, it can be passed to the recipe-script generation process 222 to formulate the sequence and steps within a recipe.
[0205] The data-reduction and abstraction engine (set of software routines) 226 is intended to reduce the larger three-dimensional data sets and extract from them key geometric and associative information. A first step is to extract from the large three-dimensional data point-cloud only the specific workspace area of importance to the recipe at that particular point in time. Once the data-set has been trimmed, key geometric features will be identified by a process known as template matching; this allows for the identification of such items as horizontal table-tops, cylindrical pots and pans, arm and hand locations, etc. Once typical known (template) geometric entities are determined in a data-set a process of object identification and matching proceeds to differentiate all items (pot vs. pan, etc.) and associates the proper dimensionality (size of pot or pan, etc.) and orientation of the same, and places them within the three-dimensional world model being assembled by the computer. All this abstracted/extracted information is then also shared with the data-extraction and mapping engine 224, prior to all being fed to the recipe-script generation engine 222.
[0206] The recipe-script generation engine process 222 is responsible for melding (blending/combining) all the available data and sets into a structured and sequential cooking script with clear process-identifiers (prepping, blanching, frying, washing, plating, etc.) and process-specific steps within each, which can then be translated into robotic-kitchen machine-executable command-scripts that are synchronized based on process-completion and overall cooking time and cooking progress. Data melding will at least involve, but will not solely be limited to, the ability to take each (cooking) process step and populating the sequence of steps to be executed with the properly associated elements (ingredients, equipment, etc.), methods and processes to be used during the process steps, and the associated key control- (set oven/cooktop temperatures/settings) and monitoring-variables (water or meat temperature, etc.) to be maintained and checked to verify proper progress and execution. The melded data is then combined into a structured sequential cooking script that will resemble a set of minimally descriptive steps (akin to a recipe in a magazine) but with a much larger set of variables associated with each element (equipment, ingredient, process, method, variable, etc.) of the cooking process at any one point in the procedure. The final step is to take this sequential cooking script and transform it into an identically structured sequential script that is translatable by a set of machines/robot/equipment within a robotic kitchen 48. It is this script the robotic kitchen 48 uses to execute the automated recipe execution and monitoring steps.
[0207] All raw (unprocessed) and processed data as well as the associated scripts (both structure sequential cooking-sequence script and the machine-executable cooking-sequence script) are stored in the data and profile storage unit/process 228 and time-stamped. It is from this database that the user, by way of a GUI, can select and cause the robotic kitchen to execute a desired recipe through the automated execution and monitoring engine 230, which is continually monitored by its own internal automated cooking process, with necessary adaptations and modifications to the script generated by the same and implemented by the robotic-kitchen elements, in order to arrive at a completely plated and served dish.
[0208]
[0209] The mini-manipulation library is a command-software repository, where motion behaviors and processes are stored based on an off-line learning process, where the arm/wrist/finger motions and sequences to successfully complete a particular abstract task (grab the knife and then slice; grab the spoon and then stir; grab the pot with one hand and then use other hand to grab spatula and get under meat and flip it inside the pan; etc.). This repository has been built up to contain the learned sequences of successful sensor-driven motion-profiles and sequenced behaviors for the hand/wrist (and sometimes also arm-position corrections), to ensure successful completions of object (appliance, equipment, tools) and ingredient manipulation tasks that are described in a more abstract language, such as grab the knife and slice the vegetable, crack the egg into the bowl, flip the meat over in the pan, etc. The learning process is iterative and is based on multiple trials of a chef-taught motion-profile from the chef studio, which is then executed and iteratively modified by the offline learning algorithm module, until an acceptable execution-sequence can be shown to have been achieved. The mini-manipulation library (command software repository) is intended to have been populated (a-priori and offline) with all the necessary elements to allow the robotic-kitchen system to successfully interact with all equipment (appliances, tools, etc.) and main ingredients that require processing (steps beyond just dispensing) during the cooking process. While the human chef wore gloves with embedded haptic sensors (proximity, touch, contact-location/-force) for the fingers and palm, the robotic hands are outfitted with similar sensor-types in locations to allow their data to be used to create, modify and adapt motion-profiles to successfully execute desired motion-profiles and handling-commands.
[0210] The object-manipulation portion of the robotic-kitchen cooking process (robotic recipe-script execution software module for the interactive manipulation and handling of objects in the kitchen environment) 252 is further elaborated below. Using the robotic recipe-script database 254 (which contains data in raw, abstracted cooking-sequence and machine-executable script forms), the recipe script executor module 256 steps through a specific recipe execution-step. The configuration playback module 258 selects and passes configuration commands through to the robot arm system (torso, arm, wrist and hands) controller 270, which then controls the physical system to emulate the required configuration (joint-positions/-velocities/-torques, etc.) values.
[0211] The notion of being able to faithfully carry out proper environment interaction manipulation and handling tasks is made possible through a real-time process-verification by way of (i) 3D world modeling as well as (ii) mini-manipulation. Both the verification and manipulation steps are carried out through the addition of the robot wrist and hand configuration modifier 260. This software module uses data from the 3D world configuration modeler 262, which creates a new 3D world model at every sampling step from sensory data supplied by the multimodal sensor(s) unit(s), in order to ascertain that the configuration of the robotic kitchen systems and process matches that required by the recipe script (database); if not, it enacts modifications to the commanded system-configuration values to ensure the task is completed successfully. Furthermore, the robot wrist and hand configuration modifier 260 also uses configuration-modifying input commands from the mini-manipulation motion profile executor 264. The hand/wrist (and potentially also arm) configuration modification data fed to the configuration modifier 260 are based on the mini-manipulation motion profile executor 264 knowing what the desired configuration playback should be from 258, but then modifying it based on its 3D object model library 266 and the a-priori learned (and stored) data from the configuration and sequencing library 268 (which was built based on multiple iterative learning steps for all main object handling and processing steps).
[0212] While the configuration modifier 260 continually feeds modified commanded configuration data to the robot arm system controller 270, it relies on the handling/manipulation verification software module 272 to verify not only that the operation is proceeding properly but also whether continued manipulation/handling is necessary. In the case of the latter (answer N to the decision), the configuration modifier 260 re-requests configuration-modification (for the wrist, hands/fingers and potentially the arm and possibly even torso) updates from both the world modeler 262 and the mini-manipulation profile executor 264. The goal is simply to verify that a successful manipulation/handling step or sequence has been successfully completed. The handling/manipulation verification software module 272 carries out this check by using the knowledge of the recipe script database F2 and the 3D world configuration modeler 262 to verify the appropriate progress in the cooking step currently being commanded by the recipe script executor 256. Once progress has been deemed successful, the recipe script index increment process 274 notifies the recipe script executor 256 to proceed to the next step in the recipe-script execution.
[0213]
[0214] The multimodal sensor-unit(s) 302, comprising, but not limited to, video cameras 304, IR cameras and rangefinders 306, stereo (or even trinocular) camera(s) 308 and multi-dimensional scanning lasers 310, provide multi-spectral sensory data to the main software abstraction engines 312 (after being acquired & filtered in the data acquisition and filtering module 314). The data is used in a scene understanding module 316 to carry out multiple steps such as (but not limited to) building high- and lower-resolution (laser: high-resolution; stereo-camera: lower-resolution) three-dimensional surface volumes of the scene, with superimposed visual and IR-spectrum color and texture video information, allowing edge-detection and volumetric object-detection algorithms to infer what elements are in a scene, allowing the use of shape-/color-/texture- and consistency-mapping algorithms to run on the processed data to feed processed information to the Kitchen Cooking Process Equipment Handling Module 318. In the module 318, software-based engines are used for the purpose of identifying and three-dimensionally locating the position and orientation of kitchen tools and utensils and identifying and tagging recognizable food elements (meat, carrots, sauce, liquids, etc.) so as to generate data to let the computer build and understand the complete scene at a particular point in time so as to be used for next-step planning and process monitoring. Engines required to achieve such data and information abstraction include, but are not limited to, grasp reasoning engines, geometry reasoning engines, physical reasoning engines and task reasoning engines. Output data from both engines 316 and 318 are then used to feed the scene modeler and content classifier 320, where the 3D world model is created with all the key content required for executing the robotic cooking script executor. Once the fully-populated model of the world is understood, it can be used to feed the motion and handling planner 322 (if robotic-arm grasping and handling are necessary, the same data can be used to differentiate and plan for grasping and manipulating food and kitchen items depending on the required grip and placement) to allow for planning motions and trajectories for the arm(s) and attached end-effector(s) (grippers, multi-fingered hands). A follow-on Execution Sequence planner 324 creates the proper sequencing of task-based commands for all individual robotic/automated kitchen elements, which are then used by the robotic kitchen actuation systems 326. The entire sequence above is repeated in a continuous closed loop during the robotic recipe-script execution and monitoring phase.
[0215]
[0216]
[0217]
[0218]
[0219]
[0220]
[0221] In some embodiments a chef performs the same food preparation operation multiple times, yielding values of the sensor reading, and parameters in the corresponding robotic instructions that vary somewhat from one time to the next. The set of sensor readings for each sensor across multiple repetitions of the preparation of the same food dish provides a distribution with a mean, standard deviation and minimum and maximum values. The corresponding variations on the robotic instructions (also called the effector parameters) across multiple executions of the same food dish by the chef also define distributions with mean, standard deviation, minimum and maximum values. These distributions may be used to determine the fidelity (or accuracy) of subsequent robotic food preparations.
[0222] In one embodiment the estimated average accuracy of a robotic food preparation operation is given by:
[0223] Where C represents the set of Chef parameters (1.sup.st through n.sup.th) and R represents the set of Robotic Apparatus parameters (correspondingly (1.sup.st through n.sup.th). The numerator in the sum represents the difference between robotic and chef parameters (i.e. the error) and the denominator normalizes for the maximal difference). The sum gives the total normalized cumulative error
and multiplying by 1/n gives the average error. The complement of the average error corresponds to the average accuracy.
[0224] Another version of the accuracy calculation weighs the parameters for importance, where each coefficient (each .sub.i) represents the importance of the i.sup.th parameter, the normalized cumulative error is
and the estimated average accuracy is given by:
[0225]
[0226]
[0227] In order to operate a mechanical robotic mechanism such as the ones described in the embodiments of this invention, a skilled artisan realizes that many mechanical and control problems need to be addressed, and the literature in robotics describes methods to do just that. The establishment of static and/or dynamic stability in a robotics system is an important consideration. Especially for robotic manipulation, dynamic stability is a strongly desired property, in order to prevent accidental breakage or movements beyond those desired or programmed. Dynamic stability is illustrated in
[0228] The cited literature addresses conditions for dynamic stability that are imported by reference into the present invention to enable proper functioning of the robotic arms. These conditions include the fundamental principle for calculating torque to the joints of a robotic arm:
[0229] where T is the torque vector (T has n components, each corresponding to a degree of freedom of the robotic arm), M is the inertial matrix of the system (M is a positive semi-definite n-by-n matrix), C is a combination of centripetal and centrifugal forces, also an n-by-n matrix, G(q) is the gravity vector, and q is the position vector. And they include finding stable points and minima, e.g. via the LaGrange equation if the robotic positions (x's) can be described by twice-differentiable functions (y's).
J[y]=.sub.x1.sup.x2L[x,y(x),y(x)]dx,
J[f]J[f+c].
[0230] In order for the system comprised of the robotic arms and hands/grippers to be stable, it is important that the system be properly designed and built and have an appropriate sensing and control system which operates within the boundary of acceptable performance. The reason that this is important is that one wants to achieve the best (highest speed with highest position/velocity and force/torque tracking and all under stable conditions) performance possible given the physical system and what its controller is asking it to do.
[0231] When one speaks of proper design, the notion is one of achieving proper observability and controllability of the system. Observability implies that the key variables of the system (joint/finger positions and velocities, forces and torques) are measurable by the system, which implies one needs to have the ability to sense these variables, which in turn implies the presence and use of the proper sensing devices (internal or external). Controllability implies that one (computer in this case) have the ability to shape or control the key axes of the system based on observed parameters from internal/external sensors; this usually implies an actuator or direct/indirect control over a certain parameter by way of a motor or other computer-controlled actuation system. The ability to make the system as linear in its response as possible, thereby negating the detrimental effects of nonlinearities (stiction, backlash, hysteresis, etc.), allows for control schemes like PID gain-scheduling and nonlinear controllers like sliding-mode control to guarantee system stability and performance even in the light of system-modeling uncertainties (errors in mass/inertia estimates, dimensional geometry discretization, sensor/torque discretization anomalies, etc.) which are always present in any higher-performance control system.
[0232] Furthermore, the use of a proper computing and sampling system is significant, as the system's ability to follow rapid motions with a certain maximum frequency content is clearly related to what control bandwidth (closed-loop sampling rate of the computer control system) the entire system is able to achieve and thus the frequency-response (ability to track motions of certain speeds and motion-frequency content) the system is able to exhibit.
[0233] All the above characteristics are significant when it comes to ensuring that a highly redundant system can actually carry out the complex and dexterous tasks a human chef requires for a successful recipe-script execution, in both a dynamic and a stable fashion.
[0234] Machine learning in the context of robotic manipulation of relevance to the invention can involve well known methods for parameter adjustment, such as reinforcement learning. An alternate and preferred embodiment for this invention is a different and more appropriate learning technique for repetitive complex actions such as preparing and cooking a meal with multiple steps over time, namely case-based learning. Case-based reasoning, also known as analogical reasoning, has been developed over time.
[0235] As a general overview, case-based reasoning comprises the following steps:
[0236] A.
[0237] Constructing and remembering cases. A case is a sequence of actions with parameters that are successfully carried out to achieve an objective. The parameters include distances, forces, directions, positions, and other physical or electronic measures whose values are required to successfully carry out the task (e.g. a cooking operation). First, [0238] 1. storing aspects of the problem that was just solved together with: [0239] 2. the method(s) and optionally intermediate steps to solve the problem and its parameter values, and [0240] 3. (typically) storing the final outcome.
B. Applying Cases (at a Later Point of Time)
[0241] 4. Retrieving one or more stored cases whose problems bear strong similarity to the new problem, [0242] 5. Optionally adjusting the parameters from the retrieved case(s) to apply to the current case (e.g. an item may weigh somewhat more, and hence a somewhat stronger force is needed to lift it), [0243] 6. Using the same methods and steps from the case(s) with the adjusted parameters (if needed) at least in part to solve the new problem.
Hence, case-based reasoning consists of remembering solutions to past problems and applying them with possible parametric modification to new very similar problems. However, in order to apply case-based reasoning to the robotic manipulation challenge, something more is needed. Variation in one parameter of the solution plan will cause variation in one or more coupled parameters. This requires transformation of the problem solution, not just application. We call the new process case-based robotic learning since it generalizes the solution to a family of close solutions (those corresponding to small variations in the input parameterssuch as exact weight, shape and location of the input ingredients). Case-based robotic learning operates as follows:
C. Constructing, Remembering and Transforming Robotic Manipulation Cases
[0244] 1. Storing aspects of the problem that was just solved together with: [0245] 2. The value of the parameters (e.g. the inertial matrix, forces, etc. from equation 1), [0246] 3. Perform perturbation analysis by varying the parameter(s) pertinent to the domain (e.g. in cooking, vary the weight of the materials or their exact starting position), to see how much parameter values can vary and still obtain the desired results, [0247] 4. Via perturbation analysis on the model, record which other parameter values will change (e.g. forces) and by how much they should change, and [0248] 5. If the changes are within operating specification of the robotic apparatus, store the transformed solution plan (with the dependencies among parameters and projected change calculations for their values).
D. Applying Cases (at a Later Point of Time)
[0249] 6. Retrieve one or more stored cases with the transformed exact values (now ranges, or calculations for new values depending on values of the input parameters), but still whose initial problems bear strong similarity to the new problem, including parameter values and value ranges, and [0250] 7. Use the transformed methods and steps from the case(s) at least in part to solve the new problem.
As the chef teaches the robot (the two arms and the sensing devices, such as haptic feedback from fingers, force-feedback from joints, and one or more observation cameras), the robot learns not only the specific sequence of movements, and time correlations, but also the family of small variations around the chef's movements to be able to prepare the same dish regardless of minor variations in the observable input parametersand thus it learns a generalized transformed plan, giving it far greater utility than rote memorization. For additional information on case-based reasoning and learning, see materials by Leake, 1996 Book, Case-Based Reasoning: Experiences, Lessons and Future Directions, http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=4068324&fileld=50269888 900006585dl.acm.org/citation.cfm?id=524680; Carbonell, 1983, Learning by Analogy: Formulating and Generalizing Plans from Past Experience, http://link.springer.com/chapter/10.1007/978-3-662-12405-5_5, which these references are incorporated by reference herein in their entireties.
[0251] As depicted in
[0252] A person of skill in the art will appreciate that the probability of overall success can be low even if the probability of success of individual stages is relatively high. For instance, given 10 stages and a probability of success of each stage being 90%, the probability of overall success is (0.9).sup.100.28 or 28%.
[0253] A stage in preparing a food dish comprises one or more mini-manipulations, where each mini-manipulation comprises one or more robotic actions leading to a well-defined intermediate result. For instance, slicing a vegetable can be a mini-manipulation consisting of grasping the vegetable with one hand, grasping a knife with the other, and applying repeated knife movements until the vegetable is sliced. A stage in preparing a dish can comprise one or multiple slicing mini-manipulations.
[0254] The probability of success formula applies equally well at the level of stages and at the level of mini-manipulations, so long as each mini-manipulation is relatively independent of other mini-manipulations.
[0255] In one embodiment, in order to mitigate the problem of reduced certainty of success due to potential compounding errors, standardized methods for most or all of the mini-manipulations in all of the stages are recommended. Standardized operations are ones that can be pre-programmed, pretested, and if necessary pre-adjusted to select the sequence of operations with the highest probability of success. Hence, if the probability of standardized methods via the mini-manipulations within stages is very high, so will be the overall probability of success of preparing the food dish, due to the prior work, until all of the steps have been perfected and tested. For instance, to return to the above example, if each stage utilizes reliable standardized methods, and its success probability is 99% (instead of 90% as in the earlier example), then the overall probability of success will be (0.99).sup.10=90.4%, assuming there are 10 stages as before. This is clearly better than 28% probability of an overall correct outcome.
[0256] In another embodiment, more than one alternative method is provided for each stage, wherein, if one alternative fails, another alternative is tried. This requires dynamic monitoring to determine the success or failure of each stage, and the ability to have an alternate plan. The probability of success for that stage is the complement of the probability of failure for all of the alternatives, which mathematically is written as:
[0257] In the above expression s.sub.i is the stage and A(s.sub.i) is the set of alternatives for accomplishing s.sub.i. The probability of failure for a given alternative is the complement of the probability of success for that alternative, namely 1d P(s.sub.i|a.sub.i), and the probability of all the alternatives failing is the product in the above formula. Hence, the probability that not all will fail is the complement of the product. Using the method of alternatives, the overall probability of success can be estimated as the product of each stage with alternatives, namely:
[0258] With this method of alternatives, if each of the 10 stages had 4 alternatives, and the expected success of each alternative for each stage was 90%, then the overall probability of success would be (1(1(0.9)).sup.4).sup.100.99 or 99% versus just 28% without the alternatives. The method of alternatives transforms the original problem from a chain of stages with multiple single points of failure (if any stage fails) to one without single points of failure, since all the alternatives would need to fail in order for any given stage to fail, providing more robust outcomes.
[0259] In another embodiment, both standardized stages comprising standardized mini-manipulations, and alternate means of the food dish preparation stages are combined, yielding even more robust behavior. In such a case, the corresponding probability of success can be very high, even if alternatives are only present for some of the stages or mini-manipulations.
[0260] In another embodiment only the stages with lower probability of success are provided alternatives, in case of failure, for instance stages for which there is no very reliable standardized method, or for which there is potential variability, e.g. depending on odd-shaped materials. This embodiment reduces the burden of providing alternatives to all stages.
[0261]
[0262]
[0263] A predefined mini-manipulation is available to achieve each functional result (e.g., the egg is cracked). Each mini-manipulation comprises of a collection of action primitives which act together to accomplish the functional result. For example, the robot may begin by moving its hand towards the egg, touching the egg to localize its position and verify its size, and executing the movements and sensing actions necessary to grasp and lift the egg into the known and predetermined configuration.
[0264] Multiple mini-manipulations may be collected into stages such as making a sauce for convenience in understanding and organizing the recipe. The end result of executing all of the mini-manipulations to complete all of the stages is that a food dish has been replicated with a consistent result each time.
[0265]
[0266] The robotic hand 72 has the RGB-D sensor 500 placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The RGB-D sensor 500 provides guidance to the robotic hand 72 in moving the robotic hand 72 toward the direction of the object and to make necessary adjustments to grab an object. Second, a sonar sensor 502f and/or a tactile pressure sensor are placed near the palm of the robotic hand 72, for detecting the distance and shape, and subsequent contact, of the object. The sonar sensor 502f can also guide the robotic hand 72 to move toward the object. Additional types of sensors in the hand may include ultrasonic sensors, lasers, radio frequency identification (RFID) sensors, and other suitable sensors. In addition, the tactile pressure sensor serves as a feedback mechanism so as to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to safely lift the object. In addition, the sonar sensor 502f in the palm of the robotic hand 72 provides a tactile sensing function to grab and handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand exerts on the knife and applies to the beef can be detected by the tactile sensor when the knife finishes slicing the beef, i.e. when the knife has no resistance, or when holding an object. The pressure distributed is not only to secure the object, but also not to break it (e.g. an egg).
[0267] Furthermore, each finger on the robotic hand 72 has haptic vibration sensors 502a-e and sonar sensors 504a-e on the respective fingertips, as shown by a first haptic vibration sensor 502a and a first sonar sensor 504a on the fingertip of the thumb, a second haptic vibration sensor 502b and a second sonar sensor 504b on the fingertip of the index finger, a third haptic vibration sensor 502c and a third sonar sensor 504c on the fingertip of the middle finger, a fourth haptic vibration sensor 502d and a fourth sonar sensor 504d on the fingertip of the ring finger, and a fifth haptic vibration sensor 502e and a fifth sonar sensor 504e on the fingertip of the pinky. Each of the haptic vibration sensors 502a, 502b, 502c, 502d and 502e can simulate different surfaces and effects by varying the shape, frequency, amplitude, duration and direction of a vibration. Each of the sonar sensors 504a, 504b, 504c, 504d and 504e provides sensing capability on the distance and shape of the object, sensing capability for the temperature or moisture, as well as feedback capability. Additional sonar sensors 504g and 504h are placed on the wrist of the robotic hand 72.
[0268]
[0269]
[0270]
[0271]
[0272] Together the thenar eminence 532 and hypothenar eminence 534 support application of large forces from the robot arm to an object in the working space such that application of these forces puts minimal stress on the robot hand joints (e.g., picture of the rolling pin). Extra joints within the palm 520 themselves are available to deform the palm. The palm 520 should deform in such a way as to enable the formation of an oblique palmar gutter for tool grasping in a way similar to a chef (typical handle grasp). The palm 520 should deform in such a way as to enable cupping, for conformable grasping of convex objects such as dishes and food materials in a manner similar to the chef, as shown by a cupping posture 542 in
[0273] Joints within the palm 520 that may support these motions include the thumb carpometacarpal joint (CMC), located on the radial side of the palm near the wrist, which may have two distinct directions of motion (flexion/extension and abduction/adduction). Additional joints required to support these motions may include joints on the ulnar side of the palm near the wrist (the fourth finger F4 528 and the fifth finger F5 530 CMC joints), which allow flexion at an oblique angle to support cupping motion at the hypothenar eminence 534 and formation of the palmar gutter.
[0274] The robotic palm 520 may include additional/different joints as needed to replicate the palm shape observed in human cooking motions, e.g., a series of coupled flexure joints to support formation of an arch 540 between the thenar and hypothenar eminences 532 and 534 to deform the palm 520, such as when the thumb F1 522 touches the pinky finger F5 530, as illustrated in
[0275] When the palm is cupped, the thenar eminence 532, the hypothenar eminence 534, and the MCP pads 536 form ridges around a palmar valley that enable the palm to close around a small spherical object (e.g., 2 cm).
[0276] The shape of the deformable palm will be described using locations of feature points relative to a fixed reference frame, as shown in
[0277] Feature points are measured by calibrated cameras mounted in the workspace as the chef performs cooking tasks. Trajectories of feature points in time are used to match the chef motion with the robot motion, including matching the shape of the deformable palm. Trajectories of feature points from the chef's motion may also be used to inform robot deformable palm design, including shape of the deformable palm surface and placement and range of motion of the joints of the robot hand.
[0278] In the embodiment as depicted in
[0279]
[0280] The visual pattern consists of surface markings 552 on the robot hand or on a glove worn by the chef. These surface markings may be covered by a food safe transparent glove 554, but the surface markings 552 remain visible through the glove.
[0281] When the surface markings 552 are visible in a camera image, two-dimensional feature points may be identified within that camera image by locating convex or concave corners within the visual pattern. Each such corner in a single camera image is a two-dimensional feature point.
[0282] When the same feature point is identified in multiple camera images, the three-dimensional location of this point can be determined in a coordinate frame which is fixed with respect to the standardized robotic kitchen 50. This calculation is performed based on the two-dimensional location of the point in each image and the known camera parameters (position, orientation, field of view, etc.).
[0283] A reference frame 556 fixed to the robotic hand 72 can be obtained using a reference frame visual pattern. In one embodiment, the reference frame 556 fixed to the robotic hand 72 comprises of an origin and three orthogonal coordinate axes. It is identified by locating features of the reference frame's visual pattern in multiple cameras, and using known parameters of the reference frame visual pattern and known parameters of the cameras to extract the origin and coordinate axes.
[0284] Three-dimensional shape feature points expressed in the coordinate frame of the food preparation station can be converted into the reference frame of the robot hand once the reference frame of the robot hand is observed.
[0285] The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame fixed to the hand of the robot or the chef.
[0286] As illustrated in
[0287]
[0288] Shape feature point locations are determined based on sensor signals. The sensors provide an output which allows calculation of distance in a reference frame which is attached to the magnet, which furthermore is attached to the hand of the robot or the chef.
[0289] The three-dimensional location of each shape feature point is calculated based on the sensor measurements and known parameters obtained from sensor calibration. The shape of the deformable palm is comprised of a vector of three-dimensional shape feature points, all of which are expressed in the reference coordinate frame, which is fixed to the hand of the robot or the chef. For additional information on common contact regions on the human hand and function in grasping, see the material from Kamakura, Noriko, Michiko Matsuo, Harumi Ishii, Fumiko Mitsuboshi, and Yoriko Miura. Patterns of static prehension in normal hands. American Journal of Occupational Therapy 34, no. 7 (1980): 437-445, which this reference is incorporated by reference herein in its entirety.
[0290]
[0291]
[0292]
[0293]
[0294]
[0295]
[0296]
[0297]
[0298]
[0299] The robotic hand 72 includes a camera senor 684, such as an RGB-D sensor, an imaging sensor or a visual sensing device, placed in or near the middle of the palm for detecting the distance and shape of an object, as well as the distance of the object, and for handling a kitchen tool. The imaging sensor 682f provides guidance to the robotic hand 72 in moving the robotic hand 72 towards the direction of the object and to make necessary adjustments to grab an object. In addition, a sonar sensor, such as a tactile pressure sensor, may be placed near the palm of the robotic hand 72, for detecting the distance and shape of the object. The sonar sensor 682f can also guide the robotic hand 72 to move toward the object. Each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g includes ultrasonic sensors, laser, radio frequency identification (RFID), and other suitable sensors. In addition, each of the sonar sensors 682a, 682b, 682c, 682d, 682e, 682f, 682g serves as a feedback mechanism to determine whether the robotic hand 72 continues to exert additional pressure to grab the object at such point where there is sufficient pressure to grab and lift the object. In addition, the sonar sensor 682f in the palm of the robotic hand 72 provides tactile sensing function to handle a kitchen tool. For example, when the robotic hand 72 grabs a knife to cut beef, the amount of pressure that the robotic hand 72 exerts on the knife and applies to the beef, allows the tactile sensor to detect when the knife finishes slicing the beef, i.e., when the knife has no resistance. The distributed pressure is not only to secure the object, but also so as not to exert too much pressure so as to, for example, not to break an egg). Furthermore, each finger on the robotic hand 72 has a sensor on the finger tip, as shown by the first sensor 682a on the finger tip of the thumb, the second sensor 682b on the finger tip of the index finger, the third sensor 682c on the finger tip of the middle finger, the fourth sensor 682d on the finger tip of the ring finger, and the fifth sensor 682f on the finger tip of the pinky. Each of the sensors 682a, 682b, 682c, 682d, 682e provide sensing capability on the distance and shape of the object, sensing capability for temperature or moisture, as well as tactile feedback capability.
[0300] The RGB-D sensor 684 and the sonar sensor 682f in the palm, plus the sonar sensors 682a, 682b, 682c, 682d, 682e in the finger tip of each finger, provide a feedback mechanism to the robotic hand 72 as a means to grab a non-standardized object, or a non-standardized kitchen tool. The robotic hands 72 may adjust the pressure to a sufficient degree to grab a hold of the non-standardized object. A program library 690 that stores sample grabbing functions 692, 694, 696 according to a specific time interval for which the robotic hand 72 can draw from in performing a specific grabbing function, is illustrated in
[0301]
[0302]
[0303]
[0304]
[0305]
[0306]
[0307]
[0308] To create the mini-manipulation that results in cracking an egg with a knife, multiple parameter combinations must be tested to identify a set of parameters that ensure the desired functional resultthat the egg is crackedis achieved. In this example, parameters are identified to determine how to grasp and hold an egg in such a way so as not to crush it. An appropriate knife is selected through testing, and suitable placements are found for the fingers and palm so that it may be held for striking. A striking motion is identified that will successfully crack an egg. An opening motion and/or force are identified that allows a cracked egg to be opened successfully.
[0309] The teaching/learning process for the robotic apparatus involves multiple and repetitive tests to identify the necessary parameters to achieve the desired final functional result.
[0310] These tests may be performed over varying scenarios. For example, the size of the egg can vary. The location at which it is to be cracked can vary. The knife may be at different locations. The mini-manipulation must be successful in all of these variable circumstances.
[0311] Once the learning process has been completed, results are stored as a collection of action primitives that together are known to accomplish the desired functional result.
[0312]
[0313] As an example of the operative relationship between the creation of a mini-manipulation in
[0314]
[0315]
[0316]
[0317] At step 892, the computer 16 tests and validates the specific successful parameter combination for X number of times (such as one hundred times). At step 894, the computer 16 computes the number of failed results during the repeated test of the specific successful parameter combination. At step 896, the computer 16 selects the next one-time successful parameter combination from the temporary library, and returns the process back to step 892 for testing the next one-time successful parameter combination X number of times. If no further one-time successful parameter combination remains, the computer 16 stores the test results of one or more sets of parameter combinations that produce a reliable (or guaranteed) result at step 898. If there are more than one reliable sets of parameter combinations, at step 899, the computer 16 determines the best or optimal set of parameter combinations and stores the optimal set of parameter combination which is associated with the specific mini-manipulation for use in the mini-manipulation library database by the robotic apparatus in the standardized robotic kitchen 50 during the food preparation stages of a recipe.
[0318]
[0319]
[0320]
[0321]
[0322]
[0323]
[0324]
[0325]
[0326]
[0327]
[0328]
[0329]
[0330]
[0331]
[0332] One embodiment of the flow charts in functioning as a recipe filter, an ingredient filter, an equipment filter, an account and social network access, a personal partner page, a shopping cart page, and the information on the purchased recipe, registration setting, create a recipe are illustrated in
[0333] In
[0334] An example of partnership between users of the platform is demonstrated in
[0335]
[0336]
[0337]
[0338] The standardized robotic kitchen 50 in
[0339] Based on the proper placement of the augmented sensor system 1854 placed somewhere in the robotic kitchen, such as on a computer-controllable railing, or on the torso of a robot with arms and hands, allows for 3D-tracking and raw data generation, both during chef-monitoring for machine-specific recipe-script generation, and monitoring the progress and successful completion of the robotically-executed steps in the stages of the dish replication in the standardized robotic kitchen 50.
[0340] The standardized robotic kitchen 50 in
[0341]
[0342] The standardized robotic kitchen 50 depicts another possible configuration for the use of one or more augmented sensor systems 20. The standardized robotic kitchen 50 shows a multitude of augmented sensor systems 20 placed in the corners above the kitchen work-surface along the length of the kitchen axis with the intent to effectively cover the complete visible three-dimensional workspace of the standardized robotic kitchen 50.
[0343] The proper placement of the augmented sensor system 20 in the standardized robotic kitchen 50, allows for three-dimensional sensing, using video-cameras, lasers, sonars and other two- and three-dimensional sensor systems to enable the collection of raw data to assist in the creation of processed data for real-time dynamic models of shape, location, orientation and activity for robotic arms, hands, tools, equipment and appliances, as they relate to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50.
[0344] Raw data is collected at each point in time to allow the raw data to be processed to be able to extract the shape, dimension, location and orientation of all objects of importance to the different steps in the multiple sequential stages of dish replication in the standardized robotic kitchen 50 in a step 1162. The processed data is further analyzed by the computer system to allow the controller of the standardized robotic kitchen to adjust robotic arm and hand trajectories and mini-manipulations, by modifying the control signals defined by the robotic script. Adaptations to the recipe-script execution and thus control signals is essential in successfully completing each stage of the replication for a particular dish, given the potential for variability for many variables (ingredients, temperature, etc.). The process of recipe-script execution based on key measurable variables is an essential part of the use of the augmented (also termed multi-modal) sensor system 20 during the execution of the replicating steps for a particular dish in a standardized robotic kitchen 50.
[0345]
[0346]
[0347]
[0348]
[0349] To access the ingredients storage-and-supply unit, part of the countertop with sliding doors can be opened, where the recipe software controls the doors and moves designated containers and ingredients to the access location where the robotic arm(s) may pick up the containers, open the lid, remove the ingredients out of the containers to a designated place, reseal the lid and move the containers back into storage. The container is moved from the access location back to its default location in the storage unit, and a new/next container item is then uploaded to the access location to be picked up.
[0350] An alternative embodiment for an ingredient storage-and-supply unit 1210 is depicted in
[0351] In
[0352]
[0353]
[0354]
[0355]
[0356]
[0357]
[0358]
[0359] The top level 1292-1 contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc.
[0360] The counter level 1292-2 not only houses the robotic arms 70, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher.
[0361] The lower level 1292-3 houses the combination convection oven and microwave 1316, the dish-washer 1318 and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware and packing materials and cutlery.
[0362]
[0363] The perspective view of the robotic kitchen 50 clearly identifies one of the many possible layouts and locations for equipment at all three levels, including the top level 1292-1 (storage pantry 1304, standardized cooking tools and ware 1320, storage ripening zone 1298, chilled storage zone 1300, and frozen storage zone 1302, the counter level 1292-2 (robotic arms 70, sink 1308, chopping/cutting area 1310, charcoal grill 1312, cooking appliances 1314 and serving counter 1306) and the lower level (dish-washer 1318 and oven and microwave 1316).
[0364]
[0365]
[0366]
[0367]
[0368]
[0369]
[0370]
[0371]
[0372]
[0373]
[0374]
[0375]
[0376]
[0377]
[0378]
[0379]
[0380]
[0381]
[0382]
[0383]
[0384]
[0385]
[0386]
[0387]
[0388]
[0389]
[0390]
[0391]
[0392]
[0393]
[0394]
[0395]
[0396]
[0397]
[0398]
[0399]
[0400]
[0401]
[0402]
[0403]
[0404]
[0405]
[0406]
[0407] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage area 82 is included, a cabinet volume 1320 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0408] The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0409] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0410]
[0411] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level a shelf/cabinet storage pantry volume 1294 is included, a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 88 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0412] The counter level not only houses monitoring sensors 1892 and control units 1894, but also includes a counter area with a sink and electronically controllable faucet 1308, another counter area 1310 with removable working surfaces for cutting/chopping on a board, etc., a charcoal-based slatted grill 1312, and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1892 providing data to one or more control units 1894, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0413] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1892 providing data to one or more control units 1896, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0414]
[0415]
[0416]
[0417]
[0418]
[0419] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0420] The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0421] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 82, and a larger cabinet volume 3=378 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0422]
[0423] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment.
[0424] At the simplest level this includes a cabinet volume 1294 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0425] The counter level not only houses monitoring sensors 1884 and control units 1886, but also includes the one or more robotic arms, wrists and multi-fingered hands 72, a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0426] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1315, the dish-washer 1318, the hard automation controlled ingredient dispensers 82 (not shown), and a larger cabinet volume 1310 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0427]
[0428] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 1302 for deep-frozen items, and another storage pantry zone 1304 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0429] The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 2020 while also including a serving counter 1306, a counter area with a sink 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices 2022 are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
[0430] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not shown), and a larger cabinet volume 1320 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0431]
[0432] The top level contains multiple cabinet-type modules with different units to perform specific kitchen functions by way of built-in appliances and equipment. At the simplest level this includes a cabinet volume 1296 used for storing and accessing standardized cooking tools and utensils and other cooking and serving ware (cooking, baking, plating, etc.), a storage ripening cabinet volume 1298 for particular ingredients (e.g. fruit and vegetables, etc.), a chilled storage zone 1300 for such items as lettuce and onions, a frozen storage cabinet volume 86 for deep-frozen items, and another storage pantry zone 1294 for other ingredients and rarely used spices, etc. Each of the modules within the top level contains sensor units 1884 providing data to one or more control units 1886, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0433] The counter level houses not only monitoring sensors 1884 and control units 1886, but also visual command monitoring devices 1316 while also including a counter area with a sink and electronic faucet 1308, another counter area 1310 with removable working surfaces (cutting/chopping board, etc.), a (smart) charcoal-based slatted grill 1312 and a multi-purpose area for other cooking appliances 1314, including a stove, cooker, steamer and poacher. Each of the modules within the counter level contains sensor units 1184 providing data to one or more control units 1186, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations. Additionally, one or more visual command monitoring devices (not shown) are also provided within the counter level for the purposes of monitoring the visual operations of the human chef in the studio kitchen as well as the robotic arms or human user in the standardized robotic kitchen, where data is fed to one or more central or distributed computers for processing and subsequent corrective or supportive feedback and commands sent back to the robotic kitchen for display or script-following execution.
[0434] The lower level houses the combination convection oven and microwave as well as steamer, poacher and grill 1316, the dish-washer 1318, the hard automation controlled ingredient dispensers 86 (not showed)s, and a larger cabinet volume 1309 that holds and stores additional frequently used cooking and baking ware, as well as tableware, flatware, utensils (whisks, knives, etc.) and cutlery. Each of the modules within the lower level contains sensor units 1307 providing data to one or more control units 376, either directly or by way of one or more central or distributed control computers, to allow for computer-controlled operations.
[0435]
[0436]
[0437]
[0438]
[0439]
[0440]
[0441]
[0442]
[0443]
[0444]
[0445] In one embodiment the emotional profile can be detected via machine learning methods based on statistical classifiers where the inputs are any measured levels of pheromones, hormones, or other features such as visual or auditory cues. If the set of features is {x.sub.1, x.sub.2, x.sub.3, . . . , x.sub.n} represented as a vector and y represents the emotional state, then the general form of an emotion-detection statistical classifier is:
Where the function f is a decision tree, a neural network, a logistic regressor, or other statistical classifier described in the machine learning literature. The first term minimizes the empirical error (the error detected while training the classifier) and the second term minimizes the complexitye.g. Occam's razor, finding the simplest function and set of parameters p for that function that yield the desired result.
[0446] Additionally, in order to determine which pheromones or other features make the most difference (add the most value) to predicting emotional state, an active-learning criterion can be added, generally expressed as:
Where L is a loss function, f is the same statistical classifier as in the previous equation, and y-hat is the known outcome. We measure whether the statistical classifier performs better (smaller loss function) by addition new features, and if so keep them, otherwise not.
[0447] Parameters, values and quantities that evolve over time can be assessed to create a human emotional profile by detecting the change or transformation from one moment to the next. There are identifiable qualities to an emotional expression. A robot with emotions in response to its environment could make quicker and more effective decisions, e.g. when a robot is motivated by fear or joy or desire it might make better decisions and attain the goals more effectively and efficiently.
[0448] The robotic emotion engine replicates the human hormone emotions and pheromone emotions, either individually or in combination. Hormone emotions refer to how hormones change inside of a person's body and how that affects a person's emotions. Pheromone emotions refer to pheromones that are outside a person's body, such as smell, that affect a person's emotions. A person's emotional profile can be constructed by understanding and analyzing the hormone and pheromone emotions. The robotic emotion engine attempts to understand a person's emotions such as anger and fear by using sensors to detect a person's hormone and pheromone profile.
[0449] There are nine key physiological sign parameters to be measured in order to build a person's emotional profile: (1) sets of hormones 2221, which are secreted internally and trigger various biochemical pathways that cause certain effects, e.g. adrenalin and insulin are hormones, (2) sets of pheromones 2222, which are secreted externally, and have an effect on another person in a similar way, e.g. androstenol, androstenone and androstadienone, (3) micro expression 2223, which is a brief, involuntary facial expression shown by humans according to emotions experienced, (4) the heart rate 2224 or heart beat, e.g., when a person's heart rate increases, (5) sweat 2225 (e.g., goose bumps) e.g. face blushes and palms get sweaty and in the state of being excited or nervous, (6) pupil dilation 2226 (and iris sphincter, biliary muscle), e.g. pupil dilation for a short time in response to feelings of fear, (7) reflex movement v7, which is the movement/action primarily controlled by the spinal arc, as a response to an external stimulus, e.g. jaw jerk reflex, (8) body temperature 2228 (9) pressure 2229. The analysis 2230 on how these parameters change over a certain time 2231 may reveal a person's emotional state and profile.
[0450]
[0451]
[0452] When a user experiences an emotion or mood swing, physiological parameters such as hormone, heart rate, sweat, pheromones can be detected and recorded with a port connecting to a person's body, above the skin and directly to the vein. The start time and end time of the mood change can be determined by the person himself or herself as the person's emotional state changes. For example, a person initiates four manual emotion cycles and creates four timelines within a week, and as determined by the person, the first one lasts 2.8 hour from the time he tags the start till the time he tags the end. The second cycle last for 2 hours, the third one last for 0.8 hours, and the fourth one last for 1.6 hours.
[0453]
[0454]
[0455]
[0456]
[0457] The computer module 1420 includes modules that include, but are not limited to, a robotic painting engine 1352 interfaced to a painting movement emulator 1422, a painting control module 1421 that acts based on visual feedback of the painting execution processes, a memory module 1380 to store painting execution program files, algorithms 1423 for learning the selection and usage of the appropriate drawing tools, as well as an extended simulation validation and calibration module 1378.
[0458]
[0459] One embodiment of the art platform standardization is defined as follows. First, standardized position and orientation (xyz) of any kind of art tools (brushes, paints, canvas, etc.) in the art platform. Second, standardized operation volume dimensions and architecture in each art platform. Third, standardized art tools set in each art platform. Fourth, standardized robotic arms and hands with a library of manipulations in each art platform. Fifth, standardized three-dimensional vision devices for creating dynamic three-dimensional vision data for painting recording and execution tracking and quality check function in each art platform. Sixth, standardized type/producer/mark/of all using paints during particular painting execution. Seventh, standardized type/producer/mark/size of canvas during particular painting execution.
[0460] One main purpose to have Standardized Art Platform is to achieve the same result of the painting process (i.e., the same painting) executing by the original painter and afterward duplicated by robotic Art Platform. Several main points to emphasize in using the standardized Art Platform: (1) have the same timeline (same sequence of manipulations, same initial and ending time of each manipulation, same speed of moving object between manipulations) of Painter and automatic robotic execution; and (2) there are quality checks (3D vision, sensors) to avoid any fail result after each manipulation during the painting process. Therefore the risk of not having the same result is reduced if the painting was done at the standardized art platform. If a non-standardized art platform is used, this will increase the risk of not having the same result (i.e. not the same painting) because adjustment algorithms may be required when the painting is not executed at not the same volume, with the same art tools, with the same paint or with the same canvas in the painter studio as in the robotic art platform.
[0461]
[0462]
[0463]
[0464]
[0465] In the case the user selects the robot engine to select the title/composer in step 1482, the engine uses its own interpretation of creativity in step 1492, to offer the human user to provide input to the selection process in step 1493. Should the human decline providing input, the robotic musician engine uses settings such as manual inputs to tonality, pitch and instrumentation as well as melodic variation in step 1499, to gather the necessary input in step 1130 to generate and upload selected instrument playing execution program files in step 1501, allowing the user to select the preferred one in step 1503, after the robotic musician engine has confirmed the selection in step 1502. The choice made by the human is then stored as a personal choice in the personal profile database in step 1504. Should the human decide to provide input to the query in step 1493, the user will be able in step 1493 to provide additional emotional input to the selection process (facial expressions, photo, news article, etc.). The input from step 194 is received by the robotic musician engine in step 1495, allowing it to proceed to step 1496, where the engine carries out a sentiment analysis related to all available input data and uploads a music selection based on the mood and style appropriate to the emotional input data from the human. Upon confirmation of selection for the uploaded music selection in step 1497 by the robotic musician engine, the user may select the start button to play the program file for the selection in step 1498.
[0466] In the case where the human wants to be intimately involved in the selection of the title/composer, the system provides a list of performers for the selected title to the human on a display in step 1483. In step 1484 the user selects the desired performer, a choice input that the system receives in step 1485. In step 1486 the robotic musician engine generates and uploads the instrument playing execution program files, and proceeds in step 1487 to compare potential limitations between a human and a robotic musician's playing performance on a particular instrument, thereby allowing it to calculate a potential performance gap. A checking step 1488 decides whether there exists a gap. Should there be a gap, the system will suggest other selections based on the user's preference profile in step 1489. Should there be no performance gap, the robotic musician engine will confirm the selection in step 1490 and allow the user to proceed to step 1491, where the user may select the start button to play the program file for the selection.
[0467]
[0468]
[0469]
[0470]
[0471] The example computer system 224 includes a processor 226 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 228 and a static memory 230, which communicate with each other via a bus 232. The computer system 224 may further include a video display unit 234 (e.g., a liquid crystal display (LCD)). The computer system 224 also includes an alphanumeric input device 236 (e.g., a keyboard), a cursor control device 238 (e.g., a mouse), a disk drive unit 240, a signal generation device 242 (e.g., a speaker), and a network interface device 248.
[0472] The disk drive unit 2240 includes a machine-readable medium 244 on which is stored one or more sets of instructions (e.g., software 246) embodying any one or more of the methodologies or functions described herein. The software 246 may also reside, completely or at least partially, within the main memory 244 and/or within the processor 226 during execution thereof the computer system 224, the main memory 228, and the instruction-storing portions of processor 226 constituting machine-readable media. The software 246 may further be transmitted or received over a network 18 via the network interface device 248.
[0473] While the machine-readable medium 244 is shown in an example embodiment to be a single medium, the term machine-readable medium should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term machine-readable medium shall also be taken to include any tangible medium that is capable of storing a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present invention. The term machine-readable medium shall accordingly be taken to include, but not be limited to, solid-state memories, and optical and magnetic media.
[0474] In general terms, there may be considered a method of motion capture and analysis for a robotics system, comprising sensing a sequence of observations of a person's movements by a plurality of robotic sensors as the person prepares a product using working equipment; detecting in the sequence of observations mini-manipulations corresponding to a sequence of movements carried out in each stage of preparing the product; transforming the sensed sequence of observations into computer readable instructions for controlling a robotic apparatus capable of performing the sequences of mini-manipulations; storing at least the sequence of instructions for mini-manipulations to electronic media for the product. This may be repeated for multiple products. The sequence of mini-manipulations for the product is preferably stored as an electronic record. The mini-manipulations may be abstracted parts of a multi-stage process, such as cutting an object, heating an object (in an oven or on a stove with oil or water), or similar. Then, the method may further comprise: transmitting the electronic record for the product to a robotic apparatus capable of replicating the sequence of stored mini-manipulations, corresponding to the original actions of the person. Moreover, the method may further comprise executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
[0475] In another general aspect, there may be considered a method of operating a robotics apparatus, comprising providing a sequence of pre-programmed instructions for standard mini-manipulations, wherein each mini-manipulation produces at least one identifiable result in a stage of preparing a product; sensing a sequence of observations corresponding to a person's movements by a plurality of robotic sensors as the person prepares the product using equipment; detecting standard mini-manipulations in the sequence of observations, wherein a mini-manipulation corresponds to one or more observations, and the sequence of mini-manipulations corresponds to the preparation of the product; transforming the sequence of observations into robotic instructions based on software implemented methods for recognizing sequences of pre-programmed standard mini-manipulations based on the sensed sequence of person motions, the mini-manipulations each comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; storing the sequence of mini-manipulations and their corresponding robotic instructions in electronic media. Preferably, the sequence of instructions and corresponding mini-manipulations for the product are stored as an electronic record for preparing the product. This may be repeated for multiple products. The method may further include transmitting the sequence of instructions (preferably in the form of the electronic record) to a robotics apparatus capable of replicating and executing the sequence of robotic instructions. The method may further comprise executing the robotic instructions for the product by the robotics apparatus, thereby obtaining substantially the same result as the original product prepared by the human. Where the method is repeated for multiple products, the method may additionally comprise providing a library of electronic descriptions of one or more products, including the name of the product, ingredients of the product and the method (such as a recipe) for making the product from ingredients.
[0476] Another generalized aspect provides a method of operating a robotics apparatus comprising receiving an instruction set for a making a product comprising of a series of indications of mini-manipulations corresponding to original actions of a person, each indication comprising a sequence of robotic instructions and the robotic instructions including dynamic sensing operations and robotic action operations; providing the instruction set to a robotic apparatus capable of replicating the sequence of mini-manipulations; executing the sequence of instructions for mini-manipulations for the product by the robotic apparatus, thereby obtaining substantially the same result as the original product prepared by the person.
[0477] A further generalized method of operating a robotic apparatus may be considered in a different aspect, comprising executing a robotic instructions script for duplicating a recipe having a plurality of product preparation movements; determining if each preparation movement is identified as a standard grabbing action of a standard tool or a standard object, a standard hand-manipulation action or object, or a non-standard object; and for each preparation movement, one or more of: instructing the robotic cooking device to access a first database library if the preparation movement involves a standard grabbing action of a standard object; instructing the robotic cooking device to access a second database library if the food preparation movement involves a standard hand-manipulation action or object; and instructing the robotic cooking device to create a three-dimensional model of the non-standard object if the food preparation movement involves a non-standard object. The determining and/or instructing steps may be particularly implemented at or by a computer system. The computing system may have a processor and memory.
[0478] Another aspect may be found in a method for product preparation by robotic apparatus, comprising replicating a recipe by preparing a product (such as a food dish) via the robotic apparatus, the recipe decomposed into one or more preparation stages, each preparation stage decomposed into a sequence of mini-manipulations and active primitives, each mini-manipulation decomposed into a sequence of action primitives. Preferably, each mini manipulation has been (successfully) tested to produce an optimal result for that mini manipulation in view of any variations in positions, orientations, shapes of an applicable object, and one or more applicable ingredients.
[0479] A further method aspect may be considered in a method for recipe script generation, comprising receiving filtered raw data from sensors in the surroundings of a standardized working environment module, such as a kitchen environment; generating a sequence of script data from the filtered raw data; and transforming the sequence of script data into machine-readable and machine-executable commands for preparing a product, the machine-readable and machine-executable commands including commands for controlling a pair of robotic arms and hands to perform a function. The function may be from the group consisting of one or more cooking stages, one or more mini-manipulations, and one or more action primitives. A recipe script generation system comprising hardware and/or software features configured to operate in accordance with this method may also be considered.
[0480] In any of these aspects, the following may be considered. The preparation of the product normally uses ingredients. Executing the instructions typically includes sensing properties of the ingredients used in preparing the product. The product may be a food dish in accordance with a (food) recipe (which may be held in an electronic description) and the person may be a chef. The working equipment may comprise kitchen equipment. These methods may be used in combination with any one or more of the other features described herein. One, more than one or all of the features of the aspects may be combined, so a feature from one aspect may be combined with another aspect for example. Each aspect may be computer-implemented and there may be provided a computer program configured to perform each method when operated by a computer or processor. Each computer program may be stored on a computer-readable medium. Additionally or alternatively, the programs may be partially or fully hardware-implemented. The aspects may be combined. There may also be provided a robotics system configured to operate in accordance with the method described in respect of any of these aspects.
[0481] In another aspect, there may be provided a robotics system, comprising: a multi-modal sensing system capable of observing human motions and generating human motions data in a first instrumented environment; and a processor (which may be a computer), communicatively coupled to the multi-modal sensing system, for recording the human motions data received from the multi-modal sensing system and processing the human motions data to extract motion primitives, preferably such that the motion primitives define operations of a robotics system. The motion primitives may be mini-manipulations, as described herein (for example in the immediately preceding paragraphs) and may have a standard format. The motion primitive may define specific types of action and parameters of the type of action, for example a pulling action with a defined starting point, end point, force and grip type. Optionally, there may be further provided a robotics apparatus, communicatively coupled to the processor and/or multi-modal sensing system. The robotics apparatus may be capable of using the motion primitives and/or the human motions data to replicate the observed human motions in a second instrumented environment.
[0482] In a further aspect, there may provided a robotics system, comprising: a processor (which may be a computer), for receiving motion primitives defining operations of a robotics system, the motion primitives being based on human motions data captured from human motions; and a robotics system, communicatively coupled to the processor, capable of using the motion primitives to replicate human motions in an instrumented environment. It will be understood that these aspects may be further combined.
[0483] A further aspect may be found in a robotics system comprising: first and second robotic arms; first and second robotic hands, each hand having a wrist coupled to a respective arm, each hand having a palm and multiple articulated fingers, each articulated finger on the respective hand having at least one sensor; and first and second gloves, each glove covering the respective hand having a plurality of embedded sensors. Preferably the robotics system is a robotic kitchen system.
[0484] There may further be provided, in a different but related aspect, a motion capture system, comprising: a standardized working environment module, preferably a kitchen; plurality of multi-modal sensors having a first type of sensors configured to be physically coupled to a human and a second type of sensors configured to be spaced away from the human. One or more of the following may be the case: the first type of sensors may be for measuring the posture of human appendages and sensing motion data of the human appendages; the second type of sensors may be for determining a spatial registration of the three-dimensional configurations of one or more of the environment, objects, movements, and locations of human appendages; the second type of sensors may be configured to sense activity data; the standardized working environment may have connectors to interface with the second type of sensors; the first type of sensors and the second type of sensors measure motion data and activity data, and send both the motion data and the activity data to a computer for storage and processing for product (such as food) preparation.
[0485] An aspect may additionally or alternatively be considered in a robotic hand coated with a sensing gloves, comprising: five fingers; and a palm connected to the five fingers, the palm having internal joints and a deformable surface material in three regions; a first deformable region disposed on a radial side of the palm and near the base of the thumb; a second deformable region disposed on a ulnar side of the palm, and spaced apart from the radial side; and a third deformable region disposed on the palm and extend across the base of the fingers. Preferably, the combination of the first deformable region, the second deformable region, the third deformable region, and the internal joints collectively operate to perform a mini manipulation, particularly for food preparation.
[0486] In respect of any of the above system, device or apparatus aspects, there may further be provided method aspects comprising steps to carry out the functionality of the system. Additionally or alternatively, optional features may be found based on any one or more of the features described herein with respect to other aspects
TABLE-US-00001 TABLE A Types of Equipment Types of Equipment KITCHEN ACCESSORIES 1 Funnels 1.1. stainless steel funnel 1.2. plastic funnel 1.3 silicone funnel 1.4 convertible funnel 2 Colanders 2.1 quadratic colanders 2.2 oval ladle-vases 2.3 colanders with folding handles 2.4 flat colander 2.5 plastic colanders 2.6 small round colanders 2.7 suspended colanders 2.8 cover-colander 2.9 stainless steel and aluminum colanders 2.1 cone colanders 3 Kitchen Appliances 3.1. Whisk 3.2. scoop, spatula 3.2.1 cook spatula 3.2.2. spatula with slots 3.2.3. confectionery spatula 3.5 Spoons 3.5.1 serving spoon 3.5.2 spoon-tongs 3.5.3 spoon with slots 3.5.4 spoon for rice 3.5.5 ladle spoon 3.5.6 ice cream spoon 3.5.7 honey spoon 3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionery syringe for cookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9 skimmer 3.10 Meat fork 3.11 Brush 3.12 coffee filter 3.5.7 honey spoon 3.5.8 spaghetti spoon 3.5.9 serving spoon 3.6 confectionery syringe for cookies and cream 3.7 soup ladle 3.8 Potato Masher 3.9 skimmer 3.10 Meat fork 3.11 Brush 3.23 ties for rolls 3.2 dough mini-scraper 3.25 grill tongs 3.26 spaghetti tongs 3.27 ice tongs 3.28 sugar tongs 3.29 package clip 3.30 package clip 3.31 citrus spray 3.32 Dough press 3.33 scoop for bulk 3.34 salad serving tongs (tweezers) 3.35 accessories for tubes 3.36 Pestle 3.37 Mortar 3.38 roller for cutting of the rings 3.39 opener for caps 3.40 meat tenderizer; meat softener 3.41 egg yolk separator 3.42 Apron 3.33 scoop for bulk 3.43 tools for decoration 3.44 jar for oil and vinegar 3.45 mug for milk boiling 3.46 napkins 3.47 tablecloth 3.48 marker for glasses 3.49 potato masher 3.50 Basket 3.51 meat tenderizer 3.52 cocotte 3.53 brush for washing of the vegetables 3.54 lids for cups 3.55 rope for baking 3.56 jar for herbs storage 3.57 Mortar 3.58 scraper for glass ceramic plates 3.59 Teapot for tea 3.60 clothespin for notes on the fridge 3.61 railing systems 3.62 hanger for kitchen tools 3.63 plunger with not adhering surface 3.64 silicone plunger 3.65 rolling pin with adjustable thickness 3.66 vacuum bags with pump 3.67 gas lighter 3.68 bone forceps 4 kitchen timers, thermometers 4.1 timer for meat roasting 4.2 digital thermometer 4.3 holder for thermometer 4.4 meat thermometer 4.5 digital timer 4.6 elector. digital timer 4.7 _aramel thermometer 5 Mills for spices 5.1 mill for black pepper 5.2 electric mill 5.3 combined mill for pepper and salt (2 in 1) 5.4 mill for spices 5.5 mill for greens 6 Measuring utensils 6.1. Measuring container (plastic bottle) 6.2. measuring jar 6.3. measuring jug 6.4. measuring bowl 6.5. mechanical dispenser for ice cream 7 mechanical mixers 8 Bowl 8.1. metal bowl 8.2. stainless steel bowl 8.3. plastic bowl 8.4. plastic bowl 8.5. bowls for food 9 Sets 9.2 wine set 9.3 sets for spices 9.6. cupcakes baking set 9.7 accessory kit for baking 9.8 set of bar tools 9.9 set of kitchen tools 9.10 Set for eggs and pancakes baking 11 Slicing and cutting of products 11.1 Cutter 11.2 holder for onions cutting 11.2 cutting boards 11.3 universal professional knives 11.4 kitchen shears 11.5 hatchet 11.6 meat hatchet 11.7 Hammer for meat with hatchet 11.8 Hoe 11.9 Hammer for meat 11.10 Knives 11.11 knife for greens 11.12 knife for oranges 11.13 knife for kiwi 11.14 knife for pineapple 11.15 Spiral knife for carrots 11.16 multifunctional knife 11.17 vegetable knife 11.18 Pizza Cutter 11.19 universal knife 11.20 knife for slicing 11.21 cook knife 11.22 gastronomic knife 11.23 opener 11.24 Cheese knife 11.25 boning knife 11.26 lettuce knife 11.27 knife for steaks 11.28 butcher knife 11.29 shredding knife 11.30 bread knife 11.31 fish knife 11.32 knife for sandwiches 11.33 Santoku knife 11.34 knife for fruit coring 11.35 Butter knife 12 openers 12.1 tin-opener 12.2 corkscrew 12.3 corkscrew on a stand 12.4 lever corkscrew 12.5 folding corkscrew 12.6 opener for waiter 12.7 openers 13 stand and holders 13.1 stands for hot 13.2 stand for kitchen utensils storing 13.3 toothpick holder 13.4 Bottle holder 13.5 Holder for capsules 13.6 stand for spoon 13.7 stand for coffee capsules 13.8 Coasters 13.9 Napkin holder 13.10 stand for eggs 13.11 stand for openers 13.12 stand for scoops 13.13 stand for cooking and serving of eggs 13.14 stand for ladle 13.15 Holder for paper towels 13.16 Transforming stand for kitchen appliances 13.17 stand for mug 13.18 stand mugs and saucers 13.19 stand for kitchen knives 13.20 stand for chicken 13.21 napkin-stand 13.22 heated stand 13.23 stands for cake 14 Appliances for peeling and cutting 14.1 grater for vegetables 14.2 grater 14.3 garlic masher 14.4 egg cutter 14.5 Manual vegetable cutter 14.6 Peeler for vegetables 14.7 Nutcracker 14.8 The device for separating the yolks from the whites 14.9 grasping for carrots cleaning 14.10 scraper fish scales 14.11 cutter for fruits 14.12 roller for holes 14.13 tongs for fish bones 14.14 spiral vegetable cutter 15 Bottle Caps 15.1 champagne cork (stopper) 15.2 stoppers for wine 15.3 The opener to remove the crown corks from bottles 16 sieves 16.1 sieve for tea 16.2 sieve-tongs for tea 16.3 Strainer for spices 16.4 Strainer for tea 16.5 Universal sieve 16.6 flour sieve 16.7 sieve to form the Bird's Nest 16.8 The Chinese sieve with a mesh insert 16.9 sieve with support 16.10 Mug-sieve for flour 16.11 sieve on the handle 17 Salt and pepper shakers 17.1 container for seasoning 17.2 salt cellar 17.3 containers for oil and vinegar 18 Dish dryers 18.1 salad dryer dryer-placemat dryer for crockery and cutlery 19 Cutlery Accessories 19.1 cutlery tray 19.2 cutlery holder 19.3 cutlery container 19.4 strainer for cutlery 19.5 wall hanger for kitchen tools 19.6 cutlery organizer 19.7 mat for cutlery 19.8 sliding tray for cutlery 19.9 dryer for cutlery 19.10 glass for cutlery 19.11 napkin for the cutlery 19.12 case for cutlery 19.13 tray for cutlery 19.14 mitten-potholder 19.15 box for cutlery 19.16 full-size rack (cassette) for cutlery 19.17 Stand without containers for cutlery 19.18 cassette for cutlery 19.19 container for cutlery 19.20 station for cutlery 19.21 Shelf for cutlery 20 Decorations for cocktails 21.1 Ducts 22.2. Sticks 23 Mold 23.1 molds for ice 23.2 molds for children 23.3 Molds for shaping products 23.4 Molds for dumplings 24 Measuring container 24.1 Measuring container 24.2 A mixing container with the dispenser 24.3 Measuring container with the funnel 24.4 Beaker 24.5 Scoop 26 kitchen scissors 26.1 Scissors for BBQ 26.2 Kitchen scissors with bottle opener 26.3 Scissors for greens 26.4 Kitchen multipurpose scissors 26.5 Kitchen scissors for poultry 27 utensil for storage 27.1 container for storage 27.2 Bottles for liquid spices, oils 27.3 jars for storage 27.4 lunchbox 27.5 foldable lunchbox 27.6 jar for hermetic storage of bulk products 27.7 Sprayer for oil/vinegar 27.8 jar for bulk products 27.9 containers for spices 27.10 container for seasoning 27.11 Container for tea 28 potholders 28.1 oven-glove 28.2 silicone potholders 28.3 dishcloth railing with hooks 29 silicone mats 29.1 baking mat 29.2 mat for baking cakes 29.3 mat for drying of the glasses 29.4 cooking mat 29.5 Mat for drying of the dishes 30 graters, presses, rubbing machines 30.1 grater with a handle 30.2 grater 30.3 multifunction grater 30.4 grater shredder 30.5 grind for the green 30.6 grind for the garlic 30.7 Slicer for tomatoes 30.8 grater with rotating drums 30.9 universal device for grinding 30.10 mechanical grater 30.11 garlic peeling tube 30.12 rubbing machine 30.13 press for vegetables 30.14 press for garlic 30.2 press for hamburgers 31 knife sharpener 31.1 electric sharpener 31.2 sharpening stone 31.3 ceramic sharpener 32 breadbox 33 lattice with legs Kitchen dishes 1 for alcohol 1.2 Brandy set with dispenser 1.3 souvenir cups 1.4 stemware 1.5 pail of ice 1.6 stemware 1.7 champagne bucket 1.8 stemware 1.9 carafe 1.10 server 1.11 bottle holder 2 tableware 2.1 first course dish 2.2 dish for bouillon 2.3 bouillon bowl 2.4 oiler 2.5 round dish 2.6 duck pan 2.7 Set for making chocolate fondue 2.8 Set for making cheese fondue 2.9 salad bowl 2.10 dish for cake 2.11 compartmental dish 2.12 set of cutlery 2.13 serving spoon and fork 2.14 dish with lid 2.15 steam table 2.16 ice-cream bowl 2.17 Flatware 2.18 saucer 2.19 saucer for jam 2.20 mustard-pot 2.21 pepper-pot 2.22 ash-pot 2.23 deep table plate 2.24 dinner plate 2.25 snack plate 2.26 deep dessert plate 2.27 dessert plate 2.28 plate for pies 2.29 horseradish-pot 3 Utensils for table 3.1 Pad for tableware 3.2 serving mat 3.3. serving tray 3.4 glass burner 4 Dishes for tea, coffee, dessert 4.1 sugar-bowl 4.2 mug 4.3 mug with teapot 4.4 mug with stand 4.5 mug with lid 4.6 tea set 4.7 dish 4.8 french-press 4.9 teapot 4.10 teapot with strainer 4.11 glass teapot 4.12 ice-cream bowl 4.13 multifunctional vase 4.14 glasses 4.15 soup bowl 4.16 wicker basket 4.17 vase 3-tier 4.18 tea set 4.19 napkin rings 4.20 pannier for fruits 4.21 table trash basket 4.22 biscuit dish 4.23 candy dish 4.24 coffee sets 5 CUTLERY 5.1 Table fork 5.2 fork for sprat 5.3 fork for crayfish 5.4 fork for oysters 5.6 fork for lemons 5.7 big spoon 5.8 dessert spoon 5.9 tea spoon 5.10 coffee spoon 5.11 lemonade spoon 5.12 ladle-spoon 5.13 spoon for hot snacks 5.14 ice cream spoon 5.15 mustard spoon 5.16 salt spoon 5.17 spatula for cakes 5.18 spatula for caviar 5.19 spatula for fish 5.20 table knife 5.21 knife and fork for the fish 5.22 knife and fork snack 5.23 knife and fork dessert 5.24 Butterknife 5.25 tool kits for lobster, crayfish 5.26 devices for spices 5.27 grille and asparagus tongs 5.28 salad unit (salad fork and spoon) 5.29 sugar-tongs 5.30 tongs for cakes and sugar 5.31 ice tongs 5.32 can-opener 5.33 fork oyster 5.34 plug for crayfish 5.36 cocotte fork 5.37 fork for canned fish in oil (sprat, sardines) 5.38 spinner for champagne 5.39 spoon to mix whiskey with soda water Kitchen appliances 1 aerogrill 2 blenders, grinder 3 coffee Maker 4 coffee grinder (coffee mill) 5 Food Processor 6 mixer 7 mini oven 8 multicooker 9 meat grinders 10 steamers 11 Raclette grill 12 Juicers 13 toasters 14 egg cooker 15 electric range 15.1 electric induction stove 16 electric kettle 16.1 thermopots 17 bread makers 18 microwaves 19 weights for kitchen 20 electric driers 21 weights for kitchen Children's dishes 1 Children Sets for baking 2 Children cutlery 3 Children thermoses 4 Children Sets of dishes List of ingredient data 1 Ingredient name 2 Ingredient Photo 3 Manufacturer 4 Country 5 Type of Ingredient 6 Type of cuisine 7 Relation to Vegetarianism 8 Spice 9 Energy value 10 Description of the Ingredient 11 Status 12 Price List of equipment data 1 Equipment name 2 Equipment photo 3 Manufacturer 4 Brand name 5 Dimensions 6 Weight 7 Connectivity 8 Type of cuisine 9 Type of equipment 10 Description of equipment 11 Year 12 Status 13 Price List of recipe data 1 Name of the recipe 2 Recipe author 3 Recipe Photo 4 Preparation time 5 Basis of the dish 6 Type of cuisine 7 Type of the dish 8 Relation to Vegetarianism 9 Spice 10 Energy value 11 Number of persons 12 Description of the recipe 13 Description of the stages of cooking 14 Ingredients 15 Type of equipment 16 Video of recipe cooking 17 User Rating 18 Expert Rating 19 Amount of sales 21 Automatic cooking 22 Price
TABLE-US-00002 TABLE B Types of Ingredients 1 MEAT and MEAT PRODUCTS 1 Basturma 2 Fat 3 brisket cooked and smoked 4 Hare 5 leather duck 6 Sausage 7 Sausages 8 Sausages Hunting party 9 Horsemeat 10 Bones with bone marrow 11 Roe 12 Rabbit 13 Meat 14 Moosemeat 15 Venison 16 Liver 17 Kidney 18 Smoked ribs 19 Salami 20 Sausages 21 Cervelat 22 Sausages 23 Hungarian smoked bacon 24 bacon fat-tailed 25 Steak 26 ribeye steak 27 Farce 28 crocodile fillet 29 Jamon 30 Choriso (spanish sausage) 31 Skewers 32 Sowbelly 33 Deer tongue 34 Frog legs LAMB, VEAL 1 breast of lamb 2 loin of lamb 3 blade lamb 4 veal brains 5 mutton ham 6 veal ham 7 leg of lamb 8 Heel muscle mutton 9 lamb offal 10 veal kidneys 11 lamb chops 12 gras cow 13 veal heart 14 lamb testicles 15 veal fillet 16 veal cheeks 17 minced lamb 18 minced veal 19 veal tail 20 veal tongue 21 eggs bullish BEEF 1 beef brisket 2 beef fillet 3 beef (sirloin) 4 beef on the bone 5 beef eye muscle 6 legs of beef 7 ham beef 8 gras beef 9 beef ribs 10 beef heart 11 minced beef 12 tail beef 13 beef tongue PORK 1 bacon 2 smoked bacon 3 Pork 4 Ham 5 pork brisket 6 smoked pork belly 7 Pork Intestine 8 legs of pork 9 boar ham 10 pork ham 11 pork ribs 12 knuckle of pork 13 Fat 14 pork (pork neck or loin) 15 pork ears 16 minced pork 17 pig tail 18 pork tongue 2 BIRDS 1 garshnep 2 turkey breast 3 chicken breast 4 chicken breast, smoked 5 duck breast 6 Goose 7 chicken ventricles 8 turkey 9 turkey wings 10 chicken wings 11 chicken 12 smoked chicken 13 grouse 14 Coot 15 duck leg 16 crow's feet 17 chicken legs 18 chicken ham 19 Quail 20 gras chicken 21 chicken giblets 22 grouse 23 chicken hearts 24 Duck 25 smoked duck 26 Pheasant 27 minced chicken 28 chicken fillet 29 foie gras 30 chicken 31 chicken gutted 32 neck duck 3 FISH and SEAFOOD 1 anchovies 2 arctic char 3 mullet 4 Black Sea goby 5 shrimp head 6 Butterfish 7 scallops 8 dorado 9 Ruff 10 caviar 11 red caviar 12 Tobiko caviar 13 Squid 14 flounder 15 cuttlefish 16 Carp 17 Sprat 18 Smelt 19 crab sticks 20 Shrimps 21 King shrimps 22 Salad shrimps 23 Tiger prawn 24 Bream 25 salmon 26 Smoked salmon 27 Mussels 28 Mussels with shells 29 Pollock 30 Molluscs 31 Sea food 32 Sea fish 33 sole (fish) 34 Crab meat 35 Krill meat 36 Burbot 37 Perch 38 Lobster 39 Cisco 40 sturgeon 41 octopus 42 baby octopus 43 shrimp broth 44 halibut 45 Pangasius 46 cod liver oil 47 Haddock 48 Crayfish 49 dried crustaceans 50 Hot smoked fish 51 red fish salted 52 Swordfish 53 Saury 54 Sardines 55 Herring 56 Salmon 57 smoked salmon 58 salted salmon 59 Seabass 60 Whitefish 61 Ramp 62 Mackerel 63 smoked mackerel 64 Sheatfish 65 Starlet 66 Walleye 67 Dried seaweed 68 Tilapia 69 Carp 70 Cod 71 Hot smoked cod 72 black cod 73 Tuna 74 Turbot 75 Eel 76 smoked eel 77 Snails 78 Oysters 79 white fish fillets 80 catfish fillets 81 fillet of carp 82 fish fillet 83 salmon fillet 84 salted herring fillets 85 perch fillet 86 Trout 87 smoked trout 88 Squid Ink 89 cervical shrimp 90 cervical cancers 91 Sprats 92 Pike 4 VEGETABLES 2 Artichokes 3 Eggplant 4 Yam 5 broccoli tops 6 beet tops 7 Broccoli 8 Rutabaga 9 Galangal 10 Peas 11 pea sprouts 12 pea pods 13 green peas 14 Daikon 15 Melon 16 Ginseng 17 Ginger 18 Zucchini 19 Feces 20 Cabbage 21 Brussels sprouts 22 Sauerkraut 23 Chinese cabbage 24 Cabbage 25 Romanesco cabbage 26 savoy cabbage 27 Cauliflower 28 Potatoes 29 young potatoes 30 Kohlrabi 31 root anise 32 salsify root 33 parsley root 34 celery root 35 fresh corn 36 white onion 37 pearl bow 38 onion 39 red onion 40 dry onion 41 small onion 42 Shallots 43 cassava 44 mini corn 45 mini peppers 46 mini-tomatoes 47 carrots 48 cucumber 49 parsnips 50 squash 51 bell peppers 52 cayenne pepper 53 fresh chili pepper 54 jalapeno peppers 55 tomato 56 pickled tomatoes 57 cherry tomatoes 58 sunflower sprouts 59 wheat germ 60 soybean seedlings 61 germinated soybeans 62 rhubarb 63 Radish 64 wild radish 65 Turnip 66 beansprouts 67 Beet 68 Asparagus 69 chopped tomatoes 70 Sweet 71 Pumpkin 72 green beans 73 Fennel 74 physalis 75 horseradish 76 zucchini 77 Garlic 78 endive 5 FRUITS 1 Apricot 2 Avocado 3 quince 4 fresh pineapple 5 Orange 6 banana 7 Hawthorn 8 cranberries 9 grapes 10 Cherry 11 Dried cherries 12 blueberries 13 Garnet 14 Grapefruit 15 Pear 16 Blackberry 17 strawberries 18 pomegranate seeds 19 carambola 20 Kiwi 21 Strawberry 22 Cranberry 23 coconut 24 gooseberry 25 kumquat 26 Lime 27 lemon 28 Litchi 29 raspberries 30 mango 31 Mandarin 32 Passionfruit 33 mini pineapple 34 Nectarine 35 buckthorn 36 papaya 37 Peach 38 Pomelo 39 Rowan 40 Drain 41 red currants 42 black currant 43 tamarind 44 Feijoa 45 fruit to taste 46 persimmon 47 cherries 48 Cherry 49 blueberries 50 Apple 51 frozen berries 52 juniper berries 53 fresh berries 6 GROCERY 1 Agar 2 Adjika 3 rice paper 4 vanilla extract 5 vermicelli rice 6 egg noodles 7 Algae 8 Glucose 9 Jam 10 raspberry jam 11 fresh yeast 12 Gelatin 13 liquid Smokehouse 14 Sweetener 15 corn muffins 16 Ketchup 17 citric acid 18 Candy 19 Confiture 20 strawberry jam 21 food dye 22 Starch 23 potato starch 24 corn starch 25 bread crumbs 26 Noodles 27 buckwheat noodles 28 Pad Thai noodles 29 rice noodles 30 glass noodles 31 noodles harusame 32 egg noodles 33 Mayonnaise 34 poppy sweet 35 Pasta 36 cannelloni pasta 37 pasta lumakoni 38 pasta feathers 39 fusilli pasta 40 pumpkin marmalade 41 jujube fruit 42 Marzipan 43 Mirin 44 coconut milk 45 almond milk 46 soy milk 47 Muesli 48 Pasta 49 peanut paste 50 red curry paste 51 tamarind paste 52 Tom Yam Paste 53 chili paste 54 Molasses 55 Pectin 56 Penne 57 Jam 58 elderberry syrup 59 vanilla syrup 60 syrup vishnevny 61 ginger syrup 62 caramel syrup 63 maple syrup 64 strawberry syrup 65 coffee syrup 66 corn syrup 67 raspberry syrup 68 mango syrup 69 honey syrup 70 almond syrup 71 walnut syrup 72 blackcurrant syrup 73 chocolate syrup 74 cranberry sauce 75 worcestershire sauce 76 pomegranate sauce 77 kimchi sauce 78 Pesto 79 fish sauce 80 fish sauce nampla 81 Tabasco sauce 82 teriyaki sauce 83 sauce tkemali 84 oyster sauce 85 sweet chili sauce 86 Japanese walnut sauce 87 spaghetti 88 crumbs of white bread 89 breadcrumbs 90 pastry decorations 91 candied 7 MILK PRODUCTS and EGGS 1 yogurt 2 natural yoghurt 3 Kefir 4 margarine 5 butter 6 melted butter 7 Milk 8 baked milk 9 buttermilk 10 curdled 11 cream 12 sour cream 13 Whey 14 Thane 15 Curd 16 curd beaded 17 quail eggs 18 Egg 8 MUSHROOMS 2 mushrooms 3 Ceps 4 Enoki mushrooms 5 Chinese dried mushrooms 6 portobello mushrooms 7 dried mushrooms 8 shiitake mushrooms 9 milkmushrooms 10 chanterelles 11 boletus 12 honey fungus 13 saffron milk cap 14 morels 15 truffles 16 meadow mushrooms 9 CHEESE 1 cheese 2 cheese Adyghe 3 brie cheese 4 feta cheese 5 Burrata cheese 6 Gouda cheese 7 Dutch cheese 8 blue cheese 9 Gorgonzola 10 grana padano cheese 11 Gruyere cheese 12 Dor Blue cheese 13 Camembert 14 goat cheese 15 cheese sausage 16 mascarpone cheese 17 Monterey Jack cheese 18 mozzarella cheese 19 soft cheese 20 goat cheese 21 parmesan cheese 22 pecorino cheese 23 processed cheese 24 cheese Poshehonsky 25 ricotta cheese 26 Roquefort cheese 27 blue cheese 28 cream cheese 29 suluguni 30 cheese curd 31 feta cheese 32 philadelphia cheese 33 cheddar cheese 34 edam cheese 35 Emmentaler cheese 10 NUTS and DRIED FRUITS 1 peanuts 2 barberry 3 walnuts (peeled) 4 raisins 5 Figs 6 Chestnut 7 Dried cranberries 8 coconut 9 dried apricots 10 Filbert (hazelnut) 11 almonds 12 Nuts 13 pine nuts 14 cashew nuts 15 Dried peaches 16 sunflower seeds 17 pumpkin seeds 18 Dried Fruits 19 Dates 20 Pistachios 21 Hazelnuts 22 Prunes 11 BEVERAGES 1 Water 2 water orange 3 mineral water 4 water pink 5 GABA-tea 6 Hibiscus 7 Kvass 8 bread kvass 9 Coke 10 Kuding 11 Lemonade 12 Mate 13 Juice 14 carbonated drink 15 Bitter Brandy 16 Rooibos 17 pineapple juice 18 orange juice 19 birch juice 20 grape juice 21 cherry juice 22 pomegranate juice 23 strawberry juice 24 cranberry juice 25 gooseberry juice 26 lime juice 27 mango juice 28 tangerine juice 29 peach juice 30 currant juice 31 tomato juice 32 apple juice 33 Sprite 34 Tonic 35 tea white 36 tea yellow 37 green tea 38 red tea 39 Puer tea 40 Puer tea in Mandarin 41 oolong tea 42 black tea 43 Espresso 12 ALCOHOL 1 Balm 2 Bitter 3 Brandy 4 Bourbon 5 Vermouth 6 Wine 7 white wine 8 sparkling wine 9 red wine 10 dry red wine 11 wine sangria 12 Whiskey 13 Vodka 14 anise vodka 15 Grappa 16 Gin 17 Irish cream liqueur 18 Calvados 19 Cachaca 20 Brandy 21 Liqueur 22 orange liqueur 23 coffee liqueur 24 chocolate liqueur 25 Madeira 26 Marsala 27 Martini 28 Beer 29 cherry beer 30 Port 31 Rum 32 white rum 33 black rum 34 Sake 35 sambuca 36 Cider 37 tequila 38 sherry 39 Champagne (Brut) 40 schnapps 13 GREENS AND HERBS 1 Basil 2 basil red 3 bouquet garni 4 oregano 5 greens 6 dried herbs 7 cabbage pak choi 8 chervil 9 cilantro 10 oxalis 11 oat root 12 fresh coriander 13 nettle 14 Watercress 15 watercress 16 rose petals 17 lemongrass 18 bamboo leaves 19 banana leaves 20 grape leaves 21 Grape leaves (salty) 22 kaffir lime leaves 23 lime leaves 24 dandelion leaves 25 green onion 26 Leek 27 marjoram 28 Chard 29 melissa 30 lemon balm 31 Mint 32 oregano 33 parsley 34 dried parsley 35 plantain 36 wormwood 37 chopped camomile 38 arugula 39 iceberg lettuce 40 green salad 41 corn salad 42 lettuce 43 leaf lettuce 44 salad Mizuno 45 Oakleaf lettuce 46 radicchio salad 47 romaine lettuce 48 salad Friess 49 salad mix 50 celery 51 Lemon grass (lemon grass) 52 Italian herbs 53 spicy herbs 54 Dill 55 dandelion flowers 56 flowers 57 lavender flowers 58 chicory 59 thyme 60 Ramson 61 saffron 62 rosehips 63 chives 64 spinach 65 sorrel 66 tarragon 14 Cereals, legumes and flours 1 beans 2 mung beans 3 bulgur 4 puffed rice 5 buckwheat green 6 Quinoa 7 buckwheat 8 corn grits 9 semolina 10 Oats 11 pearl barley 12 cereal wheat 13 couscous 14 Flour 15 buckwheat flour 16 chestnut flour 17 corn flour 18 almond flour 19 Chickpea flour 20 oat flour 21 wheat flour 22 rye flour 23 rice flour 24 Chickpeas 25 Bran 26 Millet 27 Figure 28 Figure baya 29 basmati rice 30 brown rice 31 wild rice 32 Round grain rice 33 semola (flour made from durum wheat) 34 Beans 35 white beans 36 red beans 37 buckwheat flakes 38 cereal grains 39 oat flakes 40 Lentils 41 Barley 15 Spices and Seasonings 1 star anise 2 white pepper 3 Vanillin 4 Vanilla 5 vanilla essence 6 vanilla powder 7 Wasabi 8 Caltrop 9 garam masala 10 Carnation 11 cloves minced 12 Mustard 13 sweet mustard 14 allspice peas 15 grain mustard 16 Cumin 17 ground ginger 18 Capers 19 Cardamom 20 Curry 21 Coriander 22 ground coriander 23 Cinnamon 24 coffee essence 25 balsamic cream 26 Sesame 27 Turmeric 28 bay leaf 29 lemon pepper 30 poppy seed 31 Olives 32 olives dry 33 avocado oil 34 anchovy butter 35 peanut oil 36 mustard oil 37 oil for frying 38 scented oil 39 grapeseed oil 40 canola oil 41 corn oil 42 sesame oil 43 linseed oil 44 olive oil 45 Peanut butter 46 sunflower oil 47 lean oil 48 vegetable oil 49 oil, refined 50 oil seed-bearing 51 soybean oil 52 truffle oil 53 oil pumpkin 54 almonds hammers 55 miso paste 56 sea salt 57 Nutmeg 58 Olives 59 Ligurian olives 60 hot red pepper 61 hot peppers 62 Fenugreek 63 Paprika 64 lemongrass paste 65 peperoncini 66 pepper pink polka dots 67 Chili 68 Dried chili peppers 69 mustard powder 70 seasoning fish 71 baking powder 72 rosemary 73 pink ground pepper 74 Sugar 75 vanilla sugar 76 brown sugar 77 sugar muskovado 78 sugar cane 79 powdered sugar 80 nasturtium seeds 81 Nigella seeds 82 fennel seeds 83 spice mix taco 84 Soda 85 ginger juice squeezed 86 lemon juice 87 Salt 88 citrate 89 grape sauce 90 sauce narsharab 91 ponzu sauce 92 soy sauce 93 tomato sauce 94 chili sauce 95 Spices 96 sumac 97 thyme 98 cumin 99 Mediterranean herbs 100 French herbs 101 vinegar 102 balsamic vinegar 103 wine vinegar 104 white wine vinegar 105 red wine vinegar 106 cherry vinegar 107 raspberry vinegar 108 rice vinegar 109 apple cider vinegar 110 hops suneli 111 Savory 112 chutney 113 black pepper 114 black pepper peas 115 dry garlic 116 Sage 16 PREPARED PRODUCTS 1 canned pineapple 2 canned artichokes 3 Marinated artichokes 4 baguette 5 Loaf 6 Bars of chocolate 7 meringue 8 biscuit 9 beans, canned 10 Bun 11 buns for hamburgers 12 Broth 13 beef broth 14 chicken broth 15 fish broth 16 Jam 17 Apricot jam 18 lingonberry jam 19 cherry jam 20 black currant jam 21 raspberry jam 22 blueberry jam 23 Wafer 24 canned cherry 25 Glaze 26 Dijon mustard 27 croutons 28 marinated mushrooms 29 Demiglas apple 30 Yeast 31 Jelly 32 leaven 33 marshmallows 34 crushed tomatoes in juice 35 pickled ginger 36 Cocoa 37 marinated cactus 38 Pickled capers 39 sour cabbage 40 sea kale 41 Kimchi 42 wafer cakes 43 gherkins 44 natural coffee 45 instant coffee 46 Crackers 47 Chocolate Crumb 48 Croissant 49 bouillon cubes 50 canned corn 51 marinated corn 52 Pita 53 Lanspik 54 Ice 55 Letcho 56 lasagna sheets 57 canned salmon 58 pickled onions 59 canned mandarins 60 marshmallow 61 hazelnut oil 62 sweet curd 63 Yoghurt 64 Honey 65 honey in the comb 66 Mix ginger 67 condensed milk 68 condensed milk boiled 69 milk powder 70 pickled carrots 71 ice cream 72 vanilla ice cream 73 chocolate ice cream 74 salted cucumber 75 pickled cucumbers 76 pickled cucumbers 77 Pecans 78 beet broth 79 corn sticks 80 bread sticks 81 tomato paste 82 Pasta Chocolate 83 Pate 84 frozen dumplings 85 hot pepper pickled 86 canned peaches 87 Cookies 88 Biscuit 89 Cookies Savoiardi 90 chocolate cookies 91 Pita 92 Supplements 93 tomatoes in juice 94 canned tomatoes 95 Popcorn 96 Prosciutto 97 Gingerbread 98 mango puree 99 mashed potatoes 100 tomato puree 101 apple puree 102 pickle cucumber 103 Roll 104 Pickled beets 105 pork jerky 106 sugar syrup 107 whipped cream 108 cream of coconut 109 Malt 110 Sorbet 111 barbecue sauce 112 sauce bearnez 113 Bchamel 114 Worcestershire sauce 115 sauce Demiglas 116 sauce for soups Bright udon 117 sweet and sour sauce 118 Salsa 119 sweet sauce 120 chocolate sauce 121 berry sauce 122 asparagus, soya 123 caramel chips 124 crushed crackers 125 Tartlets 126 Tahini 127 pasta for lasagna 128 dough for ravioli 129 pizza dough 130 yeast dough 131 dough kataifi 132 shortbread dough 133 pastry dough 134 puff pastry 135 dough dry 136 filo pastry 137 dried tomatoes 138 Tortilla 139 Toast 140 Tofu 141 tuna fish oil 142 tuna canned in its own juice 143 Tahini 144 Rice Stuffing 145 Canned beans 146 white bread 147 toast bread 148 rye bread 149 sweet bread 150 black bread 151 rye bread 152 corn flakes 153 ciabatta 154 tea Away 155 potato chips 156 corn chips 157 Marinated mushrooms 158 chocolate corn balls 159 Chocolate 160 white chocolate 161 bitter chocolate 162 milk chocolate 163 dark chocolate
TABLE-US-00003 TABLE C Lists of Food Preparation Methods and Equipment, Cuisine and Bases A list of food preparation methods 1; 0; The fried 2; 0; The boiled 3; 0; The stewed 4; 0; The baked 5; 0; The cut A list of Equipment 1; 0; KITCHEN ACCESSORIES 2; 1; funnels 3; 2; stainless steel funnel 4; 2; plastic funnel 5; 2; silicone funnel 6; 2; convertible funnel 7; 1; colanders 8; 7; quadratic colanders 9; 7; oval ladle-vases 10; 7; colanders with folding handles 11; 7; flat colander 12; 7; plastic colanders 13; 7; small round colanders 14; 7; suspended colanders 15; 7; cover-colander 16; 7; stainless steel and aluminum colanders 17; 7; cone colanders 18; 1; Kitchen Appliances 19; 18; whisk 20; 18; scoop spatula 21; 20; cook spatula 22; 20; spatula with slots 23; 20; confectionery spatula 24; 18; spoons 25; 24serving spoon 26; 24; spoon-tongs 27; 24; spoon with slots 28; 24; spoon for rice 29; 24; ladle spoon 30; 24; ice cream spoon 31; 24; honey spoon 32; 24; spaghetti spoon 33; 24; serving spoon 34; 18; confectionery syringe for cookies and cream 35; 18; soup ladle 36; 18; potato masher 37; 18; skimmer 38; 18; Meat fork 39; 18; brush 40; 18; coffee filter 41; 18; whisk 42; 18; silicone brush 43; 18; silicone juicer 44; 18; earthen saucer 45; 18; tea filter 46; 18; pump dispenser for oil and vinegar 47; 18; clip for silicone spoon for the edge of the pan 48; 18; transformed spoons for salad 49; 18; device for cherry seeds removing 50; 18; sink mat 51; 18; ties for rolls 52; 18; dough mini-scraper 53; 18; grill tongs 54; 18; spaghetti tongs 55; 18; ice tongs 56; 18; sugar tongs 57; 18; package clip 58; 18; package clip 59; 18; citrus spray 60; 18; Dough press 61; 18; scoop for bulk 62; 18; salad serving tongs (tweezers) 63; 18; accessories for tubes 64; 18; pestle 65; 18; mortar 66; 18; roller for cutting of the rings 67; 18; opener for caps 68; 18; meat tenderizer; meat softener 69; 18; egg yolk separator 70; 18; apron 71; 18; tools for decoration 72; 18; jar for oil and vinegar 73; 18; mug for milk boiling 74; 18; napkins 75; 18; tablecloth 76; 18; marker for glasses 78; 18; basket 79; 18; meat tenderizer 80; 18; cocotte 81; 18; brush for washing of the vegetables 82; 18; lids for cups 83; 18; rope for baking 84; 18; jar for herbs storage 86; 18; scraper for glass ceramic plates 87; 18; Teapot for tea 88; 18; clothespin for notes on the fridge 89; 18; railing systems 90; 18; hanger for kitchen tools 91; 18; plunger with not adhering surface 92; 18; silicone plunger 93; 18; rolling pin with adjustable thickness 94; 18; vacuum bags with pump 95; 18; gas lighter 96; 18; bone forceps 97; 1; kitchen timers thermometers 98; 97; timer for meat roasting 99; 97; digital thermometer 100; 97; holder for thermometer 101; 97; meat thermometer 102; 97; digital timer 103; 97; electr. digital timer 104; 97; aramel thermometer 105; 1; Mills for spices 106; 105; mill for black pepper 107; 105; electric mill 108; 105; combined mill for pepper and salt (2 in 1) 109; 105; mill for spices 110; 105; mill for greens 111; 1; Measuring utensils 112; 111; Measuring container (plastic bottle) 113; 111; measuring jar 114; 111; measuring jug 115; 111; measuring bowl 116; 111; mechanical dispenser for ice cream 117; 1; mechanical mixers 118; 1; bowl 119; 118; metal bowl 120; 118; stainless steel bowl 121; 118; plastic bowl 122; 118; plastic bowl 123; 118; bowls for food 124; 1; sets 125; 124; wine set 126; 124; sets for spices 127; 124; cupcakes baking set 128; 124; accessory kit for baking 129; 124; set of bar tools 130; 124; set of kitchen tools 131; 124; Set for eggs and pancakes baking 132; 1; Slicing and cutting of products 133; 132; cutter 134; 132; holder for onions cutting 135; 132; cutting boards 136; 132; universal professional knives 137; 132; kitchen shears 138; 132; hatchet 139; 132; meat hatchet 140; 132; Hammer for meat with hatchet 141; 132; hoe 142; 132; Hammer for meat 143; 132; knives 144; 143; knife for greens 145; 143; knife for oranges 146; 143; knife for kiwi 147; 143; knife for pineapple 148; 143; Spiral knife for carrots 149; 143; multifunctional knife 150; 143; vegetable knife 151; 143; Pizza Cutter 152; 143; universal knife 153; 143; knife for slicing 154; 143; cook knife 155; 143; gastronomic knife 156; 143; opener 157; 143; Cheese knife 158; 143; boning knife 159; 143; lettuce knife 160; 143; knife for steaks 161; 143; butcher knife 162; 143; shredding knife 163; 143; bread knife 164; 143; fish knife 165; 143; knife for sandwiches 166; 143; Santoku knife 167; 143; knife for fruit coring 168; 143; Butter knife 169; 169; openers 170; 169; tin-opener 171; 169; corkscrew 172; 169; corkscrew on a stand 173; 169; lever corkscrew 174; 169; folding corkscrew 175; 169; opener for waiter 178; 494; stands for hot 179; 494; stand for kitchen utensils storing 180; 494; toothpick holder 181; 494; Bottle holder 182; 494; Holder for capsules 183; 494; stand for spoon 184; 494; stand for coffee capsules 185; 494; Coasters 186; 494; Napkin holder 187; 494; stand for eggs 188; 494; stand for openers 189; 494; stand for scoops 190; 494; stand for cooking and serving of eggs 191; 494; stand for ladle 192; 494; Holder for paper towels 193; 494; Transforming stand for kitchen appliances 194; 494; stand for mug 195; 494; stand mugs and saucers 196; 494; stand for kitchen knives 197; 494; stand for chicken 198; 494; napkin-stand 199; 494; heated stand 200; 494; stands for cake 201; 1; Appliances for peeling and cutting 202; 201; grater for vegetables 203; 305; grater 204; 201; garlic masher 205; 201; egg cutter 206; 201; Manual vegetable cutter 207; 201; Peeler for vegetables 208; 201; Nutcracker 209; 201; The device for separating the yolks from the whites 210; 201; grasping for carrots cleaning 211; 201; scraper fish scales 212; 201; cutter for fruits 213; 201; oller for holes 214; 201; tongs for fish bones 215; 201; spiral vegetable cutter 216; 1; Bottle Caps 217; 216; champagne cork (stopper) 218; 216; stoppers for wine 219; 216; The opener to remove the crown corks from bottles 220; 1; sieves 221; 220; sieve for tea 222; 220; sieve-tongs for tea 223; 220; Strainer for spices 224; 220; Strainer for tea 225; 220; Universal sieve 226; 220; flour sieve 228; 220; The Chinese sieve with a mesh insert 229; 220; sieve with support 230; 220; Mug-sieve for flour 231; 220; sieve on the handle 232; 1; Salt and pepper shakers 233; 282; container for seasoning 234; 232; salt cellar 235; 232; containers for oil and vinegar 236; 1; Dish dryers 237; 236; salad dryer 238; 236; dryer-placemat 239; 236; dryer for crockery and cutlery 240; 1; Cutlery Accessories 241; 240; cutlery tray 242; 240; cutlery holder 243; 240; cutlery container 244; 240; strainer for cutlery 245; 240; wall hanger for kitchen tools 246; 240; cutlery organizer 247; 240; mat for cutlery 248; 240; sliding tray for cutlery 249; 240; dryer for cutlery 250; 240; glass for cutlery 251; 240; napkin for the cutlery 252; 240; case for cutlery 253; 240; tray for cutlery 254; 240; mitten-potholder 255; 240; box for cutlery 256; 240; full-size rack (cassette) for cutlery 257; 240; Stand without containers for cutlery 258; 240; cassette for cutlery 259; 240; container for cutlery 260; 240; station for cutlery 261; 240; Shelf for cutlery 262; 1; Decorations for cocktails 263; 262; ducts 264; 262; sticks 266; 496; molds for ice 267; 496; molds for children 268; 496; Molds for shaping products 269; 496; Molds for dumplings 271; 497; Measuring container 272; 497; A mixing container with the dispenser 273; 497; Measuring container with the funnel 274; 497; beaker 275; 497; scoop 276; 1; kitchen scissors 277; 276; Scissors for BBQ 278; 276; Kitchen scissors with bottle opener 279; 276; Scissors for greens 280; 276; Kitchen multipurpose scissors 281; 276; Kitchen scissors for poultry 282; 1; utensil for storage 283; 282; container for storage 284; 282; Bottles for liquid spices oils 285; 282; jars for storage 286; 282; lunchbox 287; 282; foldable lunchbox 288; 282; jar for hermetic storage of bulk products' 289; 282; Sprayer for oil/vinegar 290; 282; jar for bulk products 291; 282; containers for spices 293; 282; Container for tea 294; 1; potholders 295; 294; oven-glove 296; 294; silicone potholders 297; 294; dishcloth 298; 1; railing with hooks 299; 1; silicone mats 300; 299; baking mat 301; 299; mat for baking cakes 302; 299; mat for drying of the glasses 303; 299; cooking mat 304; 299; Mat for drying of the dishes 305; 1; graters presses rubbing machines 306; 305; grater with a handle 308; 305; multifunction grater 309; 305; grater shredder 310; 305; grind for the green 311; 305; grind for the garlic 312; 305; Slicer for tomatoes 313; 305; grater with rotating drums 314; 305; universal device for grinding 315; 305; mechanical grater 316; 305; garlic peeling tube 317; 305; rubbing machine 318; 305; press for vegetables 319; 305; press for garlic 320; 305; press for hamburgers 321; 1; knife sharpener 322; 321; electric sharpener 323; 321; sharpening stone 324; 321; ceramic sharpener 325; 1; breadbox 326; 1; lattice with legs 327; 339; Flatware 328; 327; for alcohol 329; 540; Cognac set with the batcher 330; 540; Glasses souvenir 331; 540; Glasses 332; 540; Bucket for ice 333; 540; Shot glasses 334; 540; Bucket for champagne 335; 540; Wine glasses 336; 540; decanter 337; 540; tray 338; 540; Support under a bottle 339; 327; tableware 340; 339; first course dish 341; 339; dish for bouillon 342; 339; bouillon bowl 343; 339; oiler 344; 339; round dish 345; 339; duck pan 346; 339; Set for making chocolate fondue 347; 339; Set for making cheese fondue 348; 339; salad bowl 349; 339; dish for cake 350; 339; compartmental dish 351; 339; set of cutlery 352; 339; serving spoon and fork 353; 339; dish with lid 354; 339; steam table 355; 374; ice-cream bowl 357; 339; saucer 358; 339; saucer for jam 359; 339; mustard-pot 360; 339; pepper-pot 361; 339; ash-pot 362; 339; deep table plate 363; 339; dinner plate 364; 339; snack plate 365; 339; deep dessert plate 366; 339; dessert plate 367; 339; plate for pies 368; 339; horseradish-pot 369; 327; Utensils for table 370; 369; Pad for tableware 371; 369; serving mat 372; 369; serving tray 373; 369; glass burner 374; 327; Dishes for tea coffee desert 375; 374; sugar-bowl 376; 374; mug 377; 374; mug with teapot 378; 374; mug with stand 379; 374; mug with lid 380; 374; tea set 381; 374; dish 382; 374; french-press 383; 374; teapot 384; 374; teapot with strainer 385; 374; glass teapot 387; 374; multifunctional vase 388; 540; Glasses 389; 374; soup bowl 390; 374; wicker basket 391; 374; vase 3-tier 393; 374; napkin rings 394; 374; pannier for fruits 395; 374; table trash basket 396; 374; biscuit dish 397; 374; candy dish 398; 374; coffee sets 399; 327; CUTLERY 437; 0; Kitchen appliances 438; 437; aerogrill 439; 437; blenders grinder 440; 437; coffee Maker 441; 437; coffee grinder (coffee mill) 442; 437; Food Processor 443; 437; mixer 444; 437; mini oven 445; 437; multicooker 446; 437; meat grinders 447; 437; steamers 448; 437; Raclette grill 449; 437; Juicers 450; 437; toasters 451; 437; egg cooker 452; 437; electric range 453; 437; electric induction stove 454; 437; electric kettle 455; 437; thermopots 456; 437; bread makers 457; 437; microwaves 458; 437; weights for kitchen 459; 437; electric driers 461; 0; Children's dishes 462; 461; Children Sets for baking 463; 461; Children cutlery 464; 461; Children thermoses 465; 461; Children Sets of dishes 488; 437; deep fryer 491; 339; baking sheet 494; 1; stand and holders 495; 220; sieve to form the Bird's Nest 496; 1; mold 497; 1; Measuring container 498; 339; pan 499; 339; frying pan 500; 437; Cookware for induction cookers 501; 437; Juice cookers 502; 437; Milk cooker 503; 437; Covers/splash screens 504; 437; Microwave cookware 505; 437; Braziers roasters 506; 437; Turk 507; 437; Dumpling (manti) cookers 508; 437; Sets 509; 437; Samovars 510; 437; Kasans 511; 437; Electric stove 512; 437; Casseroles (pans) 513; 512; casseroles (pans) with non-stick coating 514; 512; aluminum casseroles (pans) 515; 512; Stainless steel casseroles (pans) 516; 512; Enameled casseroles (pans) 517; 512; Teflon coated casseroles (pans) 518; 512; Heat-proof glass casseroles (pans) 519; 512; Ladles 520; 512; Ceramic casseroles (pans) 521; 512; Set of casseroles (pans) 522; 512; Pressure cooker 523; 512; Pan-steamer 524; 512; casseroles for induction cookers 525; 512; Pan-fryer 526; 512; Cast iron casserole (pot) 527; 512; Titanium casserole 528; 437; Frying pans skillet 529; 528; Frying pan with ceramic coating 530; 528; Frying pans with non-stick coating 531; 528; Frying pan with removable handle 532; 528; Stewpots 533; 528; Frying pans for grill 534; 528; Wok 535; 528; Pancake pans 536; 528; Electric frying pans 537; 528; Cast iron skillet 538; 528; Multifunctional frying pan 539; 528; Titanium frying pan 540; 437; Drinkware 541; 540; Wine glasses 542; 540; Water glasses 543; 540; Beer glasses 544; 540; Kegs 545; 540; Carafes 546; 540; Decanters 547; 540; Jugs 548; 540; Shots 549; 540; Wine glasses for champagne 550; 540; Glasses for brandy/cognac 551; 540; Wine glasses for a cocktail/martini A list of Cuisine 1; 0; Abkhaz 2; 0; Australian 3; 0; Austrian 4; 0; Azerbaijan 5; 0; Albanian 6; 0; Algerian 7; 0; American 8; 0; English 9; 0; Arabic 10; 0; Argentine 11; 0; Armenian 12; 0; Bashkir 13; 0; Belarusian 14; 0; Belgian 15; 0; Bulgarian 16; 0; Bosnian 17; 0; Brazilian 18; 0; Hungarian 19; 0; Venezuelan 20; 0; Vietnamese 21; 0; Greek 22; 0; Georgian 23; 0; Danish 24; 0; Jewish 25; 0; Israeli 26; 0; Indian 27; 0; Indonesian 28; 0; Jordanian 29; 0; Iraqi 30; 0; Iranian 31; 0; Irish 32; 0; Icelandic 33; 0; Spanish 34; 0; Italian 35; 0; Cambodian 36; 0; Canadian 37; 0; Cypriot 38; 0; Chinese 39; 0; Colombian 40; 0; Korean 41; 0; Creole 42; 0; Costa Rica 43; 0; Latvian 44; 0; Lebanese 45; 0; Libyan 46; 0; Lithuanian 47; 0; Macedonian 48; 0; Malaysian 49; 0; Moroccan 50; 0; Mexican 51; 0; Moldavian 52; 0; Mongolian 53; 0; German 54; 0; Dutch 55; 0; New Zealand 56; 0; Norwegian 57; 0; Ossetian 58; 0; Pakistani 59; 0; Palestinian 60; 0; Panamanian 61; 0; Peruvian 62; 0; Polish 63; 0; Portuguese 64; 0; Romanian 65; 0; Russian 66; 0; Serbian 67; 0; Singaporean 68; 0; Syrian 69; 0; Slovak 70; 0; Slovenian 71; 0; Thai 72; 0; Tatar 73; 0; Tibetan 74; 0; Tunisian 75; 0; Turkish 76; 0; Turkmen 77; 0; Ukrainian 78; 0; Philippine 79; 0; Finnish 80; 0; French 81; 0; Croatian 82; 0; Montenegrin 83; 0; Czech 84; 0; Chilean 85; 0; Chuvash 86; 0; Chukotka 87; 0; Swedish 88; 0; Swiss 89; 0; Scottish 90; 0; Ecuadorian 91; 0; Estonian 92; 0; Japanese 93; 0; Raw food diet 94; 0; European 95; 0; International 96; 0; Multinational 97; 0; Lean 98; 0; Caucasian 99; 0; Children A list of bases: 1; 0; Meat and meat products 2; 1; Basturma 3; 1; Fat 4; 1; brisket cooked and smoked 5; 1; Hare 6; 1; leather duck 7; 1; Sausage 8; 1; Sausages 9; 1; Sausages Hunting party 10; 1; Horsemeat 11; 1; Bones with bone marrow 12; 1; Roe 13; 1; Rabbit 14; 1; Meat 15; 1; Moosemeat 16; 1; Venison 17; 1; Liver 18; 1; Kidney 19; 1; Smoked ribs 20; 1; Salami 21; 1; Sausages 22; 1; Cervelat 23; 1; Sausages 24; 1; Hungarian smoked bacon 25; l; bacon fat-tailed 26; 1; Steak 27; 1; ribeye steak 28; 1; Farce 29; 1; crocodile fillet 30; 1; Jamon 31; 1; Choriso (spanish sausage) 32; 1; Skewers 33; 1; Sowbelly 34; 1; Deer tongue 35; 1; LAMB 36; 35; breast of lamb 37; 35; loin of lamb 38; 35; blade lamb 39; 35; veal brains 40; 35; mutton ham 41; 35; veal ham 42; 35; leg of lamb 43; 35; Heel muscle mutton 44; 35; lamb offal 45; 35; veal kidneys 46; 35; lamb chops 47; 35; gras cow 48; 35; veal heart 49; 35; lamb testicles 50; 35; VEAL 51; 35; veal cheeks 52; 35; minced lamb 53; 35; minced veal 54; 35; veal tail 55; 35; veal tongue 56; 35; eggs bullish 57; 1; BEEF 58; 57; beef brisket 59; 57; 60; 57; beef (sirloin) 61; 57; beef on the bone 62; 57; beef eye muscle 63; 57; legs of beef 64; 57; ham beef 65; 57; gras beef 66; 57; beef ribs 67; 57; beef heart 68; 57; minced beef 69; 57; tail beef 70; 57; beef tongue 71; 1; PORK 72; 71; bacon 73; 71; smoked bacon 74; 71; pork 75; 71; ham 76; 71; pork brisket 77; 71; smoked pork belly 78; 71; Pork Intestine 79; 71; legs of pork 80; 71; boar ham 81; 71; pork ham 82; 71; pork ribs 83; 71; knuckle of pork 84; 71; fat 85; 71; pork (pork neck or loin) 86; 71; pork ears 87; 71; minced pork 88; 71; pig tail 89; 71; pork tongue 90; 0; Birds 91; 90; garshnep 92; 90; turkey breast 93; 90; chicken breast 94; 90; chicken breast smoked 95; 90; duck breast 96; 90; Goose 97; 90; chicken ventricles 98; 90; turkey 99; 90; turkey wings 100; 90; chicken wings 101; 90; chicken 102; 90; smoked chicken 103; 90; grouse 104; 90; coot 105; 90; duck leg 106; 90; crow's feet 107; 90; chicken legs 108; 90; chicken ham 109; 90; quail 110; 90; gras chicken 111; 90; chicken giblets 112; 90; grouse 113; 90; chicken hearts 114; 90; duck 115; 90; smoked duck 116; 90; Pheasant 117; 90; minced chicken 118; 90; chicken fillet 119; 90; foie gras 120; 90; chicken 121; 90; chicken gutted 122; 90; neck duck 123; 0; FISH and SEAFOOD 124; 123; anchovies 125; 123; arctic char 126; 123; mullet 127; 123; Black Sea goby 128; 123; shrimp head 129; 123; Butterfish 130; 123; scallops 131; 123; dorado 132; 123; ruff 133; 123; caviar 134; 123; red caviar 135; 123; Tobiko caviar 136; 123; squid 137; 123; flounder 138; 123; cuttlefish 139; 123; carp 140; 123; sprat 141; 123; smelt 142; 123; crab sticks 143; 123; Shrimps 144; 123; King shrimps 145; 123; Salad shrimps 146; 123; Tiger prawn 147; 123; Bream 148; 123; salmon 149; 123; Smoked salmon 150; 123; Mussels 151; 123; Mussels with shells' 152; 123; Pollock 153; 123; Molluscs 154; 123; Sea food 155; 123; Sea fish 156; 123; sole (fish) 157; 123; Crab meat 158; 123; Krill meat 159; 123; Burbot 160; 123; Frog legs 161; 123; Perch 162; 123; Lobster 163; 123; cisco 164; 123; sturgeon 165; 123; octopus 166; 123; baby octopus 167; 123; shrimp broth 168; 123; halibut 169; 123; Pangasius 170; 123; cod liver oil 171; 123; haddock 172; 123; crayfish 173; 123; dried crustaceans 174; 123; Hot smoked fish 175; 123; red fish salted 176; 123; swordfish 177; 123; saury 178; 123; sardines 179; 123; herring 180; 123; salmon 181; 123; smoked salmon 182; 123; salted salmon 183; 123; seabass 184; 123; whitefish 185; 123; ramp 186; 123; mackerel 187; 123; smoked mackerel 188; 123; sheatfish 189; 123; starlet 190; 123; walleye 191; 123; Dried seaweed 192; 123; tilapia 193; 123; carp 194; 123; cod 195; 123; Hot smoked cod 196; 123; black cod 197; 123; tuna 198; 123; turbot 199; 123; eel 200; 123; smoked eel 201; 123; snails 202; 123; oysters 203; 123; white fish fillets 204; 123; catfish fillets 205; 123; fillet of carp 206; 123; fish fillet 207; 123; salmon fillet 208; 123; salted herring fillets 209; 123; perch fillet 210; 123; trout 211; 123; smoked trout 212; 123; Squid Ink 213; 123; cervical shrimp 214; 123; cervical cancers 215; 123; sprats 216; 123; pike 217; 0; VEGETABLES 218; 217; watermelon 219; 217; Artichokes 220; 217; eggplant 221; 217; yam 222; 217; broccoli tops 223; 217; beet tops 224; 217; broccoli 225; 217; rutabaga 226; 217; galangal 227; 217; peas 228; 217; pea sprouts 229; 217; pea pods 230; 217; green peas 231; 217; daikon 232; 217; melon 233; 217; Ginseng 234; 217; Ginger 235; 217; zucchini 236; 217; feces 237; 217; cabbage 238; 217; Brussels sprouts 239; 217; sauerkraut 240; 217; Chinese cabbage 241; 217; Cabbage 242; 217; Romanesco cabbage 243; 217; savoy cabbage 244; 217; cauliflower 245; 217; potatoes 246; 217; young potatoes' 247; 217; kohlrabi 248; 217; root anise 249; 217; salsify root 250; 217; parsley root 251; 217; celery root 252; 217; fresh corn 253; 217; white onion 254; 217; pearl bow 255; 217; onion 256; 217; red onion 257; 217; dry onion 258; 217; small onion 259; 217; Shallots 260; 217; cassava 261; 217; mini corn 262; 217; mini peppers 263; 217; mini-tomatoes 264; 217; carrots 265; 217; cucumber 266; 217; parsnips 267; 217; squash 268; 217; bell peppers 269; 217; cayenne pepper 270; 217; fresh chili pepper 271; 217; jalapeno peppers 272; 217; tomato 273; 217; pickled tomatoes 274; 217; cherry tomatoes 275; 217; sunflower sprouts 276; 217; wheat germ 277; 217; soybean seedlings 278; 217; germinated soybeans 279; 217; rhubarb 280; 217; Radish 281; 217; wild radish 282; 217; Turnip 283; 217; beansprouts 284; 217; beet 285; 217; Asparagus 286; 217; chopped tomatoes 287; 217; Sweet 288; 217; Pumpkin 289; 217; green beans 290; 217; Fennel 291; 217; physalis 292; 217; horseradish 293; 217; zucchini 294; 217; garlic 295; 217; endive 296; 0; FRUITS 297; 296; Apricot 298; 296; Avocado 299; 296; quince 300; 296; fresh pineapple 301; 296; Orange 302; 296; banana 303; 296; Hawthorn 304; 296; cranberries 305; 296; grapes 306; 296; Cherry 307; 296; Dried cherries 308; 296; blueberries 309; 296; Garnet 310; 296; Grapefruit 311; 296; pear 312; 296; Blackberry 313; 296; strawberries 314; 296; pomegranate seeds 315; 296; carambola 316; 296; kiwi 317; 296; Strawberry 318; 296; Cranberry 319; 296; coconut 320; 296; gooseberry 321; 296; kumquat 322; 296; Lime 323; 296; lemon 324; 296; Litchi 325; 296; raspberries 326; 296; mango 327; 296; Mandarin 328; 296; Passionfruit 329; 296; mini pineapple 330; 296; Nectarine 331; 296; buckthorn 332; 296; papaya 333; 296; Peach 334; 296; Pomelo 335; 296; Rowan 336; 296; drain 337; 296; red currants 338; 296; black currant 339; 296; tamarind 340; 296; Feijoa 341; 296; fruit to taste 342; 296; persimmon 343; 296; cherries 344; 296; Cherry 345; 296; blueberries 346; 296; apple 347; 296; frozen berries 348; 296; juniper berries 349; 296; fresh berries 350; 0; GROCERY 351; 350; agar 352; 350; adjika 353; 350; rice paper 354; 350; vanilla extract 355; 350; vermicelli rice 356; 350; egg noodles 357; 350; algae 358; 350; glucose 359; 350; jam 360; 350; raspberry jam 361; 350; fresh yeast 362; 350; gelatin 363; 350; liquid Smokehouse 364; 350; sweetener 365; 350; corn muffins 366; 350; ketchup 367; 350; citric acid 368; 350; candy 369; 350; confiture 370; 350; strawberry jam 371; 350; food dye 372; 350; starch 373; 350; potato starch 374; 350; corn starch 375; 350; bread crumbs 376; 350; Noodles 377; 350; buckwheat noodles 378; 350; Pad Thai noodles 379; 350; rice noodles 380; 350; glass noodles 381; 350; noodles harusame 382; 350; egg noodles 383; 350; mayonnaise 384; 350; poppy sweet 385; 350; pasta 386; 350; cannelloni pasta 387; 350; pasta lumakoni 388; 350; pasta feathers 389; 350; fusilli pasta 390; 350; pumpkin marmalade 391; 350; jujube fruit 392; 350; marzipan 393; 350; mirin 394; 350; coconut milk 395; 350; almond milk 396; 350; soy milk 397; 350; muesli 398; 350; Pasta 399; 350; peanut paste 400; 350; red curry paste 401; 350; tamarind paste 402; 350; Tom Yam Paste 403; 350; chili paste 404; 350; molasses 405; 350; pectin 406; 350; Penne 407; 350; jam 408; 350; elderberry syrup 409; 350; vanilla syrup 410; 350; syrup vishnevny 411; 350; ginger syrup 412; 350; caramel syrup 413; 350; maple syrup 414; 350; strawberry syrup 415; 350; coffee syrup 416; 350; corn syrup 417; 350; raspberry syrup 418; 350; mango syrup 419; 350; honey syrup 420; 350; almond syrup 421; 350; walnut syrup 422; 350; blackcurrant syrup 423; 350; chocolate syrup 424; 350; cranberry sauce 425; 350; worcestershire sauce 426; 350; pomegranate sauce 427; 350; kimchi sauce 428; 350; pesto 429; 350; fish sauce 430; 350; fish sauce nam pla 431; 350; Tabasco sauce 432; 350; teriyaki sauce 433; 350; sauce tkemali 434; 350; oyster sauce 435; 350; sweet chili sauce 436; 350; Japanese walnut sauce 437; 350; spaghetti 438; 350; ; crumbs of white bread 439; 350; breadcrumbs 440; 350; pastry decorations 441; 350; candied 442; 0; MILK PRODUCTS and EGGS 443; 442; yogurt 444; 442; natural yoghurt 445; 442; kefir 446; 442; margarine 447; 442; butter 448; 442; melted butter 449; 442; milk 450; 442; baked milk 451; 442; buttermilk 452; 442; curdled 453; 442; cream 454; 442; sour cream 455; 442; whey 456; 442; Thane 457; 442; curd 458; 442; curd beaded 459; 442; quail eggs 460; 442; egg 461; 0; mushrooms 462; 461; oyster mushrooms 463; 461; 464; 461; ceps 465; 461; Enoki mushrooms 466; 461; Chinese dried mushrooms 467; 461; portobello mushrooms 468; 461; dried mushrooms 469; 461; shiitake mushrooms 470; 461; milkmushrooms 471; 461; chanterelles 472; 461; boletus 473; 461; honey fungus 474; 461; saffron milk cap 475; 461; morels 476; 461; truffles 477; 461; meadow mushrooms 478; 0; CHEESE 479; 478; cheese 480; 478; cheese Adyghe 481; 478; brie cheese 482; 478; feta cheese 483; 478; Burrata cheese 484; 478; Gouda cheese 485; 478; Dutch cheese 486; 478; blue cheese 487; 478; Gorgonzola 488; 478; ; grana padano cheese 489; 478; Gruyere cheese 490; 478; ; Dor Blue cheese 491; 478; Camembert 492; 478; goat cheese 493; 478; cheese sausage 494; 478; mascarpone cheese 495; 478; Monterey Jack cheese 496; 478; mozzarella cheese 497; 478; soft cheese 498; 478; goat cheese 499; 478; parmesan cheese 500; 478; pecorino cheese 501; 478; processed cheese 502; 478; cheese Poshehonsky 503; 478; ricotta cheese 504; 478; Roquefort cheese 505; 478; blue cheese 506; 478; cream cheese 507; 478; suluguni 508; 478; cheese curd 509; 478; feta cheese 510; 478; philadelphia cheese 511; 478; cheddar cheese 512; 478; edam cheese 513; 478; Emmentaler cheese 514; 0; NUTS and DRIED FRUITS 515; 514; peanuts 516; 514; barberry 517; 514; walnuts (peeled) 518; 514; raisins 519; 514; figs 520; 514; Chestnut 521; 514; Dried cranberries 522; 514; coconut 523; 514; dried apricots 524; 514; Filbert (hazelnut) 525; 514; almonds 526; 514; nuts 527; 514; pine nuts 528; 514; cashew nuts 529; 514; Dried peaches 530; 514; sunflower seeds 531; 514; pumpkin seeds 532; 514; Dried Fruits 533; 514; Dates 534; 514; pistachios 535; 514; hazelnuts 536; 514; prunes 537; 0; BEVERAGES 538; 537; water 539; 537; water orange 540; 537; mineral water 541; 537; water pink 542; 537; GABA-tea 543; 537; Hibiscus 544; 537; kvass 545; 537; bread kvass 546; 537; ; Coke 547; 537; Kuding 548; 537; lemonade 549; 537; mate 550; 537; juice 551; 537; carbonated drink 552; 537; Bitter Brandy 553; 537; Rooibos 554; 537; pineapple juice 555; 537; orange juice 556; 537; birch juice 557; 537; grape juice 558; 537; cherry juice 559; 537; pomegranate juice 560; 537; strawberry juice 561; 537; cranberry juice 562; 537; gooseberry juice 563; 537; lime juice 564; 537; mango juice 565; 537; tangerine juice 566; 537; peach juice 567; 537; currant juice 568; 537; tomato juice 569; 537; apple juice 570; 537; sprite 571; 537; tonic 572; 537; tea white 573; 537; tea yellow 574; 537; green tea 575; 537; red tea 576; 537; Puer tea 577; 537; Puer tea in Mandarin 578; 537; oolong tea 579; 537; black tea 580; 537; Espresso 581; 0; ALCOHOL 582; 581; Balm 583; 581; Bitter 584; 581; brandy 585; 581; bourbon 586; 581; vermouth 587; 581; wine 588; 581; white wine 589; 581; sparkling wine 590; 581; red wine 591; 581; dry red wine 592; 581; wine sangria 593; 581; whiskey 594; 581; Vodka 595; 581; anise vodka 596; 581; grappa 597; 581; gin 598; 581; ; Irish cream liqueur 599; 581; Calvados 600; 581; Cachaca 601; 581; brandy 602; 581; liqueur 603; 581; orange liqueur 604; 581; coffee liqueur 605; 581; chocolate liqueur 606; 581; Madeira 607; 581; Marsala 608; 581; Martini 609; 581; beer 610; 581; cherry beer 611; 581; port 612; 581; rum 613; 581; white rum 614; 581; black rum 615; 581; Sake 616; 581; sambuca 617; 581; cider 618; 581; tequila 619; 581; sherry 620; 581; ( ); Champagne (Brut) 621; 581; schnapps 622; 0; GREENS AND HERBS 623; 622; basil 624; 622; basil red 625; 622; bouquet garni 626; 622; oregano 627; 622; greens 628; 622; dried herbs 629; 622; ; cabbage pak choi 630; 622; chervil 631; 622; cilantro 632; 622; oxalis 633; 622; oat root 634; 622; fresh coriander 635; 622; nettle 636; 622; Watercress 637; 622; watercress 638; 622; rose petals 639; 622; lemongrass 640; 622; bamboo leaves 641; 622; banana leaves 642; 622; grape leaves 643; 622; Grape leaves (salty) 644; 622; kaffir lime leaves 645; 622; lime leaves 646; 622; dandelion leaves 647; 622; green onion 648; 622; ; Leek 649; 622; marjoram 650; 622; chard 651; 622; melissa 652; 622; lemon balm 653; 622; Mint 654; 622; oregano 655; 622; parsley 656; 622; dried parsley 657; 622; plantain 658; 622; wormwood 659; 622; chopped camomile 660; 622; arugula 661; 622; iceberg lettuce 662; 622; green salad 663; 622; corn salad 664; 622; lettuce 665; 622; leaf lettuce 666; 622; salad Mizuno 667; 622; : Oakleaf lettuce 668; 622; radicchio salad 669; 622; romaine lettuce 670; 622; salad Friess 671; 622; salad mix 672; 622; celery 673; 622; Lemon grass (lemon grass) 674; 622; Italian herbs 675; 622; spicy herbs 676; 622; dill 677; 622; dandelion flowers 678; 622; flowers 679; 622; lavender flowers 680; 622; chicory 681; 622; thyme 682; 622; Ramson 683; 622; saffron 684; 622; rosehips 685; 622; chives 686; 622; spinach 687; 622; sorrel 688; 622; tarragon 689; 0; Cereals legumes and flours 690; 689; beans 691; 689; mung beans 692; 689; bulgur 693; 689; puffed rice 694; 689; buckwheat green 695; 689; Quinoa 696; 689; buckwheat 697; 689; corn grits 698; 689; semolina 699; 689; oats 700; 689; pearl barley 701; 689; cereal wheat 702; 689; couscous 703; 689; flour 704; 689; buckwheat flour 705; 689; chestnut flour 706; 689; corn flour 707; 689; almond flour 708; 689; Chickpea flour 709; 689; oat flour 710; 689; wheat flour 711; 689; rye flour 712; 689; rice flour 713; 689; chickpeas 714; 689; bran 715; 689; millet 716; 689; Figure 717; 689; Figure baya 718; 689; basmati rice 719; 689; brown rice 720; 689; wild rice 721; 689; Round grain rice 722; 689; semola (flour made from durum wheat) 723; 689; Beans 724; 689; white beans 725; 689; red beans 726; 689; buckwheat flakes 727; 689; cereal grains 728; 689; oat flakes 729; 689; lentils 730; 689; barley 731; 0; Spices and Seasonings 732; 731; star anise 733; 731; white pepper 734; 731; vanillin 735; 731; vanilla 736; 731; vanilla essence 737; 731; vanilla powder 738; 731; wasabi 739; 731; caltrop 740; 731; garam masala 741; 731; Carnation 742; 731; cloves minced 743; 731; mustard 744; 731; sweet mustard 745; 731; allspice peas 746; 731; grain mustard 747; 731; Cumin 748; 731; ground ginger 749; 731; capers 750; 731; cardamom 751; 731; curry 752; 731; coriander 753; 731; ground coriander 754; 731; cinnamon 755; 731; coffee essence 756; 731; balsamic cream 757; 731; sesame 758; 731; turmeric 759; 731; bay leaf 760; 731; lemon pepper 761; 731; poppy seed 762; 731; olives 763; 731; olives dry 764; 731; avocado oil 765; 731; anchovy butter 766; 731; peanut oil 767; 731; mustard oil 768; 731; oil for frying 769; 731; scented oil 770; 731; grapeseed oil 771; 731; canola oil 772; 731; corn oil 773; 731; sesame oil 774; 731; linseed oil 775; 731; olive oil 776; 731; Peanut butter 777; 731; sunflower oil 778; 731; lean oil 779; 731; vegetable oil 780; 731; oil refined 781; 731; oil seed-bearing 782; 731; soybean oil 783; 731; truffle oil 784; 731; oil pumpkin 785; 731; almonds hammers 786; 731; miso paste 787; 731; sea ??salt 788; 731; nutmeg 789; 731; olives 790; 731; Ligurian olives 791; 731; hot red pepper 792; 731; hot peppers 793; 731; fenugreek 794; 731; paprika 795; 731; lemongrass paste 796; 731; peperoncini 797; 731; pepper pink polka dots 798; 731; chili 799; 731; Dried chili peppers 800; 731; mustard powder 801; 731; seasoning fish 802; 731; baking powder 803; 731; rosemary 804; 731; pink ground pepper 805; 731; sugar 806; 731; vanilla sugar 807; 731; brown sugar 808; 731; sugar muskovado 809; 731; sugar cane 810; 731; powdered sugar 811; 731; nasturtium seeds 812; 731; Nigella seeds 813; 731; fennel seeds 814; 731; ; spice mix taco 815; 731; Soda 816; 731; ginger juice squeezed 817; 731; lemon juice 818; 731; salt 819; 731; citrate 820; 731; grape sauce 821; 731; sauce narsharab 822; 731; ponzu sauce 823; 731; soy sauce 824; 731; tomato sauce 825; 731; chili sauce 826; 731; Spices 827; 731; sumac 828; 731; thyme 829; 731; cumin 830; 731; Mediterranean herbs 831; 731; French herbs 832; 731; vinegar 833; 731; balsamic vinegar 834; 731; wine vinegar 835; 731; white wine vinegar 836; 731; red wine vinegar 837; 731; cherry vinegar 838; 731; raspberry vinegar 839; 731; rice vinegar 840; 731; apple cider vinegar 841; 731; ; hops suneli 842; 731; Savory 843; 731; chutney 844; 731; black pepper 845; 731; black pepper peas 846; 731; dry garlic 847; 731; sage 848; 0; PREPARED PRODUCTS 849; 848; canned pineapple 850; 848; canned artichokes 851; 848; Marinated artichokes 852; 848; baguette 853; 848; loaf 854; 848; Bars of chocolate 855; 848; meringue 856; 848; biscuit 857; 848; beans canned 858; 848; bun 859; 848; buns for hamburgers 860; 848; broth 861; 848; beef broth 862; 848; chicken broth 863; 848; fish broth 864; 848; Jam 865; 848; Apricot jam 866; 848; lingonberry jam 867; 848; cherry jam 868; 848; black currant jam 869; 848; raspberry jam 870; 848; blueberry jam 871; 848; Wafer 872; 848; canned cherry 873; 848; Glaze 874; 848; Dijon mustard 875; 848; croutons 876; 848; marinated mushrooms 877; 848; Demiglas apple 878; 848; yeast 879; 848; Jelly 880; 848; leaven 881; 848; marshmallows 882; 848; crushed tomatoes in juice 883; 848; pickled ginger 884; 848; Cocoa 885; 848; marinated cactus 886; 848; Pickled capers 887; 848; sour cabbage 888; 848; sea ??kale 889; 848; Kimchi 890; 848; wafer cakes 891; 848; gherkins 892; 848; natural coffee 893; 848; instant coffee 894; 848; crackers 895; 848; Chocolate Crumb 896; 848; croissant 897; 848; bouillon cubes 898; 848; canned corn 899; 848; marinated corn 900; 848; pita 901; 848; lanspik 902; 848; ice 903; 848; letcho 904; 848; lasagna sheets 905; 848; canned salmon 906; 848; pickled onions 907; 848; canned mandarins 908; 848; marshmallow 909; 848; hazelnut oil 910; 848; sweet curd 911; 848; yoghurt 912; 848; honey 913; 848; honey in the comb 914; 848; Mix ginger 915; 848; condensed milk 916; 848; condensed milk boiled 917; 848; milk powder 918; 848; pickled carrots 919; 848; ice cream 920; 848; vanilla ice cream 921; 848; chocolate ice cream 922; 848; salted cucumber 923; 848; pickled cucumbers 924; 848; pickled cucumbers 925; 848; pecans 926; 848; beet broth 927; 848; corn sticks 928; 848; bread sticks 929; 848; tomato paste 930; 848; Pasta Chocolate 931; 848; pate 932; 848; frozen dumplings 933; 848; hot pepper pickled 934; 848; canned peaches 935; 848; Cookies 936; 848; Biscuit 937; 848; Cookies Savoiardi 938; 848; chocolate cookies 939; 848; pita 940; 848; supplements 941; 848; tomatoes in juice 942; 848; canned tomatoes 943; 848; popcorn 944; 848; prosciutto 945; 848; gingerbread 946; 848; mango puree 947; 848; mashed potatoes 948; 848; tomato puree 949; 848; apple puree 950; 848; pickle cucumber 951; 848; roll 952; 848; Pickled beets 953; 848; pork jerky 954; 848; sugar syrup 955; 848; whipped cream 956; 848; cream of coconut 957; 848; malt 958; 848; sorbet 959; 848; barbecue sauce 960; 848; sauce bearnez 961; 848; bechamel 962; 848; Worcestershire sauce 963; 848; sauce Demiglas 964; 848; 965; 848; sweet and sour sauce 966; 848; salsa 967; 848; sweet sauce 968; 848; chocolate sauce 969; 848; berry sauce 970; 848; asparagus soya 971; 848; caramel chips 972; 848; crushed crackers 973; 848; tartlets 974; 848; tahini 975; 848; pasta for lasagna 976; 848; dough for ravioli 977; 848; pizza dough 978; 848; yeast dough 979; 848; dough kataifi 980; 848; shortbread dough 981; 848; pastry dough 982; 848; puff pastry 983; 848; dough dry 984; 848; filo pastry 985; 848; dried tomatoes 986; 848; Tortilla 987; 848; toast 988; 848; tofu 989; 848; tuna fish oil 990; 848; tuna canned in its own juice 991; 848; Tahini 992; 848; Rice Stuffing 993; 848; Canned beans 994; 848; white bread 995; 848; toast bread 996; 848; rye bread 997; 848; sweet bread 998; 848; black bread 999; 848; rye bread 1000; 848; corn flakes 1001; 848; ciabatta 1002; 848; tea Away 1003; 848; potato chips 1004; 848; corn chips 1005; 848; Marinated mushrooms 1006; 848; chocolate corn balls 1007; 848; Chocolate 1008; 848; white chocolate 1009; 848; bitter chocolate 1010; 848; milk chocolate 1011; 848; dark chocolate 1012; 50; veal fillet 1013; 57; beef fillet 1014; 848; sauce for soups Bright udon 1015; 296; Lemon 1016; 217; Carrots 1017; 217; Tomato
TABLE-US-00004 TABLE D Types of Cuisine and Dishes Types of Cuisine 1 Abkhaz cuisine 2 Australian cuisine 3 Austrian cuisine 4 Azerbaijan cuisine 5 Albanian cuisine 6 Algerian cuisine 7 American cuisine 8 English cuisine 9 Arabic cuisine 10 Argentine cuisine 11 Armenian cuisine 12 Bashkir cuisine 13 Belarusian cuisine 14 Belgian cuisine 15 Bulgarian cuisine 16 Bosnian cuisine 17 Brazilian cuisine 18 Hungarian cuisine 19 Venezuelan cuisine 20 Vietnamese cuisine 21 Greek cuisine 22 Georgian cuisine 23 Danish cuisine 24 Jewish cuisine 25 Israeli cuisine 26 Indian cuisine 27 Indonesian cuisine 28 Jordanian cuisine 29 Iraqi cuisine 30 Iranian cuisine 31 Irish cuisine 32 Icelandic cuisine 33 Spanish cuisine 34 Italian cuisine 35 Cambodian cuisine 36 Canadian cuisine 37 Cypriot cuisine 38 Chinese cuisine 39 Colombian cuisine 40 Korean cuisine 41 Creole cuisine 42 Costa Rica cuisine 43 Latvian cuisine 44 Lebanese cuisine 45 Libyan cuisine 46 Lithuanian cuisine 47 Macedonian cuisine 48 Malaysian cuisine 49 Moroccan cuisine 50 Mexican cuisine 51 Moldavian cuisine 52 Mongolian cuisine 53 German cuisine 54 Dutch cuisine 55 Zealand cuisine 56 Norwegian cuisine 57 Ossetian cuisine 58 Pakistani cuisine 59 Palestinian cuisine 60 Panamanian cuisine 61 Peruvian cuisine 62 Polish cuisine 63 Portuguese cuisine 64 Romanian cuisine 65 Russian cuisine 66 Serbian cuisine 67 Singaporean cuisine 68 Syrian cuisine 69 Slovak cuisine 70 Slovenian cuisine 71 Thai cuisine 72 Tatar cuisine 73 Tibetan cuisine 74 Tunisian cuisine 75 Turkish cuisine 76 Turkmen cuisine 77 Ukrainian cuisine 78 Philippine cuisine 79 Finnish cuisine 80 French cuisine 81 Croatian cuisine 82 Montenegrin cuisine 83 Czech cuisine 84 Chilean cuisine 85 Chuvash cuisine 86 Chukotka cuisine 87 Swedish cuisine 88 Swiss cuisine 89 Scottish cuisine 90 Ecuadorian cuisine 91 Estonian cuisine 92 Japanese cuisine 93 Raw food diet 94 Estonian cuisine 95 Japanese cuisine 96 Raw food diet 1 Types of Dishes 1 Snacks 2 Salads 3 Entrees 4 Main Dishes 5 Desserts 6 Drinks 7 Sauces and marinades 8 Baking 9 Semimanufactures and preservatives
TABLE-US-00005 TABLE E List of Robotic Food Preparation System (One Embodiment) Sys Responsible Major Level of No Category Category System Party(s) Challenges Completion Notes 01 Hardware Robot Hands Productionization, Robustness, Cost, Weight 02 Hardware Robot Arms 03 Hardware Robot Armature Rails 04 Hardware Capture/ Dynamic 3D Vision Training System 05 Hardware Capture/ Data Input Training 06 Hardware Capture/ Editing System Training 07 Hardware Kitchen Cabinets Module 08 Hardware Kitchen Fixtures Module 09 Hardware Kitchen Lighting with ability to Module computer-operating mode 10 Hardware Kitchen Protection/Safety Module Screen with ability to computer-operating mode 11 Hardware Kitchen Appliances Module 12 Hardware Kitchen Automatic Storage Module device with ability to computer-operating mode 13 Hardware Kitchen Automatic modular Module dispenser for flowing, liquid ingredients and water with ability to computer-operating mode Hardware Kitchen Freshness ingredients Module analytical device Hardware Kitchen Built-in electronic scales Module (in the tabletop) with ability to computer- operating mode 14 Hardware Kitchen Cleaning Module 15 Hardware Kitchen Waste Disposal Module Hardware Kitchen Multi-functional Module professional steam- oven with ability to computer-operating mode Hardware Kitchen Multi-functional Module professional kitchen processor with ability to computer-operating mode Hardware Kitchen Top-loaded dishwasher Module with ability to computer-operating mode Hardware Kitchen Professional Stove with Module turning control regulators/buttons operated with ability to computer-operating mode Hardware Kitchen Standard dimension Module layout Hardware Kitchen Anti-wieting, smoke, Module steam ventilation system autonomous or be connected to the duct with ability to computer-operating mode Hardware Kitchen Kitchen sink with tap Module with ability to computer-operating mode 16 Hardware Control/ CPU Power Modules 17 Hardware Control/ I/O Touchscreen Power Modules 18 Hardware Control/ Power Supply Power Modules 19 Hardware Accessories Utensils 20 Hardware Accessories Food Containers/Cartridges 21 Software Robot OS Module Hardware Kitchen Professional Stove with Module turning control regulators/buttons operated with ability to computer-operating mode 21 Robot Module OS Software 22 Software Robot Apps Module 23 Software Robot hand firmware Module 24 Software Robot Arm firmware Module 25 Software Robot Rail Control Module 26 Software Capture/ OS Training 27 Software Capture/ apps Training 28 Software Capture/ Vision Training 29 Software Capture/ Data Input Training 30 Software Capture/ Editing System Training 31 Software Kitchen OS Module 32 Software Kitchen App Module 33 Software Kitchen Controller Module Protection/Safety 34 Software Kitchen Controller, Appliances Module 35 Software Kitchen Controller, Storage Module 36 Software Kitchen Controller, Cleaning Module 36 Software Kitchen Controller, Steam-oven Module 36 Software Kitchen Controller, Kitchen Module Processor 36 Software Kitchen Controller, Dishwasher Module 36 Software Kitchen Controller, Stove Module 36 Software Kitchen Controller, Ventilation Module system 36 Software Kitchen Controller, Lighting Module 37 Software Kitchen Controller, Waste Module 37 Software Kitchen Controller, Tap Module 37 Software Kitchen Controller, Dispensing Module device 37 Software Kitchen Controller, Scales Module 37 Software Kitchen Controller, Freshness Module Indicator 38 Software Control/ OS Power Modules 39 Software Control/ I/O Touchscreen Power Modules 40 Software Control/ Control Apps Power Modules 41 Other Food Food Recipe Development 42 Other Food Food Container Prep 43 Other Food Food Order/Delivery 44 Other Logistics Safety/Regulatory 45 Other Logistics Sales/Distribution 46 Other Logistics Installation/Maintenance 47 Other Logistics Packaging/Shipping Container 48 Other Logistics Return Management 49 Other Logistics Technical Training 50 Other Logistics Manuals 51 Other Logistics Warranty 52 Production Robot 53 Production Kitchen Module 54 Production Integration/ Shipping 55 Production 56 Production 57 Production 58 Production 59 Production 60 Production
[0487] The present invention has been described in particular detail with respect to possible embodiments. Those skilled in the art will appreciate that the invention may be practiced in other embodiments. The particular naming of the components, capitalization of terms, the attributes, data structures, or any other programming or structural aspect is not mandatory or significant, and the mechanisms that implement the invention or its features may have different names, formats, or protocols. The system may be implemented via a combination of hardware and software, as described, or entirely in hardware elements, or entirely in software elements. The particular division of functionality between the various systems components described herein is merely example and not mandatory; functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component
[0488] In various embodiments, the present invention can be implemented as a system or a method for performing the above-described techniques, either singly or in any combination. The combination of any specific features described herein is also provided, even if that combination is not explicitly described. In another embodiment, the present invention can be implemented as a computer program product comprising a computer-readable storage medium and computer program code, encoded on the medium, for causing a processor in a computing device or other electronic device to perform the above-described techniques.
[0489] As used herein, any reference to one embodiment or to an embodiment means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the invention. The appearances of the phrase in one embodiment in various places in the specification are not necessarily all referring to the same embodiment.
[0490] Some portions of the above are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is generally perceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared, transformed, and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times to refer to certain arrangements of steps requiring physical manipulations of physical quantities as modules or code devices, without loss of generality.
[0491] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that, throughout the description, discussions utilizing terms such as processing or computing or calculating or displaying or determining or the like refer to the action and processes of a computer system, or similar electronic computing module and/or device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission, or display devices.
[0492] Certain aspects of the present invention include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the present invention could be embodied in software, firmware, and/or hardware, and, when embodied in software, can be downloaded to reside on and be operated from different platforms used by a variety of operating systems.
[0493] The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers and/or other electronic devices referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
[0494] The algorithms and displays presented herein are not inherently related to any particular computer, virtualized system, or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description provided herein. In addition, the present invention is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any references above to specific languages are provided for disclosure of enablement and best mode of the present invention.
[0495] In various embodiments, the present invention can be implemented as software, hardware, and/or other elements for controlling a computer system, computing device, or other electronic device, or any combination or plurality thereof. Such an electronic device can include, for example, a processor, an input device (such as a keyboard, mouse, touchpad, trackpad, joystick, trackball, microphone, and/or any combination thereof), an output device (such as a screen, speaker, and/or the like), memory, long-term storage (such as magnetic storage, optical storage, and/or the like), and/or network connectivity, according to techniques that are well known in the art. Such an electronic device may be portable or non-portable. Examples of electronic devices that may be used for implementing the invention include a mobile phone, personal digital assistant, smartphone, kiosk, desktop computer, laptop computer, consumer electronic device, television, set-top box, or the like. An electronic device for implementing the present invention may use an operating system such as, for example, iOS available from Apple Inc. of Cupertino, Calif., Android available from Google Inc. of Mountain View, Calif., Microsoft Windows 7 available from Microsoft Corporation of Redmond, Wash., webOS available from Palm, Inc. of Sunnyvale, Calif., or any other operating system that is adapted for use on the device. In some embodiments, the electronic device for implementing the present invention includes functionality for communication over one or more networks, including for example a cellular telephone network, wireless network, and/or computer network such as the Internet.
[0496] Some embodiments may be described using the expression coupled and connected along with their derivatives. It should be understood that these terms are not intended as synonyms for each other. For example, some embodiments may be described using the term connected to indicate that two or more elements are in direct physical or electrical contact with each other. In another example, some embodiments may be described using the term coupled to indicate that two or more elements are in direct physical or electrical contact. The term coupled, however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments are not limited in this context.
[0497] As used herein, the terms comprises, comprising, includes, including, has, having or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, or refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
[0498] The terms a or an, as used herein, are defined as one or more than one. The term plurality, as used herein, is defined as two or more than two. The term another, as used herein, is defined as at least a second or more.
[0499] An ordinary artisan should require no additional explanation in developing the methods and systems described herein but may find some possibly helpful guidance in the preparation of these methods and systems by examining standardized reference works in the relevant art.
[0500] While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments may be devised which do not depart from the scope of the present invention as described herein. It should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. The terms used should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims but should be construed to include all methods and systems that operate under the claims set forth herein below. Accordingly, the invention is not limited by the disclosure, but instead its scope is to be determined entirely by the following claims.