A control and learning module for controlling a robotic arm includes at least one learning module including at least one neural network. The at least one neural network is configured to receive and be trained by both state measurements based on measurements of current state and observation measurements based on observation data during an initial learning phase. The at least one learning module is further configured to be re-tuned by updated observation data for improved performance during an operations and secondary learning phase when the robotic arm is in normal operation and after the initial learning phase.

Embodiments provide functionality to prevent collisions between robots and objects. An example embodiment detects a type and a location of an object based on a camera image of the object, where the image has a reference frame. Motion of the object is then predicted based on at least one of: the detected type of the object, the detected location of the object, and a model of object motion. To continue, a motion plan for the robot is generated that avoids having the robot collide with the object based on the predicted motion of the object and a transformation between the reference frame of the image and a reference frame of the robot. The robot can be controlled to move in accordance with the motion plan or a signal can be generated that controls the robot to operate in accordance with the motion plan.

A method and system of controlling an appliance includes: receiving, from a first home appliance, a request to start video image processing for detecting a motion gesture of a user; processing a sequence of image frames captured by a camera corresponding to the first home appliance to identify a first motion gesture; selecting a second home appliance as a target home appliance for the first motion gesture in accordance with one or more target selection criteria, including first target selection criteria based on a location of the user relative to the first home appliance and second target selection criteria based on a level of match between the first motion gesture and a first control gesture corresponding to the second home appliance; and generating a control command to control the second home appliance in accordance with the first control gesture corresponding to the second home appliance.

Current technologies allow a robot to acquire a tool only if the tool is in a set known location, such as in a rack. In an embodiment, a method and corresponding system, can determine the previously unknow pose of a tool freely placed in an environment. The method can then calculate a trajectory that allows for a robot to move from its current position to the tool and attach with the tool. In such a way, tools can be located and used by a robot when placed at any location in an environment.

In an embodiment, a method includes identifying a force and torque for a robot to accomplish a task and identifying context of a portion of a movement plan indicating motion of the robot to perform the task. Based on the identified force, torque, and context, a context specific torque is determined for at least one aspect of the robot while the robot executes the portion of the movement plan. In turn, a control signal is generated for the at least one aspect of the robot to operate in accordance with the determined context specific torque.

In an embodiment, a method and system use various sensors to determine a shape of a collection of materials (e.g., foodstuffs). A controller can determine a trajectory which achieves the desired end-state, possibly chosen from a set of feasible, collision-free trajectories to execute, and a robot executes that trajectory. The robot, executing that trajectory, scoops, grabs, or otherwise acquires the desired amount of material from the collection of materials at a desired location. The robot then deposits the collected material in the desired receptacle at a specific location and orientation.

Embodiments provide functionality to prevent collisions between robots and objects. An example embodiment detects a type and a location of an object based on a camera image of the object, where the image has a reference frame. Motion of the object is then predicted based on at least one of: the detected type of the object, the detected location of the object, and a model of object motion. To continue, a motion plan for the robot is generated that avoids having the robot collide with the object based on the predicted motion of the object and a transformation between the reference frame of the image and a reference frame of the robot. The robot can be controlled to move in accordance with the motion plan or a signal can be generated that controls the robot to operate in accordance with the motion plan.

Embodiments provide methods and systems to modify motion of a robot based on sound and context. An embodiment detects a sound in an environment and processes the sound. The processing includes comparing the detected sound to a library of sound characteristics associated with sound cues and/or extracting features or characteristics from the detected sound using a model. Motion of a robot is modified based on a context of the robot and at least one of: (i) the comparison, (ii) the features extracted from the detected sound, and (iii) the characteristics extracted from the detected sound.

In an embodiment, a method during execution of a motion plan by a robotic arm includes determining a voice command from speech of a user said during the execution of the motion plan, determining a modification of the motion plan based on the voice command from the speech of the user, and executing the modification of the motion plan by the robotic arm.

Robots, including robot arms, can interface with other modules to affect the world surrounding the robot. However, designing modules from scratch when human analogues exist is not efficient. In an embodiment, a mechanical tool, converted from human use, to be used by robots includes a monolithic adaptor having two interface components. The two interface components include a first interface component cabal be of mating with an actuated mechanism on the robot side, the second interface capable of clamping to an existing utensil. In such a way, utensils that are intended for humans can be adapted for robots and robotic arms.