A mobile phone device includes a housing having a substantially rectangular shape wherein its height dimension substantially corresponds to a distance between an ear and a mouth of a user and wherein its width dimension is less than its height dimension. A display unit is integrated within the front surface of the mobile phone device. The display unit substantially entirely covers the front surface of the mobile phone device. The mobile phone device does not include a physical key on the front surface.

A detection unit (30) detects an input start timing of a command by a gesture on an operation target with a temporal change. A command processing unit (31) performs processing of a command recognized from the gesture based on the state of the operation target at the input start timing detected by the detection unit (30).

In one embodiment, a system includes an automatic speech recognition (ASR) module, a natural-language understanding (NLU) module, a dialog manager, one or more agents, an arbitrator, a delivery system, one or more processors, and a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to receive a user input, process the user input using the ASR module, the NLU module, the dialog manager, one or more of the agents, the arbitrator, and the delivery system, and provide a response to the user input.

A portable terminal device in an information processing system and method includes a camera and a microphone. Data of obtained images and voice are transmitted to a server that identifies operations to be executed based on the received voice and image data. The server transmits an identification of one or more results of the plurality of operations to the portable terminal device. When the portable terminal device receives only one result from the server, an operation corresponding to the one result is executed, and when a plurality of results is received, the portable terminal device displays information corresponding to the plurality of results as candidates. Additional voice is captured for selecting one of the plurality of results during the displaying of the information. A determination of one result from the plurality of results is made based on the captured voice, and an operation corresponding to the determined result is executed.

Invoking one or more previously dormant functions of an automated assistant in response to detecting, based on processing of vision data from one or more vision components: (1) a particular gesture (e.g., of one or more “invocation gestures”) of a user; and/or (2) detecting that a gaze of the user is directed at an assistant device that provides an automated assistant interface (graphical and/or audible) of the automated assistant. For example, the previously dormant function(s) can be invoked in response to detecting the particular gesture, detecting that the gaze of the user is directed at an assistant device for at least a threshold amount of time, and optionally that the particular gesture and the directed gaze of the user co-occur or occur within a threshold temporal proximity of one another.

The embodiments of the present disclosure disclose an augmented reality system that supports customized multi-channel interaction. One embodiment of the augmented reality system comprises: a head-mounted sensor assembly, a computing device, and a display module; the head-mounted sensor assembly is used to capture the user's multi-channel interactive input information and transmit the interactive input information to the computing device; the computing device is used to generate or modify the display content of the augmented reality according to the interactive input information; the display module is used to overlay display the background content with the display content of the augmented reality. The augmented reality system, by arranging the display module to the far end of the head-mounted sensor assembly, can simplify the structure of the head-mounted sensor assembly, and reduce the weight of the head-mounted sensor assembly, providing convenience for installing other sensors.

Technologies for monitoring a health-risk condition of a user include a virtual reality compute device having one or more near infrared (NIR) sensors. The virtual reality compute device presents a virtual reality (VR) presentation to the user. The virtual reality compute device produces sensor data through the one or more NIR sensors that is indicative of a heart rate of the user and a blood pressure of the user while the VR presentation is presented to the user. The virtual reality compute device determines whether the user is in a health-risk condition based on a comparison of the heart rate of the user to a heart rate safety threshold and a comparison of the blood pressure of the user to a blood pressure safety threshold. The virtual reality compute device performs a health-risk condition response in response to a determination that the user is in the health-risk condition.

A framework for generating and presenting verbal command suggestions to facilitate discoverability of commands capable of being understood and support users exploring available commands. A target associated with a direct-manipulation input is received from a user via a multimodal user interface. A set of operations relevant to the target is selected and verbal command suggestions relevant to the selected set of operations and the determined target are generated. At least a portion of the generated verbal command suggestions is provided for presentation in association with the multimodal user interface in one of three interface variants: one that presents command suggestions as a list, one that presents command suggestions using contextual overlay windows, and one that presents command suggestions embedded within the interface. Each of the proposed interface variants facilitates user awareness of verbal commands that are capable of being executed and teaches users how available verbal commands can be invoked.

Systems, methods, and apparatus for presenting information and performing a task using an electronic device. In some aspects, a device shows content items when a gaze or a shaking act plus a gaze are detected. In some aspects, a device performs a task when a name, a code, and the task are detected in voice input. In some aspects, a user communicates with a selected vehicle via a user device.

A focus group system for determining the user’s mood or reaction as the user views content based at least in part on physiological indicators measured by a physiological monitoring system and feedback from the user. The system may receive sensor data including physiological data of a user captured while the user is consuming content and receive feedback of the user associated with a reaction of the user when the sensor data was captured. The system may then determine, based at least in part on the sensor data and the feedback, a direction and magnitude of the reaction of the user to the content.