Control and/or registration of smart devices, locally by an assistant client device

Abstract

Various implementations relate to generating, locally at an assistant client device, specific control commands that, when transmitted to a corresponding smart device, are directly interpretable by the corresponding smart device to effectuate a state change at the corresponding smart device, or at a corresponding additional smart device directly controlled by the corresponding smart device. Various implementations additionally or alternatively relate to utilizing local assistant client devices in discovering, provisioning, and/or registering smart devices for an account of a user.

Claims

1. A method implemented by one or more processors of a client device executing an automated assistant client, the method comprising: identifying a generic smart device control command that specifies at least one smart device of a third-party (3P) and at least one state to be altered at the smart device, the generic smart device control command generated responsive to user interface input received at the client device; determining a reliable communications channel is not established between the client device and the smart device; responsive to determining the reliable communications channel is not established between the client device and the smart device: accessing a client devices to smart devices mapping, stored locally at the client device, to resolve an additional client device with an established reliable communications channel to the smart device; and transmitting, over a local network, the generic smart device control command to the additional client device to cause the additional client device to: select, from a plurality of 3P adapters, a particular 3P adapter for the generic smart device control command, wherein the particular 3P adapter is tailored by the 3P and integrates custom logic of the 3P, and wherein in selecting the particular 3P adapter the additional client device selects the particular 3P adapter responsive to the particular 3P adapter being assigned to the 3P and the generic smart device control command specifying the smart device of the 3P, and locally process the generic smart device control command, using the selected particular 3P adapter, to generate a specific command, that corresponds to the generic smart device control command, and that is directly interpretable by the smart device to effectuate alteration of the state at the smart device, and transmit the specific command over the established reliable communications channel to effectuate the alteration of the state at the smart device; wherein, prior to identifying the generic smart device control command, the additional client device preemptively receives and locally stores the particular 3P adapter responsive to the reliable communications channel being established with the smart device and the smart device being of the 3P, and wherein, prior to identifying the generic smart device control command, the additional client device preemptively executes, locally at the additional client device, the selected particular 3P adapter based on occurrence of a temporal period and one or more past occurrences of utilization of the particular 3P adapter during the temporal period.

2. The method of claim 1, further comprising: identifying an additional generic smart device control command that specifies at least one additional smart device and at least one additional state to be altered at the additional smart device, the additional generic smart device control command generated responsive to additional user interface input received at the client device; determining an additional reliable communications channel is established between the client device and the additional smart device; responsive to determining the additional reliable communications channel is established between the client device and the additional smart device: selecting an additional particular 3P adapter from a plurality of 3P adapters locally executable at the client device, the selecting based on the additional generic smart device control command; processing the additional generic smart device control command using the selected additional 3P adapter to generate an additional specific command, that corresponds to the additional generic smart device control command, and that is directly interpretable by the additional smart device to effectuate alteration of the additional state at the additional smart device; and transmitting, over the additional reliable communications channel established between the client device and the additional smart device, the additional specific command to effectuate the alteration of the additional state at the additional smart device.

3. The method of claim 2, further comprising, prior to the identifying the additional generic smart device control command: preemptively executing, locally at the client device, the additional particular 3P adapters to reduce latency in processing the additional generic smart device control command using the selected additional particular 3P adapter to generate the additional specific command.

4. The method of claim 1, wherein accessing the client devices to smart devices mapping, stored locally at the client device, to resolve the additional client device with the established reliable communications channel to the smart device comprises: determining, based on the client devices to smart devices mapping, that a given signal strength, between the additional client device and the smart device, is greatest amongst all client devices of the client devices to smart devices mapping.

5. The method of claim 1, wherein the established reliable communications channel to the smart device is a BLUETOOTH radio channel.

6. The method of claim 1, wherein the generic smart device control command is generated locally at the client device by locally processing user interface input received at the client device.

7. The method of claim 6, wherein the user interface input is interaction, at a touch-screen of the client device, with a rendered graphical user interface element corresponding to the smart device.

8. The method of claim 1, wherein the user interface input is a voice input received via at least one microphone of the client device, and wherein locally processing the user interface input comprises using a speech-to-text processor to convert the voice input into text, and performing natural language understanding on the text to generate the generic smart device control command.

9. The method of claim 1, further comprising: receiving voice input via at least one microphone of the client device; and streaming the voice input to a remote automated assistant system; wherein identifying the generic smart device control command comprises receiving the generic smart device control command from the remote automated assistant system responsive to streaming the voice input to the remote automated assistant system.

10. The method of claim 1, further comprising: receiving, at the additional client device, an additional generic smart device control command that specifies an additional smart device, wherein the additional client device also has the established reliable communications channel to the additional smart device; selecting, by the additional client device, from the plurality of 3P adapters assigned to the reliable communications channel, an alternate 3P adapter for the additional generic smart device control command; locally processing the additional generic smart device control command, using the selected alternate 3P adapter, to generate an additional specific command, that corresponds to the additional generic smart device control command, and that is directly interpretable by the additional smart device; and transmitting the additional specific command over the established reliable communications channel to the additional smart device.

11. The method of claim 1, wherein in selecting, from the plurality of 3P adapters, the particular 3P adapter, the additional client device selects the particular 3P adapter based on the generic smart device control command indicating a particular 3P and the particular 3P adapter being assigned to the particular 3P.

12. The method of claim 1, wherein the particular 3P adapter includes one or more layers that do not conform with any industry standard protocol suite.

13. The method of claim 1, wherein the smart device firmware is created by a particular 3P and wherein the particular 3P adapter is proprietary to the particular 3P.

14. The method of claim 1, wherein determining whether the reliable communications channel is established between the client device and the smart device comprises one or both of: determining whether any communications channel is established between the client device and the smart device; or determining whether a communications channel, established between the client device and the smart device, has at least a threshold signal strength.

15. The method of claim 14, wherein determining whether any communications channel is established and determining whether the communications channel has at least a threshold signal strength are each based on the client devices to smart devices mapping.

16. The method of claim 14, wherein determining whether any communications channel is established and whether the communications channel has at least a threshold signal strength are each based on one or more signals received via the communications channel subsequent to a most recent updating of the client devices to smart devices mapping.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

(2) FIG. 2 illustrates an example environment that includes multiple smart devices, from a plurality of disparate parties, and that includes multiple assistant devices, each of which can locally control one or more of the smart devices according to implementations disclosed herein.

(3) FIG. 3 illustrates a state diagram of an example of controlling two smart light bulbs of FIG. 2, using one of the client devices of FIG. 2.

(4) FIG. 4 illustrates a state diagram of an example of routing a generic smart device control command, for controlling another light bulb of FIG. 2, from a first client device of FIG. 2 to an alternate client device of FIG. 2.

(5) FIG. 5 is a flow chart illustrating an example method of controlling a smart device, using a local assistant client device, according to various implementations disclosed herein.

(6) FIG. 6 is a flow chart illustrating another example method of controlling a smart device, using a local assistant client device, according to various implementations disclosed herein.

(7) FIG. 7 is a flow chart illustrating an example method of registering a smart device, using a local assistant client device, according to various implementations disclosed herein.

(8) FIG. 8 illustrates an example architecture of a computing device.

DETAILED DESCRIPTION

(9) There is a proliferation of smart network connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices) such as smart home alarms, smart door locks, smart cameras, smart lights, smart thermostats, smart weight scales, smart beds, smart irrigation systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart fire alarms, smart moisture detectors, etc. Often, multiple smart devices are located within the confines of a structure, such as a home—or located within multiple related structures, such as a user's primary residence and the user's secondary residence and/or work location.

(10) Further, there is a proliferation of assistant client devices that each include an assistant client that can optionally interact with one or more remote automated assistant components to form a logical instance of an automated assistant. An assistant client device can be devoted solely to assistant functionality (e.g., a standalone speaker and/or standalone audio/visual device including only an assistant client and associated interface, and devoted solely to assistant functionality) or can perform assistant functionality in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). Moreover, some smart devices can also be assistant client devices. For example, some smart devices can include an assistant client and at least speaker(s) and/or microphone(s) that serve (at least in part) as user interface output and/or input devices for an assistant interface of the assistant client.

(11) Various techniques have been proposed for associating smart devices with corresponding logical instances of automated assistants (and optionally with individual assistant client devices). For example, a user, group of users, an assistant client device, and/or a group of assistant client devices (e.g., all within a structure) can be linked (e.g., in one or more databases) with a plurality of disparate smart devices to enable interaction with (e.g., control of) the smart devices via automated assistants. For instance, each of multiple assistant client devices in a household can be linked to each of multiple disparate smart devices in the household to enable any user (or a restricted group of users) to interface with any one of the assistant client devices to interact with any one of the multiple disparate smart devices.

(12) One example of such linking is a device topology representation (also referred to herein as a “home graph”) that can be user created, and/or automatically created, and that may define various assistant client devices, various smart devices, identifier(s) for each, and/or attribute(s) for each. For example, the identifier(s) for a device can specify a room (and/or other area(s)) of a structure in which the device is located (e.g., living room, kitchen) and/or can specify nickname(s) and/or alias(es) for the device (e.g. couch lamp, front door lock, bedroom speaker, kitchen assistant, etc.). In this manner, the identifiers of devices can be names, aliases, and/or locations of the respective devices that the user is likely to associate with the respective devices.

(13) The device topology representation can further specify one or more device attributes associated with the respective devices. The device attributes for an assistant client device can be, for example, associated with one or more input and/or output modalities supported by the assistant client device. For instance, a device attribute for a standalone speaker-only assistant client device can indicate that it is capable of providing audible output, but incapable of providing visual output. The device attributes of a smart device can, for example, identify one or more states, of the smart device, that can be controlled; identify a party (e.g., a 3P) that manufactures, distributes, and/or creates the firmware for the smart device; and/or identify a unique identifier for the smart device, such as a 1P or 3P provided fixed identifier. According to various implementations disclosed herein, the device topology representation can optionally further specify: which smart devices can be controlled locally by which assistant client devices; local addresses for locally controllable smart devices (or local addresses for hubs that can directly locally control those smart devices); local signal strengths and/or other preference indicators between assistant client devices and smart devices. Further, according to various implementations disclosed herein, the device topology representation (or a variation thereof) can be locally stored at each of a plurality of assistant client devices for utilization in locally controlling and/or locally registering smart devices.

(14) Now turning to FIG. 1, an example environment in which techniques disclosed herein may be implemented is illustrated. The example environment includes a plurality of client computing devices 110.sub.1-N (also referred to herein simply as “client devices”), cloud-based automated assistant components 120, smart device systems 140.sub.A-N, smart devices 145.sub.A-N, 3P adapters database 150, and a home graph 152 for a user of the client devices 110.sub.1-N. The client devices 110.sub.1-N and smart devices 145.sub.1-N of FIG. 1 represent client devices and smart devices that are at least selectively associated with one another (e.g., via the home graph 152 and/or other device topology). For example, the smart devices 145.sub.1-N can all be at a home (e.g., in the interior and/or exterior of the home), the client devices 110.sub.1-N can be at least occasionally in the same home, and the smart devices 145.sub.1-N and the client devices 110.sub.1-N can be linked to one another utilizing one or more techniques, such as those described herein. Through such linking, the client devices 110.sub.1-N can be utilized to locally control the smart devices 145.sub.1-N according to implementations described herein.

(15) One or more (e.g., all) of the client devices 110.sub.1-N can execute a respective instance of an automated assistant client 117.sub.1-N. However, in some implementations one or more of the client devices 110.sub.1-N can optionally lack an instance of an automated assistant client 117.sub.1-N, and still include engine(s) and hardware components for locally controlling and/or registering one or more smart devices (e.g., an adapter interaction engine, adapter(s), radio(s), routing engine, and/or a registration engine). An instance of the automated assistant client 117.sub.1-N can be an application that is separate from an operating system of the corresponding client device 110.sub.1-N (e.g., installed “on top” of the operating system)—or can alternatively be implemented directly by the operating system of the corresponding client device 110.sub.1-N. As described further below, each instance of the automated assistant client 117.sub.1-N can optionally interact with cloud-based automated assistant component(s) 120 in responding to various requests provided by a user via I/O components 111 of any one of the client devices 110.sub.1-N. Further, and as also described below, other engine(s) of the client devices 110.sub.1-N can optionally interact with cloud-based automated assistant component(s) 120.

(16) One or more cloud-based automated assistant components 120 can be implemented on one or more computing systems (collectively referred to as a “cloud” or a “remote” computing system) that are communicatively coupled to client devices 110.sub.1-N via one or more wide area networks (e.g., the Internet), indicated generally by 105.sub.1 of FIG. 1. For example, cloud-based automated assistant component(s) 120 can be implemented by one or more clusters of high-performance servers. It is noted that the client devices 110.sub.1-N can utilize one or more local area networks in accessing the wide-area networks 105.sub.1 and/or in locally communicating with one another. Such local area networks can include a Wi-Fi network and/or a mesh network between the client devices 110.sub.1-N.

(17) The cloud-based automated assistant components 120 can also be communicatively coupled with smart device systems 140.sub.A-N via one or more wide area networks. The communicative coupling of the cloud-based automated assistant components 120 with the smart device systems 140 is indicated generally by 105.sub.2 of FIG. 1. Further, the smart device systems 140 can each be communicatively coupled to a corresponding group of one or more smart devices 145.sub.A-N via one or more wide area networks, indicated generally by 105.sub.3 of FIG. 1. It is noted that the smart devices 145.sub.A-N can utilize one or more local area networks in accessing the wide-area networks 105.sub.3.

(18) Each of the smart device systems 140.sub.A-N can be either a 1P or a 3P system, and each can be communicatively coupled with one or more corresponding smart devices 145.sub.A-N. For example, a first smart device system 140.sub.A-N can be controlled by a first 3P and communicatively coupled with a first smart device 145.sub.A1, a second smart device system 140 can be controlled by a second 3P and communicatively coupled with a second smart device 145.sub.B1 and a third smart device 145.sub.B2, etc.

(19) The smart device systems 140 can communicate with the devices 145.sub.A-N via the wide-area networks 105.sub.3 to control their respective smart devices 145.sub.A-N, to deliver firmware updates to their respective smart devices 145.sub.A-N, to ascertain the status of their respective smart devices 145.sub.A-N, etc. For example, a given one of the smart device systems 140 can communicate with one of the smart devices 145.sub.A-N to control the smart device in response to user input being received via a mobile application, for the smart device system, that enables control of the smart device.

(20) Also, for example, a given one of the smart device systems 140 can communicate with one of the smart devices 145.sub.A-N to control the smart device in response to a request from cloud-based automated assistant component(s) 120. For example, according to some techniques a user can provide, via one or more I/O components 111.sub.1 of a client device 110.sub.1, a request to control a smart device, such as spoken input of “turn off the couch light” provided via microphone(s) of I/O components 111.sub.1. The request (e.g., the spoken input or a conversion thereof) can be transmitted by the automated assistant client 117.sub.1 of the client device 110.sub.1, to the cloud-based automated assistant component(s) 120 via the wide-area networks 105.sub.1. The cloud-based automated assistant component(s) 120 can process the request to determine a smart device to be controlled based on the request, and transmit, via the wide-area networks 105.sub.2, a request to a respective one of the smart device systems 140.sub.A-N which, in turn transmits, via wide-area networks 105.sub.3, corresponding command(s) to the smart device. However, as described herein such techniques present drawbacks such as high latency, low reliability, and/or excessive consumption of network resources.

(21) While implementations disclosed herein can still utilize such techniques for some smart device control requests and/or in some situations, various implementations disclosed herein can at least selectively bypass, in controlling a smart device, communications between the cloud-based automated component(s) 120 and a respective one of the smart device system(s) 140 and/or communication(s) between the smart device system(s) 140 and the smart device. Rather, those various implementations generate, locally at one of the client devices 110.sub.1-N, specific control commands that are transmitted locally (e.g., via a local-area-network or a local peer-to-peer connection) to smart device(s) to effectuate a desired state change at the smart device(s). This can mitigate the high latency, low reliability, excessive consumption of network resources, and/or other drawbacks of the above-mentioned techniques.

(22) In some implementations, the plurality of client computing devices 110.sub.1-N and smart devices 145.sub.A-N can be associated with each other in various ways in order to facilitate performance of techniques described herein. For example, in some implementations, the plurality of client devices 110.sub.1-N and smart devices 145.sub.A-N may be associated with each other by virtue of being communicatively coupled via one or more LANs and/or via one or more peer-to-peer networks. This may be the case, for instance, where plurality of client computing devices 110.sub.1-N are deployed across a particular area or environment, such as a home, a building, and so forth. Additionally or alternatively, in some implementations, plurality of client devices 110.sub.1-N and smart devices 145.sub.A-N may be associated with each other by virtue of them being members of a coordinated ecosystem of client devices 110.sub.1-N and smart devices 145.sub.A-N that are at least selectively accessible by one or more users (e.g., an individual, a family, employees of an organization, other predefined groups, etc.). In some of those implementations, the ecosystem of client devices 110.sub.1-N and smart devices 145.sub.1-N can be manually and/or automatically associated with each other in the home graph 152 and/or other device topology. For example, one or more of the client devices 110.sub.1-N and smart devices 145.sub.1-N can be associated with one another through registration techniques described herein.

(23) An instance of an automated assistant client 117.sub.1-N, by way of its interactions with one or more cloud-based automated assistant components 120, may form what appears to be, from a user's perspective, a logical instance of an automated assistant with which the user may engage in a human-to-computer dialog. For example, a user can engage with the same logical instance of an automated assistant using either client device 110.sub.1 and automated assistant client 117.sub.1 or using client device 110.sub.N and automated assistant client 117.sub.N. While the particular instances of the automated assistant client 117.sub.1 and 117.sub.N can vary and/or can provide user interface output via different I/O components 111.sub.1 and 111.sub.N and/or accept different user interface input via different I/O components 111.sub.1 and 111.sub.N (e.g., I/O components 111.sub.1 can include a touch-screen, while I/O components 111.sub.N do not), the user may still effectively engage with the same logical instance of the automated assistant. For the sakes of brevity and simplicity, the term “automated assistant”, as used herein will refer to the automated assistant client 117 executing on a client device 110 and optionally to one or more cloud-based automated assistant components 120 (which may be shared amongst multiple automated assistant clients 117). Although two client devices 110.sub.1 and 110.sub.N of a coordinated ecosystem are illustrated in FIG. 1, it is understood that many additional client devices can be included in the ecosystem. Further, it is understood that separate coordinated ecosystems of client devices will also be provided, each associated with different user(s) (e.g., account(s)) and/or environments, and that such separate coordinated ecosystems can also interact with cloud-based automated assistant component(s) 120 (but with interactions tailored to the account(s) of those separate ecosystems).

(24) The client devices 110.sub.1-N may include, for example, one or more of: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone assistant-centric interactive speaker, a standalone assistant-centric interactive display with speaker(s), a smart appliance such as a smart television, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.

(25) As mentioned above, one or more of the client devices 110.sub.1-N can optionally include a respective instance of an automated assistant client 117.sub.1-N. The automated assistant clients 117.sub.1-N can each process user inputs received from input device(s) of respective I/O components 111.sub.1-N, such as spoken inputs detected via microphone(s) of I/O components 111.sub.1-N, touch inputs received via touch-screen displays of I/O components 111.sub.1-N, images detected via camera(s) of I/O components 111.sub.1-N, etc. Further, each of the automated assistant clients 117.sub.1-N can optionally render various outputs via output device(s) of respective I/O components 111.sub.1-N, such as speaker(s) and/or touch-screen displays of I/O components 111.sub.1-N.

(26) In various implementations, one or more instances of the automated assistant client 117.sub.1-N can interface with the cloud-based automated assistant component(s) 120 in processing inputs and/or in generating outputs based on the inputs and/or in generating smart device control commands based on the inputs. For example, automated assistant client 117.sub.1 can transmit, to cloud-based automated assistant components 120, audio data corresponding to spoken input received after invocation of the automated assistant client 117.sub.1 at the client device 110.sub.1. The invocation of the automated assistant client 117.sub.1 at the client device 110.sub.1 can be based on detecting an invocation phrase (e.g., “OK Assistant”), interaction of a hardware button or graphical button that invokes the automated assistant client 117.sub.1, in response to a gesture detected via a camera of the I/O components 111.sub.1, and/or other invocation signal(s). The cloud-based automated assistant components 120 can convert the audio data to text using speech-to-text (STT) processor 121, and perform natural language understanding (NLU) on the text using NLU engine 122 to determine an appropriate response. For example, the appropriate response can be a textual response that can optionally be converted to generated speech using text-to-speech (US) processor, and transmitted to the automated assistant client 117.sub.1 for rendering of the generated speech via speaker(s) of the I/O components 111.sub.1.

(27) As another example, the appropriate response can be a generic smart device control command, such as a structured command that defines an intent (e.g., turn on/off, dim, increase, decrease), defines a smart device (e.g., a unique identifier of the smart device), defines a third-party for the smart device (e.g., a unique identifier for the third-party), and/or one or more further parameters for the intent (e.g., how much to dim, what to increase/decrease, etc.). The generic smart device control command can be transmitted to the assistant client 117.sub.1, or another assistant client of another client device 110, for local implementation of the generic smart device control command. In some implementations and/or for some assistant clients 117.sub.1-N, the assistant client can include a local STT processor, local NLU engine, and/or other component(s) for locally processing received inputs, without necessitating interfacing with cloud-based automated assistant component(s) 120. For example, assistant client 117.sub.1 can include one or more local component(s) for locally generating generic smart device control commands, without interfacing with cloud-based automated assistant component(s) 120.

(28) One or more of the client devices 110 also optionally includes a respective adapter interaction engine 114.sub.1-N, one or more respective locally stored adapters 113.sub.A-N, and one or more respective available radios 112.sub.A-N. The same adapters 113.sub.A-N and available radios 112.sub.A-N can optionally be locally provided on each of the client devices 110.sub.1-N, or the adapters 113.sub.A-N, and/or radios 112.sub.A-N can optionally vary among client devices 110.sub.1-N. Further, as described herein, one or more of the adapters 113.sub.A-N can be preemptively executed at a client device 110.sub.1-N, and which adapters are preemptively executing at a given time for a given client device 110.sub.1-N can be based on one or more criteria, such as historical usage of the client device, temporal criteria, etc.

(29) Further description of adapter interaction engines 114.sub.1-N and locally stored adapters 113.sub.A-N is now provided with reference to client device 110.sub.1. A generic smart device control command can be passed from the automated assistant client 117.sub.1 to the adapter interaction engine 114.sub.1. Although illustrated separately in FIG. 1 for clarity, it is noted that in various implementations adapter interaction engine 114.sub.1, adapters 113.sub.A-N, routing engine 115.sub.1, and/or registration engine 116.sub.1 can logically be incorporated as part of automated assistant client 117.sub.1. The generic smart device control command can be generated responsive to voice input, touch input, triggering of an automated assistant routine, a signal from another non-assistant device, and/or other signal(s). For example, the generic smart device control command can be generated responsive to voice input received at the client device 110.sub.1. For instance, the automated assistant client 117.sub.1 can transmit audio data, corresponding to the voice input, to remote automated assistant component(s) 120, and receive, in response, the generic smart device control command. Also, for instance, the automated assistant client 117.sub.1 can itself process the voice input to locally generate the generic smart device control command. As another example, automated assistant client 117.sub.1 can render, on a touch sensitive screen of I/O components 111.sub.1, an interactive graphical interface element for control of smart device(s) and can directly interpret interactions with the interactive graphical interface element into corresponding generic smart device control commands. The generic smart device control command can be a structured command that defines an intent, defines smart device(s) to be controlled, optionally defines a third-party for the smart device, and/or one or optionally one or more further parameters for the intent.

(30) The adapter interaction engine 114.sub.1 can select, based on the generic smart device control command, a corresponding one of the adapters 113.sub.A-N based on that adapter being provided by a party that corresponds to the generic smart device control command. The adapter interaction engine 114.sub.1 provides the generic smart device control command to the selected one of the adapters 113.sub.A-N, which processes the generic smart device control command to generate specific control commands that are directly interpretable by corresponding smart device(s) to effectuate state change(s) at the corresponding smart device(s), or at corresponding additional smart device(s) directly controlled by the corresponding smart device. In various implementations, each of adapters 113.sub.A-N is implemented by JavaScript (or other interpreted programming language) provided by a party corresponding to the adapter, and each of the adapters 113.sub.A-N can translate generic smart device control commands into specific control commands that conform to a protocol suite of the party. Each of adapters 113.sub.A-N can be sandboxed, executing within a container within the client device 110.sub.1 or the automated assistant client 117.sub.1, can process generic smart device control commands as described, and can further optionally report events to the automated assistant client 117.sub.1 using the schema of the automated assistant. The adapters 113.sub.A-N can be transmitted to the client device 110.sub.1 by the cloud-based automated assistant component(s) 120, which can select the adapters 113.sub.A-N from 3P adapters database 150. The adapters of 3P adapters database 150 can be updated by smart device systems 140.sub.A-N (e.g., to conform to firmware updates and/or to improve functionality) and updated adapters pushed to the client device 110.sub.1. In some implementations, additional adapter(s) can be provided and utilized to process certain generic smart device control commands, optionally for certain smart device(s). For example, a given additional adapter can be utilized to process certain generic smart device control commands, and generate corresponding specific control commands that are common for multiple 3P protocol suites (e.g., common to an industry standard base protocol to which the 3P protocol suite(s) conform or are variations). For instance, a generic wake-on-Ian (WOL) command can be processed using the given additional adapter to generate a specific WOL control command that is applicable to all protocol suites that are variation(s) of the IP protocol industry standard. In these and other manners, additional adapter(s) can be utilized to process certain generic smart device control command(s) (e.g., for smart devices of multiple 3Ps), while 3P specific adapter(s) can be utilized to process certain other generic smart device control command(s).

(31) The generated specific control commands are provided to the adapter interaction engine 114.sub.1. The adapter interaction engine 114.sub.1 selects one of the radios 112.sub.A-N for locally transmitting the specific control command to effectuate alteration of state(s) of the smart device(s) indicated by the specific control command (and the generic smart device control command on which the specific control command is based). The adapter interaction engine 114.sub.1 can select the radio based on it being assigned, locally, to the party and/or adapter corresponding to the generic smart device control command. The adapter interaction engine 114.sub.1 then causes the specific control commands to be transmitted via the selected one of the radios 112.sub.A-N, and addressed to the smart device(s), to thereby effectuate a state change at the addressed smart device(s), or at smart device(s) connected to the addressed smart device(s) (e.g., when the addressed smart device(s) is a hub controlling additional smart device(s)).

(32) One or more of the client devices 110 further optionally include a respective routing engine 115.sub.1-N and/or a respective registration engine 116.sub.1-N. Each routing engine 115.sub.1-N can selectively determine to route, via a local network, a received generic smart device control command to an alternate client device, of the ecosystem of client devices 110.sub.1-N, based on one or more criteria. For example, the client device 110.sub.1 can initially identify a smart device control command based on, for example, the smart device control command being generated locally at the client device 110.sub.1 (e.g., responsive to touch-interaction with a corresponding graphical element displayed at the client device 110.sub.1) or being generated at remote automated assistant components 120, and initially being provided to the client device 110.sub.1 based on being generated based on spoken or other input received at the client device 110.sub.1.

(33) The generic smart device control command can specify a particular smart device to be controlled, and the routing engine 115.sub.1 can determine that a reliable communications channel is not established between the client device 110.sub.1 and the particular smart device. For example, the particular smart device can be one to be controlled via a Bluetooth communications channel and the routing engine 115.sub.1 can determine that the particular smart device is not detectable via a Bluetooth radio of radios 112.sub.A-N of client device 110.sub.1, or a signal strength between the client device 110.sub.1 and the particular smart device, via the Bluetooth radio, fails to satisfy a threshold. Such a determination can be based on, for example, current readings via the Bluetooth radio and/or based on local version of the home graph 152.sub.1 indicating that the client device 110.sub.1 does not have a reliable connection with the particular smart device.

(34) In response, the routing engine 115.sub.1 can transmit the generic smart device control command, over a local network (e.g., Wi-Fi or a mesh network established between assistant client devices 110.sub.1-N) to an alternate client device 110.sub.2-N that has an established reliable communications channel to the smart device. The routing engine 115.sub.1 can determine such an alternate client device utilizing, for example, local version of the home graph 152.sub.1, which can indicate that the alternate client device has a reliable connection with the smart device. For example, the routing engine 115.sub.1 can select the alternate client device based on the local version of the home graph 152.sub.1 indicating the alternate client device has the best signal strength, for the smart device, amongst all client devices 110.sub.1-N. The local version of the home graph 152.sub.1 is locally stored at the client device 110.sub.1 and can be the same or a variant of the home graph 152 maintained by the remote automated assistant component(s) 120 (e.g., a compressed version, a slightly less up to date version). The alternate client device, in response to receiving the generic smart device control command, can then use a corresponding local adapter to translate the generic smart device control command to a corresponding specific command, and transmit the corresponding specific command to the particular smart device. Alternatively, the adapter interaction engine 114.sub.1 can generate the corresponding specific command using a corresponding local one of the adapters 113.sub.A-N, and transmit the corresponding specific command to the alternate client device, in lieu of transmitting the generic smart device control command.

(35) Each of the registration engines 116.sub.1-N is utilized in locally discovering, provisioning, and/or registering smart devices for an account of a user. Each of the registration engines 116.sub.1-N can interface with a respective locally stored home graph 152.sub.A-N, respective adapters 113.sub.A-N (e.g., via a respective adapter interaction engine 114.sub.1-N), and respective radios 112.sub.A-N in discovering, provisioning, and/or registering smart devices. Further, each of the registration engines 116.sub.1-N can interface with a cloud registration engine 127 of cloud-based automated assistant component(s) 120.

(36) In some implementations, a smart device that is not yet registered can be discovered by causing each of the registration engines 116.sub.1-N to scan one or more respective communications channels (e.g., Wi-Fi, Bluetooth, and/or other) for smart device(s) that aren't registered. For example, for Wi-Fi, each of the registration engines 116.sub.1-N can cause a scan to be performed via a respective Wi-Fi radio, of radios 112.sub.A-N, to identify any unregistered device(s), optionally using service set identifiers (SSIDs) and/or basic service set identifiers (BSSIDs) to filter for particular smart devices for which corresponding adapters are available. As another example, for Bluetooth, each of the registration engines 116.sub.1-N can cause a scan to be performed via a respective Bluetooth radio to identify any un-paired smart device(s), optionally utilizing identifier(s) to filter for particular smart devices for which corresponding adapters are available. The discovery can occur at periodic or non-periodic intervals, or in response to a user request (e.g., a voice request, a request initiated via a smart phone app for the assistant, etc.).

(37) When one or more of the registration engines 116.sub.1-N discovers a smart device, the routing engine of one of the smart devices that discovered the smart device can provision the smart device by pairing with it (e.g., in the case of Bluetooth) and/or causing it to be connected to a secured Wi-Fi network to which the client devices 110.sub.1-N are already connected. Which of the registration engines 116.sub.1-N provisions the smart device can be determined based on negotiation directly between the client devices 110.sub.1-N and/or can be determined by cloud routing engine 125, which can receive indications, from registration engines.sub.1-N, of which registration engines.sub.1-N discovered the smart device.

(38) After provisioning, data provided by the provisioned smart device can be received, at one of the client devices 110.sub.1-N, and processed using a corresponding local adapter 113.sub.A-N of the assistant client device, to generate registration data in a schema of the automated assistant. The corresponding local adapter can already be locally provided at the client device or can optionally be pushed, to the client device, by cloud-based automated assistant component(s) 120 responsive to the provisioning of the smart device. For example, the cloud registration engine 127 can access 3P adapters database 150 to retrieve a 3P adapter that corresponds to the smart device that is provisioned, and push the 3P adapter to one or more (e.g., all) of the client devices 110.sub.1-N for local storage and utilization at the client devices 110.sub.1-N The registration data generated by the local adapter 113.sub.A-N can be utilized by the automated assistant to add the smart device to a home graph or other structure that defines smart devices, assistant client devices, and various properties (e.g., name(s), location(s), etc.) of the smart devices and assistant client devices. For example, the registration data can be transmitted to the cloud registration engine 127, which can add the registration data to the home graph 152, and optionally push updated local versions of the home graph 152.sub.1-N to each of the client devices 110.sub.1-N. The cloud registration engine 127 can optionally also interface with a corresponding one of the smart device systems 140.sub.A-N, to report the registration to the smart device system to thereby enable the smart device system to locally correlate the smart device registration.

(39) In some implementations, the cloud registration engine 127 registers a smart device with the home graph 152, based on biometric data, detected via one or more of the client devices 110.sub.1-N, corresponding to an account associated with the home graph 152. For example, the smart device can be assigned to the home graph 152 (e.g., in lieu of another home graph associated with another account of one or more of the client devices 110.sub.1-N) based on voice data, received via one or more of the client devices 110.sub.1-N during the discovering, provisioning, and/or registering, matching voice authentication data stored in association with the account. Other biometric identification can additionally or alternatively be utilized, such as facial, fingerprint, etc. In some implementations, the cloud registration engine 127 can associate a registered smart device (e.g., in home graph 152 and local versions) with those client device(s) 110.sub.1-N that can locally transmit commands to the smart device. For example, if communications with the smart device occur via a short-range radio (e.g., Bluetooth radio), those client devices 110.sub.1-N having at least a corresponding signal strength (for the corresponding connection to the smart device via the short-range radio), can be designated in the home graph 152, optionally along with a designation of the corresponding signal strengths. Such designations can be used by routing engines 115.sub.1-N and/or by cloud routing engine 125 in determining which assistant client device(s) should be utilized in generating and/or transmitting specific control commands to the smart device.

(40) In some implementations, a smart device is discovered that is broadcasting an access point and that smart device is not yet provisioned as it is not connected to a secured Wi-Fi network utilized by the client devices 110.sub.1-N, and lacks the credentials for connecting to the secured Wi-Fi network. In some of those implementations, each of a plurality of the client devices 110.sub.1-N, detected the availability of the access point through scanning of its respective Wi-Fi radio. In some versions of those implementations, a single assistant client device is selected for connecting to the smart device via the access point of the smart device, and securely sharing, with the smart device via the connection, the credentials for connecting to the secured Wi-Fi network. The single client device can be selected through negotiation of the registration engines 116.sub.1-N of the client devices, or by the cloud registration engine 127. Once the credentials are shared, the smart device connects to the secured Wi-Fi network and can be registered through communications via the Wi-Fi network. In various versions where the single assistant client device is selected, it is selected based at least in part on human interaction data for the single assistant client device. For example, it can be selected based on the human interaction data indicating that no user interface input has been received at the single assistant client device within a threshold amount of time and/or based on the human interaction data indicating that the single assistant client device is utilized less frequently than any other of the plurality of assistant client devices that detected the access point. In these and other manners, the single client device can be selected so that disruption to ongoing and/or forthcoming assistant interactions is prevented, as the single client device will break its connection with the secured Wi-Fi network when connecting to the local access point—and the single client device is selected based on criteria that indicated it is not being used and/or is less likely (than other devices of the ecosystem) to be used.

(41) In some implementations, a smart device that is not yet registered can be discovered by causing each of the registration engines 116.sub.1-N to scan one or more respective communications channels (e.g., Wi-Fi, Bluetooth, and/or other) for smart device(s) that aren't registered. However, in some of those implementations the smart device may not be registrable locally and/or controllable locally. Nonetheless, in some versions of those implementations a notification related to the discovered smart device can be presented at one or more of the assistant client device(s) and/or other client device(s). The notification can include link(s) and/or other interactive content that can be interacted with to facilitate registration in a non-local manner (e.g., via the cloud). For example, the interactive content, when selected, can result in rendering of a single sign in interface and/or a smart device specific sign in interface that enables registering of the smart device.

(42) As mentioned above, cloud-based automated assistant component(s) 120 can optionally interface with each of the client devices 110 in performance of certain operations. The cloud-based automated assistant components 120 can include a STT module 121 configured to leverage the resources of the cloud to convert audio data (provided by a corresponding client device 110) into text (which may then be provided to NLU module 122). TTS module 123 may be configured to leverage the resources of the cloud to convert textual data into computer-generated speech output. NLU module 122 processes natural language input generated by users via client devices 110.sub.1-N and may generate annotated output for use by one or more other components of cloud-based automated assistant component(s) 120. For example, the NLU module 122 may process natural language free-form input that is generated by a user via one or more user interface input devices of client device 110.sub.1, such as text that has been converted from spoken input, or typed text. The generated annotated output includes one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input.

(43) In some implementations, the NLU module 122 is configured to identify and annotate various types of grammatical information in natural language input. For example, the NLU module 122 may include a part of speech tagger and/or an entity tagger. In some implementations, the NLU module 122 may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or “cluster,” references to the same entity based on one or more contextual cues. In some implementations, one or more components of the NLU module 122 may rely on annotations from one or more other components of the NLU module 122.

(44) Cloud routing engine 125 can optionally be utilized to select a particular client device, of client devices 110.sub.1-N, to which to provide a given generic smart device control command. The cloud routing engine 125 can utilize the home graph 152 in selecting the particular client device. For example, the cloud routing engine 125 can utilize the home graph 152 to select the particular client device based on the home graph 152 indicating that the particular client device has an established communication channel with smart device(s) of the generic smart device control command and/or has the best (e.g., highest signal strength) communications channel with smart device(s) of the generic smart device control command. The cloud routing engine 125 can then address the generic smart device control command to the selected particular client device (e.g., using an address stored in the home graph) to cause it to be transmitted directly to that client device, for locally converting, at the selected particular client device, to specific control commands and for locally transmitting, at the selected particular client device, the specific control commands to corresponding smart device(s). The cloud routing engine 125 can route a generic smart device control command to a particular client device, even when the generic smart device control command is generated responsive to user interface input received at a difference client device.

(45) As described herein, in some implementations cloud-based automated assistant component(s) 120 (e.g., STT module 121 and NLU module 122) can process audio data corresponding to spoken input captured at one of the client devices 110, generate a generic smart device control command based on such spoken input, and return the generic command to the one of the client devices 110 for local generation, at the one of the client devices, of specific commands for the corresponding smart device(s). In other implementations or situations, the cloud-based automated assistant component(s) can provide generic smart device control command(s) responsive to other signals, such as responsive to triggering of an automated assistant routine that involves the generic smart device control command.

(46) Additional description of various components of FIG. 1 is now provided with reference to the additional figures. FIG. 2 depicts a home floorplan that includes a plurality of rooms, 250, 252, 254, 256, 258, 260, and 262. A plurality of client devices 110.sub.1-3 are deployed throughout at least some of the rooms. Each of the client devices 110.sub.1-3 can optionally implement an instance of an automated assistant client configured with selected aspects of the present disclosure and can optionally include one or more input devices, such as microphones, touch-screens, etc. and/or one or more output devices such as speakers, displays, etc. For example, a first client device 110.sub.1 taking the form of a standalone interactive speaker and display device (e.g., display screen, projector, etc.) is deployed in room 250, which in this example is a kitchen. A second client device 110.sub.2 taking the form of an interactive standalone speaker is deployed in room 254, which in this example is a bedroom. A third client device 110.sub.3, also taking the form of an interactive standalone speaker, is deployed in room 256.

(47) The plurality of client devices 110.sub.1-3 may be communicatively coupled with each other and/or other resources (e.g., smart devices and the Internet) via a wireless router 101, depicted in room 252, and/or a local mesh network. Additionally, other client devices—particularly mobile devices such as smart phones, tablets, laptops, wearable devices, etc.—may also be present, e.g., carried by one or more persons (e.g., user 103) in the home and may or may not also be connected to the same LAN. It should be understood that the configuration of client devices depicted in FIG. 2 is just one example; more or fewer and/or different client devices may be deployed across any number of other rooms and/or areas other than a home.

(48) Further depicted in FIG. 2 are a plurality of smart devices. The smart devices include a smart light 145.sub.A1 that is provided by a first 3P (as indicated by the “A” designation). The smart light 145.sub.A1 is controllable via a Bluetooth radio using a protocol suite, of the first 3P, that is a variation of an industry standard. The smart devices further include smart lights 145.sub.B1 and 145.sub.B2, and a smart doorbell 145.sub.B3, that are controlled via a hub 147.sub.B that is coupled to the wireless router 101. The smart lights 145.sub.B1 and 145.sub.B2, smart doorbell 145.sub.B3, and hub 147.sub.B are provided by a second 3P (as indicated by the “B” designation). The smart lights 145.sub.B1 and 145.sub.B2, smart doorbell 145.sub.B3, and hub 147.sub.B utilize an industry standard protocol, and the hub 147.sub.B is addressable via TCP/IP, utilizing a Wi-Fi radio and via its hardwired connection to wireless router 101. The smart devices further include smart washer 145.sub.C1 that is provided by a third 3P (as indicated by the “C” designation). The smart washer 145.sub.C1 is controllable via a low-rate wireless personal area network radio channel using a proprietary protocol suite of the third 3P. The smart devices further include smart thermostat 145.sub.D1 that is provided by a fourth 3P (as indicated by the “D” designation). The smart thermostat 145.sub.D1 is controllable via a Wi-Fi radio using a variation, of the fourth 3P, of the UDP protocol suite. It should be understood that the configuration of smart devices 145 depicted in FIG. is just one example; more or fewer and/or different smart devices may be deployed across any number of other rooms and/or areas other than a home.

(49) FIG. 2 and the above description of FIG. 2 will now be utilized in describing various aspects of FIGS. 3-7. FIG. 3 illustrates an example state diagram 300 of an example of controlling smart lights 145.sub.B1 and 145.sub.B2 of FIG. 2, using client device 110.sub.1 of FIG. 2. In FIG. 3, adapter interaction engine 114.sub.1 of client device 110.sub.1 identifies a generic smart device control command at 352. The generic smart device control command can be generated by client device 110.sub.1, or provided to client device 110.sub.1 by remote cloud-based automated assistant component(s) 120 and/or by another client device. The generic smart device control command identifies smart lights 145.sub.B1 and 145.sub.B2, indicates that their color should be adjusted to “bright red”, and optionally identifies the second 3P.

(50) At 354, the adapter interaction engine 114.sub.1 selects 3P adapter 113.sub.B that corresponds to the second 3P. The adapter interaction engine 114.sub.1 can select the 3P adapter 113.sub.B, from a plurality of available 3P adapters stored locally at the client device 110.sub.1, based on the 3P adapter 113.sub.B being assigned to the second 3P and/or to the smart lights 145.sub.B1 and 145.sub.B2 specified in the generic smart device control command.

(51) At 356, the interaction engine 114.sub.1 provides the generic smart device control command to the 3P adapter 113.sub.B. In some implementations, the 3P adapter 113.sub.B is locally stored, and optionally already executing, at the client device 110.sub.1 based on the smart lights 145.sub.B1 and 145.sub.B2 and/or other smart device(s) of the second 3P being registered for an account associated with the client device 110.sub.1. In some implementations, the 3P adapter 113.sub.B is already executing at the client device 110.sub.1 further based on, for example, the 3P adapter 113.sub.B being recently utilized at the client device 110.sub.1, being utilized in the past at dates/times that are similar to a current date/time, and/or based on one or more other criteria.

(52) At 362, the 3P adapter 113.sub.B generates a specific command based on the provided generic command and, at 364, the 3P adapter 113.sub.B returns the generated specific command to the adapter interaction engine 114.sub.1. It is noted that, even though the second 3P utilizes a standard protocol, the generated specific command may differ from that which would otherwise be generated utilizing the standard protocol since, for example, the 3P adapter 113.sub.B may integrate custom logic, of the second 3P, that will result in specific commands, for the “bright red” state change request, that are specifically tailored to the second 3P. The specific command can be addressed to the smart hub 147.sub.B that can directly forward the specific command to the smart lights 145.sub.B1 and 145.sub.B2, which can directly interpret the specific command to cause a change of the color state of the lights to the “bright red” state. Alternatively, the smart hub 147.sub.B can interpret the specific command to thereby directly control the smart lights 145.sub.B1 and 145.sub.B2 in accordance with the specific command.

(53) At 358, the adapter interaction engine 114.sub.1 selects a radio 112.sub.B for the 3P adapter 113.sub.B and provides the specific command to cause it to be transmitted using the selected radio at 372. The radio 112.sub.B is selected, from a plurality of available radios at the client device 110.sub.1, based on being assigned (locally at the client device 110.sub.1) to the 3P adapter 113.sub.B. For example, the radio 112.sub.B can be a Wi-Fi radio and the specific command can be transmitted to the smart hub 147.sub.B via TCP/IP.

(54) As noted above, the smart hub 147.sub.B can directly forward the specific command to the smart lights 145.sub.B1 and 145.sub.B2, which can directly interpret the specific command to cause a change of the color state of the lights to the “bright red” state. Alternatively, the smart hub 147.sub.B can interpret the specific command and send further interpreted commands to thereby directly control the smart lights 145.sub.B1 and 145.sub.B2 in accordance with the specific command. Regardless, at 382 the state change is implemented. Optionally, at 384 the smart hub 147.sub.B transmits a response back to the radio 112.sub.B and at 374 the response is provided to the 3P adapter 113.sub.B, which at 366 interprets the response (e.g., into a schema of the automated assistant) and provides the interpreted response to the adapter interaction engine 114.sub.1. The adapter interaction engine 114.sub.1 can optionally, at 359, provide the interpreted response to one or more additional components, as a response to the identified generic command. For example, where the interpreted response indicates the state change was successful it can be provided to update a display of client device 110.sub.1 to indicate the success, can be provide to cloud-based automated assistant component(s) 120 to update the home graph 152 to reflect the state change, provided to a system 140 associated with the second 3P to reflect the state change, and/or provided to other component(s).

(55) FIG. 4 illustrates an example state diagram 400 of an example of routing a generic smart device control command, for controlling smart light 145.sub.A1 of FIG. 2, from client device 110.sub.1 of FIG. 2 to client device 110.sub.2 of FIG. 2. In FIG. 4, routing engine 115.sub.1 of client device 110.sub.1 identifies a generic smart device control command at 452. The generic smart device control command can be generated by client device 110.sub.1, or provided to client device 110.sub.1 by remote cloud-based automated assistant component(s) 120 and/or by another client device. For example, the generic smart device control command can be generated by client device 110.sub.1 responsive to touch-interaction with a graphical interface element, displayed via a touch-screen of the client device 110.sub.1, that enables adjusting a dimming level of the smart light 145.sub.A1. The generic smart device control command identifies smart light 145.sub.A1, indicates that it should be adjusted to a “50% dimmed” state, and optionally identifies the first 3P.

(56) At 454, the routing engine 115.sub.1 determines that client device 110.sub.1 has an unreliable communication channel with the smart light 145.sub.A1. For example, as mentioned above the smart light 145.sub.A1 can be controllable via a Bluetooth radio. Due to the distance between the client device 110.sub.1 and the smart light 145.sub.A1, and/or due to obstructions, the Bluetooth radio of the client device 110.sub.1 may not have any communication with the smart light 145.sub.A1 or may have communication, but with a signal strength that fails to satisfy a threshold. The routing engine 115.sub.1 can determine the unreliable communication channel based on current data from the Bluetooth radio, or based on past historical data (e.g., stored in a local mapping of client devices to smart devices).

(57) At 456, the routing engine 115.sub.1 resolves assistant client device 110.sub.2 that has a reliable communication channel with the smart light 145.sub.A1. In resolving the client device 110.sub.2, the routing engine 115.sub.1 can optionally rely on a client devices to smart devices mapping, stored locally at the client device 110.sub.1, to determine client device 110.sub.2 is one having an established reliable communications channel to the smart light 145.sub.A1 (e.g., as a result of being proximate to the smart light 145.sub.A1). The client devices to smart devices mapping can define client devices of the ecosystem and, for each of the assistant client devices, can define corresponding smart device(s) for which the assistant client device has a corresponding reliable communications channel, and can optionally define signal strength(s) between the assistant client device and smart device(s). Such mapping can optionally be included in a locally stored home graph as described herein.

(58) At 458, the routing engine 115.sub.1 transmits, using a local network, the generic smart device control command to the resolved client device 110.sub.2.

(59) At 460, the adapter interaction engine 114.sub.2 of the client device 110.sub.2 selects 3P adapter 113.sub.A that corresponds to the first 3P. The adapter interaction engine 114.sub.2 can select the 3P adapter 113.sub.A, from a plurality of available 3P adapters stored locally at the client device 110.sub.2, based on the 3P adapter 113.sub.A corresponding to the first 3P and/or the smart light 145.sub.A1 specified in the generic smart device control command.

(60) At 462, the interaction engine 114.sub.2 provides the generic smart device control command to the 3P adapter 113.sub.A. In some implementations, the 3P adapter 113.sub.A is locally stored, and optionally already executing, at the client device 110.sub.2 based on the smart light 145.sub.A1 being registered for an account associated with the client device 110.sub.2. In some implementations, the 3P adapter 113.sub.A is already executing at the client device 110.sub.1 further based on, for example, there being a strong strength of signal between the client device 110.sub.2 and the smart light 145.sub.A1, 3P adapter 113.sub.A being recently utilized at the client device 110.sub.1, and/or based on one or more other criteria.

(61) At 472, the 3P adapter 113.sub.A generates a specific command based on the provided generic command and, at 474, the 3P adapter 113.sub.A returns the generated specific command to the adapter interaction engine 114.sub.2. It is noted that the first 3P utilizes a protocol suite that is a variation of an industry standard and, as a result, the generated specific command may not conform to the industry standard. The specific command can be addressed to the smart light 145.sub.A1, which can directly interpret the specific command to cause a change of the dimming level of the smart light 145.sub.A1 to “50%”.

(62) At 464, the adapter interaction engine 114.sub.2 selects a radio 112.sub.A for the 3P adapter 113.sub.A and causes it to be transmitted using the selected radio at 482. The radio 112.sub.A is selected, from a plurality of available radios at the client device 110.sub.2, based on being assigned (locally at the client device 110.sub.2) to the 3P adapter 113.sub.A. For example, the radio 112.sub.A can be a Bluetooth radio.

(63) At 492 the smart light 145.sub.A1 directly interprets the specific command to cause a change of the dimming level of the smart light 145.sub.A1 to “50%”. Optionally the smart light 145.sub.A1 transmits a response back to the radio 112.sub.A and the response is provided to the 3P adapter 113.sub.A, which interprets the response (e.g., into a schema of the automated assistant) and provides the interpreted response to the adapter interaction engine 114.sub.2. The adapter interaction engine 114.sub.2 can optionally provide the interpreted response to one or more additional components, as a response to the identified generic command.

(64) Turning now to FIGS. 5-7, various methods are described with reference to operations of flow charts of those figures. For convenience, the operations of flow charts described below are described with reference to a system that performs the operations. The system can include one or more components of one or more client devices (e.g., client devices that execute an automated assistant client) and/or one or more components of a separate computing system (e.g., remote automated assistant server(s)). Moreover, different systems can perform the operations of different flowcharts. Additionally, while the operations of the flowcharts are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

(65) FIG. 5 is a flow chart illustrating an example method of 500 controlling a smart device, using a local assistant client device, according to various implementations disclosed herein. FIG. 5 can be implemented using one or more processors of the local assistant client device.

(66) At block 502, the system identifies a generic smart device control command that specifies smart device(s) and state(s) of the smart device(s) to be altered.

(67) At block 504, the system determines whether there is a reliable communications channel.

(68) If, at block 504, the system determines there is not a reliable communications channel, the system proceeds to block 506.

(69) At block 506, the system selects an alternate local assistant client device with a reliable communications channel to the smart device(s).

(70) At block 508, the system transmits the generic smart device control command to the selected alternate local assistant client device for locally controlling the smart device(s). The selected alternate local assistant client device, upon receiving the generic smart device control command, can locally process the generic smart device control command to generate a specific command, that corresponds to the generic smart device control command, and that is directly interpretable by the smart device to effectuate alteration of the state at the smart device. The selected alternate local assistant client device can further transmit the specific command over the established reliable communications channel to effectuate the alteration of the state at the smart device.

(71) If, at an iteration of block 504, the system determines there is a reliable communications channel, the system proceeds to block 510.

(72) At block 510, the system selects a particular adapter from a plurality of available adapters and based on the generic smart device control command.

(73) At block 512, the system processes the generic smart device control command using the selected adapter to generate specific commands.

(74) At block 514, the system transmits the specific command over the reliable communications channel to effectuate an alteration of the state(s) of the smart device(s).

(75) FIG. 6 is a flow chart illustrating another example 600 method of controlling a smart device, using a local assistant client device, according to various implementations disclosed herein. FIG. 6 can be implemented using one or more processors of the local assistant client device.

(76) At block 602, the system preemptively executes each of a plurality of adapters, such as a plurality of 3P adapters and optionally a 1P adapter.

(77) At block 604, the system identifies a generic smart device control command that specifies smart device(s) and state(s) of the smart device(s) to be altered.

(78) At block 606, the system selects, based on the generic smart device control command, a particular adapter from the plurality of executing adapters.

(79) At block 608, the system processes the generic smart device control command using the selected adapter to generate a specific command.

(80) At block 610, the system selects, from a plurality of available communication channels, a particular communication channel assigned to the selected adapter.

(81) At block 612, the system transmits the specific command using the selected particular communication channel to cause the state(s) to be altered at the smart device(s).

(82) FIG. 7 is a flow chart illustrating an example method 700 of registering a smart device, using a local assistant client device, according to various implementations disclosed herein. FIG. 7 can be implemented using one or more processors of the local assistant client device and/or one or more processors of remote automated assistant server(s).

(83) At block 702, the system causes each assistant client device, of an ecosystem of assistant client devices, to scan communication channel(s) for an unprovisioned smart device(s).

(84) At block 704, the system determines whether any new unprovisioned smart devices are detected by the scanning.

(85) If, at an iteration of block 704, the system determines there aren't any new unprovisioned smart devices, the system proceeds to block 706 and the method 700 ends.

(86) If at an iteration of block 704, the system determines there is at least one new unprovisioned smart device, the system proceeds to block 708.

(87) At block 708, the system causes a selected assistant device to interface with the new smart device, via a communications channel, to establish local connection(s) of the new smart device to the assistant client device(s). Block 708 may optionally include one or more sub-blocks, such as sub-block 709.

(88) At sub-block 709, the system can select the assistant client device for interfacing based on human interaction data for the assistant client device. For example, at sub-block 709 the new smart device can be the smart thermostat 145.sub.D1 of FIG. 2 and it can be broadcasting an access point, but not yet connected to the secured wireless router 101 utilized by the assistant client devices, and can lack the credentials for connecting to the secured wireless router 101. In some of those implementations, each of a plurality of assistant client devices, of the ecosystem of client devices, detected the availability of the access point through scanning of its respective Wi-Fi radio. For example, all three of the client devices 110.sub.1-3 of FIG. 2 may have detected the availability of the access point. In some versions of those implementations, a single assistant client device is selected for connecting to the smart device via the access point of the smart device, and securely sharing, with the smart device via the connection, the credentials for connecting to the secured Wi-Fi network. Once the credentials are shared, the smart device connects to the secured Wi-Fi network and can be registered through communications via the Wi-Fi network. The single assistant client device can be selected based at least in part on human interaction data for the single assistant client device. For example, client device 110.sub.3 can be selected based on the human interaction data indicating that no user interface input has been received at client device 110.sub.3 within a threshold amount of time and/or based on the human interaction data indicating that the client device 110.sub.3 is utilized less frequently than client device 110.sub.1 and client device 110.sub.2.

(89) At block 710, the system causes the selected or an additional assistant client device to use an adapter, for the new smart device(s), in generating registration data for the new smart device. Block 710 may optionally include one or more sub-blocks. At sub-block 711, the system can provide the adapter to the assistant client device(s) in response to the assistant client device(s) detecting the new smart device. For example, the adapter may not yet be locally stored at the assistant client device(s) if other smart device(s), for the party associated with the new smart device(s), are not yet linked with the assistant client devices. In such a situation, the adapter for that party can be retrieved from an adapter database, and provided to the assistant client device(s).

(90) At block 712, the system uses the registration data to incorporate the new smart device into a home graph. Block 712 may optionally include one or more sub-blocks. At sub-block 713, the system may store data indicating the assistant client device(s) that can locally control the new smart device, and optionally indicating preferences for locally controlling the new smart device. At sub-block 714, the system may optionally select the home graph based on biometric data, detected via one or more of the assistant client devices, corresponding to an account associated with the home graph.

(91) At block 715, the system may optionally transmit registration information to a third-party for the new smart device.

(92) FIG. 8 is a block diagram of an example computing device 810 that may optionally be utilized to perform one or more aspects of techniques described herein. Computing device 810 typically includes at least one processor 814 which communicates with a number of peripheral devices via bus subsystem 812. These peripheral devices may include a storage subsystem 824, including, for example, a memory subsystem 825 and a file storage subsystem 828, user interface output devices 820, user interface input devices 822, and a network interface subsystem 818. The input and output devices allow user interaction with computing device 810. Network interface subsystem 818 provides an interface to outside networks and is coupled to corresponding interface devices in other computing devices.

(93) User interface input devices 822 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen incorporated into the display, audio input devices such as voice recognition systems, microphones, and/or other types of input devices. In general, use of the term “input device” is intended to include all possible types of devices and ways to input information into computing device 810 or onto a communication network.

(94) User interface output devices 820 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term “output device” is intended to include all possible types of devices and ways to output information from computing device 810 to the user or to another machine or computing device.

(95) Storage subsystem 824 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 824 may include the logic to perform selected aspects of one or more methods described herein.

(96) These software modules are generally executed by processor 814 alone or in combination with other processors. Memory 825 used in the storage subsystem 824 can include a number of memories including a main random access memory (RAM) 830 for storage of instructions and data during program execution and a read only memory (ROM) 832 in which fixed instructions are stored. A file storage subsystem 828 can provide persistent storage for program and data files, and may include a hard disk drive, a floppy disk drive along with associated removable media, a CD-ROM drive, an optical drive, or removable media cartridges. The modules implementing the functionality of certain implementations may be stored by file storage subsystem 828 in the storage subsystem 824, or in other machines accessible by the processor(s) 814.

(97) Bus subsystem 812 provides a mechanism for letting the various components and subsystems of computing device 810 communicate with each other as intended. Although bus subsystem 812 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

(98) Computing device 810 can be of varying types including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computing device 810 depicted in FIG. 8 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computing device 810 are possible having more or fewer components than the computing device depicted in FIG. 8.

(99) In situations in which certain implementations discussed herein may collect or use personal information about users (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), users are provided with one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected about the user, stored and used. That is, the systems and methods discussed herein collect, store and/or use user personal information only upon receiving explicit authorization from the relevant users to do so.

(100) For example, a user is provided with control over whether programs or features collect user information about that particular user or other users relevant to the program or feature. Each user for which personal information is to be collected is presented with one or more options to allow control over the information collection relevant to that user, to provide permission or authorization as to whether the information is collected and as to which portions of the information are to be collected. For example, users can be provided with one or more such control options over a communication network. In addition, certain data may be treated in one or more ways before it is stored or used so that personally identifiable information is removed. As one example, a user's identity may be treated so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger region so that the user's particular location cannot be determined.

(101) In some implementations, a method implemented by one or more processors of a client device executing an automated assistant client is provided, and includes identifying a generic smart device control command. The generic smart device control command specifies at least one smart device and at least one state to be altered at the smart device. The method further includes determining whether a reliable communications channel is established between the client device and the smart device and, responsive to determining the reliable communications channel is not established between the client device and the smart device: accessing a client devices to smart devices mapping, stored locally at the client device, to resolve an additional client device with an established reliable communications channel to the smart device; and transmitting, over a local network, the generic smart device control command to the additional client device. Transmitting the generic smart device control command to the additional client device causes the additional client device to: locally process the generic smart device control command to generate a specific command, that corresponds to the generic smart device control command, and that is directly interpretable by the smart device to effectuate alteration of the state at the smart device, and transmit the specific command over the established reliable communications channel to effectuate the alteration of the state at the smart device.

(102) These and other implementations of the technology can optionally include one or more of the following features.

(103) In some implementations, the method further includes, responsive to determining the reliable communications channel is established between the client device and the smart device: selecting a particular third-party (3P) adapter from a plurality of 3P adapters locally executable at the client device, the selecting based on the smart device control command; processing the smart device control command using the selected 3P adapter to generate the specific command, that corresponds to the generic smart device control command, and that is directly interpretable by the smart device to effectuate alteration of the state at the smart device; and transmitting, over the reliable communications channel established between the client device and the smart device, the specific command to effectuate the alteration of the state at the smart device. In some versions of those implementations, the method further includes, prior to the identifying the generic smart device control command: receiving the particular 3P adapter at the client device and from a remote server, where receiving the particular 3P adapter is responsive to the smart device, or an additional smart device of the 3P, being registered for the client device. In some additional or alternative versions of those implementations, the method further includes, prior to the identifying the generic smart device control command: preemptively executing, locally at the client device, the particular 3P adapter to reduce latency in processing the smart device control command using the selected 3P adapter to generate the specific command.

(104) In some implementations, determining whether the reliable communications channel is established between the client device and the smart device includes determining whether any communications channel is established between the client device and the smart device and/or determining whether a communications channel, established between the client device and the smart device, has at least a threshold signal strength. Determining whether any communications channel is established and determining whether the communications channel has at least a threshold signal strength can be based on the client devices to smart devices mapping and/or can be based on one or more signals received via the communications channel subsequent to a most recent updating of the client devices to smart devices mapping.

(105) In some implementations, accessing the client devices to smart devices mapping, stored locally at the client device, to resolve the additional client device with the established reliable communications channel to the smart device includes: determining, based on the client devices to smart devices mapping, that a given signal strength, between the additional client device and the smart device, is greatest amongst all client devices of the client devices to smart devices mapping.

(106) In some implementations, the established reliable communications channel to the smart device is a Bluetooth radio channel.

(107) In some implementations, the generic smart device control command is generated locally at the client device by locally processing user interface input received at the client device. In some of those implementations, the user interface input is interaction, at a touch-screen of the client device, with a rendered graphical user interface element corresponding to the smart device.

(108) In some implementations, the user interface input is a voice input received via at least one microphone of the client device, and locally processing the user interface input includes using a speech-to-text processor to convert the voice input into text, and performing natural language understanding on the text to generate the generic smart device control command.

(109) In some implementations, the method further includes receiving voice input via at least one microphone of the client device and streaming the voice input to a remote automated assistant system. In some versions of those implementations, identifying the generic smart device control command includes receiving the generic smart device control command from the remote automated assistant client responsive to streaming the voice input to the remote automated assistant system.

(110) In some implementations, a method implemented by one or more processors of a client device executing an automated assistant client is provided and includes preemptively executing, locally at the client device, each of a plurality of third-party (3P) adapters each generated by a corresponding 3P. Each of the 3P adapters converts each of a plurality of corresponding generic smart device control commands into corresponding specific commands that are each tailored, when locally transmitted to at least one corresponding smart device of the 3P, to be directly interpretable by the corresponding smart device to effectuate a state change at the corresponding smart device, or at a corresponding additional smart device directly controlled by the corresponding smart device. The plurality of 3P adapters are optionally preemptively executed locally at the client device responsive to smart devices of the 3P being previously registered for the client device. The method further includes identifying a particular generic smart device control command that specifies at least one particular smart device and at least one state to be altered at the particular smart device. The method further includes: selecting, from the plurality of 3P adapters and based on the particular smart device control command, a particular 3P adapter; processing the particular smart device control command using the selected particular 3P adapter to generate a particular specific command, of the specific commands, that corresponds to the smart device control command; selecting, from a plurality of communication channels available at the automated assistant client device, a particular communication channel assigned to the particular 3P adapter; and transmitting, using the selected particular communication channel, the particular specific command to cause the at least one state to be altered at the particular smart device.

(111) These and other implementations of the technology can optionally include one or more of the following features.

(112) In some implementations, the plurality of 3P adapters preemptively executing at the client device are a subset of available 3P adapters stored at the client device. In some of those implementations, the method further includes selecting the particular 3P adapter, from the available 3P adapters, for preemptively executing at the client device. The selecting can be based on one or more dynamic criteria, such as a corresponding recency of utilization of the particular 3P adapter at the client device; a signal strength for a communication channel between the client device and the particular smart device corresponding to the particular 3P adapter; a time of day; and/or a day of the week.

(113) In some implementations, the method further includes, prior to the preemptively executing: receiving the particular 3P adapter at the client device and from a remote server, optionally responsive to the particular smart device, or an additional smart device of the 3P, being registered for the client device.

(114) In some implementations, an additional 3P adapter, of the 3P adapters preemptively executed at the client device, is assigned to an additional particular communication channel that is distinct from the particular communication channel. In some of those implementations, the particular communication channel is a first radio channel of the client device and the additional particular communication channel is a second radio channel of the client device. For example, the first radio channel can be a Bluetooth channel and the second radio channel can be a Wi-Fi radio channel.

(115) In some implementations, the particular specific command generated by the 3P adapter conforms to a particular protocol suite of the 3P adapter, and an additional 3P adapter, of the 3P adapters preemptively executed at the client device, generates commands that conform to an additional communication protocol suite that is distinct from the particular communication protocol suite. In some of those implementations the additional 3P adapter and the 3P adapter are both assigned to the particular communication channel. In some versions of those implementations, the particular protocol suite of the 3P adapter does not conform to an industry adopted standard, and can optionally be proprietary to the 3P.

(116) In some implementations, the particular generic smart device control command is generated responsive to user interface input received at an additional client device, and identifying the particular generic smart device control command includes: receiving, via the local network communication, the particular generic smart device command from the additional client device. In some of those implementations, the method further includes, prior to identifying the particular generic smart device control command: determining a signal strength, for the particular communication channel, between the client device and the particular smart device; and transmitting data indicative of the signal strength. The particular generic smart device command can be received from the additional client device based on the transmitted data indicative of the signal strength.

(117) In some implementations, selecting, from the plurality of 3P adapters and based on the particular smart device control command, the particular 3P adapter includes selecting the 3P adapter based on: the generic smart device control command further specifying an identifier for the 3P adapter, and/or the particular smart device specified by the generic smart device control command being assigned, at the client device, to the 3P adapter.

(118) In some implementations, the method further includes: identifying, based on the generic smart device control command, an address for the particular smart device; and verifying that the particular specific command is addressed to only the address for the particular smart device. Transmitting the particular smart device control command using the selected particular communication channel can be contingent on the verifying.

(119) In some implementations, a method implemented by one or more processors is provided and includes causing each assistant client device, of an ecosystem of assistant client devices connected to a secured Wi-Fi network that provides Internet access, to scan one or more communications channels for availability of an access point being broadcast by a smart device that lacks credentials for connecting to the secured Wi-Fi network. The method further includes determining, based on data generated by the scanning, that each of a plurality of assistant client devices, of the ecosystem of client devices, detected the availability of the access point. The method further includes selecting a single assistant client device from the plurality of assistant client devices that detected the availability of the access point. Selecting the single assistant client device from the plurality of assistant client devices can optionally be based at least in part on human interaction data for the single assistant client device. The method further includes causing the single assistant client device to: break a connection with the secured Wi-Fi network; after breaking the connection, establish a connection to the smart device via the access point, and securely share, with the smart device via the connection, the credentials for connecting to the secured Wi-Fi network.

(120) These and other implementations of the technology disclosed herein can include one or more of the following features.

(121) In some implementations, selecting the single assistant client device can be based at least in part on the human interaction data for the single assistant client device and can include: selecting the single assistant client device based on the human interaction data indicating that no user interface input has been received at the single assistant client device within a threshold amount of time; selecting the single assistant client device based on the human interaction data indicating that no audible output is being rendered by the assistant client device responsive to previously received user interface input; and/or selecting the single assistant client device based on the human interaction data indicating that the single assistant client device is utilized less frequently than any other of the plurality of assistant client devices that detected the setup access point.

(122) In some implementations, the method further includes: receiving, via the secured Wi-Fi network, registration data from the smart device; and using the registration data to assign the smart device in a stored home graph associated with the ecosystem of assistant client devices. In some of those implementations, using the registration data to assign the smart device in the stored home graph includes: assigning the smart device in the stored home graph based on determining that biometric data, detected via one or more of the assistant client devices of the ecosystem, corresponds to an account associated with the stored home graph. For example, the stored home graph can be selected, based on the biometric data, in lieu of an additional home graph associated with the ecosystem of assistant client devices and associated with an additional account. In various implementations the biometric data includes voice data detected via one or more microphones of the one or more of the assistant client devices of the ecosystem.

(123) In some implementations, a method implemented by one or more processors is provided and includes receiving, from each of a plurality of assistant client devices each having a corresponding short-range radio, a corresponding signal strength for a corresponding connection, via the short-range radio, to a smart device. The method further includes determining, based on the received signal strengths, that a given assistant device, of the plurality of assistant devices, is preferred for locally controlling the smart device. The method further includes identifying a generic smart device control command that specifies at least one smart device and at least one state to be altered at the smart device. The method further includes, based on determining the given assistant device is preferred for locally controlling the smart device, transmitting the generic smart device control command to the given assistant device to cause the given assistant device to: locally process the generic smart device control command to generate a specific command, that corresponds to the generic smart device control command, and that is directly interpretable by the smart device to effectuate alteration of the state at the smart device and transmit the specific command over the short-range radio to effectuate the alteration of the state at the smart device.

(124) In some implementations, a method implemented by one or more processors is provided and includes causing each assistant client device, of an ecosystem of assistant client devices, to scan one or more communications channels for any unprovisioned smart devices. The method further includes determining, based on data generated by the scanning, that at least a given assistant client device, of the ecosystem of client devices, detected a given smart device that is unprovisioned. The method further includes causing the given assistant client device to utilize a locally stored third-party (3P) adapter to process registration data, provided by the given smart device in a format for the 3P, to generate registration information that is in a schema for an automated assistant associated with the assistant client device. The method further includes utilizing the registration information to register the given assistant client device in association with an account associated with the ecosystem of assistant client devices.

(125) In some implementations, a method implemented by one or more processors is provided and includes receiving, from each of a plurality of assistant client devices each having a corresponding short-range radio, a corresponding signal strength for a corresponding connection, via the short-range radio, to a smart device. The method further includes determining, based on the received signal strengths, that a given assistant device, of the plurality of assistant devices, is preferred for locally controlling the smart device. The method further includes, based on determining the given assistant device is preferred for locally controlling the smart device, causing a particular third-party (3P) adapter, that corresponds to the smart device, to be downloaded to, and locally stored at, the given assistant device. The particular 3P adapter can be, for example, configured to locally process generic smart device control commands to generate specific commands that correspond to the generic smart device control commands and that are directly interpretable by the smart device to effectuate alteration of the state at the smart device.

(126) These and other implementations of the technology disclosed herein can include one or more of the following features.

(127) The method can further include, based on determining the given assistant device is preferred for locally controlling the smart device, causing the particular 3P adapter to be preemptively executed at the given assistant device.

(128) In some implementations, the particular 3P adapter is downloaded to only the given assistant device, to the exclusion of other of the smart devices, based on determining that the given assistant device is preferred for locally controlling the smart device.