A BRAIN CONTROL INTERFACE SYSTEM FOR CONTROLLING A CONTROLLABLE DEVICE

Abstract

A brain control interface system for controlling a controllable device located in an environment is disclosed. The brain control interface system comprising: a brain control interface configured to detect brain activity of a user indicative of a control command for controlling the controllable device, and to derive the control command from the brain activity, a sensor configured to detect changes of an environmental characteristic in the environment, a processor configured to: determine if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user, and if the temporal correlation is not present, control the controllable device according to the control command, if the temporal correlation is present, refrain from controlling the controllable device according to the control command.

Claims

1. A brain control interface system for controlling a controllable device located in an environment, the brain control interface system comprising: a brain control interface configured to detect brain activity of a user indicative of a control command for controlling the controllable device, and to derive the control command from the brain activity, a sensor configured to detect changes of an environmental characteristic in the environment, a processor configured to determine if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user, and if the temporal correlation is not present, control the controllable device according to the control command, if the temporal correlation is present, refrain from controlling the controllable device according to the control command.

2. The brain control interface system of claim 1, wherein the processor is configured to determine if the detected change of the environmental characteristic exceeds a threshold, and to refrain from controlling the controllable device according to the control command only if the environmental characteristic exceeds the threshold.

3. The brain control interface system of claim 1, wherein the processor is configured to determine the presence of the temporal correlation between the detected change of the environmental characteristic and the detected brain activity of the user by determining if the detected change of the environmental characteristic and the detected brain activity of the user occur within a predetermined time window.

4. The brain control interface of claim 3, wherein the predetermined time window is less than 1 second.

5. The brain control interface system of claim 1, wherein the processor is further configured to determine a presence of a second correlation between the type of environmental characteristic and the control command, and, if the temporal correlation is not present and the second correlation is present, control the controllable device according to the control command, if the second correlation is not present and the temporal correlation is present, control the controllable device according to the control command, if the temporal correlation and the second correlation are not present, control the controllable device according to the control command, if the temporal correlation and the second correlation are present, refrain from controlling the controllable device according to the control command.

6. The brain control interface system of claim 1, wherein the processor is further configured to: when the processor has refrained from controlling the controllable device according to the control command, request a user operating the brain control interface system to control the controllable device based on the control command, and when the user has approved the request via a user interface, control the controllable device according to the control command, and when the user has disapproved the request via the user interface, again refrain from controlling the controllable device according to the control command.

7. The brain control interface system of claim 1, wherein the processor is further configured to control a further device when the temporal correlation is present, wherein the control of the further device is based on the brain activity as a response to the detected change of the environmental characteristic.

8. The brain control interface system of claim 1, wherein the sensor is a light sensor, and wherein the environmental characteristic is the environmental light level.

9. The brain control interface system of claim 1, wherein the sensor is a temperature sensor, and wherein the environmental characteristic is the environmental temperature.

10. The brain control interface system of claim 1, wherein the brain control interface and the sensor are comprised in a brain control interface device.

11. The brain control interface system of claim 1, wherein the processor is further configured to: obtain first data indicative of the location of the change of the environmental characteristic, obtain second data indicative of the location of the user, determine, based on the location of the change of the environmental characteristic and the location of the user, if the change of the environmental characteristic occurred within a predefined proximity of the user, and, if the change of the environmental characteristic has occurred outside the predefined proximity and if the temporal correlation is present, control the controllable device according to the control command.

12. The brain control interface system of claim 11, wherein the second data is further indicative of an orientation of the user, and wherein the processor is further configured to: determine, based on the location of the change of the environmental characteristic and the orientation of the user, if the change of the environmental characteristic occurred within a field of view of the user, and, if the change of the environmental characteristic has occurred outside the field of view of the user and if the temporal correlation is present, control the controllable device according to the control command.

13. The brain control interface system of claim 1, wherein the controllable device is a lighting device.

14. A method of controlling a controllable device located in an environment, the method comprising: detecting, by a brain control interface, brain activity of a user indicative of a control command for controlling the controllable device, deriving the control command from the brain activity, detecting, by a sensor, changes of an environmental characteristic in the environment, determining, by a processor, if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user, and if the temporal correlation is not present, controlling the controllable device according to the control command, if the temporal correlation is present, refraining from controlling the controllable device according to the control command.

15. A computer program product for a computing device, the computer program product comprising computer program code to perform the method of claim 14 when the computer program product is run on a processing unit of the computing device, the computing device comprising a brain control interface, a sensor and a processor.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above, as well as additional objects, features and advantages of the disclosed systems, devices and methods will be better understood through the following illustrative and non-limiting detailed description of embodiments of devices and methods, with reference to the appended drawings, in which:

[0039] FIG. 1 shows schematically an example of a brain control interface system for controlling a controllable device;

[0040] FIG. 2a shows schematically an example of a sensor signal and brain activity wherein a temporal correlation is present between the sensor signal and the brain activity;

[0041] FIG. 2b shows schematically an example of a sensor signal and brain activity wherein no temporal correlation is present between the sensor signal and the brain activity;

[0042] FIG. 2c shows schematically an example of a sensor signal and its threshold, and brain activity;

[0043] FIGS. 3a-3c show schematically various examples of determining a location of a user with respect to an environmental change; and

[0044] FIG. 4 shows schematically a method of controlling a controllable device located in an environment.

[0045] All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the invention, wherein other parts may be omitted or merely suggested.

DETAILED DESCRIPTION

[0046] FIG. 1 shows schematically an overview of a brain control interface system 100. The brain control interface system 100 comprises a brain control interface 120 (e.g. a head-worn device). The brain control interface 120 (BCI) is configured to detect brain signals indicative of brain activity of a user 160 in an environment 150. The BCI 120 may comprise one or more electrodes 122 in contact with the user's scalp, which electrodes 122 are used for detecting EEG signals of the user. It should be understood that such a BCI 120 is an example, and that other types of brain signal detection may be used. The system 100 further comprises a sensor 102 configured to detect (current/actual) changes of an environmental characteristic in the environment. The system 100 further comprises one or more processors 106 configured to determine if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user. The processor 106 is configured to, if the temporal correlation is not present, control a controllable device 130 according to a control command derived from the brain activity of the user 160. The processor 106 is further configured to, if the temporal correlation is present, refrain from controlling the controllable device 130 according to the control command.

[0047] The processor 106 is configured to control the controllable device 130. The processor 106 may be configured to control the controllable device 130 according to the control command by communicating control signals to the controllable device 130 (e.g. via Zigbee, BLE, Ethernet, etc.). The processor 106 may be comprised in the controllable device 130, and control the controllable device 130 directly. Alternatively, the processor 106 may be comprised in a remote device 170, or for example in the BCI 120, and the processor 106 may control the controllable device via a communication unit 104 configured to communicate the control signals to the controllable device 130.

[0048] The processor 106 may be comprised in a single device or distributed across multiple devices, which may depend on the system architecture of the BCI system 100. For instance, in the example of FIG. 1, the one or more processors 106 and the input 102 are comprised in a single device 170, which device 170 is communicatively coupled with the BCI 120 and the controllable device 130. It should be understood that this system architecture is merely an example, and that the skilled person is able to design alternative system architectures without departing from the scope of the appended claims. For instance, a first processor 106 may be comprised in the BCI 120, and a second processor 106 on a remote server or in the controllable device 130. In another example, the processor may be comprised in the BCI 120, in the controllable device 130, in a sensor device comprising the sensor 102, etc.

[0049] The system 100 comprises a brain control interface configured to detect brain activity of a user indicative of a control command for controlling the controllable device 130, and to derive the control command from the brain activity. The controllable device 130 may be a device configured to adjust an environmental characteristic that corresponds to the environmental characteristic detected by the sensor. The controllable device 130 may, for example, be a connected (home) appliance or connected (office) equipment. The controllable device 130 may comprise a receiver configured to receive control signals indicative of the control command, for instance via a wireless network. The controllable device 130 may, for example, be a connected speaker, a lighting device comprising one or more LED light sources, a thermostat, a tv, a (tablet) pc, a smartphone, a game console, etc.

[0050] The sensor 102 is configured to detect changes of an environmental characteristic in the environment. The sensor 102 may for example be a light sensor, and the environmental characteristic may be the environmental light level. The sensor 102 may for example be a temperature sensor, and the environmental characteristic may be the environmental temperature. The sensor 102 may for example be an audio sensor, and the environmental characteristic may be environmental audio. The sensor 102 may for example be a humidity sensor, and the environmental characteristic may be the environmental humidity. The sensor 102 may comprise multiple sensors each configured to detect a change of a respective different environmental characteristic.

[0051] The processor 106 is configured to determine if there is a temporal correlation between a detected change of the environmental characteristic (as detected by the sensor 102) and the detected brain activity of the user 160. The processor 106 may be configured to determine the presence of the temporal correlation between the detected change of the environmental characteristic and the detected brain activity of the user by determining if the detected change of the environmental characteristic and the detected brain activity of the user occur at substantially the same time or at least within a predetermined time window. The predetermined time window may be less than 1 second. Depending on the application of the brain control interface system, the predetermined time window may for example be less than 500 ms, or even less than 200 ms. For instance, the sensor 102 may be a light sensor and the controllable device 130 may be a lighting device. The lighting device may be configured to receive control signals from the processor 106 and comprise driver configured to adjust the light output of one or more (LED) light sources accordingly. The sensor 102 may detect a change in light level (e.g. due to switching on of a device such display, due to the sun appearing from behind the clouds, due to blinds opening, etc.) or a change in color (e.g. due to a display changing from red to blue, due to an automated change in color temperature of the ceiling lights). If the brain control interface 120 detects brain activity indicative of a control command for the controllable device 130 (e.g. a control command to switch the light on) at substantially the same time, the processor 106 may determine that there is a temporal correlation between the detected change of the environmental characteristic (the light level) and the detected brain activity of the user 160. If the temporal correlation is present, the processor 106 refrains from controlling the controllable device 130 according to the control command. If the temporal correlation is not present, the processor 106 controls the controllable device 130 according to the control command (e.g. the control command to switch the lighting device on).

[0052] FIGS. 2a and 2b show schematically examples of sensor signals ss (indicative of sensor readings of the sensor 102) and brain signals bs (indicative of the brain activity of the user 160) over time t. The brain signals bs are illustrated as a single line. This may, for example, correspond to the signal provided by a single electrode. It should be understood that multiple of such signals may be detected by different electrodes, and that the schematical examples provided in FIGS. 2a and 2b are provided for illustrative purposes. In FIG. 2a the increase of the sensor signal ss (i.e. the change of the environmental characteristic) and the peak in the brain signal bs (i.e. the brain activity indicative of the control command for the controllable device 130) occur substantially simultaneously. The processor 106 may thus determine that there is a temporal correlation between the between a detected change of the environmental characteristic and the detected brain activity of the user 160. The processor 106 may therefore refrain from controlling the controllable device 130 according to the control command. Alternatively, the detected change may be inverse. Hence, the increase of the sensor signal as depicted in FIG. 2a may be a similar shaped decrease of the sensor signal, which occurs substantially simultaneously with the peak in the brain signal. The sensor may then also detect a change of the environmental characteristic. The processor may also in such alternative situations determine that there is a temporal correlation between the detected change of the environmental characteristic and the detected brain activity of the user. In case of a gradual transition of the environmental characteristic (e.g. ceiling light fading in within 200 ms from off to 50% and then within 800 ms from 50% to 100%), the temporal correlation between the timeseries detected change of the environmental characteristic and the timeseries detected brain activity of the user will be analyzed. In FIG. 2b the increase of the sensor signal ss (i.e. the change of the environmental characteristic) and the peak in the brain signal bs (i.e. the brain activity indicative of the control command for the controllable device 130) at different times (e.g. outside a predetermined time window). The processor 106 may thus determine that there is no temporal correlation between the between a detected change of the environmental characteristic and the detected brain activity of the user 160. The processor 106 may therefore control the controllable device 130 according to the control command.

[0053] The processor 106 may be configured to determine if the detected change of the environmental characteristic exceeds a threshold, and to refrain from controlling the controllable device 130 according to the control command only if the environmental characteristic exceeds the threshold. The processor 106 may be further configured to, if the detected change of the environmental characteristic does not exceed the threshold, control the controllable device 130 according to the control command. FIG. 2c illustrates an example wherein the detected change of the environmental characteristic (signal ss) does not exceed a threshold th. The processor 106 may determine there is a temporal correlation between the between a detected change of the environmental characteristic and the detected brain activity of the user 160. So initially the processor 106 would refrain from controlling the controllable device 130 according to the control command, but since the change of the environmental characteristic (signal ss) does not exceed a threshold th, the processor 106 may control the controllable device 130 according to the control command. In examples, the threshold may be a threshold range, having an upper threshold limit and a lower threshold limit, wherein the signal does not exceed the threshold if the signal remains within said threshold range (i.e. between the lower and upper threshold limit).

[0054] The processor 106 may be further configured to determine a presence of a second correlation between the type of environmental characteristic and the (type of) detected brain activity of the user. The processor 106 may thus determine if the type of the environmental characteristic detected by the sensor 102 corresponds to an environmental characteristic that is to be changed by the brain activity. The controllable device 130 may be a device configured to adjust an environmental characteristic that corresponds to the environmental characteristic detected by the sensor 102. The processor 106 may be further configured to control the controllable device according to the control command if the temporal correlation is not present and the second correlation is present. The processor 106 may be further configured to control the controllable device according to the control command if the second correlation is not present and the temporal correlation is present. The processor 106 may be further configured to control the controllable device according to the control command if the temporal correlation and the second correlation are not present. The processor 106 may be further configured to refrain from controlling the controllable device according to the control command if the temporal correlation and the second correlation are present. For instance, if the environmental characteristic is of a first type (e.g. a light level in the environment) and the brain activity (and therewith the control command) is of a correlated type (e.g. a lighting control command), the processor 106 may determine that the second correlation is present. For instance, if the environmental characteristic is of a first type (e.g. a light level in the environment) and the brain activity (and therewith the control command) is of a non-correlated type (e.g. an audio control command), the processor 106 may determine that the second correlation is not present. For instance, if the environmental characteristic is of a first type (e.g. a temperature in the environment) and the brain activity (and therewith the control command) is of a correlated type (e.g. a color temperature of the light control command), the processor 106 may determine that the second correlation is present. The processor 106 may be configured to access a (local or remote) memory configured to store correlations between types of environmental characteristics and respective (types of) brain activities (and therewith (types of) control commands) of the user, and to determine the presence of the second correlation between the type of environmental characteristic and the (type of) detected brain activity of the user based on the stored correlations.

[0055] The processor 106 may be further configured to request the user 160 operating the brain control interface system 100 to control the controllable device 130 based on the control command when the processor 106 has refrained from controlling the controllable device according to the control command. The user 160 may then approve or disapprove the request via a user interface (e.g. a voice assistant, a touch screen, one or more buttons on a switch, etc.). The processor 106 may be communicatively coupled (e.g. wirelessly, directly) to the user interface. When the user 160 has approved the request via the user interface, the processor 106 may control the controllable device 130 according to the control command, and when the user has disapproved the request via the user interface, the processor 106 may again refrain from controlling the controllable device 130 according to the control command.

[0056] The processor 106 may be further configured to control a further device when the temporal correlation is present, wherein the control of the further device is based on the brain activity as a response to the detected change of the environmental characteristic. The processor 106 may be configured to communicate with the further device in a similar manner as with the controllable device 130. The further device may be the device that caused the change of the environmental characteristic. The processor 106 may be configured to determine which further device (of a plurality of further devices) has caused the change of the environmental characteristic, for instance based on sensor data from the sensor or based on a signal received from the further device. For instance, when an environmental light level changes, the processor 106 may determine that a light source in the environment caused this change, and control the light output of the light source based on the brain activity of the user (the manner in which the user responded to the change of the environmental characteristic).

[0057] The processor 106 may be further configured to determine whether to control the controllable device 130 further based on the location of the change of the environmental characteristic. The processor 106 may be configured to obtain first data indicative of the location of the change of the environmental characteristic. The processor 106 may, for example, obtain information about the location and/or orientation of the sensor 102 relative to the environment 150, and determine the location of the change of the environmental characteristic based thereon. This has been illustrated in FIG. 3a, wherein the location of the sensor 302a, and therewith the location of the change of the environmental characteristic 304a may be obtained by the processor 106. Alternatively, the sensor 102 may be configured to detect the location of the change of the environmental characteristic, and communicate this location to the processor 106. This has been illustrated in FIG. 3a, wherein the sensor 302b (e.g. a camera, a thermopile camera, an audio sensor, etc.) may have a field of view 312b, and the sensor 302b may be configured to determine the location of the change of the environmental characteristic 304b in its field of view 312b based on sensor signals (e.g. based on an image of the environment, based on the signal strength of the sensor signals, etc.), and determine the location of the change of the environmental characteristic relative to the environment based on the sensor's 302b location relative to the environment. The location of the sensor 102, 302a, 302b, 302c may be determined based on data from an (indoor) positioning system. Examples of such a positioning system include a radio frequency (RF) beacon system, a coded light positioning system, etc. Alternatively, the location of the sensor 102, 302a, 302b, 302c may have been defined by the user via a user interface. It should be understood that techniques for determining a location of a sensor/device are known in the art and will therefore not be discussed in detail.

[0058] The processor 106 may be further configured to obtain second data indicative of the location of the user 160 relative to the environment 150 or relative to the sensor 102. The second data may be obtained from an (indoor) positioning system. Examples of such a positioning system include a radio frequency (RF) beacon system, a coded light positioning system, etc. Alternatively, the location of the user 160 may have been defined by the user 160 via a user interface. It should be understood that techniques for determining a location of a user relative to an environment 150 are known in the art and will therefore not be discussed in detail. The processor 106 may be further configured to determine, based on the location of the change of the environmental characteristic and the location of the user, if the change of the environmental characteristic occurred within a predefined proximity of the user. If the change of the environmental characteristic has occurred outside the predefined proximity and if the temporal correlation is present, control the controllable device according to the control command. The predefined proximity may, for example, be defined as a distance, or be defined as an area (e.g. a (part of) a room in the environment 150). FIGS. 3a and 3b show examples of environments. In the example of FIG. 3a, the user 160a is located outside predefined proximity range 310a. The processor 106 may therefore control-even if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the userthe controllable device 130 according to the control command, for instance because the detected change of the environmental characteristic may be imperceivable by the user. In the example of FIG. 3b, the user 160b is located inside predefined proximity range 310b. The processor 106 may therefore refrain from controlling the controllable device 130 according to the control command, because the detected change of the environmental characteristic may be perceivable by the user.

[0059] The second data may be further indicative of an orientation of the user 160. The orientation may be obtained from an orientation sensor (e.g. from a magnetometer comprised in a user-worn or held device, from a remote camera, etc.) or the orientation of the user 160 may have been defined by the user via a user interface. The processor 106 may be further configured to determine whether to control the controllable device 130 further based on the orientation of the user relative to the change of the environmental characteristic. The processor 106 may be further configured to determine, based on the location of the change of the environmental characteristic and the orientation of the user, if the change of the environmental characteristic occurred within a field of view (FoV) of the user 160. If the change of the environmental characteristic has occurred outside the field of view of the user 160 and if the temporal correlation is present, control the controllable device 130 according to the control command. FIG. 3c shows two user-locations 160c and 160c. If the user would be located at location 160c, the processor 106 may determine that the change of the environmental characteristic (e.g. the environmental light level) has occurred within the FoV of the user 160c, and therefore determine to refrain from controlling the controllable device 130 based on the control command. If the user would be located at location 160c, the processor 106 may determine that the change of the environmental characteristic (e.g. the environmental light level) has occurred outside the FoV of the user 160c, and therefore determine to control the controllable device 130 based on the control command, because the detected change of the environmental characteristic may be imperceivable by the user.

[0060] FIG. 4 shows schematically a method 400 of controlling a controllable device located in an environment. The method comprises: detecting 402, by a brain control interface, brain activity of a user indicative of a control command for controlling the controllable device, deriving 404 the control command from the brain activity, detecting 406, by a sensor, changes of an environmental characteristic in the environment, determining 408, by a processor, if there is a temporal correlation between a detected change of the environmental characteristic and the detected brain activity of the user, and, if the temporal correlation is not present, controlling 410 the controllable device according to the control command, if the temporal correlation is present, refraining 412 from controlling the controllable device according to the control command.

[0061] The method 400 may be executed by computer program code of a computer program product when the computer program product is run on a computing system, such as the system 100.

[0062] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims.

[0063] In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. Use of the verb comprise and its conjugations does not exclude the presence of elements or steps other than those stated in a claim. The article a or an preceding an element does not exclude the presence of a plurality of such elements. The invention may be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer or processing unit. In the device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0064] Aspects of the invention may be implemented in a computer program product, which may be a collection of computer program instructions stored on a computer readable storage device which may be executed by a computer. The instructions of the present invention may be in any interpretable or executable code mechanism, including but not limited to scripts, interpretable programs, dynamic link libraries (DLLs) or Java classes. The instructions can be provided as complete executable programs, partial executable programs, as modifications to existing programs (e.g. updates) or extensions for existing programs (e.g. plugins). Moreover, parts of the processing of the present invention may be distributed over multiple computers or processors or even the cloud.

[0065] Storage media suitable for storing computer program instructions include all forms of nonvolatile memory, including but not limited to EPROM, EEPROM and flash memory devices, magnetic disks such as the internal and external hard disk drives, removable disks and CD-ROM disks. The computer program product may be distributed on such a storage medium, or may be offered for download through HTTP, FTP, email or through a server connected to a network such as the Internet.