COMPUTER-IMPLEMENTED CONTINUOUS CONTROL METHOD, SYSTEM AND COMPUTER PROGRAM

Abstract

The present disclosure relates to improving ways of making decisions made by, or with the assistance of, computing systems. One aspect relates to a computer-implemented continuous control method, the method comprising: obtaining a policy decision according to an application of a policy to an inference; the inference being of an authorised user's state of mind over a time interval, determined according to a plurality of biometric readings obtained passively from the user over the time interval for continuous authentication of the user; the user's identity having been continuously authenticated according to the plurality of biometric readings; and causing one or more actions in accordance with the policy decision. Further aspects relate to a data processing system comprising means for carrying out such a method, a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out such a method, a computer-readable data carrier having stored thereon such a computer program and a data carrier signal carrying such a computer program.

Claims

1. A computer-implemented continuous control method comprising: obtaining a policy decision according to an application of a policy to an inference; the inference being of an authorised user's state of mind over a time interval, determined according to a plurality of biometric readings obtained passively from the user over the time interval for continuous authentication of the user; the user's identity having been continuously authenticated according to the plurality of biometric readings; and causing one or more actions in accordance with the policy decision.

2. The method of claim 1, further comprising determining the inference prior to obtaining the policy decision.

3. The method of either of claims 1, further comprising obtaining the plurality of biometric readings prior to obtaining the policy decision.

4. The method of claim 3, wherein the plurality of biometric readings comprise a plurality of biometric readings obtained from each of a plurality of user devices.

5. The method of any of claim 1, wherein the plurality of biometric readings comprise a respective plurality of biometric readings obtained for each of a plurality of biometrics.

6. The method of claim 5, wherein determining the inference comprises: for each of the plurality of biometrics, making a biometric-specific inference of the user's state of mind over the time interval in dependence on the respective plurality of biometric readings; associating each of the biometric-specific inferences with a confidence score; and determining the inference by combining all of the biometric-specific inferences, each weighted in dependence on its associated confidence score.

7. The method of claim 6, further comprising, for each of the biometric-specific inferences: obtaining contextual data; and determining the confidence score in dependence thereon.

8. The method of claim 1, further comprising authenticating the user's identity according to the plurality of biometric readings prior to obtaining the policy decision.

9. The method of claim 1, wherein the one or more actions are one or more of: denying a request for functionality received from the user through a user interface; granting a request for functionality received from the user through a user interface; providing feedback to the user through a user interface; communicating feedback to a separate computing device; and scheduling a future action.

10. The method of claim 1, wherein the inference is that the user is one or more of: in a particular emotional state selected from a plurality of emotional states; and in a particular physiological condition selected from a plurality of physiological conditions.

11. The method of claim 2, wherein determining the inference is performed by a classifier, the classifier optionally being trained on biometric data collected from a plurality of individuals, the classifier optionally comprising a support vector machine or artificial neural network.

12. A data processing system comprising means for carrying out the method of claim 1.

13. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 1.

14. A computer-readable data carrier having stored thereon the computer program of claim 13.

15. A data carrier signal carrying the computer program of claim 13.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0039] Aspects of the present disclosure will now be described by way of example with reference to the accompanying figures. In the figures:

[0040] FIG. 1 illustrates an example system 100 in which the approach of the present disclosure can be employed;

[0041] FIG. 2 is a flowchart illustrating an example computer-implemented continuous control method; and

[0042] FIG. 3 schematically illustrates an example data processing system capable of performing the method of FIG. 2.

DETAILED DESCRIPTION OF THE FIGURES

[0043] The following description is presented to enable any person skilled in the art to make and use the system, and is provided in the context of a particular application. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art.

[0044] The present disclosure provides a computer-implemented method in which an inference of a user's state of mind is used to determine a suitable action in accordance with a policy. The inference is based on biometric measurements already taken from the user to continuously authenticate their identity. In this way, in addition to the security provided by continuous authentication, improvements to the functioning of the computer system and/or the user's experience of it can be made by automatically adapting to/taking into account the user's mood or other mental state. The method is performed “in the background” from the user's perspective, with no unnecessary action being required of them and no unnecessary delays being introduced. The continuous nature of the biometric data collection can also increase the difficulty of spoofing the data, enabling increased security.

[0045] Biometrics are measurable, distinctive characteristics of a human which can be used to label and describe individuals. Individuals can therefore be identified using one, or a combination, of their biometrics. Biometrics include physiological characteristics and behavioural characteristics.

[0046] The biometric measurements on which the inference is based can for example comprise one or more of: [0047] facial images, [0048] pulse measurements, [0049] gait measurements, [0050] breathing pattern measurements, [0051] chemical signature measurements (e.g. from breath and/or perspiration), [0052] voice recordings, [0053] handwriting scans, [0054] handling signature measurements (e.g. one or more of orientation, direction and/or speed and/or acceleration of translational and/or rotational motion, holding pressure, frequency of interaction and/or changes in and/or patterns of changes in one or more of these) [0055] user interface interaction signature measurements (e.g. characteristic ways of one or more of typing, pressing buttons, interacting with a touch sensitive or gesture control device and viewing a display, for example determined through one or more of: force and pressure on a tactile interface; speed, rhythm, frequency, style and duration of interaction with a tactile or gesture based interface; and visual tracking of a display), and [0056] linguistic analysis measurements (e.g. from free text type and/or voice recordings).

[0057] Continuous authentication refers to authentication which takes place on an on-going basis. This is in contrast to traditional authentication, which is prompted by a specific external stimulus indicating a request for functionality requiring authentication. (In the traditional case, the request for functionality could be specific, for example requesting access to a protected file, or more general, for example requesting log-in to a device which then enables multiple functions of that device.) Continuous authentication is based on measurements obtained from a user passively, i.e. without the user being required to knowingly perform any particular prompted or remembered action.

[0058] Measurements to achieve continuous biometric authentication can be taken by sampling one or more continuous sensor outputs and/or by triggering one or more sensors as required. Measurements can be taken continually; i.e. one after another, as quickly as the measurement apparatus allows. Alternatively, measurements can be taken on a routine basis. For example a measurement or series of measurements could accompany any action or any of a class of actions (as opposed to a specific action) implemented on or by the device, e.g. handling of the device and/or use of any user input device comprised in the device and/or receipt or transmission of a communication by the device. Measurements could alternatively be taken on a particular temporal basis, for example a regular (e.g. periodic) basis, according to some other temporal pattern or randomly triggered (e.g. according to a stochastic variable).

[0059] FIG. 1 illustrates an example system 100 in which the approach of the present disclosure can be employed. A smartphone 110 comprises a front-facing camera 111 for capturing images of a user's face. A smartwatch 120, in wireless communication with the smartphone 110 (for example via Bluetooth™) comprises a pulse monitor, an accelerometer and a gyroscope (not shown). A pair of earbuds 130, connected to the smartphone 110 via a wire 131, comprises a microphone 132. The smartphone 110 is also in wireless (e.g. cellular) communication with one or more other computing devices such as servers (not shown) via a network 140 (e.g. the Internet).

[0060] Biometric measurements can be taken from one or more of the illustrated devices. The camera 111 of the smartphone 110 can be used to perform facial recognition for continuous authentication of the user's identity, as well as expression recognition to infer an emotional or physiological state of the user. The pulse monitor of the smartwatch 120 can be used to perform both continuous authentication and state of mind inference based on the user's heart rate. The accelerometer and gyroscope of the smartwatch 120 can feed into gait analysis to both continuously authenticate the user and infer their emotional and/or physiological state. The combination of data from the pulse monitor, accelerometer and gyroscope can be used to infer when the user is exercising, and modify inferences made based on their biometrics accordingly. Finally, the microphone can be used to continuously authenticate the user via voice recognition and infer their state of mind from one of more of their vocal pitch, volume, rhythm and speed.

[0061] It can be seen from the system 100 shown in FIG. 1 and described above that one or more biometrics can be used to both continuously authenticate a user and infer their state of mind. Where multiple biometrics are used, these can be collected by a single device or multiple devices. Processing of the raw biometric data collected can be performed by the device the data is collected by, or by another device in communication with it as appropriate in dependence on relative processing power, electrical power and memory available at each device. Some or all of the processing could be done by a device not illustrated, such as a remote server, communicably coupled to the smartphone 110 via the network 140.

[0062] Once the user has been continuously authenticated and their state of mind inferred, a policy is applied to determine an action to take. The policy could be stored on the device determining the action or remotely, but accessible by it. The policy could be set by the user or by a third party such as their employer. The policy could be fixed or open to modification. Such modification could be according to administrator action and/or automatic according to a machine learning algorithm. By way of example, a policy could be defined as follows.

TABLE-US-00001 State of mind Action(s) Angry Divert outgoing email to Drafts folder & notify user Inebriated Disable connected car ignition & notify user breath sample must be provided (and alcohol level test passed) to reenable Divert outgoing email to Drafts folder & notify user Calm Enable all functions Remind user of tasks on to do list Log time interval over which calm state of mind can be inferred to inform automatic scheduling of future tasks Frightened Deny access to banking application Notify emergency contact Stressed Disable incoming email notifications & turn on auto- reply with alternative contact details for urgent matters Fatigued Suggest taking a break

[0063] FIG. 2 is a flowchart illustrating an example computer-implemented continuous control method 200. The method 200 comprises obtaining a policy decision according to an application of a policy to an inference at step 240, then causing one or more actions in accordance with the policy decision at step 260. The inference is of an authorised user's state of mind over a time interval, determined according to a plurality of biometric readings obtained passively from the user over the time interval for continuous authentication of the user (the user's identity having been continuously authenticated according to the plurality of biometric readings). Obtaining the policy decision at step 240 can comprise determining the policy decision or receiving a policy decision made by a separate computing device.

[0064] The time interval can for example be between a tenth of a second and one minute. It can depend on the type of biometric readings. For example, photographs to determine facial expression are typically sampled at high frequency but over short time intervals, since facial expression can change from second to second. In contrast, heart rate measurements are typically taken over an interval of around a minute to reduce the risk of brief spikes being mischaracterised.

[0065] The method 200 can optionally comprise determining the inference at step 230, prior to obtaining the policy decision at step 240. Alternatively, the inference can be determined by a separate computing device and communicated to the computing device conducting the method 200.

[0066] The inference can optionally be that the user is one or more of: [0067] in a particular emotional state selected from a plurality of emotional states, such as stressed, afraid, angry or calm; and [0068] in a particular physiological condition selected from a plurality of physiological conditions, such as under the influence of a mind-altering substance, suffering from a medical condition or experiencing fatigue.

[0069] Determination of the inference at optional step 230 can optionally be performed by a classifier. The classifier can optionally comprise a support vector machine or artificial neural network. The classifier can optionally be trained on biometric data collected from a plurality of individuals. It can be refined for use by a particular user by means of a calibration process in which inferences are made and presented to the user for them to confirm or refute.

[0070] Determining the inference at optional step 230 can optionally comprise: [0071] for each of the plurality of biometrics, making a biometric-specific inference of the user's state of mind over the time interval in dependence on the respective plurality of biometric readings at step 232; [0072] associating each of the biometric-specific inferences with a confidence score at step 236; and [0073] determining the inference by combining all of the biometric-specific inferences, each weighted in dependence on its associated confidence score at step 238.

[0074] Weighting factors applied to each of the biometric-specific inferences can be proportional to the respective associated confidence scores or could be determined from the confidence scores in some other way. For example, confidence scores could be grouped into bands, with each band corresponding to a single weighting factor value. In either of these schemes, or any other, a confidence threshold could be applied such that the weighting factor value is set to zero if the confidence score is below a predetermined minimum.

[0075] The confidence scores can for example be based on known accuracies of biometric-specific inferences drawn from particular biometrics and/or known error bars associated with the biometric readings used to determine the biometric-specific inferences. Confidence scores can alternatively or additionally depend on the age of the associated biometric readings. For example, if a first type of biometric reading is updated more frequently then a second type of biometric reading, the confidence score associated with the first type of biometric reading could be higher than that associated with the second type of biometric reading.

[0076] The confidence scores could be fixed, or updateable by the user performing a calibration process in which biometric-specific inferences are made and presented to the user for them to confirm or refute.

[0077] The use of multiple biometrics to determine the inference can thus allow the inference to be drawn with reasonable confidence even if an individual biometric would not generally be sufficient. For example, an elevated heart rate alone could indicate several different physiological and emotional states, e.g. angry, stressed, frightened, excited or under the influence of certain drugs. However, an elevated heart rate could be used to corroborate an inference of one of those states from expression recognition in a facial image.

[0078] Optionally, optional step 230 can further comprise, for each of the biometric-specific inferences: [0079] obtaining contextual data, for example comprising one or more contextual measurements over the time interval at step 234; and [0080] determining the confidence score in dependence thereon at step 235.

[0081] For example, in the system 100 of FIG. 1, a contextual measurement associated with a facial image taken by the camera 111 of the smartphone 110 could be ambient light level. A confidence level associated with a biometric-specific inference drawn from expression detection based on that image could be lowered if the light level is below a predetermined threshold. Similarly, the accelerometer and gyroscope of the smartwatch 120 can be used to provide contextual measurements for the heart rate obtained from its pulse monitor. If the accelerometer and gyroscope measurements suggest the user is, or has recently been, exercising then a confidence level associated with a biometric-specific inference drawn from an elevated heart rate (e.g. that the user is stressed) can be reduced relative to a confidence level associated with that biometric-specific inference if the accelerometer and gyroscope measurements suggest that the user is stationary. Other contextual data to inform the inference could also be retrieved from the computing device's memory or received from another computing device. For example, the confidence level associated with the biometric-specific inference drawn from the elevated heart rate that the user is stressed could be reduced further if the user's calendar includes a recently completed gym class.

[0082] The use of contextual measurements to adjust confidence levels associated with biometric-specific inferences allows one or more secondary biometrics to be used as “back-ups” for a primary biometric-specific inference, in the event that the contextual measurements suggest that the primary biometric-specific inference may not be as accurate as usual at that time, or is for some reason unavailable. In some implementations, resources such as memory, processing and electrical power can be saved by only determining secondary biometric-specific inferences if the confidence level(s) associated with the primary biometric-specific inference is below a predetermined threshold.

[0083] The method 200 can optionally comprise obtaining the plurality of biometric readings at step 210 prior to obtaining the policy decision at step 240. Alternatively, if a separate computing device determines the inference then that same separate computing device obtains the plurality of biometric readings in order to do so. (If the same computing device is determining the inference and determining the policy decision, then it must obtain the plurality of biometric readings prior to performing either of those steps.) Obtaining the biometric readings at optional step 210 can comprise directly making biometric measurements or receiving them from a separate computing device. The plurality of biometric readings can comprise a plurality of biometric readings obtained from each of a plurality of user devices. The plurality of biometric readings can comprise a plurality of biometric readings obtained for each of a plurality of biometrics.

[0084] The method 200 can optionally comprise authenticating the user's identity according to the plurality of biometric readings at step 220 prior to obtaining the policy decision at step 240. Alternatively, the authentication can be performed by a separate computing device. The device which authenticates the user's identity must obtain the plurality of biometric readings in order to do so. Authentication of the user's identity and determination of the inference can therefore be performed by the same computing device. Alternatively, these actions can be performed by separate computing devices provided that both are able to obtain the requisite plurality of biometric readings.

[0085] The one or more actions caused in accordance with the policy decision at step 260 can optionally comprise one or more of: [0086] denying a request for functionality received from the user through a user interface at step 261; [0087] granting a request for functionality received from the user through a user interface at step 262; [0088] providing feedback to the user through a user interface at step 263; [0089] communicating feedback to a separate computing device at step 264; and [0090] scheduling a future action at step 265.

[0091] FIG. 3 schematically illustrates a data processing system 300 capable of performing the method 200. It comprises a processor 310 operably coupled to both a memory 320 and an interface 330.

[0092] The memory 320 can optionally comprise instructions which, when the program is executed by the processor 310, cause the data processing system 300 to carry out the method 200. Alternatively or additionally, the interface 330 can optionally comprise one or both of a physical interface 331 configured to receive a data carrier having such instructions stored thereon and a receiver 332 configured to receive a data carrier signal carrying such instructions.

[0093] The interface 330 can optionally comprise a receiver 332 configured to receive the inference.

[0094] The interface 330 can optionally comprise one or more biometric devices 333 configured to directly measure one or more of the plurality of biometric readings. Alternatively or additionally, the interface 330 can optionally comprise a receiver 332 configured to receive one or more of the plurality of biometric readings.

[0095] The interface 330 can optionally comprise one or more contextual measurement devices 334 configured to directly take one or more contextual measurements. Alternatively or additionally, the interface 330 can optionally comprise a receiver 332 configured to receive contextual data.

[0096] The interface 330 can optionally comprise one or more user interface devices 335 which can for example be configured to receive requests for functionality from the user and/or provide feedback to the user. Alternatively or additionally, the interface 330 can optionally comprise a transmitter 336 configured to communicate feedback to a separate computing device.

[0097] The receiver 332, when present, can comprise one or more wireless receiver modules and/or one or more wired receiver modules. Similarly, the transmitter 336, when present, can comprise one or more wireless transmitter modules and/or one or more wired transmitter modules.

[0098] Other embodiments will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. It is intended that the specification and examples be considered as exemplary only.

[0099] In addition, where this application has listed the steps of a method or procedure in a specific order, it could be possible, or even expedient in certain circumstances, to change the order in which some steps are performed, and it is intended that the particular steps of the method or procedure claims set forth herein not be construed as being order-specific unless such order specificity is expressly stated in the claim. That is, the operations/steps may be performed in any order, unless otherwise specified, and embodiments may include additional or fewer operations/steps than those disclosed herein. It is further contemplated that executing or performing a particular operation/step before, contemporaneously with, or after another operation is in accordance with the described embodiments.

[0100] The methods described herein may be encoded as executable instructions embodied in a computer readable medium, including, without limitation, non-transitory computer-readable storage, a storage device, and/or a memory device. Such instructions, when executed by a processor (or one or more computers, processors, and/or other devices) cause the processor (the one or more computers, processors, and/or other devices) to perform at least a portion of the methods described herein. A non-transitory computer-readable storage medium includes, but is not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, compact discs (CDs), digital versatile discs (DVDs), or other media that are capable of storing code and/or data.

[0101] Where a processor is referred to herein, this is to be understood to refer to a single processor or multiple processors operably connected to one another. Similarly, where a memory is referred to herein, this is to be understood to refer to a single memory or multiple memories operably connected to one another.

[0102] The methods and processes can also be partially or fully embodied in hardware modules or apparatuses or firmware, so that when the hardware modules or apparatuses are activated, they perform the associated methods and processes. The methods and processes can be embodied using a combination of code, data, and hardware modules or apparatuses.

[0103] Examples of processing systems, environments, and/or configurations that may be suitable for use with the embodiments described herein include, but are not limited to, embedded computer devices, personal computers, server computers (specific or cloud (virtual) servers), hand-held or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, network personal computers (PCs), minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. Hardware modules or apparatuses described in this disclosure include, but are not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated or shared processors, and/or other hardware modules or apparatuses.

[0104] User devices can include, without limitation, static user devices such as PCs and mobile user devices such as smartphones, tablets, laptops and smartwatches.

[0105] Receivers and transmitters as described herein may be standalone or may be comprised in transceivers. A communication link as described herein comprises at least one transmitter capable of transmitting data to at least one receiver over one or more wired or wireless communication channels. Wired communication channels can be arranged for electrical or optical transmission. Such a communication link can optionally further comprise one or more relaying transceivers.

[0106] User input devices can include, without limitation, microphones, buttons, keypads, touchscreens, touchpads, trackballs, joysticks and mice. User output devices can include, without limitation, speakers, buzzers, display screens, projectors, indicator lights, haptic feedback devices and refreshable braille displays. User interface devices can comprise one or more user input devices, one or more user output devices, or both.