Identifying sensory inputs affecting working memory load of an individual

Abstract

In an aspect of the invention, a method of identifying sensory inputs affecting working memory load of an individual is provided. The method comprises monitoring (S101) working memory load of the individual using a sensor device, detecting (S102) an increase in the working memory load of the individual, and identifying (S103), in response to the detected increase, at least one sensory input affecting the working memory load of the individual.

Claims

1. A device comprising: processing circuitry, wherein the device is configured to: monitor working memory load of an individual; identify a first sensory input that is impacting the working memory load of the individual; and after identifying the first sensory input, diminish the impact that the first sensory input has on the working memory load of the individual.

2. The device of claim 1, wherein identifying the first sensory input comprises detecting a sensory input coinciding in time with an increase in the working memory load of the individual.

3. The device of claim 1, wherein diminishing the impact that the first sensory input has on the working memory load of the individual comprises: controlling a source of the first sensory input to diminish the impact that the first sensory input has on the working memory load of the individual.

4. The device of claim 1, wherein the diminishing the impact that the first sensory input has on the working memory load of the individual comprises: initiating a countermeasure to the first sensory input to diminish the impact that the first sensory input has on the working memory load of the individual.

5. The device of claim 1, wherein identifying the first sensory input comprises: identifying a plurality of sensory inputs; and for each identified sensory input, determining a measure with which said identified sensory input is impacting the working memory load of the individual.

6. The device of claim 5, wherein the device is further configured to, based on said determined measure, determine the sensory input having the largest impact on the working memory load of the individual, wherein the sensory input determined to have the largest impact on the working memory load of the individual is the first sensory input.

7. The device of claim 5, wherein the device is further configured to: store said measures in a database; and use the stored measures to determine one or more high-impact sensory input types.

8. The device of claim 5, wherein identifying the first sensory input further comprises subjecting the individual to the plurality of sensory inputs.

9. The device of claim 5, wherein the plurality of sensory inputs comprises two or more of: an audible input, a visual input, a change in temperature, a change in humidity, or an odor.

10. The device of claim 1, wherein monitoring the working memory load of an individual comprises one or more of: monitoring one or both eyes of the individual; monitoring the individual's heartbeat; or monitoring the individual's brain activity.

11. The device of claim 1, wherein the device is further configured to detect an increase in the working memory load of the individual.

12. The device of claim 11, wherein the identifying is performed as a result of detecting the increase in the working memory load of the individual.

13. The device of claim 11, wherein detecting the increase in the working memory load of the individual comprises detecting an increase in a pupil size.

14. The device of claim 1, wherein the first sensory input is a noise, and diminishing the impact that the first sensory input has on the working memory load of the individual comprises producing an out-of-phase representation of the noise.

15. The device of claim 1, wherein the device is further configured to: identify a second sensory input that is impacting the working memory load of the individual; and after identifying the second sensory input, diminish the impact that the second sensory input has on the working memory load of the individual.

16. The device of claim 1, wherein the first sensory input is an audible input, a visual input, a change in temperature, a change in humidity, or an odor.

17. A method comprising: monitoring working memory load of an individual; identifying a first sensory input that is impacting the working memory load of the individual; and after identifying the first sensory input, diminishing the impact that the first sensory input has on the working memory load of the individual.

18. The method of claim 17, wherein identifying the first sensory input comprises detecting a sensory input coinciding in time with an increase in the working memory load of the individual.

19. The method of claim 17, wherein diminishing the impact that the first sensory input has on the working memory load of the individual comprises: controlling a source of the first sensory input to diminish the impact that the first sensory input has on the working memory load of the individual.

20. The method of claim 17, wherein the diminishing the impact that the first sensory input has on the working memory load of the individual comprises: initiating a countermeasure to the first sensory input to diminish the impact that the first sensory input has on the working memory load of the individual.

21. The method of claim 17, wherein monitoring the working memory load of an individual comprises one or more of: monitoring one or both eyes of the individual; monitoring the individual's heartbeat; or monitoring the individual's brain activity.

22. The method of claim 17, further comprising detecting an increase in the working memory load of the individual.

23. The method of claim 22, wherein the identifying is performed as a result of detecting the increase in the working memory load of the individual.

24. The method of claim 17, wherein the first sensory input is a noise, and diminishing the impact that the first sensory input has on the working memory load of the individual comprises producing an out-of-phase representation of the noise.

25. The method of claim 17, wherein the first sensory input is an audible input, a visual input, a change in temperature, a change in humidity, or an odor.

26. A non-transitory computer readable medium storing a computer program comprising computer-executable instructions which when executed by processing circuitry of a device causes the device to perform the method of claim 17.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The invention is now described, by way of example, with reference to the accompanying drawings, in which:

(2) FIG. 1 illustrates a device for identifying sensory inputs affecting working memory load of an individual, according to an embodiment of the invention;

(3) FIG. 2 shows a top view of the device of FIG. 1 with a user seated in front of the device;

(4) FIG. 3 illustrates a flowchart of a method of identifying sensory inputs affecting working memory load of an individual, according to an embodiment of the invention;

(5) FIG. 4 illustrates a flowchart of a method of identifying sensory inputs affecting working memory load of an individual, according to another embodiment of the invention;

(6) FIG. 5 shows a top view of the device of FIG. 1 with a user seated in front of the device equipped with a pair of headphones with noise-reducing capability;

(7) FIG. 6 illustrates a flowchart of a method of identifying sensory inputs affecting working memory load of an individual, according to yet another embodiment of the invention;

(8) FIG. 7 illustrates a flowchart of a method of identifying sensory inputs affecting working memory load of an individual, according to a further embodiment of the invention;

(9) FIG. 8 illustrates a flowchart of a method of identifying sensory inputs affecting working memory load of an individual, according to yet a further embodiment of the invention; and

(10) FIG. 9 illustrates a device for identifying sensory inputs affecting working memory load of an individual, according to another embodiment of the invention.

DETAILED DESCRIPTION

(11) The invention will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Like numbers refer to like elements throughout the description.

(12) FIG. 1 illustrates a device 10 for identifying sensory inputs affecting working memory load of an individual according to an embodiment of the invention. The device is and shows a device exemplified in the form of a desktop computer 10 in a front view, having a screen 11, a camera 12, a microphone 13, and a loudspeaker 14.

(13) FIG. 2 shows a top view of the desktop computer 10, with a user 20 seated in front of it. As can be seen, the camera 12, the microphone 13, and the loudspeaker 14 are operatively coupled to a processing unit 15 embodied in the form of one or more microprocessors arranged to execute a computer program 16 downloaded to a suitable storage medium 17 associated with the microprocessor 15, such as a Random Access Memory (RAM), a Flash memory, a hard disk drive, a cloud service or other information storage devices. The processing unit 15 is arranged to control operation of the desktop computer 10 when the appropriate computer program 16 comprising computer-executable instructions is downloaded to the storage medium 17 and executed by the processing unit 15. The storage medium 17 may also be a computer program product comprising the computer program 16. Alternatively, the computer program 16 may be transferred to the storage medium by means of a suitable computer program product, such as a Digital Versatile Disc (DVD) or a memory stick. As a further alternative, the computer program may be downloaded to the storage medium 17 over a network. The processing unit 15 may alternatively be embodied in the form of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

(14) As discussed hereinbefore, the computer 10 is in an embodiment configured to identify events that have an unwanted effect on the focus of the user, e.g., whilst the user performs a cognitively demanding task.

(15) As can be seen in FIG. 2, the user 20 of the computer 10 is, when seated in front of the screen 11, positioned in a field of view of the camera 12. A method of identifying sensory inputs affecting working memory load of an individual—i.e., the user 20—will be described in the following with reference to FIG. 2 and further to FIG. 3 illustrating a flowchart of the method.

(16) A sensor device is used to monitor working memory load of an individual in step S101. In this particular exemplifying embodiment, the sensor device is embodied by the camera 12 which monitors one or both eyes of the user 20. Now, if the user 20 becomes distracted by a sensory input, the working memory load will increase, which typically results in an increase of the pupils of the user's eyes.

(17) In an embodiment of the invention, it is envisaged that if the diameter of the pupil of one of the user's eyes increase above a threshold value, for instance 0.5 mm, the working memory load of the user 20 is considered to have increased. It is further envisaged that different threshold values are used; for example, a 0.2 mm increase represents a first working memory load value A, a 0.4 mm increase represents a second working memory load value B, a 0.6 mm increase represents a third working memory load value C, and so on.

(18) In step S102, the camera 12 (or the processing unit 15 analysing images captured by the camera 12), detects an increase in the working memory load of the user 20, for instance by concluding that the diameter of the pupils of the user's eyes has increased above a certain threshold value.

(19) In response to the detected increase in the user's working memory load, a sensory input detection device—in this particular embodiment being exemplified by the processing unit 15 receiving signals from the microphone 13—advantageously identifies one or more sensory inputs affecting the working memory load of the user 20 in step S103. In this example, the built-in microphone 13 of the computer 20 registers a sound being likely to be the sensory input causing the detected change in working memory load of the user 20.

(20) In an embodiment, the sensory input, in this example being a sound, is advantageously identified by the processing unit 15 to coincide in time with the detected increase in the working memory load, thereby being considered to be the sensory input causing the increase. Thus, if the sound was recorded just before the increase in working memory load was detected, the sound is considered be the sensory input causing the increase.

(21) In an embodiment, upon having identified a sensory input affecting the working memory load of the user 20, actions are taken to diminish the impact that the sensory input has on the working memory load, as will be discussed in the following.

(22) In the embodiment illustrated in the flowchart of FIG. 4, after the microphone 13 has recorded a sound being considered to be the sensory input causing the increase in working memory load in step S103, the processing unit 15 of the computer 10 concludes that the user 20 herself just started an audio player of the computer 10 and advantageously lowers a sound level being output by the audio player via the loudspeaker 14 of the computer 10 in step S104a, since the selected audio level is too high and thus affects the working memory load of the user 20. Hence, in this particular embodiment, the impact that the identified sensory input has on the working memory load of the user is diminished using the processing unit 15 controlling the very source of the identified sensory input, in this case being the computer loudspeaker 14.

(23) In another embodiment illustrated with reference to FIGS. 5 and 6, the impact of the sensory input on the user's working memory load is diminished by initiating a countermeasure to the identified one sensory input.

(24) As can be seen in FIG. 5, the user 20 wears a pair of headphones 21 which in this example is equipped with noise-reducing capability. Now, the camera 12 of the computer 10 in cooperation with the processing unit 15 monitor and detect an increase in the working memory load of the user 20 in steps S101 and S102 as illustrated in FIG. 6, and a processing unit and microphone (not shown) of the headphones 21 serve as a sensory input detection device for registering disturbing background noise in step S103, as illustrated with the identified sound 22.

(25) After receiving a wireless or wired signal 23 from the computer 10 indicating that an increase in working memory load has been detected in step S102, the processing unit of the noise-reducing headphones 21 advantageously initiates a countermeasure to the identified sensory input by subjecting the individual to a signal 24 which is an out-of-phase representation of the background noise 23 in step S104b, thereby effectively cancelling out the background noise.

(26) In the above discussed embodiments, the camera 12 is used as a sensor device for monitoring and detecting an increase in working memory load of the user 20 in cooperation with the processing unit 15, by detecting changes in the pupil size of the user's eyes. It may further be envisaged that the camera 12 is used as a gaze detector with capability to track a direction of the user's visual attention and its duration.

(27) Generally, people have a tendency to look at an object when using it for a task—someone working at a laptop will spend most of the task time looking at the laptop. Hence, if their gaze is suddenly shifted, this may be an indication that they have been distracted. When gaze is shifted to another object that is providing a sensory input (e.g., making a sound, displaying changing images, etc.), this may indicate that that object is causing distraction.

(28) However, people may also shift gaze to concentrate, or for inspiration, such as looking skywards.

(29) By tracking the user's gaze, and in particular recording the times she changes gaze from a device she is using for her task and toward an object providing a sensory input, a measurement can be made as to how often the user is likely being distracted. A signature ‘inspiration’ or ‘concentration’ gaze can be learnt for a given user and discounted from this measurement.

(30) Further, it is envisaged that a worn device equipped with a camera is used as a sensor device for monitoring and detecting an increase in working memory load of the user in cooperation with a processing unit of the worn device, such as a virtual reality (VR) headset or a Google Glass type eyewear.

(31) Moreover, activity tracking sensor devices that is able to identify specific activities the user is engaged in and possibly the user's efficiency in these activities is envisaged.

(32) Similar to gaze, a person's motion can be used to assess the likelihood they are being distracted. Hence if the sensor device is a motion tracking sensor, and if the user shifts position often, particular in response to an object providing sensory input (e.g., a television) then the sensor's output can provide an indication that object is distracting. Therefore, by monitoring the person's motion, and if possible correlating this with objects in the room, then a measure can be made as to whether they are being distracted and by what.

(33) An activity tracking sensor device may also track the progress of a task, for example number of words written in a document, increase in size of a file of a drawing file, number of cells adjusted in an Excel file etc.

(34) Various sensor devices other than cameras can be envisaged, e.g., EEG sensors, EKG sensors, heart rate meters, etc.

(35) Different properties may further be combined to detect an increase in the working memory load of the user, for instance by considering a combination of two or more of pupil size, gaze, heart rate, activity, EEG, etc.

(36) Further, a number of types of devices for identifying sensory inputs affecting working memory load of an individual can be envisaged, such as laptops, tablets, smart phones, smart watches (using for instance heart rate as a measure of working memory load), television sets, etc.

(37) In yet another embodiment, the user 20 is during a learning phase of the device 10 purposely subjected to different sensory inputs while changes in the working memory load of the individual are monitored. Further, a measure associated with each particular sensory input is estimated and stored in a database for subsequent use.

(38) For instance, for any given user, a nominal working memory load may be recorded when the user practically is not subjected to any (disturbing) sensory input in her work environment. This lowest working memory load is denoted “Load.sub.NOM”, and corresponds to pupil size denoted “PupilSize.sub.NOM”.

(39) Now, the user is subjected to different sensory inputs, and an increase in pupil size of the user's eyes is detected and a corresponding increase in working memory load, as will be illustrated in Table 1 herein below.

(40) Hence, in an embodiment of the invention, a representation of any identified sensory input, and the corresponding measure with which said any identified sensory input affects the working memory load of the individual, is entered in a database as illustrated in Table 1.

(41) Reference is further made to the flowchart of FIG. 7.

(42) In a first round, the user 20 is subjected to three different sound levels of music selected from one of her playlists (i.e., music that the user indeed appreciates) and played through the loudspeaker 14 of her computer, while the camera 12 monitors the pupil size of the user's eyes in step S101 and the processing unit 15 detects a 0.1 mm increase, a 0.3 mm increase and a 0.5 mm increase, respectively, for the three (increasing) sound levels Sound level 1, Sound level 2, and Sound level 3, in step S102, which are considered in step S105 to correspond to a 10%, 30% and 50% increase in working memory load with respect to Load.sub.NOM. The different sounds are identified by the microphone 13 and the processing unit 15 in step S103.

(43) Hence, for each identified sensory input of the plurality of sensory inputs affecting the working memory load of the user 20, a measure with which each identified sensory input affects the working memory load of the user 20 is determined and entered in the database. This assessment is typically performed by the processing unit 15, but could alternatively be performed by the camera 12 itself. The three recorded sounds and their respective impact on the working memory load correspond to Items 1, 2, and 3, in Table 1.

(44) In a second round, the user 20 is subjected to lighting conditions corresponding to indoor lighting of the office during a winterday. Again, the camera 12 monitors the pupil size of the user's eyes and the processing unit 15 detects a 0.2 mm increase, which is considered to correspond to a 20% increase in working memory load with respect to Load.sub.NOM. This corresponds to Item 4 in Table 1.

(45) In a third round, the user 20 is subjected to the sound of the office air conditioning system starting. The camera 12 monitors the pupil size of the user's eyes and the processing unit 15 detects a 0.1 mm increase, which is considered to correspond to a 10% increase in working memory load with respect to Load.sub.NOM. This corresponds to Item 5 in Table 1.

(46) TABLE-US-00001 TABLE 1 Recorded sensory inputs vs. increase in working memory load. Increase in working Change in pupil size memory load from Sensory input from nominal size nominal load Sound level 1 of audio PupilSize.sub.NOM + 0.1 mm Load.sub.NOM + 10% player Sound level 2 of audio PupilSize.sub.NOM + 0.3 mm Load.sub.NOM + 30% player Sound level 3 of audio PupilSize.sub.NOM + 0.5 mm Load.sub.NOM + 50% player Wintertime indoor PupilSize.sub.NOM + 0.2 mm Load.sub.NOM + 20% lighting Air condition sound PupilSize.sub.NOM + 0.1 mm Load.sub.NOM + 10%

(47) Table 1 exemplifies five different items, while in a real-life scenario, tens of different sensory inputs may be recorded in order to cover an abundance of situations having potential to occur and thus increase the working memory load of the user.

(48) It should be noted that a database such as that of Table 1 may be built with purposely subjecting the user 20 to sensory inputs, but can be built while the user 20 is “naturally” subjected to the sensory inputs. Further, the naturally occurring sensory inputs can be added to a database comprising sensory inputs to which the user 20 purposely has been subjected.

(49) In a further embodiment illustrated with reference to the flowchart of FIG. 8, the database of Table 1 is utilized to select which out of a number of sensory inputs that the user 20 is subjected to should be diminished to effectively decrease the working memory load of the user 20.

(50) If an increase in working memory load of the user 20 is detected, some sensory inputs (i.e., sounds, visual inputs, and potentially even odours) may be removed or obscured in order to reduce the load on the user's working memory.

(51) In other words—as distractions taking place within the locality of the user (i.e., within a distance that those activities can impact the senses of the user) provide sensory inputs that do not contribute to the execution of the user's task, the presence of these sensory inputs may be mitigated or even eliminated.

(52) Assuming that that the user 20 of FIG. 2 is subjected to a plurality of sensory inputs, for instance those listed in Table 1, as identified by the microphone 13 as regards the audible sensory inputs and a photometer (not shown) identifying the indoor lighting in step S103 after an increase in load has been detected by the processing unit 15 in step S102. In such scenario, a problem which may arise is related to distinguishing which of the sensory inputs affects the user 20 the most.

(53) By turning to the database of Table 1 in step S103a—assuming that the audio player of the computer 10 outputs music at Sound level 2—the processing unit 15 concludes that the audio player playing at Sound level 2 affects the user 20 as much as the indoor lighting and the sound of the air condition do jointly.

(54) The processing unit 15 of the computer 10 may hence determine that the audio player is to be turned off in step S104a, or at least that its output sound level should be reduced, thereby advantageously decreasing the working memory load of the user 20.

(55) FIG. 9 illustrates a device 10 for identifying identify sensory inputs affecting working memory load of an individual. The device 10 comprises monitoring means 30 adapted to monitor working memory load of the individual, detecting means 31 adapted to detect an increase in the working memory load of the individual, and identifying means 32 adapted to identify, in response to detecting the increase, at least one sensory input affecting the working memory load of the individual.

(56) The monitoring means 30, detecting means 31 and identifying means 32 may comprise communications interface(s) for receiving and providing information, and further a local storage for storing data, and may (in analogy with that previously discussed) be implemented by a processor embodied in the form of one or more microprocessors arranged to execute a computer program downloaded to a suitable storage medium associated with the microprocessor, such as a RAM, a Flash memory or a hard disk drive.

(57) The invention has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.