MULTI-INPUT AUTOMATIC MONITORING OF MOTION TRACKING SYSTEM AND ACTUATION

Abstract

Methods for automatically monitoring operation of a motion tracking system and actuating based on the operation, the motion tracking system comprising both a computing device, and a plurality of inertial measurement units adapted to be arranged on a body of a user. The computing device processes measurements of the units and sound captured by at least one microphone to provide flags related to the motion tracking procedure with regards to the motion input and the sound input, and actuates when motion-related flags and sound-related flags have been provided.

Claims

1. A method for automatically monitoring operation of a motion tracking system and actuating based on the operation, the motion tracking system comprising both a computing device, and a plurality of inertial measurement units adapted to be arranged on a body of a user, the computing device comprising both a data communications module and at least one microphone, the method comprising: commanding, by the computing device, to provide one or more user perceptible signals indicative of at least one predetermined movement to be performed by the user; providing, by each unit of the plurality of units, measurements of body members of the user both while each unit is arranged on a respective body member and after the user has been instructed to perform the at least one predetermined movement; processing, by the computing device, the measurements of the plurality of units to provide a motion sequence, MS, of at least part of at least some body members of the user; storing, by the computing device and associated with both the user and a movement of the at least one predetermined movement, the MS and/or a range of motion, ROM, of one or more of the some body members of the user, the ROM being computed by the computing device based on the MS; processing, by the computing device, the MS and/or the ROM to check fulfillment thereof of one or more predetermined criteria associated with a respective movement of the at least one predetermined movement and counting, by the computing device, a number of times that each of the one or more predetermined criteria are unfulfilled; providing, by the computing device, a motion-related flag each time a counted number of times does not meet a respective predetermined threshold; processing, by the computing device, sound captured with the at least one microphone to detect human voice therein; processing, by the computing device, the sound captured to detect human gasps therein based on the detected human voice; providing, by the computing device, a sound-related flag when it determines that the user has gasped a number of times exceeding a respective predetermined threshold; and actuating, by the computing device, when both at least one motion-related flag and at least one sound-related flag have been provided, wherein the step of actuating comprises at least one of the following: commanding to provide one or more user perceptible signals indicative of the at least one motion-related flag provided and/or indicative of a corrective action to be performed by the user based on the at least one motion-related flag provided; and communicating the at least one motion-related flag provided to one or more electronic devices.

2. A method for tracking a motion of a subject, the method comprising: (a) providing a motion tracking system comprising: (i) a computing device; (ii) a plurality of inertial measurement units, wherein an inertial measurement unit of the plurality of inertial measurement units is arranged on a body member of the subject; and (iii) at least one microphone; (b) providing one or more indications of at least one movement to the subject; (c) measuring a plurality of signals associated with the at least one movement using at least (1) the plurality of inertial measurement units and (2) the at least one microphone; and (d) using the computing device to process the plurality of signals measured in (c), thereby tracking the motion of the subject.

3. The method of claim 2, further comprising processing the plurality of signals measured in (c) to provide a motion sequence or a range of motion of the one or more body members of the subject.

4. The method of claim 2, further comprising using the computing device to determine a presence of a motion related flag based at least in part on the plurality of signals measured in (c).

5. The method of claim 4, further comprising using the computing device to count a number of occurrences of the motion related flag.

6. The method of claim 5, wherein the computing device processes at least one signal of the plurality of signals is used to detect a number of gasps of the user.

7. The method of claim 7, wherein the detection of the number of gasps of the user is performed by determining a number of sounds which meet a signal to noise level threshold.

8. The method of claim 6, further comprising determining that greater than a threshold number of gasps or a threshold number of motions have been detected; and indicating to the user a corrective action based on the predetermined movement.

9. The method of claim 8, further comprising determining from the range of motion or the motion sequence whether one or more predetermined criteria associated with the predetermined movement are met.

10. The method of claim 9, wherein the one or more predetermined criteria comprise at least one of: the motion sequence not fulfilling one or more predetermined constraints set for each predetermined movement commanded; the range of motion being lower than a range of motion previously stored for the same user and for the same predetermined movement; and an elapsed time for the one or more body members in the motion sequence to be in the initial position of the predetermined movement exceeding a predetermined time threshold.

11. The method of claim 9, wherein the one or more predetermined criteria at least comprises the predetermined criterion of the range of motion being lower than a range of motion previously stored, and wherein said criterion comprises: an average of the range of motion being lower than an average of the range of motion previously stored, and a difference between the averages exceeding a predetermined average range of motion threshold; and/or the range of motion of the respective movement of the one or more predetermined movements in a plurality of repetitions of the respective movement being lower than the average of the range of motion previously stored.

12. The method of claim 9, further comprising recalibrating of the plurality of inertial measurement units when the motion sequence does not fulfill, a number of times, one or more of the predetermined criteria associated with the predetermined movement.

13. The method of claim 9, wherein indicating of the corrective action comprises indications of how each unit of the plurality of units shall be arranged on a body of the user.

14. The method of claim 9, wherein indicating of the corrective action comprises indicating that the motion tracking system is faulty.

15. The method of claim 9, wherein the user is instructed to perform the predetermined motion and the measured signals comprise time elapsed from when the instruction is given.

16. The method of claim 9, wherein indicating of the corrective action comprises indicating that the motion tracking procedure is complete.

17. The method of claim 9, further comprising storing a temporal or permanent motion tracking record of the user.

18. The method of claim 17, wherein the temporal or permanent motion tracking record comprises a ban associated with the user; and wherein the user is not permitted to operate the motion tracking system when there is a stored active motion tracking ban associated with the user.

19. The method of claim 9, further comprising establishing a communication link between the computing system and one or more electronic devices, wherein the communication link comprises one or more of: a voice communication link, a video communication link, and a data communication link.

20. The method of claim 9, wherein the at least one predetermined criteria comprise at least two predetermined criterion.

21. The method of claim 20, wherein the at least two predetermined criterion are counted independently.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0108] To complete the description and in order to provide for a better understanding of the disclosure, a set of drawings is provided. Said drawings form an integral part of the description and illustrate embodiments of the disclosure, which should not be interpreted as restricting the scope of the disclosure, but just as examples of how the disclosure can be carried out. The drawings comprise the following figures:

[0109] FIG. 1 diagrammatically shows a motion tracking system in accordance with embodiments.

[0110] FIG. 2 shows a user of a motion tracking system with inertial measurement units arranged thereon in accordance with embodiments.

[0111] FIGS. 3 and 4 show the user of FIG. 2 while performing two different predetermined movements in accordance with embodiments.

[0112] FIGS. 5-7 show graphs representing the unfulfillment of predetermined criteria and provision of flags in accordance with embodiments.

[0113] FIG. 8 shows an exemplary gasp detection by methods and systems in accordance with embodiments.

DESCRIPTION OF WAYS OF CARRYING OUT THE DISCLOSURE

[0114] FIG. 1 diagrammatically shows a motion tracking system 5 in accordance with embodiments. The motion tracking system 5 includes a plurality of inertial measurement units 20a-20n, IMUs, and a computing device 10, which may be e.g. a tablet, a mobile phone, a personal computer, etc.

[0115] Each IMU 20a-20n is adapted to be arranged on the body of a user so that the measurements provided by each IMU 20a-20n can be processed by the computing device 10, thereby providing a motion sequence of the user. The IMUs 20a-20c may be attached to the body members of the user in a number of ways, for instance using straps, Velcro, etc. Each IMU 20a-20n is a motion tracker that includes one or more sensing devices selected from e.g. an accelerometer 21, a gyroscope 22 and a magnetometer 23. In the embodiment of FIG. 1, each IMU 20a-20n includes all three sensing devices 21-23, but in other embodiments the IMUs only include an accelerometer 21 and a gyroscope 22, for instance. Preferably, all IMUs 20a-20n include the same sensing devices 21-23.

[0116] The IMUs 20a-20n further include at least one processor 26, at least one memory 27, and a first communications module 28 for transmitting radiofrequency signals to the computing device 10 and which include e.g. advertisement packages, data packets with identification data (e.g. one or more identities, keys, etc.), data packets with measurements of the sensing device(s) 21-23, etc. When no wireless communications connections are established with the computing device 10, the radiofrequency signals of the IMUs 20a-20n include advertisement packages for indicating their presence and that they are active. Once the wireless communications connections are established (using a technology and protocol known by a skilled person, for instance but without limitation, Bluetooth and Bluetooth Low Energy communications, cellular network communications such as GSM, UMTS or LTE, wireless LAN communications, etc.) with the computing device 10, the radiofrequency signals of the IMUs 20a-20n may include identification data and/or the measurements. An antenna for radiating electromagnetic waves is provided as part of the first communications module 28. Preferably, the same first communications modules 28 also enable the IMUs 20a-20n to receive data from the computing device 10 upon capturing electromagnetic waves with the antenna.

[0117] Preferably, at least one processor 26 of the IMUs 20a-20n runs a sensor fusion algorithm for processing the measurements of the sensing devices 21-23 within the respective sensor. The sensor fusion algorithm is intended to enhance the raw measurements of the sensing devices by correcting errors thereof due to drifts of the sensing devices and, thus, outputs processed measurements that are to be transmitted to the computing device 10.

[0118] The computing device 10 includes at least one processor 11, at least one memory 12, and a second communications module 13 for at least receiving data. The second communications module 13 includes at least one antenna whereby electromagnetic waves may be captured, and then processed by the at least one processor 11.

[0119] The computing device 10 also includes at least one microphone 14, and preferably includes a screen 15. On said screen 15 the computing device 10 is capable of showing instructions and/or information to the intended user about the operation of the system 5, for example predetermined movements that are to be performed by an intended user of the motion tracking system 5, indications of a correspondence between IMUs 20a-n and body members of the user for arrangement thereof, potential problems detected, provision of flags, etc. To this end, the computing device 10 stores, in the at least one memory 12, data indicative of a correspondence between IMUs and users assigned to those IMUs, and also data relative to the physical exercises, i.e. predetermined movements, of intended users. Any of these data can be transmitted to and/or received from another electronic device thanks to the second communications module 13. For example, a therapist is able to receive the feedback at a computing device in a hospital so as to monitor the evolution of the person. Based on the feedback received, the therapist is able to adjust the difficulty of the movement(s), the number of repetitions thereof, prescribe new movements, etc. so that the person may further exercise using the motion tracking system 5. Further, in addition to the screen 15, which provides visual feedback, the computing device 10 may also include further visual output means (e.g. LEDs, animations), audio output means (e.g. loudspeakers), vibrating means (e.g. a vibrator), etc. for providing user perceptible signals in the form of sounds, vibration, animated graphics, etc. The computing device 10 controls the information provided by the screen 15 or by any other user presenting means by way of digital commands sent to the particular device(s).

[0120] The system 5 could additionally or alternatively include other user presenting means like the ones mentioned above and which the computing device 10 can control by way of wired or wirelessly-transmitted commands.

[0121] FIG. 2 shows a user 1 of a motion tracking system with IMUs 20a-20c arranged thereon in accordance with embodiments. The user 1 is standing still.

[0122] A first IMU 20a is arranged on a right upper arm 2, a second IMU 20b is arranged on a right lower arm 3, and a third IMU 20c is arranged on a chest 4.

[0123] As known in the art, the user 1 arranges the IMUs 20a-20c in this way according to information provided by a computing device of the motion tracking system about where and how (i.e. position and angle) each IMU 20a-20c is to be arranged; alternatively, the computing device may only inform the user 1 about which body members 2-4 shall have an IMU 20a-20c arranged thereon and how, the user 1 places the IMUs 20a-20c accordingly, and then by way of an automated calibration procedure or with user input means the IMU-body member correspondence is registered by the computing device. Concerning the former, the computing device may command the provision of user perceptible signals indicative of one or more predetermined calibration movements that the user 1 is to perform to determine which IMU 20a-20c is on each body member 2-4 based on the measurements provided by the IMUs 20a-20c during reproduction of the movement, whereas in the latter forms can be shown to the user 1 to specify what IMU 20a-20c is on what body member 2-4.

[0124] FIG. 3 shows the user 1 of FIG. 2 while performing a predetermined movement in which, while standing still, an upper body is to be moved forward while maintaining the hips static relative to the floor. FIG. 4 shows the same user 1 while performing a predetermined movement in which, while standing still, an upper body is to be moved to one side while maintaining the hips static relative to the floor.

[0125] These predetermined movements and additional ones or other ones can be part of a physical rehabilitation procedure whereby the user 1 intends to improve her/his physical condition.

[0126] The physical rehabilitation typically includes a number of predetermined movements, each with its respective number of repetitions. When the rehabilitation is to be conducted with a motion tracking system, each predetermined movement also includes respective one or more predetermined criteria (including one or more predetermined constraints) so that people can undergo a rehabilitation procedure without direct supervision of a therapist or doctor. The type and number of predetermined movements, the numbers of repetitions, and the predetermined criterion or criteria are assigned to each particular user 1 based on the physical condition thereof and the recovery plan the user has to follow. All these parameters can change with the number of sessions taken by the user 1. In this sense, the computing device stores or retrieves data with all the parameters necessary to instruct the user 1 to perform the predetermined movements and monitor the activity of the user 1 while exercising.

[0127] FIGS. 5-7 show graphs representing the unfulfillment of predetermined criteria and provision of flags 60, 70 in accordance with embodiments.

[0128] The horizontal axes of the graphs represent time. In the horizontal axis of each graph are marks showing when a user of a motion tracking system ends a repetition of a predetermined movement, the coding M.sub.I,J is used, where I stands for the number of predetermined movement and J stands for the repetition of the l-th predetermined movement; for example, the mark M.sub.2,4 represents the time when the user has finished the fourth repetition of the second predetermined movement. Some sessions include a single predetermined movement, whereas other sessions include two, three, four or even more predetermined movements. Each predetermined movement typically includes a plurality of repetitions.

[0129] In the vertical axes are represented different predetermined criteria. The circles represent each time a predetermined criterion is unfulfilled as the session advances over time. For the sake of the illustration only, the dotted lines represent for which predetermined movements each predetermined criterion is defined. In this sense, some predetermined movements may have a same predetermined criterion when it comes to ROM, starting position, etc.

[0130] Referring to FIG. 5, four predetermined criteria C1, C2, C4 and G are set for a first predetermined movement M.sub.1, and four predetermined criteria C2, C3, C5 and G are set for a second predetermined movement M.sub.2. A predetermined threshold may be set for each particular predetermined criterion, and/or a predetermined threshold may be set for two or more particular predetermined criteria as will be described with reference to FIG. 6.

[0131] The criterion G illustrated in FIGS. 5 and 6 relates to the determination of the user gasping. Each time the computing device determines that the user gasps (in the manner described in FIG. 8 for example) a circle marker 55 is illustrated in the graphs.

[0132] In the example of FIG. 5, the criterion G has a threshold of four gasps for a same predetermined movement performed by the user, whereas in the example of FIG. 6 the value of the threshold is the same but is per session.

[0133] As it can be seen in the graph of FIG. 5, the fourth gasp that is detected takes place during the second repetition of the second predetermined movement M.sub.2,2, and the three previous gasps have taken place during repetitions of the first predetermined movement M.sub.1. Hence, a sound-related flag 70 is not provided until M.sub.2,5, when four gasps have occurred during M.sub.2.

[0134] Regarding the first criterion C1 set for M.sub.1 only, is unfulfilled 50 three times. The respective threshold is three per session (that is to say, the computing device counts all the times the criterion is unfulfilled 50 irrespective of whether the unfulfillments 50 occur during a single movement or different movements), and at the third unfulfillment 50 a motion-related flag 60 is provided.

[0135] The second criterion C2, set for both M.sub.1 and M.sub.2, is unfulfilled 51 four times. The respective threshold is four per session, and at the fourth unfulfillment 51 a motion-related flag 60 is provided.

[0136] The third criterion C3, set for M.sub.2 only, is not unfulfilled and, thus, no motion-related flag 60 is provided due to C3.

[0137] The fourth criterion C4, set for M.sub.1 only, is unfulfilled 52 once. The respective threshold is three per movement and no motion-related flag 60 is provided due to C4.

[0138] The fifth criterion C5, set for M.sub.2 only, is unfulfilled 53 twice. The respective threshold is four per session and no motion-related flag 60 is provided due to C5.

[0139] The motion tracking system actuates when at least one motion-related flag 60 and at least one sound-related flag 70 are provided. Therefore, when the first motion-related flag 60 is provided, no action is taken. Since six repetitions are set for M.sub.1, after the sixth repetition thereof the commanded predetermined movement becomes M.sub.2, with six repetitions as well. By the time the second motion-related flag 60 is provided no sound-related flag 70 has been provided yet, so there is no actuation by the computing device of the system. At M.sub.2,5 the sound-related flag 70 is provided, and at that moment either the motion-related flags 60 are communicated to one or more electronic devices and/or some feedback is provided to the user. Despite that, no change in the predetermined movement is made and the sixth repetition of the second predetermined movement is completed by the user. In this sense, it may occur that the actuation of the computing device is a silent communication to the therapist or doctor, and the user is not informed of any problematic situations or actuations based on the same.

[0140] In relation to FIG. 6, criteria C1-C3 and G are set. The user is instructed to perform three different predetermined movements M.sub.1-M.sub.3.

[0141] In this case, a predetermined threshold is set in cumulative form in respect of criteria C1 and C2, even if C1 is not set for M.sub.2 and M.sub.3 but C2 is. Said threshold has a value of three per session (it could be per movement too). Despite this cumulative predetermined threshold, one or both of the criteria C1 and C2 may also have respective predetermined thresholds, either per session or per movement.

[0142] The first criterion C1 is unfulfilled 50 twice. The respective threshold is three per movement, but no motion-related flag 60 is provided due to this threshold.

[0143] Before the user starts M.sub.2, a sound-related flag 70 is provided due to the number of gasps of the user. The lack of a motion-related flag 60 in M.sub.1 does not trigger any actuation.

[0144] In M.sub.2, the second criterion C2 is unfulfilled 51 twice. In spite of its respective threshold being five per session, a motion-related flag 60 is provided due to the cumulative predetermined threshold. This is so because the first criterion C1 was unfulfilled 50 twice during M.sub.1 and at the first unfulfillment 51 of the second criterion C2 during M.sub.2 the cumulative threshold was met. If the cumulative threshold had been set per movement, the unfulfillment 51 of the second criterion C2 would have to be during M.sub.1 to produce the provision of the motion-related flag 60.

[0145] Depending on the user, the computing device can actuate as soon as the motion-related flag 60 is provided owing to the existence of the sound-related flag 70. For example, when the user is an elderly person, the computing device may want to take some action to avoid the risk of the user getting injured. In this example the user is a person without a severe medical condition and, thus, there is no actuation with just one flag 60, 70 of each, but at least two motion-related flags 60, 70, or at least three flags 60, 70 of one kind and one or more of the other 70, 60. In some other examples, the actuation may take place when one flag 60, 70 of each kind is provided but with a time difference between the two that is less than a predetermined time threshold.

[0146] The third criterion C3, set for M.sub.2 only, is unfulfilled 52 three times. The respective threshold is three per session, and at the third unfulfillment 52 (during M.sub.2,4) a motion-related flag 60 is provided.

[0147] A second sound-related flag 70 is provided during M.sub.2,5. As aforesaid, the threshold for criterion G in the example of FIG. 6 is four per session, thus the second sound-related flag 70 is provided although the first of the four unfulfillments 55 took place during M.sub.1,6. As it can be appreciated, once a flag 60, 70 is provided, a respective counter of times that a criterion has been unfulfilled is reset.

[0148] The existence of two motion-related flags 60 and two sound-related flags 70 triggers an actuation, in this case the computing device commands the provision of signals to instruct the user to skip to another predetermined movement in spite of the second predetermined movement having eight repetitions set for the ongoing session.

[0149] In FIG. 7, there are three predetermined criteria C, W1 and W2. The criterion C is unfulfilled 50 whenever the computing device triggers a recalibration of the IMUs. The criterion W1 is unfulfilled 55 whenever the computing device detects that the user has pronounced a particular word that is present within a predetermined dictionary, whereas the criterion W2 is unfulfilled 56 whenever the computing device detects that the user has pronounced different words present within the predetermined dictionary. Accordingly, every time the criterion W1 is unfulfilled 55 the criterion W2 is also unfulfilled 56 since it relates to the pronunciation of an offending word.

[0150] The rehabilitation session is programmed to have four different predetermined movements, with ten repetitions of each.

[0151] The criterion C is unfulfilled 50 four times, and its predetermined threshold is two per session. The criterion W1 is unfulfilled 55 three times, and its predetermined threshold is two per session. The criterion W2 is unfulfilled 56 five times, and its predetermined threshold is three per session

[0152] While the user performs repetitions of M.sub.1, two recalibrations are triggered because multiple predetermined constraints of M.sub.1 that the user had to comply with were not met, which may be indicative of an incorrect unit arrangement or units that are not calibrated, i.e. the sensing devices thereof measure with an error that is deemed to be excessive. A motion-related flag 60 is provided. Shortly thereafter, a sound-related flag 70 is provided owing to the criterion W2. The computing device instructs the user to skip to M.sub.2 in case the problems are not due to the calibration of the units or the arrangement thereof but to incorrect reproduction of M.sub.1 by the user.

[0153] During M.sub.2, again two recalibrations are triggered and, hence, a motion-related flag 60 is again provided. By the time the second motion-related flag 60 is provided, another sound-related flag 70 has been provided due to the criterion W1. So, at M.sub.2,4, following the provision of the motion-related flag 60, the computing device signals the end of the motion tracking procedure and instructs the user to stop all activity, thereby not continuing with any one of the second, third and fourth predetermined movements.

[0154] FIG. 8 shows an exemplary gasp detection by methods and systems in accordance with embodiments.

[0155] A graph is illustrated in which sound 80 captured by at least one microphone of a computing device is plotted versus time. In this example, two level thresholds L1, L2 are set for detecting gasps, which preferably are dynamically adjustable but can be constant as well and selected based on experimental tests or assumptions (e.g. expected speech and noise power respective exceeding and not exceeding certain power levels). The sound 80 and level thresholds are represented in dB, but other units may be used too.

[0156] As sound 80 is captured, the computing device processes it so as to detect human voice in it. Based on the detection or non-detection, speech and noise levels are determined, and preferably the level thresholds L1, L2 are set based on said speech and noise levels. In order to increase the confidence in the determination of the speech level, when the computing device considers that human voice is present in certain parts of the sound 80, it runs word extraction techniques on it to establish whether said part in fact is speech.

[0157] Accordingly, the first level threshold L1 is set or adjusted based on the expected sound level when there is no human voice; said threshold is above said level by a predetermined factor. All sound level not exceeding the first level threshold L1 is deemed to be a baseline sound level.

[0158] Likewise, the second level threshold L2 is set or adjusted based on the expected sound level when there is human voice, and preferably when said human voice corresponds to speech; said threshold is below said level by a predetermined factor. All sound level exceeding the second level threshold L2 is deemed to be speech, and more particularly words 81; preferably the computing device confirms that they are words 81 by running the word extraction techniques.

[0159] As for the sound level between the first and second level thresholds L1, L2, the same is deemed to be gasps 82.

[0160] Said word and gasp detection can rely on the level of the sound 80 itself and the procedure described above, or on the level of the sound 80 in terms of signal-to-noise ratio, SNR. In the latter case, the level thresholds L1, L2 are SNR level thresholds. The SNR is greater when the user is speaking, and becomes lower when the user does not, but the SNR is still greater when the user is gasping (and especially between words 81, e.g. between T4 and T5) than the SNR when there user is neither speaking nor gasping.

[0161] In the graph represented, silence and, thus, noise is captured from an initial time instant up to T1. From T1 to T2 one or several words 81 are pronounced by the user. From T2 to T3 there is no speech nor gasps because the sound 80 does not exceed the first threshold level L1. From T3 to T4 a word 81 is pronounced by the user, and after that and up to T5 there is a gasp 82 followed by another word between T5 and T6. From T6 to T7 there is silence. And, again, between T7 and T8 and between T9 and T10, the user gasps 82 two more times.

[0162] In this text, the term “comprises” and its derivations (such as “comprising”, etc.) should not be understood in an excluding sense, that is, these terms should not be interpreted as excluding the possibility that what is described and defined may include further elements, steps, etc.

[0163] On the other hand, the invention is obviously not limited to the specific embodiment(s) described herein, but also encompasses any variations that may be considered by any person skilled in the art (for example, as regards the choice of materials, dimensions, components, configuration, etc.), within the general scope of the invention as defined in the claims.