AUGMENTED REALITY SYSTEM FOR PHANTOM LIMB PAIN REHABILITATION FOR AMPUTEES

Abstract

The present invention relates to a system for neuromuscular rehabilitation of a patient having an affected limb comprising: a feedback member arranged to give real-time visual feedback; a plurality of electrodes arranged to acquire an electric signal corresponding to an intent to move said affected limb; a control unit configured to: perform pattern recognition of said electric signals, wherein at least one feature in said electric signal is used to predict motion intent of said affected limb adjacent to at least one joint, such aggregated motions of said affected limb are predicted; based on output signals from said performed pattern recognition, control said feedback member to perform actions corresponding to said motions, whereby said actions of said feedback member are individually and simultaneously controlled by said patient via said intended motions.

Claims

1. A phantom limb pain rehabilitation augmented reality system for rehabilitation of a patient having an amputated limb, said system comprising: a video capturing device arranged to face towards said patient and to acquire a video stream capturing at least a majority of said patient including a stump of the amputated limb; a display arranged to provide a real-time visual feedback to said patient, the real-time visual feedback comprising the video stream and a virtual limb in place of a missing portion of the amputated limb, wherein the at least the majority of the patient, the stump, and the virtual limb are displayed in the real-time visual feedback; a plurality of electrodes each arranged to acquire an electric signal generated from the stump of the amputated limb, or from a portion of the patient's body on the same side as the amputated limb, said electric signal corresponding to an intent to move said missing portion being associated with phantom limb pain; and a control unit configured to: perform pattern recognition of said electric signals, wherein at least one feature in said electric signals is used to predict motion intent of said missing portion of the amputated limb adjacent to at least one joint, to thereby predict aggregated motions comprising individual motions of several joints of said missing portion of the amputated limb performed at least partly simultaneously to alleviate the phantom limb pain; track the motion of the stump in the visual feedback being displayed on said display; superimpose said virtual limb onto the stump of said patient's body in said visual feedback being displayed on said display, wherein said virtual limb follows said stump of said patient's body in said visual feedback being displayed on said display such that said virtual limb remains in an anatomically correct position; based on output signals from said performed pattern recognition, control said virtual limb on said display to perform motions corresponding to said aggregated motions to provide the phantom limb pain rehabilitation for the patient, whereby said motions of said virtual limb are individually and simultaneously controlled by said patient via said intended motions, wherein, for pattern recognition, said control unit is configured to: divide each of said electric signals into signal segments defined by time intervals; extract signal features from at least one of said segments; combine said signal features into a feature vector relating to said motion; and based on said feature vector, predict said intended motion of said missing portion of the amputated limb.

2. The system according to claim 1, wherein at least two aggregated motions of said missing portion of the amputated limb are predicted, wherein said control unit is configured to control said virtual limb to perform at least two actions corresponding to said at least two motions.

3. The system according to claim 1, wherein the control unit is configured to extract signal features from each of said segments, and wherein said features comprise, for each of said segments, an extracted cardinality of data elements within a segment of said electrical signal.

4. The system according to claim 1, wherein said control unit is configured to, based on said output signals from said pattern recognition, control a video game on said display.

5. The system according to claim 1, wherein said electrodes are implantable into a patient's body for detecting bioelectrical signals.

6. The system according to claim 1, wherein said control unit is configured to, based on said output signals from said pattern recognition, control computer unit input commands.

7. The system according to claim 1, wherein said plurality of electrodes is a high density electrode array.

8. The system according to claim 1, wherein the individual motions are at least two of flexing of a joint, extending the joint, pronation, supination, open or close hand, ulnar deviation, and radial deviation.

9. The system according to claim 7, wherein said control unit is further configured to: determine, by feature selection or signal separation, which of said plurality of electrodes are collecting useful data, and discard or process data collected from electrodes determined not to collect useful data.

10. The system according to claim 1, wherein the display is a desktop display screen.

11. The system according to claim 1, wherein tracking of the predetermined portion of the patient's body is performed by tracking markers arranged on the predetermined portion of the patient's body.

12. The system according to claim 1, wherein the control unit receives and stores a report of the progression of phantom pain felt by the patient.

13. A method for controlling a phantom limb pain rehabilitation augmented reality system for rehabilitation of a patient having an amputated limb, said method comprising the steps of: acquiring, via a plurality of electrodes, electric signals generated from a stump of the amputated limb or from a portion of the patient's body on the same side as the amputated limb, said electric signals corresponding to an intent to move a missing portion of the amputated limb being associated with phantom limb pain; performing pattern recognition of said electric signals; predicting motion intent in at least one joint using at least one feature in said electric signals, to thereby predict aggregated motions comprising individual motions of several joints of said missing portion of the amputated limb performed at least partly simultaneously to alleviate the phantom limb pain; providing real-time visual feedback to said patient on a display, the real-time visual feedback comprising a video stream of said patient, captured by a video capturing device facing towards the patient, and a virtual limb corresponding to said missing portion of the amputated limb such that at least a majority of the patient, the stump, and the virtual limb are displayed in the real-time visual feedback; tracking the motion of a stump of the patient's body in the visual feedback being displayed to the patient; superimposing said virtual limb onto the stump of said patient's body in said visual feedback, wherein said virtual limb follows said stump of said patient's body in said visual feedback such that said virtual limb remains in an anatomically correct position on the stump; and based on output signals from said performed pattern recognition, controlling the virtual limb to perform motions corresponding to said aggregated motions to provide the phantom limb pain rehabilitation for the patient, whereby said motions of said virtual limb are individually and simultaneously controlled by said patient via said intended motions, wherein said pattern recognition comprises: dividing each of said electric signals into signal segments defined by time windows, extracting signal features from at least one of said segments, combining said signal features into a feature vector, and based on the feature vector, predicting said intended motion of said stump and missing portion of the amputated limb.

14. The method according to claim 13, wherein said pattern recognition comprises extracting signal features from each of said signal segments, and wherein said features comprise, for each of said segments, an extracted cardinality of data elements within a segment of said electrical signal.

15. The method according to claim 13, comprising the steps of: executing predetermined motions predefined by said control unit; associating said features in said electric signals with said executed predetermined motions; performing rehabilitation tasks by said patient, wherein said tasks are training motions predetermined in said control unit; and reporting said patient's progress on said display.

16. The method according to claim 13, wherein reporting the patient's progress includes reporting the amount of pain experienced by the patient.

17. Use of a phantom limb pain rehabilitation system for rehabilitation of a patient having an amputated limb, said system comprising: a video capturing device arranged to face towards said patient and to acquire a video stream capturing at least a majority of said patient including a stump of the amputated limb; a display arranged to provide a real-time visual feedback to said patient, the real-time visual feedback comprising the video stream and a virtual limb in place of a missing portion of the amputated limb, wherein the at least the majority of the patient, the stump, and the virtual limb are displayed in the real-time visual feedback; a plurality of electrodes each arranged to acquire an electric signal generated from the stump of the amputated limb, or from a portion of the patient's body on the same side as the amputated limb, said electric signal corresponding to an intent to move said missing portion being associated with phantom limb pain; and a control unit configured to: perform pattern recognition of said electric signal, wherein at least one feature in said electric signal is used to predict motion intent in the missing portion of the amputated limb adjacent to at least one joint, to thereby predict aggregated motions comprising individual motions of several joints of said missing portion of the amputated limb performed at least partly simultaneously to alleviate the phantom limb pain; track the motion of a stump of the patient's body in the visual feedback being displayed on said display; superimpose said virtual limb onto the stump of said patient's body in said visual feedback being displayed on said display, wherein said virtual limb follows said stump of said patient's body in said visual feedback being displayed on said display such that said virtual limb remains in an anatomically correct position; based on output signals from said performed pattern recognition, control said virtual limb to perform motions corresponding to said aggregated motions to provide the phantom limb pain rehabilitation for the patient, whereby said actions of said virtual limb are individually and simultaneously controlled by said patient via said intended motions, wherein, for pattern recognition, said control unit is configured to: divide each of said electric signals into signal segments defined by time intervals; extract signal features from at least one of said segments; combine said signal features into a feature vector relating to said motion; and based on said feature vector, predict said intended motion of said missing portion of the amputated limb, and wherein said use comprises: executing predetermined motions predefined by said control unit; associating said features in said electric signals with said executed predetermined motions; performing rehabilitation tasks by said patient, wherein said tasks are training motions predetermined in said control unit; predicting said training motions based on the associated features and on the acquired electric signals from said portion of the patient's body, wherein said virtual limb performs said predicted training motions; and reporting said patient's progress on said display.

18. Use according to claim 17, wherein reporting includes to report the progression of phantom pain felt by the patient, the reporting is stored in the control unit.

19. The system according to claim 1, wherein said features further comprise at least one of: with respect to time domain: mean absolute value, median, standard deviation, variance, waveform length, root mean square (RMS), zero-crossings of the signal, peaks (RMS), peaks mean, mean velocity, slope changes, power, difference absolute mean, max fractal length, fractal dimension, fractal dimension Higuchi, rough entropy, correlation, and co-variance; and with respect to frequency domain: waveform length, mean, median frequency, peaks mean, peaks median, and peaks standard deviation.

20. The method according to claim 13, wherein said features further comprise at least one of: with respect to time domain: mean absolute value, median, standard deviation, variance, waveform length, root mean square (RMS), zero-crossings of the signal, peaks (RMS), peaks mean, mean velocity, slope changes, power, difference absolute mean, max fractal length, fractal dimension, fractal dimension Higuchi, rough entropy, correlation, and co-variance; and with respect to frequency domain: waveform length, mean, median frequency, peaks mean, peaks median, and peaks standard deviation.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0075] FIG. 1 illustrates a schematic drawing of a system according to an alternative embodiment of the invention;

[0076] FIG. 2 illustrates a schematic drawing of a system according to an alternative embodiment of the invention;

[0077] FIG. 3 illustrates exemplary individual motions of a limb;

[0078] FIG. 4 illustrates a flow-chart of an alternative embodiment according to the invention;

[0079] FIG. 5 illustrates a flow-chart of an alternative embodiment according to the invention; and

[0080] FIG. 6 illustrates an exemplary electrode placement.

DESCRIPTION OF EXAMPLE EMBODIMENTS OF THE PRESENT INVENTION

[0081] In the following description, the present invention is mainly described with reference to neuromuscular rehabilitation in relation to phantom limb pain in a patient with a missing limb. It should, however, be noted that this by no means limits the scope of the invention, which is equally applicable to neuromuscular rehabilitation of any other condition where myoelectric and/or neuroelectric signals can be used to drive the system and promote motor execution. For example, such as neuromuscular rehabilitation of a patient suffering from e.g. stroke, incomplete spinal cord, and/or nerve injuries.

[0082] FIG. 1 illustrates a schematic of a system 100 according to an exemplary embodiment of the invention. FIG. 1 shows a patient 102 having an affected limb 104, the patient 102 is using the system 100. In this case the affected limb 104 has a missing portion such that there is missing limb, for example as a result of an amputation. Furthermore, FIG. 1 shows a display 106 connected to a control unit 108 and a plurality of electrodes 110 also connected to the control unit 108. Although three electrodes are shown in FIG. 1, any suitable number of electrodes is possible to use. The connection between the display 106 and the control unit 108 may be enabled through physical wires or by wireless connection. The electrodes 110 are arranged to acquire an electric signal generated from a portion 103 of the patient's body, in this particular case the stump 103 of a limb having a missing portion. As illustrated in this exemplary embodiment, the electrodes 110 are attached on the surface of the patient's skin such that myoelectric signals may be recorded from the patient. However, the electrodes 110 may also be implantable electrodes 110. The control unit 108 is configured to read electric signals acquired by the electrodes 110 and perform pattern recognition of the electric signals. The control unit 108 processes the electric signals and recognizes features in the electric signals such that motion intent of aggregated motions is predicted. In other words, the control unit 108 predicts motions of the missing limb intended by the patient 102. After the intended motion has been predicted, the control unit 108 controls the virtual limb 116 on the display to perform the intended motions. For example, if a patient 102 intends to move his limb at a first joint, for example the elbow, and at the same time supinate the hand such that it rotates, the virtual limb 116 will move from a first position 112 to a second position 114 according to the aggregated intended motions. In the first position the missing limb is to the left and the palm of the hand faces the patient 102 viewing the display, then the aggregated motion of the limb intended by the patient 102 is to flex the elbow joint and to pronate the hand, the intended motion is predicted by the control unit 108 and the limb moves to position 114. Other motions are of course possible such as for example flexing of a limb, extending the limb, pronation, supination, open or close hand (in the case where a hand is part of an affected limb), or any other motion possible with a limb, and/or combinations thereof. The control unit 108 may be connected to the display 106 by a wireless connection or via physical electrical connections. Moreover, although in FIG. 1 a virtual arm is shown, a robotic device may be used instead. For example, the display with the virtual arm may be replaced by a prosthetic device or a remotely controlled vehicle or similar.

[0083] FIG. 2 shows another system 200 according to an embodiment of the invention. Similar to what is depicted in FIG. 1, FIG. 2 shows a patient 202 having an amputated limb 204, and the patient 202 is using the system 200. At the stump 203 of the amputated limb there is a plurality of electrodes 210 arranged to acquire an electric signal at the stump 203. Furthermore, the electrodes 210 are connected to a control unit 208 and there is also a display 206 connected to the control unit 208. Although three electrodes are shown in FIG. 2, any suitable number of electrodes is possible. The display 206 is configured to provide real-time feedback to the patient 202 and to show a virtual limb 214. Furthermore, there is a video capturing device in the form of a camera 212 arranged to acquire a video stream of the patient 202. The camera 212 may be a conventional webcam. For example the camera 212 may be a simple traditional webcam. Moreover, the video stream is shown on the display 206. In other words the patient 202 can see him/herself in real-time on the display 206. The control unit 208 is configured to superimpose the virtual limb 214 onto a predetermined portion 215 of the patient's body shown on the display 206. As shown in FIG. 2, the virtual limb 214 is a portion of an arm and the hand of the arm, superimposed onto the stump 215 of the affected limb being an amputated limb. Furthermore, there are markers 213 such as fiducial markers 213 arranged on the stump of the amputated limb on the patient such that the control unit 208 may track the motion of the affected limb 204 via the webcam 212. This enables the virtual limb 214 to be aligned anatomically correctly with the stump 215 of the amputated limb of the patient 216 on the display. Moreover, if the patient 202 moves around, the virtual limb 214 stays in the anatomically correct position on the display 206. Similar to what is described with reference to FIG. 1, the control unit 208 is arranged to read electric signals acquired by the electrodes 210 and to perform pattern recognition on the electric signals. The control unit 208 processes the electric signals and recognizes features in the electric signals such that motion intent of aggregated motions is predicted. In other words, the control unit 208 predicts aggregated motions of the missing limb intended by the patient. After the intended motions have been predicted, the control unit 208 controls the virtual limb 214 on the display to perform the intended motions. For example, if a patient intends to move his limb at a first joint, for example the elbow, and at the same time supinate the hand such that it rotates, the virtual limb will move from a first position 218 to a second position 220 according to the intended motion. In the first position 218 the missing limb is to the left and the palm of the hand faces away from the patient 202 viewing the display 206, then the motion of the limb intended by the patient is to flex the elbow joint and to supinate the hand, the intended motion is predicted by the control unit 208 and the virtual limb moves to position 220. Other motions are of course possible, such as for example flexing of a limb, extending the limb, pronation, supination, open or close hand (in the case where a hand is part of an affected limb), or any other motion possible with a limb, and/or combinations thereof. The control unit 208 may be connected to the display 206 by a wireless connection or via physical electrical connections.

[0084] Superimposing of the virtual limb onto a predetermined portion of the patient's body as described with reference to FIG. 2 enables an augmented reality. Furthermore, the patient may play a video game shown on the display via intended motions. In other words, the conventional mouse and keyboard are substituted by predicted motions.

[0085] FIG. 3 illustrates exemplary individual motions of a limb. In this case the limb is a hand, however, the limb may within the context of the invention be any limb such as an arm, a leg, a foot, etc. In FIG. 3 a limb, illustrated as a hand, is shown in a resting position 300, and the other positions 302-316 are relative to the resting position 300. The other positions are: open hand 302, closed hand 304, flexed position 306, extended position 308, pronation 310, supination 312, ulnar deviation 314, and radial deviation 316. Note that other motions are also possible. For example, it is possible to move the hand in any of the above motions at the same time as performing any motion with the arm adjacent to the elbow.

[0086] FIG. 4 is a flow-chart illustrating the method steps according to an embodiment of the invention. The method may be implemented using for example the system depicted in FIG. 1 or FIG. 2. In a first step S101 electrical signals are acquired from a patient. The electrical signals may for example be acquired from surface electrodes mounted on the surface of the skin of the patient or may be acquired from electrodes implanted into the patient and connected to nerves of the patient. In a second step S103, pattern recognition is performed on the electrical signals acquired by the electrodes. During pattern recognition, features of the electric signals are distinguished and determined to correspond to certain motions of the limb intended by the patient. In a final step S105, a virtual limb is controlled to perform motions based on the predicted motions. The virtual limb is displayed on a display shown to the patient. Optionally, a robotic device is controlled via the pattern recognition, for example a prosthetic device may be controlled.

[0087] FIG. 5 is a flow-chart illustrating the method steps according to another embodiment of the invention. In FIG. 5 the pattern recognition S103 steps are illustrated in more detail. In a first step S201, the electrical signals acquired from the patient are divided into segments. The length of the segments is defined in terms of predetermined time windows. For example, a time window may have 200 ms duration. Moreover, each electrical signal from the independent electrodes is divided into segments. Within each segment at least one characteristic features is extracted. A non-exhaustive list is of features of in the time domain of an electrical signal is comprised in the following: mean absolute value, median, standard deviation, variance, waveform length, root mean square (RMS), zero-crossings of the signal, peaks (RMS), peaks mean, mean velocity, slope changes, power, difference absolute mean, max fractal length, fractal dimension, fractal dimension Higuchi, rough entropy, correlation, co-variance, etc. A non-exhaustive list is of features in the frequency domain of an electrical signal is comprised in the following: waveform length, mean, median, peaks mean, peaks median, peaks standard deviation, etc. Furthermore, the cardinality of the segments is extracted as a characteristic feature. The cardinality of a signal is the number of unique elements of the signal. For example, if a signal is defined by the elements in the vector [1 1 2 3 5] the cardinality of the signals is four. Using the cardinality of the signal in each segment as a feature for pattern recognition, the accuracy of the pattern recognition is increased. As an example, the following is the average class-specific accuracy of a commonly used classifier (Linear Discriminant Analysis, LDA) in the prediction of 11 hand and wrist motions in 20 subjects when fed with the following features:

[0088] Cardinality: 90.0%

[0089] Zero crossings: 82.1%

[0090] Wave length: 82.0%

[0091] Root mean square value: 81.8%

[0092] Mean absolute value: 80.1%

[0093] Sign slope change: 74.7%

[0094] As can be seen from the above accuracies of prediction, the prediction accuracy when using the cardinality feature is higher compared to when using the other commonly used features listed above, resulting in an improved pattern recognition and thus prediction of intended motions.

[0095] In a subsequent step S205, a feature vector is formed by placing the extracted features, including the cardinality, as entries in a vector for the present segment. From the feature vector, an intended motion is predicted S207 using known algorithms. Algorithms used for pattern recognition may be e.g. supervised, unsupervised, and/or competitive, as well of a statistical nature (e.g. LDA), or biologically inspired (e.g. artificial neural networks, ANN). The algorithms may be arranged in different topologies (e.g. One-Vs.-One).

[0096] FIG. 6 illustrates an array of electrodes arranged on a limb of a person, for example the limb may be an affected limb 104. On the skin of the limb there is a high density array 602 of electrodes 603 (only one is numbered). The number of electrodes is at least six, but may be for example, 8, 10, 12 or more. The electrodes 603 are arranged close to each other in order to densely cover a large area of the limb. For example, the distance 604 between adjacent electrodes may be less than 30 mm, or even less such as less than 10 mm or less than 5 mm. With such a high density array 602, the placement of the individual electrodes 603 is less crucial, instead a control unit 108, 208, 608 may determine which signals are useful. The control unit 608 may be arranged adjacent to the electrodes 603 and may communicate wirelessly 610 with the control unit 108, 208. Alternatively, or additionally the control unit 108, 208 communicates directly with the electrodes via cables 614. In one example embodiment, the array 602 of electrodes extends around the outer circumference 606 of the limb in order to cover a large area around the limb 104. The control unit 108, 208, or 608 may determine from the signals acquired from the electrodes 603 of the array 602 which of the signals are useful. For example, the control unit 108, 208, or 608 may use a feature selection algorithm or a signal separation algorithm to determine if a signal is useful. The electrode arrangement embodiment shown in FIG. 6 may be used separate from the other embodiments described herein.

[0097] Additionally, variations to the disclosed embodiments can be understood and effected by the skilled person in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. For example, virtual reality is also possible within the context of the invention although only augmented reality is shown in the exemplary embodiment of FIG. 2. The system and method may further be used for controlling a robotic device such as a remotely controlled toy vehicle, helicopter, boat, robotic arm such as for example a prosthesis, etc. Furthermore, although three electrodes are shown in FIG. 1 and FIG. 2, any suitable number of electrodes is possible within the scope of the invention.

[0098] Furthermore, the methods disclosed and described may be used by a person not having an affected limb, thus the method and the system is equally applicable to a person having only non-affected limbs.

[0099] Furthermore, the communication between electrodes and the control unit may be either via wires of via wireless communication. In case of wireless communication, there may be a second control unit arranged nearby the electrodes.

[0100] In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage.