LOWER LIMB REHABILITATION SYSTEM BASED ON AUGMENTED REALITY AND BRAIN COMPUTER INTERFACE

Abstract

A lower limb rehabilitation system based on augmented reality and a brain computer interface includes a display, a plurality of motion sensors, a brain wave monitor, and an analysis platform. The display is configured to receive and play a virtual scene video to guide a user to perform gait rehabilitation training. The plurality of motion sensors is configured to sense gait data. The brain wave monitor is configured to record an electroencephalogram signal by detecting an electric current change in a brain wave of the user. The analysis platform is configured to compare the gait data with the virtual scene video to determine the accuracy of footsteps of the user and provide feedback. The analysis platform inputs the electroencephalogram signal to a machine learning model to quantify the electroencephalogram signal into an index value representing a lower limb motor function of the user.

Claims

1. A lower limb rehabilitation system based on augmented reality and a brain computer interface, comprising: a display for a user to wear and configured to receive and play a virtual scene video for the user to watch, to guide the user to perform gait rehabilitation training; a plurality of motion sensors respectively disposed at a plurality of parts of a lower limb of the user and configured to sense gait data; a brain wave monitor configured to record an electroencephalogram signal by detecting an electric current change in a brain wave of the user, wherein the electroencephalogram signal is a brain wave signal in a brain motor area of the user; and an analysis platform coupled to the display, the plurality of motion sensors, and the brain wave monitor, wherein the analysis platform is configured to: store a plurality of virtual scene videos by using a database unit, and select the virtual scene videos from the database unit and transmit the virtual scene videos to the display; receive the gait data sensed by the plurality of motion sensors and compare the gait data with the virtual scene videos, to determine the accuracy of footsteps of the user according to a virtual sign generated by the virtual scene videos and provide the user with feedback; input the electroencephalogram signal to a machine learning model, so that the machine learning model quantifies the electroencephalogram signal into an index value, wherein the index value is used for representing a lower limb motor function of the user; and output the index value.

2. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, wherein the analysis platform has a display screen, and the display screen is configured to visualize an index value result determined by the machine learning model, for a rehabilitation therapist to observe a brain electrophysiological activity during the training of the user.

3. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, wherein the plurality of virtual scene videos has different rehabilitation difficulty levels, and the analysis platform is configured to select the virtual scene video having the corresponding difficulty level according to the index value of the user, for the user to perform gait rehabilitation training in conformity with a current status of the user.

4. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, wherein each virtual scene video has a music rhythm, and the analysis platform is configured to control the display to synchronously play the virtual scene video and the music rhythm, so that the user performs the gait rehabilitation training with beats of the music rhythm.

5. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, wherein the plurality of motion sensors is respectively disposed on a waist, two thighs, two calves, and at least one instep of the user, and a plurality of reference planes is defined by positions of the plurality of motion sensors.

6. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, wherein the display is configured to project and superimpose, onto the real world, a plurality of virtual signs in the virtual scene video, for the user to walk along the plurality of virtual signs.

7. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 1, the lower limb rehabilitation system further comprising a functional electrical stimulator coupled to the analysis platform, wherein the functional electrical stimulator is disposed on the lower limb of the user, and is configured to electrically stimulate a tibialis anterior muscle of the user, to cause the tibialis anterior muscle of the user to contract.

8. The lower limb rehabilitation system based on augmented reality and the brain computer interface as claimed in claim 7, the lower limb rehabilitation system further comprising an alarm coupled to the analysis platform, wherein the analysis platform is configured to evaluate whether the index value is greater than an index threshold, and if an evaluation result is no, the analysis platform controls the alarm to transmit a warning signal to remind a rehabilitation therapist to adjust a parameter of the functional electric stimulator.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will become more fully understood from the detailed description given hereinafter and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, and wherein:

[0022] The sole FIGURE is a block diagram of a system according to a preferred embodiment of the present invention.

[0023] In the various FIGURES of the drawings, the same numerals designate the same or similar parts. Furthermore, when the terms “inner”, “outer”, “top”, “bottom”, “front”, “rear” and similar terms are used hereinafter, it should be understood that these terms have reference only to the structure shown in the drawings as it would appear to a person viewing the drawings, and are utilized only to facilitate describing the invention.

DETAILED DESCRIPTION OF THE INVENTION

[0024] Referring to the FIGURE, a preferred embodiment of a lower limb rehabilitation system based on augmented reality and a brain computer interface according to the present invention includes a display 1, a plurality of motion sensors 2, a brain wave monitor 3, and an analysis platform 4. The display 1, the plurality of motion sensors 2, and the brain wave monitor 3 are coupled to the analysis platform 4.

[0025] The display 1 is provided for a user to wear, and is configured to receive and play a virtual scene video for the user to watch, to guide the user to perform gait rehabilitation training. In this embodiment, the display 1 may project and superimpose, onto the real world, a plurality of virtual signs in the virtual scene video, for the user to walk along the plurality of virtual signs. For example, the display 1 may be smart glasses such as Microsoft HoloLens, and have functions such as augmented reality (AR), gesture recognition, voice recognition, iris recognition, and the like. The display 1 may also be other head-up or head-mounted displays having the same functions. The present invention is not limited in this regard.

[0026] The plurality of motion sensors 2 is respectively disposed at a plurality of parts of a lower limb of the user and is configured to sense gait data. In this embodiment, each motion sensor 2 may be a six-axis sensor. The six-axis sensor includes a three-axis accelerometer and a three-axis gyroscope, such as MPU6050 launched by InvenSense Inc. Preferably, each motion sensor 2 may be a nine-axis sensor. The nine-axis sensor may be a combination of a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer, a combination of a six-axis accelerometer and a three-axis gyroscope, or a combination of a three-axis accelerometer and a six-axis gyroscope.

[0027] Specifically, a quantity of the plurality of motion sensors 2 is preferably six to seven. The motion sensors may be respectively disposed at a waist, two thighs, two calves, and at least one instep of the user. Each two motion sensors 2 may form a pair and define a reference plane to conclude a coordinate coefficient of the knee joint of the user, thereby comprehensively measuring a changing angle at joints of the lower limb. For example, on one lower limb, the motion sensors 2 at the waist and the thigh may form a first pair, the motion sensors 2 at the thigh and the calf can form a second pair, and the motion sensors 2 at the calf and the instep can form a third pair. For instance, two motion sensors 2 are respectively disposed at the thigh and the calf to record a change of position coordinate of the thigh and the calf on the same plane, thereby concluding the coordinate coefficient of the knee joint. Namely, the detection of the changing angle at joints is performed by analyzing the position coordinate of two adjacent parts of the lower limb. The gait data may include information such as a position, an angle, a speed, and an acceleration of the joints of the lower limb of the user when walking. Therefore, data such as a step speed, a step frequency, a step distance, and symmetry of the user can be calculated accordingly.

[0028] The brain wave monitor 3 is configured to record an EEG signal by detecting an electric current change in a brain wave of the user. The EEG signal refers to an EEG signal in a brain motor area of the user. In this embodiment, the brain wave monitor 3 may be a wearable brain wave electrode cap, and is configured to record brain wave power values in frequency bands such as α, β, δ and θ in the EEG signal of the user.

[0029] The analysis platform 4 is coupled to the display 1, the plurality of motion sensors 2, and the brain wave monitor 3. In this embodiment, a Raspberry Pi 3/4 may be used as the analysis platform 4. The analysis platform 4 stores a plurality of virtual scene videos by using a database unit 41. The analysis platform 4 selects one of the virtual scene videos from the database unit 41, and transmits the virtual scene video to the display 1, so that the display 1 plays the virtual scene video for the user to perform gait rehabilitation training according to the virtual scene video. The analysis platform 4 receives the gait data sensed by the plurality of motion sensors 2 to compare the gait data with the virtual scene video, to determine the accuracy of footsteps of the user according to a virtual sign generated by the virtual scene video and provide the user with feedback. The form of feedback may include a voice prompt or a video prompt, and the present invention is not limited thereto.

[0030] The analysis platform 4 inputs the EEG signal to a machine learning model 42, so that the machine learning model 42 quantifies the EEG signal into an index value. The index value is used for representing a lower limb motor function of the user. In this embodiment, a larger index value indicates that the lower limb motor function of the user approximates that of a healthy person. The analysis platform 4 outputs the index value. The machine learning model 42 is, for example, but not limited to being trained by using a support vector machine (SVM). One of ordinary skill in the art may understand the technology of the SVM, and details will not be described herein.

[0031] It is to be noted that the plurality of virtual scene videos may have different rehabilitation difficulty levels such as elementary, intermediate, advanced and a customized rehabilitation difficulty level. The analysis platform 4 may select the virtual scene videos having the corresponding difficulty level according to the index value of the user, for the user to perform gait rehabilitation training in conformity with a current status of the user. On the other hand, each virtual scene video may have a music rhythm. The display 1 synchronously plays the music rhythm while playing the virtual scene video, so that the user can perform the gait rehabilitation training with beats of the music rhythm.

[0032] The analysis platform 4 of the lower limb rehabilitation system based on augmented reality and a brain computer interface according to the present invention may further include a display screen 43. The display screen 43 is configured to visualize an index value result determined by the machine learning model 42, for a rehabilitation therapist to observe a brain electrophysiological activity during the training of the user. The display screen 43 may be, for example, but is not limited to a common computer screen, or mobile devices having a display function, such as a smart phone, a tablet, or a laptop. The user may capture a picture displayed on the display screen 43 and transmit the picture to the rehabilitation therapist, for the rehabilitation therapist to observe the brain electrophysiological activity of the user.

[0033] The lower limb rehabilitation system based on augmented reality and a brain computer interface according to the present invention may further include a functional electric stimulator (FES) 5. The FES 5 is disposed on the lower limb of the user and is coupled to the analysis platform 4. The analysis platform 4 may control the FES 5 to electrically stimulate a tibialis anterior muscle of the user, to cause the tibialis anterior muscle of the user to contract. In this way, the system can avoid foot drop when the user performs the gait rehabilitation training, and can assist the user in walking. Specifically, the analysis platform 4 may analyze, according to the gait data such as ankle joint angles and hip joint angles of the user, whether the user has the foot drop. If an analysis result is “Yes”, the FES 5 is controlled to electrically stimulate the tibialis anterior muscle of the user. If the analysis result is “No”, no extra operation is performed.

[0034] The lower limb rehabilitation system based on augmented reality and a brain computer interface according to the present invention may further include an alarm 6 coupled to the analysis platform 4. The analysis platform 4 can evaluate whether the index value is greater than an index threshold. If an evaluation result is “Yes”, the analysis platform 4 does not need to perform an extra operation. If the evaluation result is “No”, the analysis platform 4 may control the alarm 6 to transmit a warning signal to remind the rehabilitation therapist to adjust a parameter of the FES 5, thereby ensuring that the user can finish the gait rehabilitation training as scheduled. The alarm 6 may be, for example, but is not limited to a light-emitting diode, a buzzer, or a combination thereof, and is configured to transmit a warning signal such as warning light, a warning sound, or a combination thereof. The present invention is not limited in this regard.

[0035] In use of the lower limb rehabilitation system based on augmented reality and a brain computer interface according to the present invention, the user (for example, a stroke patient) wears the display 1 and the brain wave monitor 3 on the head, and the plurality of motion sensors 2 is disposed at body parts such as a waist, a thigh, a calf, an instep, and the like. The user or the rehabilitation therapist controls the analysis platform 4 to select a virtual scene video in conformity with the current rehabilitation difficulty level of the user, so that the analysis platform 4 transmits the virtual scene video to the display 1. The virtual scene video may be constructed by Unity. The display 1 projects and superimposes, onto the real world, two virtual channels and a plurality of virtual signs in the virtual scene video. The plurality of virtual signs is respectively located in one of the virtual channels, and move toward the user along the virtual channels with the music rhythms. In this way, the user can perform gait rehabilitation training with the beats of the music rhythms according to the plurality of virtual signs. When the user performs gait rehabilitation training, the analysis platform 4 receives the gait data sensed by the plurality of motion sensors 2, and analyzes whether an angle of bending the knee joint of the user reaches a predetermined threshold (for example, 30 degrees), and the virtual sign does not move to the rear of the user yet. If the analysis result is “Yes”, the analysis platform 4 controls the virtual scene video to generate a virtual object, and causes the virtual object and the virtual sign on the corresponding virtual channel to offset each other, thereby obtaining a rehabilitation score. If the analysis result is “No”, no extra operation is performed.

[0036] The analysis platform 4 inputs the EEG signal sensed by the brain wave monitor 3 to the machine learning model 42, so that the machine learning model 42 quantifies the EEG signal into an index value (for example, in a range of 1 to 100) used for representing a lower limb motor function. In this way, the user can learn the rehabilitation level of the lower limb. Further, during the gait rehabilitation training, the user may attach the FES 5 to the tibialis anterior muscle to facilitate contraction of the tibialis anterior muscle through electrical stimulation, to avoid the foot drop. Further, the analysis platform 4 may evaluate whether the index value is greater than an index threshold (for example, 70). If the evaluation result is “No”, the alarm 6 is controlled to transmit a warning signal, to remind the user and the rehabilitation therapist to adjust a parameter of the FES 5, thereby improving the effectiveness of rehabilitation of the user.

[0037] Based on the above, according to the lower limb rehabilitation system based on augmented reality and a brain computer interface of the present invention, the display can be used to play the virtual scene videos for the user to watch, to guide the user to perform gait rehabilitation training. Gait data sensed by the plurality of motion sensors is compared with the virtual scene videos to determine the accuracy of footsteps of the user according to the virtual sign generated by the virtual scene videos, and provide the user with the feedback on the rehabilitation training. The analysis platform detects, by using the brain wave monitor, the EEG signal of the user after performing the gait rehabilitation training, and inputs the EEG signal to the machine learning model to evaluate and quantify the effectiveness of the gait rehabilitation training of the user, thereby obtaining and outputting the index value representing the lower limb motor function of the user. In this way, according to the present invention, the user may directly use the lower limb rehabilitation system based on augmented reality and a brain computer interface at home without the need to go to the hospital. Therefore, the time and costs for commuting between the home and the hospital can be saved, and the effectiveness of the gait rehabilitation training of the user can be learned immediately.

[0038] Although the invention has been described in detail with reference to its presently preferable embodiments, it will be understood by one of ordinary skill in the art that various modifications can be made without departing from the spirit and the scope of the invention, as set forth in the appended claims.