SYSTEM AND METHOD FOR MONITORING AND TEACHING CHILDREN WITH AUTISTIC SPECTRUM DISORDERS

Abstract

The invention relates to methods and systems for monitoring and teaching children with autistic spectrum disorders and can be used for effectively managing special educational work with children with autistic spectrum disorders (ASDs). According to the invention, the system comprises a remote server, personal computer devices of parents and specialists that are connected by an integrated network to the remote server, and a neuro-interface module for tracking a child's brain activity, said module being placed on the child and being connected by means of the integrated network to the remote server and comprising EEG sensors, wherein the neuro-interface module comprises an accelerometer and a gyroscope, and sensors for detecting a gaze direction, and the remote server is capable of collecting and analyzing visual data about the child's activity.

Claims

1. A system for monitoring and teaching children with autistic spectrum disorders, comprising: a teaching robot comprising at least a microprocessor of the robot, which microprocessor is connected to a data exchange module of the robot for data exchange with another device and with external devices by means of an integrated network, a personal computer device of a child comprising at least a microprocessor of the computer device of the child connected to a video camera of the computer device of the child, a display of the computer device of the child, a data exchange module of the computer device of the child for data exchange with another device and with external devices by means of the integrated network, wherein the system comprises a remote server comprising at least a microprocessor of the remote server, connected to the data exchange module of the remote server for data exchange with external devices by means of the integrated network personal computer devices of parents and specialists connected to the remote server by means of the integrated network, a neuro-interface module for tracking a child's brain activity placed on the child and connected by means of the integrated network to the remote server and comprising EEG sensors, wherein the neuro-interface module comprises an accelerometer and a gyroscope, sensors for detecting a gaze direction located on the personal computer device of the child, wherein the remote server is configured to collect and analyze visual data about the child's activity: facial expressions, gaze direction, and physical activity, and is also configured to automatically transmit data about the current and undesirable states of the child to the personal computer devices of parents and specialists.

2. The system of claim 1, wherein the remote server is configured to send signals in the form of push notifications with information about the current state of the child accompanied by a sound signal to the personal computer devices of parents and specialists.

3. The system of claim 1, wherein the EEG sensors include five frontal EEG sensors and two behind-the-ear contacts.

4. The system of claim 1, wherein the remote server is configured to process the incoming data in a neural network mode for accurate recognition of the child's emotions.

5. The system of claim 1, wherein the teaching robot has a drive for movement and/or simulation of movement.

6. A method for monitoring and teaching children with autistic spectrum disorders comprising the following: a teaching robot is used to communicate with a child, a personal computer device of a child is used to give tasks to the child, the device is connected by means of an integrated network for data exchange, wherein the method comprises the following steps: a remote server is used, which is connected by means of the integrated network to other devices, with all the learning tasks for the child stored on the server, personal computer devices of parents and specialists are used, which are connected to other devices by means of the integrated network, a child's brain activity is tracked using a neuro-interface module placed on the child, the body position of the child is monitored using an accelerometer and a gyroscope installed in the neuro-interface module, the child's emotions is monitored using sensors for detecting a gaze direction, all data are processed on the remote server with transmitting data about the current and undesirable states of the child from the server to the personal computer devices of parents and specialists.

7. The method of claim 9, wherein the method comprises sending push notifications with information about the current state of the child accompanied by a sound signal to the personal computer devices of parents and specialists.

8. The method of claim 9, wherein the method comprises processing the incoming data on the remote server in a neural network mode for accurate recognition of the child's emotions.

9. The method of claim 9, wherein the method comprises simulation of movements for demonstration to the child using a drive of the teaching robot.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Other distinctive features and advantages of the invention will be obvious from the description given below by way of illustration and not limiting, with reference to the accompanying drawings, in which:

[0087] FIG. 1 schematically illustrates layout of the system for monitoring and teaching children with autistic spectrum disorders according to the invention,

[0088] FIG. 2 shows the appearance of the neuro-interface,

[0089] FIG. 3 schematically illustrates the relationship of the components of the system according to the invention,

[0090] According to FIG. 1-3, the monitoring system, the system for monitoring and teaching children with autistic spectrum disorders comprises a training robot 1 comprising at least a microprocessor 11 of the robot, which microprocessor is connected to an optional video camera 12 of the robot, an optional speaker 13, an optional display 14 of the robot, and a data exchange module 15 of the robot for data exchange with external devices by means of the integrated network 2. The system comprises a personal computer device 3 of a child comprising at least a microprocessor 31 of the computer device of the child, which microprocessor is connected to a video camera 32 of the computer device of the child, a display 33 of the computer device of the child, and a data exchange module 34 of the computer device of the child for data exchange with external devices by means of the integrated network 2. The system comprises a remote server 4 comprising at least a microprocessor 41 of the remote server, which microprocessor is connected to a data exchange module 42 of the remote server for data exchange with external devices by means of the integrated network 2. Also, the remote server 4 has a module 43 for storing databases with all tasks and statistics thereon.

[0091] The system also comprises personal computer devices 5 of parents and specialists, connected by means of the integrated network 2 to the remote server 4. Additionally, the system comprises a neuro-interface module 6 for tracking a child's brain activity placed on the child and connected by means of the integrated network 2 with the remote server 4 and comprising EEG sensors 61, wherein the neuro-interface module comprises an accelerometer 62 and a gyroscope 63. The system also comprises sensors for detecting a gaze direction 7 located on the child's personal computer device 3. The remote server 4 is configured to collect and analyze visual data about the child's activity: facial expressions, gaze direction, and motor activity, and is also configured to automatically transmit data about the current and undesirable states of the child to the personal computer devices 5 of parents and specialists.

[0092] The remote server 4 is configured to send signals in the form of push notifications with information about the current state of the child accompanied by a sound signal to the personal computer devices 5 of parents and specialists.

[0093] EEG sensors include five frontal EEG sensors and two behind-the-ear contacts. See FIG. 2.

[0094] The remote server 4 is configured to process the incoming data in a neural network mode for accurate recognition of the child's emotions.

[0095] The teaching robot 1 has a drive 16 for movement and/or simulation of movement. In the general case, the robot may not have a drive, a video camera 12, a loudspeaker 13, and a display 14.

[0096] Dotted arrows indicate a connection by means of the integrated network for data exchange.

[0097] Implementation of the Invention

[0098] The system for monitoring and teaching children with autistic spectrum disorders operates as follows.

[0099] Software is installed on the child's personal computer device, which software is a learning environment and a program for processing data about the child's state. A state monitoring application is installed on the personal computer devices 5 of parents and specialists (mobile phone or other suitable device).

[0100] In accordance with the recommendations of specialists, blocks of tasks for the child's independent work are selected. The blocks are enabled by means of the learning environment, from the base of exercises 43 connected thereto.

[0101] The behavior of the teaching robot 1 is programmed in the tasks. The teaching robot 1 is able to maintain a conversation with the child (ask questions, give advice, give “feedback” on tasks, answer questions, etc.), and also can show various movements and postures.

[0102] To monitor the state, the appropriate sensors 6 are put on the child. The psychophysiological state is monitored using a neuro-interface 6. This device tracks brain activity. Normal and undesirable states are programmed. The system can be customized for a particular child, which consists in obtaining the child's EEG data in a normal state and recording them into the database. If there is a significant deviation from the normal state, alerting system is triggered and signals are transmitted to the personal computer devices 5 of parents and specialists.

[0103] Motor activity can be monitored using motion sensors 62 and/or 63 or a motion detection program using an image from the camera. The use of sensors is more reliable, but this method is not convenient in the conditions of daily work with a child. Monitoring of the motor activity is needed to complete some gamified tasks, as well as to track undesirable states (excessive motor activity is a warning sign).

[0104] An additional means of state monitoring is an emotion recognition program. For its correct operation it is necessary to use a web camera. When strong negative emotions are registered, alerting system is triggered and signals are transmitted to the personal computer devices 5 of parents and specialists.

[0105] Monitoring of the child's behavior using a gaze direction tracking system is also provided. It makes it possible to monitor such states when the child stops looking at the robot or display and/or stares into space.

[0106] All the means or some of them can be used at once.

[0107] Information about the child's state, including notifications about the onset of undesirable or dangerous states, is transmitted to the personal computer devices 5 of parents and specialists.

[0108] Reporting on the work done, tasks completed and child's success is implemented by the built-in means of the learning environment.

[0109] Management of the learning environment, including access to reporting, can be carried out remotely.

[0110] There are four levels of access to the educational platform hosted on the remote server 4:

[0111] System administrator. Can add both courses and tasks therein, create users, assign rights. This is the role of the system administrator, an employee of the development company.

[0112] Author of the course. Manages the course, can hide unnecessary blocks and customize repetition of the necessary ones, customize the calendar of lessons, and monitor the results. This is a role for an ABA training specialist, an employee of an educational center.

[0113] Assistant. Cannot edit the content of the course, but can monitor student's activities, check tasks, give comments. This role is for the parents of the child.

[0114] Student. Has access to tasks to be performed. The role is for the child.

[0115] The Main Functions of the Platform for the Teacher: [0116] To select a list of tasks for the student, depending on the individual plan. [0117] To track student's progress and adjust the learning plan. [0118] To track learning statistics.

[0119] The Main Functions of the Platform for the Parents: [0120] To select a list of tasks for the student, depending on the individual plan. [0121] To organize independent activities of the child. [0122] To track child's progress and adjust the learning plan.

[0123] Main Functions of the Platform for the Developers: [0124] Possibility to add new tasks for students on a regular basis. [0125] Creation of new courses, assignment of rights.

[0126] The Main Functions of the Platform for the Student: [0127] Constant access to the training platform. [0128] Developing lessons according to the individual plan under the supervision of the teacher. [0129] Constant online monitoring by parents and/or specialists.

[0130] Specific Example of Operation

[0131] Tasks are divided into seven modules in accordance with the technologies adopted in the ABA therapy, each of the blocks has its own learning goal and is implemented as a separate software set of tasks:

[0132] 1. Copying Movements

[0133] Movements, which the student should repeat, are shown to the student. Movements include actions with objects and selection from several objects, successive touching of objects, and also small motor, facial, and articulatory movements.

[0134] There are three levels of difficulty of actions with objects. [0135] Positional simulation. For example, putting a cube in a cup is easier than tapping a cube on a table. [0136] Actions with objects. For example, knock on the table, wave in the air. [0137] Movements with a change of an object. (For example, the first movement with a cube, the second movement with a ball) [0138] Demonstration of more complex actions (for example, assembling a building kit) without specific instructions; the child must repeat the actions himself

[0139] Graphomotor imitation (drawing, writing, tracing patterns, etc.).

[0140] A hand that draws any lines, shapes, letters, etc. is shown to the child, and the child is invited to repeat these actions.

[0141] Demonstration of movements is done by the robot 1 (commands are sent to the robot 1 from the server of the educational platform 4) or via a video clip on the platform (web application).

[0142] 2. Vocal and Verbal Imitation

[0143] Sounds, words, and phrases of different lengths are presented to the student. The student is invited to repeat them. It is implemented in the form of web application.

[0144] 3. Receptive Speech

[0145] Training of a passive vocabulary. Images of objects are presented to the student and each of them is called aloud (a noun vocabulary is formed). Then actions are presented and voiced (a vocabulary of verbs is formed). Then adjectives (colors, shape, size, etc.) and their antagonistic pairs (big-small, dry-wet, etc.) are presented and voiced. Then combinations (noun+verb, noun+adjective, noun+verb+adjective). It is implemented in the form of web application with the artificial intelligence (AI) complex connected for processing audio data. Microphone and loudspeakers are required.

[0146] 4. Naming

[0147] An object is shown to the child, and the child should to name it. The system recognizes and analyzes what the student said. At the next level, the child must name the movements. It is implemented in the form of web application with artificial intelligence (AI) complex connected for processing audio data. A microphone is required.

[0148] 5. Expressive Speech.

[0149] A picture is shown to the child, and the child should to describe it and answer questions about it. It is important that the child should not repeat the phrases that accompanied these pictures at the previous levels. For the successful completion of the task, the child must answer the question correctly. For example, the picture shows a white dancing hare. At the previous step, the child called this picture “White hare”, “Hare is dancing”, “White hare is dancing”. Now the child is asked questions: “Who is this?”, “What is he doing?”, “What color is he?”.

[0150] The sequence of pictures and questions is not obvious to the child.

[0151] 6. Development of the Visual and Cognitive Sphere

[0152] Sorting images into categories. Mosaic, puzzles. Tasks aimed at developing the skill of quick recognition and memorization of images. Continuation of logical sequences. Story sequences, restoring the sequence of events. It is implemented in the form of web application.

[0153] 7. Development of Playing Skills

[0154] The use of substitute items. Mastering the concept of transferring the move, playing by the simplest rules. Collecting puzzles together. Performing actions on a condition (for example, the child should click on a ball when his name is called). Board games: dominoes, dice. Sea battle, chess at a higher level. It is implemented in the form of web application.

[0155] Each module has several levels of difficulty. In each lesson, tasks from several blocks can be used, but it is necessary to maintain a balance of complexity and take into account the child's capabilities.

INDUSTRIAL APPLICABILITY

[0156] The proposed system for monitoring, system for monitoring and teaching children with autistic spectrum disorders can be implemented in practice by a specialist and, when implemented, provides the achievement of the claimed result, which allows to conclude that the criterion of “industrial applicability” for the invention is met.

[0157] In accordance with the proposed invention, a pilot system for monitoring, system for monitoring and teaching children with autistic spectrum disorders has been produced.

[0158] ROBOTIS MINI was used as a training robot for testing.

[0159] Technical parameters of the ROBOTIS MINI robot: [0160] Controller: OpenCM9.04-C [0161] Control interface: Bluetooth-module BT-210 [0162] Programming interface: COM port [0163] Power supply: 2 Li-Ion batteries LB-041 [0164] Drive mechanisms: 16 servos DYNAMIXEL XL-320 [0165] Drive connection interface: DYNAMIXEL TTL Bus (UART) [0166] Dimensions: 27 cm×35.5 cm×9.5 cm

[0167] The neuro-interface Muse was also used for testing, which is a single-channel, non-invasive EEG interface equipped with seven sensors, including five frontal sensors and two behind-the-ear contacts. This provides an excellent signal with minimal noise.

[0168] The data captured by the electrodes is the electrical activity of neurons. The formation of patterns of EEG signals is manifested at the moments when a significant number of neurons are synchronized and form a significantly high electrical activity in one period, which can be registered on the surface of the human head.

[0169] Therefore, the system receives information about the potential difference between the original raw EEG signal (main electrode) and zero point (reference electrode).

[0170] Technical characteristics of the microcontroller used:

[0171] The neuro-interface is based on the microcontroller PIC24 (peripheral interface controller) developed by the American company Microchip Technology Inc. The received signal from the sensors is processed and the positions of concentration and relaxation are established, with all EEG data being interpreted and sent to the output.

[0172] A triple monitoring system was used to monitor the child's state for testing purposes. The system used two cameras: a web camera on the child's personal computer device and another camera in the room, which gives the maximum view (a camera of the robot can be used), and also a neuro-interface comprising an EEG sensor, a gyroscope, and an accelerometer.

[0173] Emotional State Control

[0174] A simple high-precision neural network was used to determine the emotional state of the child based on video from the camera. To create the neural network, we used openCV and Keras, and the fer2013 dataset was taken to train it.

[0175] The neural network received an image from the web camera installed at the child's workplace, recognized a face on it, and then recognized the emotion expressed. In the case of a negative emotional state, the system notified the parents about it.

[0176] Detection of the Body Position

[0177] Detection of the body position was carried out based on the data from cameras using recurrent neural network.

[0178] Patterns of brain activity were detected based on the data received from the neuro-interface. This includes data on brain activity, body position and movements of the head.

[0179] As a result of the test operation of the system for monitoring and teaching children with autistic spectrum disorders, it was found that it makes it possible at once: [0180] To provide the child with a wide range of ABA tasks for development of speaking and behavioral skills according to the ABA programs. [0181] To control remotely the psychoemotional state of the child during lessons. [0182] The possibility for the sensors to effectively trigger when a hazardous situation arises. [0183] To vary the content according to the current state of the child. [0184] To analyze the results of the child's work with the complex. [0185] To implement reporting on the work done. [0186] The claimed system and method make it possible to optimize educational work with children with ASDs for the children themselves and those responsible for them (parents, teachers). [0187] The claimed system and method are designed for children of different ages with various manifestations of ASDs.

[0188] Therefore, the claimed system and method solve the problem set out and provide the achievement of the technical result, namely: the possibility of working with a child without the physical presence of an adult, that is, the possibility of automatic operation of the system with monitoring of the child's state and remote monitoring of states and managing the system.

[0189] An additional useful technical result of the claimed invention is that the invention provides: [0190] the possibility that the child is not overloaded with technical devices, [0191] the child being interested in the process.

[0192] Furthermore: [0193] Independent work of the child is possible. [0194] The program can be customized for each child. [0195] The work of teachers and psychologists becomes easier and more efficient. [0196] The claimed system and method make it possible to carry out lessons at home. At the same time, a specialist of an educational institution still has the opportunity to analyze the results of the child's work on the basis of built-in reporting systems (if needed). [0197] Not only external changes in the child's behavior can be observed, but also psychophysiological state of the child can be monitored.

[0198] The claimed system and method can also be used as an educational platform for training the communication and social skills of any child, not only having autistic spectrum disorders; they can also be used not only for organizing work, but even for resting any child.