SYSTEM FOR PROVIDING VIRTUAL PARTICIPATION IN AN EDUCATIONAL SITUATION

Abstract

A system is disclosed providing active participation for persons being prevented to be physically present in educational situations by means of a robot, a server and a personal device providing audio and video from a remote environment to the user, and virtual presence by means of emitting audio from the robot captured by the personal device and providing e.g. movements, mood indications and raise hand signals to the robot from the personal device by the user.

Claims

1. A system for virtual participation of a user in a remote environment, the system comprising: a robot localized in the remote environment, including at least one head part and one body part tiltably connected to each other, including at least a camera capturing video of the remote environment, a first microphone capturing audio from the remote environment, a first loudspeaker being able to emit audio captured from the user, a wireless connection means adjusted to connect the robot to a wireless network, a processing unit at least adjusted to code and stream video and audio, a Micro Controller Unit, MCU, adjusted to control one or more motor driver circuits driving one or more electrical motors being able to tilt the head part relative to the body part and to rotate the robot relative to the ground; a mobile user device including at least a second microphone being able to capture user audio, a second loudspeaker being able to emit captured audio from the remote environment and a touch screen being able to display the captured video of the remote environment; an app installed on the mobile user device at least adjusted to transmit an audio stream and control signals and movements commands to the MCU based on user input on the touch screen, characterized in one or more LEDs for displaying user status; and a server being in communication with the robot and the mobile user device, at least adjusted to provide a pairing procedure between the robot and the mobile user device, and to authenticate and initiate a direct communication between the robot and the mobile user device only if the robot and the mobile user device are paired.

2. The system of claim 1, wherein the server on request from the user to pair with the robot is adjusted to transmit a randomly generated passcode to the user, and wherein the app is adjusted to prompt the user to enter a passcode and return the entered passcode to the server which is adjusted to pair the app and the robot only if the returned passcode equals the randomly generated passcode.

3. The system of claim 1, wherein the wireless network is a mobile phone network.

4. The system of claim 1, wherein at least one of the LEDs for displaying user status also is adapted to display user mood.

5. The system of claim 1, wherein the robot further comprises a power supply circuitry.

6. The system of claim 1, wherein the robot further comprises a battery charger circuitry.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 schematically illustrates the overall system,

[0012] FIG. 2 is a flow chart illustrating the process of a user device connecting to a paired robot by means of a personal code,

[0013] FIG. 3 is a flow chart illustrating the initial pairing process of a user device with a robot,

[0014] FIG. 4 illustrates an example of how finger movements on the touch screen may change the captured view by the robot camera,

[0015] FIG. 5-7 are schematic views of the different hardware units in the robot and how they interact,

[0016] FIGS. 8-10 are illustrations on have events are exchanged between app, robot and server.

DESCRIPTION OF EMBODIMENTS

[0017] In the embodiments herein, systems and methods providing active participation for persons being prevented to be physically present in educational situations are disclosed. A particular situation being addressed is the case where children with long term illness needs assistance to actively participate in the education taking place in a class room. However, the embodiments herein may also be used in other similar situations like remote work, virtual presence for physically disabled people etc. For exemplary purposes, we will in the following description concentrate on class room situations, where a child which is at home or at the hospital is represented by a robot standing on the child's desk at school. The robot works as the child's eyes, ears and voice in the classroom.

[0018] As illustrated in FIG. 1, according to embodiments herein there is a system having three main components: A mobile application (app) being installed on a mobile device with a touch screen, a server system (server), and an avatar robot (robot).

[0019] The robot contains means for connecting to a wireless network or a mobile network, e.g. a 4G modem, and uses the mobile network to communicate with the server and the user's app. The robot may in overall be constructed by a head part and a body part which are tiltably connected. The body part could for instance be able to twist the robot 360 degrees in relation to ground. One or more electric drives should be installed providing the abovementioned rotational and tiling movements. The robot could further at least be provided with a camera, a speaker, a microphone, a computer unit and a robotic system.

[0020] The server is the glue in communication between the app and the robot. The server is in communication with the robot on a more or less continuous basis, even when the app is not open. When the user of the app opens the app, the app will contact the server and initiate a direct connection between the app and the robot (end to end communication). As will be explained in further details later, this communication will include control signals and video and audio streams.

[0021] One important aspect of the embodiments herein is that one robot is securely paired with one personal device (e.g. a mobile phone). Only the paired units are allowed to communicate with each other. This is done to ensure the privacy of the child and the teacher in the classroom.

[0022] There will never be any doubt who is logged on to the robot.

[0023] Because the robot is paired with one, and only one, mobile phone, a mobile network would be advantageously to use for communication as there would be no configuration to be done on the robot. WiFi would require the robot to be configured for each network it is to be used on.

[0024] The mobile app is the tool the children use to interact with their robots. Each robot will only accept connections from one app. Some examples of tasks being performed by the mobile app would be: [0025] Register mobile app against the robot, via a server [0026] Stream audio from the client, receive audio/video from the robot via WebRTC [0027] Send movement commands to the robot [0028] Display the state of robot within the app [0029] Send raise hand command [0030] Change volume output on the robot [0031] Express how the child is feeling today

[0032] Referring to FIG. 2-4, the abovementioned points will in the following be described in further details.

[0033] Register

[0034] As being illustrated with a flow chart in FIG. 3, when a user sets up a subscription for a robot, he will receive a randomly generated passcode which is used to pair the mobile app with the robot. Because one need to make sure it is only the child that can access the robot, the user must complete multistep process the first time he opens the app. In this registration process, he has to enter the passcode, his age, accept terms (e.g if under 18, parents have to accept), and create a personal code which is used to unlock the app. This is the personal code being referred to in the flow chart of FIG. 2, which illustrates overall connection procedure.

[0035] Send Movement Commands

[0036] The robot may be controlled by swiping on the screen while the video stream is active, similar to panning on a large image or scrolling on a web page. The picture seen on the user's screen will follow his finger movements. An example of this is illustrated in FIG. 4. The circles in FIG. 4a represent an imaginable movement of the finger on the picture on the touch screen captured by the robot's camera, spanning from a starting circle positioned approximately in the middle of the picture to an ending circle positioned nearby a door handle in the right hand side in the picture. The result of the movement is illustrated in FIG. 4b, where the position of the door handle relative to the picture frame now is positioned approximately in the middle of the picture. This is accomplished by tilt and rotational movements of the head part related to the body part corresponding to the user's finger movement on the touch screen.

[0037] State of the Robot

[0038] The inventors have experienced that children need to know how their robot looks in the classroom. It is also common in functionality in different video communication applications to have a small image showing how the user virtually appears in other persons view. According to some embodiments, a representation of the robot overlaid the stream. As an example, when a raise hand command is sent, a top light starts blinking on the robot. This may be represented in the app by a blinking top light icon appearing on the screen.

[0039] Other examples of the state of the robot may be: [0040] On/off state [0041] Small movement animations [0042] Battery status [0043] Emotional colours

[0044] Robot Hardware and Software

[0045] As illustrated in FIG. 5, apart from the mechanics and the robotic system, the robot may have three main units: [0046] Computer unit [0047] Robotics controller (i.e. for movement and indicators) [0048] 4G or wireless module

[0049] As illustrated in FIG. 6, the computer unit is implemented for handling two main tasks, namely audio/video processing and data communication handling messages and control signaling between the robot systems, the app and server.

[0050] The audio/video processing may be implemented in several ways, but in this example, an effective standard method referred to as WebRTC (Web Real-Time Communication) is used. WebRTC facilitates the coding/decoding and streaming of the audio and video data.

[0051] The computer unit should preferably be a small embedded computer board which runs the robot's main software connected to the camera, the microphone and the loudspeaker. It is further connected to the 4G modem which enables it to communicate with the mobile app and the server system.

[0052] It is also dispatching messages from the app and the server to the Robotics System's Micro Controller Unit (MCU).

[0053] Robotics System

[0054] Referring now to FIG. 7, the robotics system may at least comprise a Micro Controller Unit (MCU), Motor driver circuits, 2 stepper motors, LEDs for displaying status and the user's mood, a Power Supply circuitry, and a Battery charger circuitry.

[0055] The robot should be able to move in the horizontal and vertical plane.

[0056] For horizontal movements, the whole robot turns around. This enables full freedom of rotation, 360 degrees. This is required for the child to look around the whole classroom, even if the robot is placed on a desk in the middle of the room.

[0057] For vertical movement, the head part is enabled to tilt up and down relative to the body part. The freedom of tilting movement may be limited, e.g. to approximately 40 degrees to prevent mechanical damages. The camera should be located in the head part, making the user able to look up and down virtually look up and down.

[0058] The LEDs may be used to indicate several things: [0059] Robot eyes switched on when the user is connected to the robot [0060] Head light switched on when the user wants to raise hand. [0061] Mood lights displaying different colours based on the indicated mood of the user.

[0062] Mobile Modem Module

[0063] The mobile modem module may be a full GSM (2G), UMTS (3G) and LTE (4G) or another similar next generation mobile modem module used to transfer data between the robot and the app and server. The module may for instance be connected to the AV system via USB to enable high speed data transfer.

[0064] Server Systems

[0065] The server system communicates with both the robot and the app. When a user wants to connect to the paired robot, it will ask the server if the robot is online, and if so, request it to set up a connection. The connection is then set up between the robot and the app with no data going through the server.

[0066] WebSockets

[0067] WebSockets may be used as a communication platform between app, robot and server. FIGS. 8-10 are illustrations on have events are exchanged. The system is meant to be flexible so that new events can be added when the software and/or hardware adds more functionality.

[0068] Authentication (FIG. 8)

[0069] authenticate is transmitted from the app after connecting to the server. The event is emitted before any other events are emitted as the server will ignore them until the client is authenticated. A JSON Web Token string containing the login information is sent.

[0070] authenticated is emitted from server after client successfully authenticates. Empty payload.

[0071] unauthorized is emitted from server when client fails authentication. Can be emitted at any time, not only after client emits authenticate.

[0072] Server disconnects client after emitting.

[0073] Webrtc (FIG. 9)

[0074] All WebRTC signalling is sent through this event. It takes only one parameter: data. Which should be an object with type and data properties.

[0075] Robotics (FIG. 10)

[0076] Robotics commands are broadcast from App to Robot.

[0077] Specifications

[0078] App Specification

[0079] 1. WebRTC

[0080] a. Send local audio

[0081] i. User should be able to mute local audio

[0082] b. Receive and display video

[0083] i. H264

[0084] ii. Video should be displayed full screen

[0085] c. Get STUN and TURN servers from communication API

[0086] d. Gather statistics about stream quality

[0087] i. Aggregated and sent to our stats server

[0088] e. Signalling

[0089] i. Should emit and listen for signalling events

[0090] ii. Messages should be send: {type: String, data: Mixed}

[0091] 2. Register client

[0092] a. User should enter a code when client is not registered

[0093] b. Code should be sent to the communication server where the server will reply with a token if code is valid

[0094] i. Retrieved token should be stored securely on the device

[0095] c. Parents have to agree to terms to not access the stream

[0096] d. Child has to create personal code

[0097] 3. Ensure that child is the person which can access the robot

[0098] a. On opening application (after it has been registered) user has to enter personal code

[0099] b. If code is valid the user can connect to robot

[0100] 4. Connect to robot

[0101] a. Authenticate client using stored token

[0102] b. Should show a connecting screen

[0103] 5. Attention light

[0104] a. Send command to robot

[0105] b. Interface/button to send command

[0106] 6. Movement

[0107] a. Send move command with touch deltas

[0108] b. Tap to move

[0109] 7. Communicating mood

[0110] a. Interface to send mood

[0111] 8. Change audio level on robot

[0112] 9. Robot self status (representation of the current state of robot)

[0113] a. Battery status

[0114] b. Should show which lights are lit and their colour

[0115] c. Wake/sleep animations

[0116] d. Mood indicator

[0117] e. Movement

[0118] 10. Report errors and exceptions

[0119] Robot Hardware Specification [0120] Dimensions [0121] Weight: <1 kg [0122] Height: <30 cm [0123] Width: <20 cm [0124] Depth: <15 cm [0125] Battery [0126] Use: >6 h in use [0127] Standby: >18 h [0128] Size: 4 cell LiIon,

[0129] 12 Ah [0130] Voltage: 3.6 or 3.7 V [0131] Power [0132] 2 A USB charger [0133] Consumption Use: <1.5 A [0134] Standby: <500 mA [0135] Lights [0136] RGB eyes [0137] RGB top LED [0138] RGB circle of lights on neck or around speaker [0139] Media [0140] 8 ohm speaker [0141] Electret microphone [0142] >=5 Mpix camera [0143] Docking station [0144] USB power adapter connects to docking station [0145] Motors [0146] X axis stepper motor, 360 degree free movement [0147] Y axis stepper motor, 40 degree movement [0148] Y axis may need end stop(s) [0149] Real time PCB [0150] ATMEGA328P (may change to NXP or STM ARM M0) [0151] 2 motor driver [0152] Amplifier 1.5 W class D [0153] 2 step up battery>

[0154] 5V @ 1 A [0155] 1 charger (1.5 A) [0156] USB audio codec [0157] Computer System [0158] SnapDragon on a DART SD410

[0159] module [0160] Mobile System [0161] LTE module, one of: [0162] UBlox

[0163] TOBI L210 [0164] Telit LE910

[0165] The above description and illustrations are merely illustrative examples of different embodiments of the present invention, and is not limiting the scope of the invention as defined in the following independent claims and the corresponding summary of the invention as disclosed above.