Computer-implemented method and system for testing or training a users cognitive functions

20170025033 ยท 2017-01-26

Assignee

Inventors

Cpc classification

International classification

Abstract

Disclosed are a computer-implemented method and system for testing and/or training visual, cognitive and/or eye-hand coordination functions or skills of a user, using a computer or data processing device having a touch-sensitive display, wherein at least one line-shaped object (100) is presented on the touch-sensitive display, wherein the user has to retrace the line(s) of the presented object (100) and wherein the user's retraced line(s) (105) is (are) evaluated in order to determine a characteristic or parameter which relates to deviations and/or conformity (120) between the presented at least one line-shaped object (100) and the user's retraced line(s) (105).

Claims

1. A computer-implemented method for testing and/or training visual, cognitive and/or eye-hand coordination functions or skills of a user, using a computer or data processing device having a touch-sensitive display, characterized in that presenting at least one line-shaped object (100) on the touch-sensitive display, wherein the user has to retrace the line(s) of the presented object (100) and wherein the retraced line(s) (105) is (are) evaluated in order to determine a characteristic which relates to deviations and/or conformity (120) between the presented at least one line-shaped object (100) and the retraced line(s) (105).

2. Method according to claim 1 wherein the determined characteristic is used to assess the user's visual, cognitive and/or eye-hand coordination functions, or the user's health status.

3. Method according to claim 1 or 2, wherein the characteristic is a scalar number or a multi-dimensional number.

4. Method according to any of the preceding claims wherein the characteristic is determined by a linear or non-linear summation or integration of single deviation values.

5. Method according to any of the preceding claims, wherein the characteristic is determined based on the following evaluation steps: Evaluating an average pixel distance for pixels that are drawn away from a nearest original pixel; evaluating an average variance that holds information about how consistently the drawing is made; evaluating a curve length in comparison with an original curve length; evaluating an average inverse pixel distance that holds information about how an original pixel is away from the nearest drawn pixel; evaluating the time required for retracing the line(s) of the presented object.

6. Method according to claim 5, wherein the average pixel distance (apd) is calculated based on the following equation (1) apd = .Math. i = 1 n .Math. .Math. distance ( P ( i ) drawing , P ( i ) original ) n ; ( 1 ) wherein the average variance (var) is calculated based on the following equation (2) var = .Math. i = 1 n .Math. .Math. ( distance ( P ( i ) drawing , P ( i ) original ) - apd ) 2 n ; ( 2 ) wherein the curve length (cl) and an according curve length difference (cld) are calculated based on the following two equations (3) cl = .Math. i = 1 m - 1 .Math. .Math. ( DP ( x ) i + 1 - DP ( x ) i ) 2 + ( DP ( y ) i + 1 - DP ( y ) i ) 2 .Math. .Math. cld = .Math. cl drawing - cl original .Math. ; ( 3 ) wherein the average inverse pixel distance (ipd) is calculated based on the following equation (4) ipd = .Math. i = 1 n .Math. .Math. distance ( P ( i ) original , P ( i ) drawing ) n ; ( 4 ) wherein the time needed the time required for retracing the line(s) of the presented object to draw is measured in seconds.

7. Method according to claim 6 wherein calculating the following modular quotients: q apd = min 100 .Math. ( max 0 .Math. 100 .Math. apd worst - apd apd worst - apd best ) q var = min 100 .Math. ( max 0 .Math. 100 .Math. var worst - var var worst - var best ) q cld = min 100 .Math. ( max 0 .Math. 100 .Math. cld worst - cld cld worst - cld best ) q ipd = min 100 .Math. ( max 0 .Math. 100 .Math. ipd worst - ipd ipd worst - ipd best ) q time = min 100 .Math. ( max 0 .Math. 100 .Math. time worst - time time worst - time best ) .

8. Method according to claim 7 wherein, based upon the modular quotients, calculating a main score as an average score (score.sub.avg) or a multiplicative score (score.sub.mult), as follows: score avg = q apd + q var + q cld + q ipd + q time 5 .Math. .Math. or score mult = 100 .Math. ( q apd 100 .Math. q var 100 .Math. q cld 100 .Math. q ipd 100 .Math. q time 100 ) .

9. Method according to any of claims 5 to 8 wherein the time elapsed between drawing two lines of the presented object are evaluated, in particular compared with provided corresponding data of test patients having a certain disease.

10. Method according to any of the preceding claims wherein a relative displacement of the position of a retraced object, in relation to the position of the presented object, is evaluated.

11. Method according to any of the preceding claims wherein the lines of the presented object are retraced using at least two moving directions.

12. Method according to any of the preceding claims wherein the order of retraced lines is evaluated.

13. Method according to any of the preceding claims wherein the line width and/or the pressure being applied by the user when retracing an object are evaluated.

14. Method according to any of the preceding claims, wherein the inside of a presented line-shaped object has to be colored by the user and wherein evaluating conformity of the coloring with the border lines of the presented object.

15. Method according to any of the preceding claims wherein presenting an at least partially hidden or masked line-shaped object.

16. Method according to claim 15 wherein presenting at least two supporting points of a presented object.

17. Method according to any of the preceding claims wherein presenting an object only temporarily.

18. A computer-implemented system for testing and/or training visual, cognitive and/or eye-hand coordination functions or skills of a user, wherein the system comprises a touch-sensitive display, characterized by processing means for executing the method according to any of the preceding claims.

19. System according to claim 18 comprising processing means for presenting at least one line-shaped object (100) on the touch-sensitive display, for detecting information about line(s) of the presented object (100) being retraced (105) by the user, using the touch-sensitive display, and for evaluating the detected information in order to determine a characteristic which relates to deviations and/or conformity (120) between the presented at least one line-shaped object (100) and the retraced line(s) (105).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The objects, features and advantages of the invention are understood within the context of the Detailed Description, as set forth below. The Detailed Description is understood within the context of the accompanying drawings, which form a material part of this disclosure, wherein:

[0032] FIGS. 1A-C depict an enlarged view of a computer-generated, pixel-oriented circular shape, together with a user's retraced line, in order to illustrate the method according to the invention;

[0033] FIGS. 2A, B depict an overview and enlarged view of a computer-generated arbitrary shape, together with a user's coloring inside the shape, according to an embodiment of the invention;

[0034] FIGS. 3-8 shall illustrate different application scenarios of the invention;

[0035] FIG. 9 is a flow diagram of a preferred embodiment of the method according to the invention.

DETAILED DESCRIPTION

[0036] FIGS. 1A und 1B show a corner section (FIG. 1A) and an enlarged view (FIG. 1B) of a pixel-oriented quadratic or rectangular shape (line 100 in FIG. 1A) being presented on a touch-sensitive display or screen of an underlying digital processing or computer device which has been re-drawn or re-traced according to line 105 by a user of the device. It can be seen that the retraced line 105 deviated from the presented line 100 wherein the deviation is highlighted by means of hatching lines.

[0037] For the part within the circle 110 shown in FIG. 1A, an enlarged or zoomed view is depicted in FIG. 1B. The mentioned pixels are shown as dots 115 in FIG. 1B. The deviation between each pixel and a corresponding point on the re-traced line are depicted as vectors 120. It can be seen that these vectors 120 comprise different signs depending on the direction of an underlying deviation.

[0038] Only for illustrative purposes, FIG. 1C shows another shape 100 and a corresponding re-traced line 105, namely a section of a circular shape. As in the previous example, the deviation is highlighted by hatching lines.

[0039] Based on the above described deviations or vectors 120, the following algorithm, in the present embodiment a linear equation, is used to calculate, as a characteristic number, the integral value of all deviations along the re-traced line 105, 105. The characteristic number in this example is scalar but can be multi-dimensional, e.g. a vector or matrix, in case of multi-dimensional equation system being used instead of the present linear equation. The described algorithm provides for an automated calculation of the conformity or deviation between an arbitrary presented shape and a re-drawn or re-traced line or shape. Thus advantageously, an immediate feedback by the system to the user is possible.

[0040] FIGS. 2A and 2B show an embodiment of a shape 200, together with an enlarged view of the shape's border, wherein a user has to color or fill in the inside area 205 of the shape using a touch pen or a user's finger. As in the previous example, the user's coloring will not be perfectly aligned with the shape's border thus causing deviations between the shape's border 200 being presented by the computer and the border line 210 of the coloring.

[0041] The above illustrated deviations or conformity between a presented shape and a retraced line or coloring drawn by the user, in the present embodiment, will be evaluated based on the following described calculation steps and corresponding mathematical formulas.

[0042] The calculation of a drawing's key quantity, compared to the presented original, is based upon the following modular steps, where it is assumed that a drawing consists of x*y=n pixels (P) and m drawing points (DP), with DPP. [0043] Step 1: Average pixel distance (apd)

[00007] apd = .Math. i = 1 n .Math. .Math. distance ( P ( i ) drawing , P ( i ) original ) n ( 1 )

[0044] The average pixel distance sets the average number of pixels that a drawn pixel is away from the nearest original pixel where a bigger distance leads to higher values. It holds information about how close the (re-)drawn lines are to the original, but gives no information about how much of the original lines are covered. [0045] Step 2: Average Variance (var)

[00008] var = .Math. i = 1 n .Math. .Math. ( distance ( P ( i ) drawing , P ( i ) original ) - apd ) 2 n ( 2 )

[0046] The average variance holds information about how consistently the drawing is made (see FIGS. 1A-C). In most cases, a varying distance between the (re-)drawn and the original line leads to higher values. [0047] Step 3: Curve Length (cl) and Curve Length Difference (cld)

[00009] cl = .Math. i = 1 m - 1 .Math. .Math. ( DP ( x ) i + 1 - DP ( x ) i ) 2 + ( DP ( y ) i + 1 - DP ( y ) i ) 2 .Math. .Math. cld = .Math. cl drawing - cl original .Math. ( 3 )

[0048] Based upon the drawing points, the curve length is calculated to be compared to the original curve length. A bigger difference between drawing and original leads to higher values of cld. [0049] Step 4: Average Inverse Pixel Distance (ipd)

[00010] ipd = .Math. i = 1 n .Math. .Math. distance ( P ( i ) original , P ( i ) drawing ) n ( 4 )

[0050] The average inverse pixel distance sets the average number of pixels that an original pixel is away from the nearest drawn pixel where a bigger distance leads to higher values. It holds information about how close the original lines are to the drawn ones and thereby gives information about how much of the original lines are covered. [0051] Step 5: The Time Needed to Draw


time=seconds needed(5)

[0052] The amount of time which is needed to perform the drawing is measured in seconds.

[0053] For calculating the key quantity, quotients based upon these values (1)-(5) are built or calculated which transcribe them to values between values 0 and 100, wherein a value of 0 means a big difference between the original and the drawing curves or lines, and wherein a value of 100 means an essentially perfect score. For this, additional worst and best values are taken into account for every quotient (1)-(5) as follows:


apd.sub.best=1, apd.sub.worst=72


var.sub.best=1,5, var.sub.worst=15


cld.sub.best=10, cld.sub.worst=cl.sub.original


ipd.sub.best=0, ipd.sub.worst=128


time.sub.best=1, time.sub.worst=24(6)

wherein, based on these value pairs (6), calculating the following modular quotients:

[00011] q apd = min 100 .Math. ( max 0 .Math. 100 .Math. apd worst - apd apd worst - apd best ) .Math. .Math. q var = min 100 .Math. ( max 0 .Math. 100 .Math. var worst - var var worst - var best ) .Math. .Math. q cld = min 100 .Math. ( max 0 .Math. 100 .Math. cld worst - cld cld worst - cld best ) .Math. .Math. q ipd = min 100 .Math. ( max 0 .Math. 100 .Math. ipd worst - ipd ipd worst - ipd best ) .Math. .Math. q time = min 100 .Math. ( max 0 .Math. 100 .Math. time worst - time time worst - time best ) ( 7 )

[0054] Based upon these modular quotients (7), the key quantities for different approaches can now be calculated. Taking all quotients (7) into account, the two main scores are an average score (8) and a multiplicative score (9), as follows:

[00012] score avg = q apd + q var + q cld + q ipd + q time 5 ( 8 ) score mult = 100 .Math. ( q apd 100 .Math. q var 100 .Math. q cld 100 .Math. q ipd 100 .Math. q time 100 ) ( 9 )

[0055] Generally speaking, the multiplicative score is somewhat stronger and thus it is required to be more accurate to get as high results as with the average score.

[0056] The present invention can be used in various different application scenarios, examples or embodiments of which are described in the following, referring to FIGS. 3 to 8.

[0057] Referring to the embodiment illustrated in FIGS. 3A and 3B, a touchpad 300 of a portable computer is shown where the user input is accomplished using a touch pen 305. The two FIGS. 3A and 3B illustrate this action or process at two consecutive time stages. Of course, user input can also be accomplished using a user's finger. The area 310 on the touch screen 300 provides information to the user regarding conformity or deviation of the user's input in relation to a presented object or structure. In this and the following embodiments, the presented information is the conformity of a user's input with a presented object or structure, namely in units of percentage [%].

[0058] In general, the presented and to be redrawn object(s) or structure(s) are basic mathematic shapes like one or more circles, squares, rectangles, or triangles. In case of one circle being presented to the user, three or more supporting points are required to clearly define the expected shape and thus at least three points are presented which have to be crossed or cut by the re-drawn (retraced) circle. In case of a square, accordingly, three supporting points to be located somewhere on different lines of the square are required to clearly define the expected shape and thus at least three points are presented which have to be crossed or cut by the drawn (retraced) square. In case of a triangle, accordingly, three supporting points to be located at different lines of the triangle are required to clearly define the expected shape and thus at least three points are presented which have to be crossed or cut by the drawn (retraced) triangle.

[0059] Alternatively, instead of presenting the mentioned required number (in the above cases three) of supporting points, in the present embodiment, only two points 315, 320 are presented asking the user to draw e.g. a circle 325, 330 wherein the two supporting points 315, 320 in this embodiment define the circle's diameter or may be located somewhere on the circle thus allowing the user to draw a circle with an arbitrary diameter. Accordingly, a square can be clearly defined by only two supporting points if they represent opposing corners of the square. In case of a triangle, the user can be asked to draw an equal-sided triangle based on only two presented points. As illustrated by the embodiment shown in FIGS. 4A and 4B, which also depicts a touch screen 400, pen 405 and information area 410, a circle 415 or the like can even be drawn free-hand by the user without any supporting points, and thus also with an arbitrary diameter. In the two cases where a circle or the like is drawn with an arbitrary diameter (or size, in cases of other regular shapes like a square), the final shape 420 drawn by the user is compared with a corresponding shape which is calculated e.g. by a least-square matching calculation (least square fit).

[0060] The above described method can be advantageously used in the fields of human medical diagnosis and/or therapy. Now referring to an embodiment not shown in a figure, the presented samples of object(s), structure(s) or pattern(s) comprise preferably standardized shapes for which other patient's data, e.g. data of test patients, are available, e.g. patients suffering from an apoplectic stroke (Apoplexia) or any other brain damage who have a reduced ability to draw line-shaped objects etc. Based on the method and system according to invention, such patients advantageously can perform therapeutic exercises without the attendance of medical personnel and particularly obtain the positive results of such exercises without any delay thus motivating them to continue with these exercises.

[0061] Referring to the embodiment illustrated in FIGS. 5A and 5B, which again depicts a touch screen 500 and an information area 505, wherein the user input, in FIG. 5A, is accomplished using a user's finger 510, and in FIG. 5B, using a digitizer or pen 520. The presented sample(s) of object(s), structure(s) or pattern(s) are not generated by the computer but the user him-/herself. In this embodiment, the user first draws an arbitrary line-shaped object, structure or pattern 515, 525 which is stored by the computer. Such a drawn object, structure or pattern is used as a kind of master and will be compared with (an) successively drawn shapes of (an) as much as possible identical object(s), structure(s) or pattern(s). Evaluation of conformity or deviation between successively drawn shapes 530, 535, 540, 545 can be used to determine the user's cognitive status in view of repeating events, as illustrated in FIGS. 5C and 5D, e.g. his/her power of concentration. The determined user's cognitive status, in the present embodiment, is presented to the user by means of different information areas 505, each of them being related to one of the shapes 535, 540, 545. Such an embodiment thus can be used to detect a mentioned ADHD syndrome. As in some of the herein described other embodiments, other parameters like the time required to retrace an object or part of an object, the applied pressure when drawing an object or part of an object, the order of drawing lines, or the direction of drawing lines can be detected or monitored thus enhancing the validity of the evaluation results.

[0062] Now referring to the embodiment illustrated in FIGS. 6A and 6B, which again shows a touch screen 600 with horizontal (auxiliary) drawing lines 605, several information areas 610 and a user's input pen 615. The objective in the present scenario is to teach a language alphabet. In this embodiment, the presented object(s) is (are) language characters. In the present embodiment, sample characters for upper case 620 and lower case letters 625 are presented by the computer on the left side of the touch screen 600 and the user draws or enters the character at a different position of the screen. In the present embodiment, the two distinct user entry areas allow a user to redraw both upper and lower case letters. However, this is only a certain application scenario, and thus two or even more entry areas or fields can be used for learning purposes as well, or any other purpose. The above described fast algorithm for evaluation of the entered characters allows for an immediate feedback which provides enhanced motivation to the user to continue with the learning exercises.

[0063] Referring to the embodiment illustrated in FIGS. 7A and 7B, again showing a touch screen 700, an input pen 705 and an information area 710, the object(s) or structure(s) are letters or characters 715 being used for calligraphy, e.g. those being used in Japanese language. The particular aspect of such characters is the varying line width and with real drawing of such characters on paper thus required different pressure to be applied by the user to the pen when drawing the character 715, 715, 715. Accordingly, in this embodiment, the user is asked to retrace a presented character wherein varying the pressure applied to the input device, or the user's finger if being used as input device. In addition, when drawing Japanese characters it is essential to consider the right order of lines to be drawn. Insofar, evaluation of conformity or deviations between the presented character and the retraced character takes the line width and order of drawing the lines into consideration. It is emphasized that the final drawing does not show the order of drawing lines but using a touch screen, graphic tablet or the like enables to monitor and/or to protocol the order of entered lines. In the present embodiment, areas of the drawn (retraced) character which are not conform or congruent with the presented character (see e.g. 715), are highlighted or marked, e.g. using a certain color scheme or the like.

[0064] Referring to another embodiment illustrated in FIGS. 8A to 8C, showing three time stages of a process according to the invention using again a touch screen 800, the samples of object(s), structure(s) or pattern(s) are presented only temporarily, i.e. only during the first stage (FIG. 8A) 805 and then disappearing as illustrated by the blank screen 810 during the second stage (FIG. 8B), and thus have to be redrawn or retraced 805 by the user during the third stage (FIG. 8C), in the present embodiment by a user's finger 815, based on the user's short-term memory, wherein the user's input is evaluated like with a memory test. This embodiment particularly allows for testing (assessing) or training the user's capacity for remembering.

[0065] Now referring to the flow diagram depicted in FIG. 9, according to the first step 900 of the shown routine, a pixel- or vector-oriented shape to be presented on a touch screen is generated. The generated shape is then presented 905 on the touch screen. In step 910 any user input is detected and the corresponding pixel coordinated being stored at least intermediately. Of course, step 910 thus may consist of a number of recurring single steps which are implemented to detect the user input on a discrete i.e. pixel-oriented basis.

[0066] In step 915 of the shown routine, the stored pixels of the user input are compared with corresponding pixels of the computer-presented shape. This comparison reveals deviation values which are summed-up or integrated in step 920. As illustrated by the dashed arrow line, this summing-up can be done on a recurring pixel-per-pixel basis until all pixels have been considered. As described above, the summing-up can be performed either using the measured deviation values including their signs, thus revealing a final score which indicates the relative conformity of the redrawn shape with regard to the center of the presented line shape, or using absolute deviation values, thus providing indication of the overall conformity of the redrawn shape with the presented shape. After all pixels have been processed, the finally calculated score is presented 925 to the user, e.g. using a pre-described information field 310.

[0067] According to the further optional steps (separated by dashed line 930), the first step 900 has to be modified in order to generate at least two shapes n wherein n=1, . . . n.sub.max, with n.sub.max being the number of (different) shapes to be generated by the computer. By means of test step 935 which checks if the mentioned maximum number of shapes n.sub.max to be presented has been reached (i.e. the condition n>n.sub.max is fulfilled), the previously described steps 900-925 are repeated until this condition is fulfilled. According to following step 940, all single scores being calculated in step 920 are used to calculate a final score or to provide an assessment about the likelihood that the user is suffering from a certain disease, in particular a mentioned brain disease. The final score or medical assessment is then presented 945 to the user, e.g. by way of an information or text box being presented on the touch screen.

[0068] The herein described method and system can be implemented in a medical or training application or appliance running on any computer or data processing system having a touch screen, like a personal (stationary) computer, a portable computer like a laptop or tablet computer, or mobile devices with a touch screen like smart phones.