METHOD AND SYSTEM FOR IMPROVING THE VISUAL EXPLORATION OF AN IMAGE DURING A TARGET SEARCH

Abstract

Object of the present invention are a method and a system for monitoring a user's attention distribution in the course of a digital image exploration, adapted to produce feedback that give the user himself an indication of the efficacy of the exploration performed, even on the basis of the relevance that the various zones of the image have in relation to a certain goal, relevance that can possibly be determined by an expert system responsible for examining the image as a function of the above-mentioned objective.

Claims

1. A system for improving the analysis of digital images by a user comprising electronic data processing means, comprising data storage means, digital image display means and interface means, adapted to allow said electronic data processing means to collect data from a user, wherein said electronic data processing means are adapted to modify the display of said digital images during the exploration thereof by the user so as to add indicators adapted to mark the zones of said digital images deemed important and which have not yet been explored by the user with the necessary care.

2. The system according to claim 1, wherein said interface means comprise devices selected from the group comprising: alphanumeric keyboards, mouse pointers and buttons and eye-tracker devices.

3. The system according to claim 1, wherein said electronic data processing means comprising data storage means further comprise a first processing module comprising a first digital image analysis expert system adapted to interact with appropriate databases so as to associate a relevance index to each point of the analysed digital image as a function of a certain predetermined target, and a second processing module comprising a second digital image analysis expert system, adapted to supply a feedback to both said first digital image analysis expert system and to said digital image display means.

4. The system according to claim 3, wherein said feedback supplied to said first digital image analysis expert system comprises an update of the value of the relevance index associated to each point of said digital image.

5. The system according to claim 3, wherein said feedback supplied to said digital image display means comprises highlighter markers superimposed on said image and adapted to direct the user's attention to a certain zone of said image.

6. The system according to claim 5, wherein said dynamic highlighter markers are such as to dissolve after a certain interval of time so as to allow display of the original image without any superimposed modification.

7. The system according to claim 5, wherein said dynamic highlighter markers are selected from the group comprising: zones of said image with contrasting colours, moving shapes superimposed on said image and adapted to attract the user's attention towards the zones of interest of said image, direction indicators facing towards the zones of interest of said image, zones of said image which intermittently blink.

8. The system according to claim 3, wherein said second digital image analysis expert system is adapted to process the data received from said eye-tracker device so as to distinguish the user's saccades from fixations; to analyse and categorise said fixations into a certain number of cases which are distinguished in relation to predetermined thresholds and related to the relevance indexes of the points of said image fixed by the user; to update the value of the relevance index associated to each point of said digital image and to transmit said updated relevance index value to said first digital image analysis expert system; to calculate an optimal scanpath of said image; to generate dynamic markers, on the basis of said optimal scanpath, that are adapted to highlight appropriate zones of said image; to send said dynamic markers to said graphic interface of said data processing means so that they are appropriately superimposed on said image and suggest said optimal scanpath to the user.

9. A method for improving the analysis of digital images by a user comprising: displaying a digital image to a user on appropriate display means; acquiring the user's scanpath of said image by means of an eye-tracker device; acquiring point markers of said image made by said user; comparing the user's scanpath of said image and the points of said image marked by the user with data relating to the relevance index of the points of said image originating from a digital image analysis expert system; producing a first feedback adapted to update said data relating to the relevance index of the points of said image originating from a digital image analysis expert system.

10. The method according to claim 9, wherein comparison of the points of said image marked by the user with said data relating to the relevance index of the points of said image originating from a digital image analysis expert system is carried out by establishing a certain number of distinction thresholds and ranges with respect to the possible values of said relevance index within which said points of said image marked by the user are categorised.

11. The method according to claim 9, comprising: calculating an optimal scanpath of said image; producing a second feedback adapted to generate dynamic markers adapted to highlight appropriate zones of said image; sending said dynamic markers to said appropriate display means so that said dynamic markers are displayed superimposed on said image and suggest said optimal scanpath to the user.

12. The method according to claim 9, wherein said first and second feedbacks are produced according to the following steps: establishing four distinction ranges by dividing the range of values of said relevance index, comprised between 0 and 1, into four sub-ranges: a) for relevance indexes between 0 and 0.2, b) between 0.2 and 0.5, c) between 0.5 and 0.8, d) between 0.8 and 1; evaluating the point fixed by the user and the type of fixation acquired with regards to two time thresholds, a first threshold S1 and a second threshold S2>S1; in the case that fixation has a greater duration than both said first threshold S1 and said second threshold S2, the display of said digital image is optimised to promote the user's inspection thereof; if said fixation has a longer duration than said first threshold S1 but lesser than said second threshold S2, checks for the presence of a manual input by the user corresponding to the user's intention to mark the concerned point as important are carried out, and if present the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said range b) feedback is produced by adding the point fixed by the user to the interest points of said image by updating the set of positive target points, if said index belongs to said range a) a graphic type feedback is provided by means of a dynamic marker superimposed on the display of said graphic image; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), a feedback is generated that is adapted to insert the point fixed by the user into the points which are not of interest of said image, if said index belongs to said ranges a) or b), no feedback is produced; in the case in which said fixation has a shorter duration than said first threshold S1, checks for the presence of manual input by the user are carried out, if present, the relevance index is compared with said distinction thresholds: if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said ranges a) or b), feedback is generated that is adapted to add the point of said digital image currently being examined to the set of positive points, removing it from the set of negative points; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds: if said index belongs to said range d), user confirmation that the examined point is a positive target is requested: if the user agrees, then it is confirmed that the point is a positive target, otherwise, if the user does not confirm, the point is removed from the set of positive targets and added to the set of negative targets, if said index does not belong to said range d), no feedback is produced.

13. A computer program comprising code programming means adapted to execute the steps of the method described in claim 9 when said program is executed by a computer.

14. A computer-readable recording medium having a program recorded thereon, said readable recording medium comprising computer program code means adapted to implement all the steps of claim 9, when said computer program is executed on the computer.

15. The method according to claim 11, wherein said second feedback is produced according to the following steps: establishing four distinction ranges by dividing the range of values of said relevance index, comprised between 0 and 1, into four sub-ranges: a) for relevance indexes between 0 and 0.2, b) between 0.2 and 0.5, c) between 0.5 and 0.8, d) between 0.8 and 1; evaluating the point fixed by the user and the type of fixation acquired with regards to two time thresholds, a first threshold S1 and a second threshold S2>S1; in the case that fixation has a greater duration than both said first threshold S1 and said second threshold S2, the display of said digital image is optimised to promote the user's inspection thereof; if said fixation has a longer duration than said first threshold S1 but lesser than said second threshold S2, checks for the presence of a manual input by the user corresponding to the user's intention to mark the concerned point as important are carried out, and if present the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), no feedback is produced, if said index belongs to said range b) feedback is produced by adding the point fixed by the user to the interest points of said image by updating the set of positive target points, if said index belongs to said range a) a graphic type feedback is provided by means of a dynamic marker superimposed on the display of said graphic image; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds, if said index belongs to said range c) or to said range d), a feedback is generated that is adapted to insert the point fixed by the user into the points which are not of interest of said image, if said index belongs to said ranges a) or b), no feedback is produced; in the case in which said fixation has a shorter duration than said first threshold Si, checks for the presence of manual input by the user are carried out, if present, the relevance index is compared with said distinction thresholds: if said index belongs to said range c) or to said range d), no feedback is produced , if said index belongs to said ranges a) or b), feedback is generated that is adapted to add the point of said digital image currently being examined to the set of positive points, removing it from the set of negative points; in the case that said manual input by the user is not present, the relevance index is compared with said distinction thresholds: if said index belongs to said range d), user confirmation that the examined point is a positive target is requested: if the user agrees, then it is confirmed that the point is a positive target, otherwise, if the user does not confirm, the point is removed from the set of positive targets and added to the set of negative targets, if said index does not belong to said range d), no feedback is produced.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG. 1 shows the block diagram of the architecture of the method according to the present invention in the case in which the expert system responsible for preliminarily examining the image as a function of a certain objective to be identified is updated and a visual feedback is also provided to the user.

[0027] FIG. 2 shows the block diagram of the architecture of the method according to the present invention in the case in which the expert system responsible for preliminarily examining the image as a function of a certain objective to be identified is updated and no visual feedback is provided to the user.

[0028] FIG. 3 shows a block diagram of the algorithm at the basis of the method according to the present invention.

[0029] FIG. 4 shows the flow chart of the evaluation algorithm for user behaviour and the generation of the expert system update and of the feedback to display to the user.

SUMMARY OF THE INVENTION

[0030] Object of the present invention are a method and a system for monitoring a user's attention distribution in the course of the exploration of an image, adapted to produce feedback that give the user himself an indication of the efficacy of the exploration carried out, even on the basis of the relevance of the various image zones in relation to a certain objective, relevance that can possibly be determined by an expert system responsible for examining the image as a function of the above-mentioned objective.

[0031] Unlike the solutions of the prior art, the main characteristics of the method and the system according to the present invention are the following: a first feedback provided to the digital image analysis expert system possibly part of the system according to the present invention, adapted to update and modify the behaviour thereof on the basis of operator choices (self-learning); a second feedback comprising the dynamic placement of the highlighter markers on the digital image being examined; a single exploratory action having two steps: a first free exploration step and a subsequent second step wherein the system, automatically and also “learning” from the choices made by the user himself, guides the user in the search for the target through appropriate visual feedback.

[0032] In greater detail, the method and the system according to the present invention are adapted to identify the information relating to the observed and unobserved areas of the examined image, using the incoming information from the eye-tracking device, which monitors user gaze in real time and effectively communicates said information to the user himself.

[0033] The algorithm at the basis of the method according to the present invention is adapted to receive real-time information on the zones observed and not observed by the user, and to present at a suitable time (for example, when the operator is looking at a point deemed interesting but is going beyond) appropriate feedback that direct the user's attention to a certain zone of the image and then dissolve to allow display of the original image without any superimposed artefact.

[0034] In addition, the method and the system according to the present invention, allow the best view of the examined image to be presented to the user, by selecting the method deemed most effective for presenting the visual feedback to the user.

DETAILED DESCRIPTION OF THE INVENTION

[0035] FIG. 1 illustrates an overview of the system according to the present invention comprising electronic data processing means 10, associated to data storage means 11, digital image display means 12 and interface means 13, adapted to allow said electronic data processing means to collect data from a user, wherein said electronic data processing means 10 are adapted, among other things, to modify the display of said digital images during the exploration thereof by the user so as to add indicators adapted to mark the zones of said digital images deemed important and which have not yet been explored by the user with the necessary care.

[0036] Said data processing means 10 can be implemented from any system able to perform calculations on the basis of a set of instructions contained in suitable data storage means. Said data processing means 10 can thus include personal computers, workstations, etc.

[0037] Said interface means 13 preferably comprise alphanumeric keyboards, mouse pointers, buttons 14 and eye-tracker devices 15.

[0038] A digital image 20, object of user analysis, is initially analysed by a first processing module comprising a first digital image analysis expert system 21 possibly but not necessarily stored in said data storage means 11 and adapted to establish a series of points of the same image deemed to be relevant as a function of a specific predefined target. Said first digital image analysis expert system 21 is adapted to interact with appropriate data bases 22 wherein are archived the results previously achieved in similar analyses, so that said first expert system 21 may progressively refine the accuracy of his detections.

[0039] The series of points identified by said expert system is generally organised according to a matrix, known as “relevance matrix” 23, wherein each point of the image is evaluated by means of a relevance index (with value between 0, no relevance, and 1, absolute relevance) with respect to the afore-mentioned default target.

[0040] At this point, said relevance matrix is provided, together with said image 20 and with the data originating from said interface means 13, at a second processing module comprising a second digital image analysis expert system 24, adapted to provide a feedback to both said first digital image analysis expert system 21 and to said digital image display means 12, preferably in the form of highlighter markers adapted to direct the user's attention to a certain zone of said image 20. Advantageously, said highlighter markers are adapted to dissolve after a certain interval of time so as to allow the user to view the original image without any permanent superimposed artefact.

[0041] In greater detail, and in reference to the accompanying FIG. 3, said second digital image analysis expert system 24 operates in the following way: the gaze of the user in the act of exploring said image 20 displayed on said digital image display means 12, is detected by said eye-tracker device 15 which, subsequently transmits 30 the collected data, relating to the gaze of the user, to said second digital image analysis expert system 24 which processes the data received so as to distinguish 31 the user's saccades from fixations. Said fixations are subsequently analysed 32 and categorised 33 in a certain number of cases 34 that are distinguished 35 in relation to some preset thresholds and relating to the relevance indexes of said relevance matrix. As a result thereof, the indexes of the relevance matrix are updated 36 by means of an updating of the training set of said first digital image analysis expert system 21 (for example, in the case of a neural network based expert system the parameters of the neural network are updated) and an optimal scanpath is calculated 37 on the basis of which dynamic markers are generated that are sent 38 to the graphic interface 16 of said data processing means 10 so as to be dynamically superimposed on said image 20, in such a way as to guide the user's attention towards the parts of the image deemed important and not yet adequately explored.

[0042] Said dynamic markers can be implemented, for example, with some zones with contrasting colours, with moving shapes adapted to attract the user's attention towards the zones of interest, with some simple direction indicators (arrows for example) pointing towards the zones of interest, or further still through zones which intermittently flash.

[0043] Advantageously, said highlighter markers are such as to dissolve after a certain interval of time so as to allow display of the original image without any superimposed modification.

[0044] The system object of the present invention can have different modes of operation, for example, it can operate in both the so-called active visual exploration mode (AVE, shown in FIG. 1 attached), in which overlap to the picture being examined highlighter markers described above according to the above described method, which in so-called mode of visual navigation passive (PVE, shown in FIG. 2 attached), in which do not overlap the image under examination highlighter markers described above, the exploration of the image 20 by the user, in this case is free and the system provides only to update the relevance matrix on the basis of the behaviour of the same user during the exploration of the image.

[0045] A further mode of operation can provide for the alternation of the above-mentioned AVE and PVE modes. For example, the system object of the present invention can be set so as to acquire the image in question and operate for a certain initial time interval T1 in PVE mode up to the time in which the user displays the image and begins his exploration and, consequently, the system object of the present invention switches to AVE mode.

[0046] In reference to the accompanying FIG. 4, an example of algorithm implemented in said second digital image analysis expert system 24 comprises the series of steps described below.

[0047] A certain number of distinction thresholds and ranges is established with respect to the possible values of said relevance index, for example, four distinction thresholds are established by dividing the range of values comprised between 0 and 1 into four sub-ranges: a) for relevance indexes between 0 and 0.2 (certain non-relevance of the observed point), b) between 0.2 and 0.5 (low probability of relevance of the observed point), c) between 0.5 and 0.8 (high probability of relevance of the observed point), d) between 0.8 to 1 (certain relevance of the observed point).

[0048] At this point, the data collected by said interface means 13 and relevant to the detection of the ocular movements of the user—by means of said Eye Tracker device 15 and the input provided by the user through said manual interface means 14—are processed in the following manner:

[0049] The point fixed by the user and the type of fixation (gaze) acquired with regards to two time thresholds, a first threshold S1 and a second threshold S2>S1, is evaluated 40, 41.

[0050] In the case that said fixation has a greater duration than both said first threshold S1 and said second threshold S2, the display of said digital image 20 is optimised 61 to promote the user's inspection thereof, possibly by displaying the current relevance index of the fixed points. Said optimisation can provide for an increase of the contrast, increase of the resolution, an increase of the zoom level, etc.

[0051] In the case of said fixation having a greater duration than said first threshold S1 but lesser than said second threshold S2, the presence of a manual input by the user (manual input corresponding to the user's intention to mark the point in question as important), is checked 42.

[0052] In the case of said manual input being present, the relevance index is compared 43, 44, 45 with said distinction thresholds; if said index belongs to said range of c) or to said range d), then the user agrees with said first digital image analysis expert system 21 and consequently said second digital image analysis expert system 24 does not produce 46 any feedback; if said index belongs to said range b) then the point set by the user between the points of interest of said image is added 47 by updating the set of positive targets; if said index belongs to said range a) then a graphic type feedback is provided 48 by means of a dynamic marker superimposed on the display of said digital image, possibly with user confirmation request.

[0053] In the case, on the other hand, in which said manual user input is not present , the relevance index is again compared 49, 50 with said distinction thresholds; if said index belongs to said range c) or to said range d), then there is generated a feedback adapted to insert 51, possibly with a user confirmation request, the point fixed by the user between the points of no interest of said image; if said index belongs to said ranges a) or b) then the user agrees with the first digital image analysis expert system 21 and consequently said second digital image analysis expert system 24 does not produce 52 any feedback.

[0054] In the case in which said fixation has a lesser duration than said first threshold S1, the presence of manual user input (manual input corresponding to the user's intention to mark the point in question as important) is again checked 53.

[0055] In the case of the presence of said manual input, the relevance index is compared 54, 55 with said distinction thresholds: if said index belongs to said range c) or to said range d), then the user agrees with said first digital image analysis expert system 21 and consequently said second digital image analysis expert system 24 does not produce 56 any feedback; if said index, on the other hand, belongs to the ranges a) or b), then a feedback is generated that is adapted to update 57, possibly with user confirmation request, the set of positive and negative so as to add the point of said digital image currently being examined to the set of positive points, while removing it from the set of negative points.

[0056] Lastly, in the case that said manual user input is not present, the relevance index is compared 58 with said distinction thresholds: if said index belongs to said range d), the algorithm is practically certain that the point in question is a positive target and user confirmation is requested 59: if the user agrees then it is confirmed that the point is a positive target, otherwise if the user does not agree, the point is removed from the set of positive targets and added to the set of negative targets. This is a case in which the point examined is a simple target of the negative type or a case in which the user has paid little attention during his analysis.

[0057] If, on the other hand, said index does not belong to the range d), no feedback is produced 60.