EYE TRACKING SYSTEM AND METHOD TO DETECT THE DOMINANT EYE

Abstract

The invention relates to an eye tracking system (10) and a method for operating an eye tracking system (10) for determining if one of a left and a right eye of a user is dominant, wherein at least one image of the left and the right eye of the user is captured (S30), based on the at least one image and according to a predefined accuracy function a left accuracy score for the left eye (S34a) and a right accuracy score for the right eye (S34c) is determined and it is determined if one of the left and the right eye of the user is dominant in dependency of at least the left and the right accuracy score (S36). Thereby user-specific properties relating to his eyes can be provided and considered when performing eye tracking so that the robustness and accuracy of eye tracking can be enhanced.

Claims

1-15. (canceled)

16. Method for operating an eye tracking system (10) for determining if one of a left and a right eye of a user is dominant, comprising the steps: a) capturing at least one image of the left and the right eye of the user (S30); b) based on the at least one image and according to a predefined accuracy function determining a left accuracy score for the left eye (S34a) and a right accuracy score for the right eye (S34c); c) determining if one of the left and the right eye of the user is dominant in dependency of at least the left and the right accuracy score (S36);and d) determining a point of regard of the left eye (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.L,V2))) and a point of regard P(M.sup.R(g.sub.i.sup.R,C)) P(M.sup.R(g.sub.i.sup.R,V1)) P(M.sup.R(g.sub.i.sup.R,V2))) of the right eye on the basis of the at least one image; characterized in the step: e) determining the left accuracy score in dependency of a deviation of the determined point of regard (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.L,V2))) of the left eye from at least one left set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2) and determining the right accuracy score in dependency of a deviation of the determined point of regard (P(M.sup.R(g.sub.i.sup.R,C)) P(M.sup.R(g.sub.i.sup.R,V1)) P(M.sup.R(g.sub.i.sup.R,V2))) of the right eye from at least one right set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2); wherein in step b) based on the at least one image and according to the predefined accuracy function a binocular accuracy score for the left and the right eye is determined (S34b), and in step c) it is determined if one of the left and the right eye of the user is dominant in dependency of the left, right and binocular accuracy score (S36), wherein the binocular accuracy score is a measure for a deviation of a binocular point of regard of the left and the right eye from a binocular set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2).

17. Method according to claim 16, characterized in that the left and right set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2) are provided by showing a finite sequence of points on a display device (14) of the eye tracking system (10) that the user is required to fixate and/or by showing a smooth and/or continuous motion of a point or a stimulus image on the display device (14) that the user is required to follow with both eyes.

18. Method according to claim 16, characterized in that the left, right and binocular set point (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2) of regard are identical.

19. Method according to claim 16, characterized in that a signal indicating the identity of the dominant eye is provided and/or outputted in dependency of the left, right and binocular accuracy score (S24; S38a, S38b, S38c).

20. Method according to claim 16, characterized in that the determination if one of the left and the right eye of the user is dominant is performed during a calibration procedure (S12a, S12b) of the eye tracking system (10) and/or during a validation procedure (S18) for validating results of the calibration procedure.

21. Method according to claim 22, characterized in that during the calibration procedure (S12a, S12b) of the eye tracking system (10) several first points of regard of the user are calculated and compared to corresponding first set points of regard (p.sub.i.sup.G,C), wherein on the basis of this comparison calibration parameters of the eye tracking system are set so that the calculated first points of regard are mapped to corresponding calibrated first points of regard (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.R(g.sub.i.sup.R,C))).

22. Method according to claim 20, characterized in that during the validation procedure (S18) several second points of regard (P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.R,V2)), P(M.sup.R(g.sub.i.sup.R,V1)), P(M.sup.R(g.sub.i.sup.R,V2))) of the user are calculated and compared to corresponding second set points of regard (p.sub.i.sup.G,V1, p.sub.i.sup.G,V2).

23. Method according to claim 21, characterized in that on the basis of the deviation of the calibrated first points of regard (P(M.sup.L(g.sub.i.sup.L,C), P(M.sup.R(g.sub.i.sup.R,C))) from the first set points of regard an error (r.sup.L, r.sup.R, r.sup.B) of the calibration procedure for the left eye, the right eye and for both eyes is calculated and/or on the basis of the deviation of the second points of regard (P(M.sup.L(g.sub.i.sup.L,V1, p.sub.i.sup.G,V2)), P(M.sup.R(g.sub.i.sup.R,V1)), P(M.sup.R(g.sub.i.sup.R,V2))) from the second set points of regard (p.sub.i.sup.G,V1, p.sub.i.sup.G,V2) an error (L.sup.L, L.sup.R, L.sup.B) of the validation procedure for the left eye, the right eye and for both eyes is calculated.

24. Method according to claim 20, characterized in that during the calibration procedure (S12b) an optimal subset of calibration points is automatically detected by excluding outliers according to a predefined criterion, preferably by evaluation all the subsets of the calibration points or by finding a randomized consensus.

25. Method according to claim 23, characterized in that the left, right and binocular accuracy score are each determined in dependency of: whether the left point of regard (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.L,V2))), right point of regard (P(M.sup.R(g.sub.i.sup.R,C)) P(M.sup.R(g.sub.i.sup.R,V1)) P(M.sup.R(g.sub.i.sup.R,V2))) and binocular point of regard, respectively, can be calculated on the basis of the at least one image; and/or the error (r.sup.L, r.sup.R, r.sup.B) of the calibration procedure (S12a, S12b) and/or validation procedure (S18) for the left eye, the right eye and both eyes, respectively.

26. Method according to claim 19, characterized in that eye tracking is performed, wherein a present point of regard is calculated in dependency of the signal.

27. Method according to claim 26, characterized in that the present point of regard is calculated with respect to the left eye if the left eye was determined as being dominant, the present point of regard is calculated with respect to the right eye if the right eye was determined as being dominant and the present point of regard is calculated with respect to both eyes if none of the left and right eye was determined as being dominant.

28. Method according to claim 26, characterized in that the present point of regard is calculated as a weighted average between left eye data and right eye data, wherein weights of the weighted average are a function of the determined left accuracy score and right accuracy score, so that the right eye data is more heavily influencing the weighted average than the left eye data if the right accuracy score is higher than the left accuracy score, and the left eye data is more heavily influencing the weighted average than the right eye data if the left accuracy score is higher than the right accuracy score.

29. Eye tracking system (10) for determining if one of a left and a right eye of a user is dominant, wherein the eye tracking system is configured: a) to capture at least one image of the left and the right eye of the user; b) based on the at least one image and according to a predefined accuracy function to determine a left accuracy score for the left eye and a right accuracy score for the right eye; c) to determine if one of the left and the right eye of the user is dominant in dependency of at least the left and the right accuracy score; and d) to determine a point of regard of the left eye (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.L,V2))) and a point of regard (P(M.sup.R(g.sub.i.sup.R,C)) P(M.sup.R(g.sub.i.sup.R,V1)) P(M.sup.R(g.sub.i.sup.R,V2))) of the right eye on the basis of the at least one image; characterized in that the eye tracking device if configured to: e) determine the left accuracy score in dependency of a deviation of the determined point of regard (P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.L(g.sub.i.sup.L,V1)), P(M.sup.L(g.sub.i.sup.L,V2))) of the left eye from at least one left set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2) and determining the right accuracy score in dependency of a deviation of the determined point of regard (P(M.sup.R(g.sub.i.sup.R,C)) P(M.sup.R(g.sub.i.sup.R,V1)) P(M.sup.R(g.sub.i.sup.R,V2))) of the right eye from at least one right set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2). f) determine a binocular accuracy score for the left and the right eye based on the at least one image and according to the predefined accuracy function(S34b), and to determine if one of the left and the right eye of the user is dominant in dependency of the left, right and binocular accuracy score (S36), wherein the binocular accuracy score is a measure for a deviation of a binocular point of regard of the left and the right eye from a binocular set point of regard (p.sub.i.sup.G,C, p.sub.i.sup.G,V1, p.sub.i.sup.G,V2).

Description

[0026] In the following, advantageous embodiments of the present invention are described in more detail with reference to the accompanying drawings.

[0027] They show in:

[0028] FIG. 1 a schematic illustration of a method for operating an eye tracking system according to an embodiment of the invention;

[0029] FIG. 2 a schematic view of five calibration points shown on a display of an eye tracking system together with a determined point of regard for the left and the right eye each;

[0030] FIG. 3a a schematic view of four validation points shown on a display of an eye tracking system during a validation procedure together with a determined point of regard of the left and the right eye each according to an embodiment of the invention;

[0031] FIG. 3b a schematic view of four validation points shown on a display of the eye tracking system during a validation procedure together with a determined point of regard of the left and the right eye each according to an embodiment of the invention;

[0032] FIG. 4 a schematic illustration of the method for operating an eye tracking system, wherein a channel of the eye tracking system is chosen in dependency on the accuracy score of each channel according to an embodiment of the invention; and

[0033] FIG. 5 a schematic view of an eye tracking system for determining the dominant eye of a user according to an embodiment of the invention.

[0034] In the following description of preferred embodiments for the previously used terms left set point of regard, right set point of regard and binocular set point of regard the term stimulus point is used as a synonym and furthermore, depending on whether these stimulus points are shown during a calibration procedure or validation procedure, these stimulus point are also called calibration points or validation points, respectively.

[0035] FIG. 1 shows a schematic view of a method for operating an eye tracking system 10 (compare FIG. 5) to determine the dominant eye of a user according to an embodiment of the invention. The method starts in step S10, where an input is received for starting a calibration procedure of the eye tracking system 10. This input can be a user input but also something different like a general starting signal. When this input is received, the eye tracking system 10 starts the calibration procedure. This calibration procedure can be performed as a standard calibration in step S12a or also optionally as a smart calibration in step S12b. Whether the standard calibration in step S12a or the smart calibration in step S12b is performed can also be indicated by the input signal in step S10.

[0036] If the user starts the standard calibration process, in step S12a stimulus points are shown at n predefined static positions on the stimulus plane and the user is supposed to look at these points. The stimulus plane can be a screen or an image plane of a display device of the eye tracking system, wherein the eye tracking system can be implemented as a remote eye tracker or also as a head mounted eye tracking device comprising head mounted displays. Thereby, the eye tracking system can also be configured to provide three dimensional images so that the stimulus plane not necessarily has to coincide with the physical screen plane of a display but can also be a virtual image plane produced by a 3D display or by stereoscopic displays. The position and orientation of this stimulus plane is known to the eye tracking system 10.

[0037] When the user looks at these stimulus points, the eye tracking system captures images of both eyes of the user and thereby, for example, can calculate the 3D position and/or orientation of each of the eyes and the gaze vectors of each of the eyes, wherein the point of regard of the left eye then is defined as the intersection of the gaze vector of the left eye with the stimulus plane, the right point of regard is defined as the intersection of the right eye gaze vector with the stimulus plane and the binocular point of regard can be defined as the intersection of a binocular gaze vector with the stimulus plane or also can be simply defined as the positional average of the left point of regard and the right point of regard on the stimulus plane. The stimulus points further are shown each at a time. The ground truth point of regards for these n stimulus points, which can also be denoted as set points of regard, are priorly known as p.sup.G.sub.i=1[n, which are normally represented by 2D vectors. From the eye tracking software accordingly n gaze samples g.sup.L.sub.i=1˜nfrom the left gaze channel, and n gaze samples g.sup.R.sub.i=1˜n from the right gaze channel are obtained. Each gaze sample g.sub.i includes all the eye and/or gaze information relevant to the i.sup.th calibration point.

[0038] The calibration has to produce a gaze correction model M: g.fwdarw.M(g) which maps the raw gaze samples to the calibrated ones. Especially there is produced a gaze correction model for the left eye M.sup.L and for the right eye M.sup.R each. The gaze correction model will ideally be able to correct all gaze samples from this user and the following gaze tracking.

[0039] Let the (priorly known) function that computes or extracts the point of regard from the gaze sample g be P: g.fwdarw.p=P(g). The gaze correction for the left eye M.sup.L is then optimized to minimize the difference between the calibrated point of regard p.sup.L=P(M.sup.L(g.sup.L)) and the ground truth p.sup.G, as

[00001] $\begin{matrix} \underset{M^{L}}{argmin} .Math. {.Math.}_{i = 1}^{n} .Math. .Math. {.Math. p_{i}^{L} - p_{i}^{G} .Math.}^{2} .Math. \underset{M^{L}}{= argmin} .Math. {.Math.}_{i = 1}^{n} .Math. {.Math. P (M (g^{L})) - p_{i}^{G} .Math.}^{2} & (1) \end{matrix}$

[0040] The residual error of the left channel on the i.sup.th calibration point is computed as

r.sub.i.sup.l=∥P(M.sup.L(g.sub.i.sup.L)−p.sub.i.sup.G∥. (2)

[0041] The calibration is always done for the two monocular channels separately. Especially the same applies for the right channel in the same way and equations (1) and (2) also apply for the right channel if the index “L” is substituted by “R”. Normally, the point of regard of the binocular gaze channel is created by averaging the left and right channels, as

[00002] $\begin{matrix} p^{P} - \frac{p^{L} + p^{R}}{2} - \frac{P (M^{L} (g^{L})) + P (M^{R} (g^{L}))}{2} . & (3) \end{matrix}$

[0042] Therefore, the residual error for the binocular channel can be computed as

r.sub.i.sup.B=∥p.sub.i.sup.B−p.sub.i.sup.G∥. (4)

[0043] Instead of static stimulus points, the calibration can also be done by showing a continuously moving stimulus point. The path of the moving point is fully specified by a path model p.sup.G, and this is the ground truth point of regard in the calibration. The gaze correction for the left eye M.sup.L is optimized to minimize the difference between the path of the calibrated point of regard and the ground truth path by doing a dynamic time warping:

[00003] $\begin{matrix} \underset{M^{L}}{argmin} .Math. \int_{t} .Math. .Math. {.Math. p^{L} (t) - \underset{- T < Δ .Math. .Math. t < T}{\min_{Δ .Math. .Math. t}} .Math. p^{G} (t + Δ .Math. .Math. t) .Math.}^{2} .Math. dt = \underset{M^{L}}{argmin} .Math. \int_{t} .Math. .Math. {.Math. P (M (g^{L} (t))) - \underset{- T < Δ .Math. .Math. t < T}{\min_{Δ .Math. .Math. t}} .Math. p^{G} (t + Δ .Math. .Math. t) .Math.}^{2} .Math. dt, .Math. & (5) \end{matrix}$

[0044] where T controls the maximal time window for matching.

[0045] The overall residual error of the left channel is exactly what we tried to minimize above

[00004] $\begin{matrix} r^{L} = \int_{t} .Math. .Math. {.Math. p^{L} (t) - \underset{- T < Δ .Math. .Math. t < T}{\min_{Δ .Math. .Math. t}} .Math. p^{G} (t + Δ .Math. .Math. t) .Math.}^{2} .Math. dt . & (6) \end{matrix}$

[0046] Again the same applies for the right channel. The residual error for the binocular channel can be computed in a similar way after

[00005] $\begin{matrix} r^{B} = \int_{t} .Math. .Math. {.Math. p^{B} (t) - \underset{- T < Δ .Math. .Math. t < T}{\min_{Δ .Math. .Math. t}} .Math. p^{G} (t + Δ .Math. .Math. t) .Math.}^{2} .Math. dt, .Math. with .Math. .Math. p^{B} = \frac{p^{L} + p^{R}}{2} . & (7) \end{matrix}$

[0047] So during the calibration process in step S12a the correction models M.sup.L and M.sup.R for the left and the right eye are calculated as well as optionally the errors r.sup.L, r.sup.R and r.sup.B for the left channel, the right channel and the binocular channel, respectively. These results of the calibration process can then be outputted or provided in step S14 for further use by the eye tracking system, for example for a subsequent eye tracking and/or for a subsequent validation procedure.

[0048] These results can also directly be used for determining the dominant eye in step S22 in form of a self validation, which is explained later.

[0049] After the calibration procedure in step S12a is finished, the user can make an input in step S16 for activating a validation procedure, which then is performed in step S18 using the results of the calibration provided in step S14. Instead of a user input in Step S16 any other activation signal can be provided for activating the validation procedure. The validation procedure can also be started automatically after the calibration procedure is finished and step 516 can be omitted.

[0050] During the validation procedure, again stimulus points are shown on the stimulus plane but preferably at positions that are different from the positions of the calibration points.

[0051] While these stimulus points are shown, images of the eyes of the user are captured by the eye tracking system and eye features of each eye, like gaze vectors, are determined. Thereby, one again gets raw gaze samples g.sub.j=1˜m.sup.L for the left eye and g.sub.j=1˜m.sup.R for the right eye for each of the ground truth points of regard p.sup.G.sub.j=1˜m, which are the positions of the stimulus points. Then the validation error for each channel, namely the error r.sup.R for the right channel the error r.sup.L for the left channel and the error r.sup.B for the binocular channel, is computed by applying the calibrated gaze correction model M, especially M.sup.L and M.sup.R:

[00006] $\begin{matrix} r_{i}^{L} = .Math. P (M^{L} (g_{j}^{L})) - p_{j}^{G} .Math., & (9) \\ r_{i}^{R} = .Math. P (M^{R} (g_{j}^{R})) - p_{j}^{G} .Math., & (10) \\ r_{i}^{B} = .Math. \frac{P (M^{L} (g_{j}^{L})) + P (M^{R} (g_{j}^{L}))}{2} - p_{j}^{G} .Math. . & (11) \end{matrix}$

[0052] Instead of showing single stimulus validation points, again a validation stimulus point can be shown in a continuously moving manner as described with regard to the calibration procedure.

[0053] Furthermore, instead of the standard calibration described in step S12a, a smart calibration can be performed in step S12b. The reason for this is that due the distraction of the user or other problems, not all detected fixations in the calibrations procedure are good for the calibration purpose. Therefore, in the smart calibration a screening procedure can be implemented to detect and remove the “bad” calibration points. This feature provides higher calibration robustness and is also relevant to the calibration quality.

[0054] The smart calibration is similar to the standard calibration in that first of all multiple calibration stimulus points are shown on a stimulus plane, one after the other, and for each stimulus point the gaze and/or point of regard of the left eye, the right eye and the binocular gaze and/or point of regard is determined. The difference to the standard calibration is that not necessarily all of the stimulus points and the corresponding eye properties, like gaze and determined points of regard, are used for providing the correction model M.

[0055] Considering a set of n calibration points, the smart calibration procedure could be as follows:

[0056] First of all, all possible and interested subsets are enumerated. For example, when we have five points, all three-point, four-point and five-point subsets will be listed and evaluated in the following steps.

[0057] In the next step, for each calibration set three measures are considered: The number of calibration points, the quality (residual) of the calibration, and the deviation of the gaze correction from translation and scaling. There is also a prior knowledge based preference towards the subset consisting of more points. For a subset under testing, the calibration is done only using the points in the set, and then the corresponding residual error on the calibration set is computed. Empirically, bad calibration points often require a large “gaze correction” to minimize the residual. Therefore, a large deviation from a model that applies only translation and scaling is a sign of bad calibration points. All theses three measures will be combined into a score by a weighted sum. In the next step, among all the subsets the best one is selected based on their score. If the subset that consists of all points wins over all the other subsets, the calibration continues with all the n points. If this is not the case, the gaze correction model from the best subset will be used. The calibration points that are not in the selected subset will be labelled as “not used for bad quality”.

[0058] This proposed approach is a reliable approach even for a small set of calibration points, e.g. five points. For a calibration with a large set of calibration points, e.g. more than eight, a more general sample consensus approach, for example RANSAC (Random Sample Consensus), can be used to detect the best subset.

[0059] So if finally the best subset is determined, the correction model M and residual errors as described for the standard calibration procedure in step S12a can be calculated, especially with regard to the chosen subset, and provided in step S14. If then the standard validation procedure is performed afterwards in step S18, the results of the smart calibration procedure are used similarly to those described above with regard to the standard calibration.

[0060] After this validation procedure in step S18, the validation results are provided in step S20 for determining the dominant eye of the user in step S22. Especially these validation results are the residual errors provided by (9), (10) and (11) or the respective overall residual errors in case of a smooth pursuit. Moreover, also the number of validly tracked points regarding each of the eyes can be provided as validation result.

[0061] With regard to determining the dominant eye, the basic idea is to evaluate the quality of each gaze channel based on the calibration and/or validation quality, and the best channel (binocular, left, right) can be chosen for eye tracking. If a validation procedure had been performed in step S18, the monocular validation error is the most natural quality measure for each monocular channel. The validation quality can also include the information about the number of well tracked validation points. So, based on the computed validation errors for each channel according to equations (9), (10) and (11), and further taking into account the number of well tracked validation points for each channel, an accuracy score can be determined for each of the channels. This accuracy score is the higher, the smaller the respective validation error and the higher the number of well tracked validation points is. Optionally, a “bonus”, e.g. in form of a predefined numerical value, can be added to the accuracy score of the binocular channel so that in doubt the binocular gaze is usually more robust than the monocular gazes. Additionally, it could also be considered whether the numbers of valid gaze samples in the validation procedure is very unbalanced between the left and right channels, so that also information about the number of well tracked validation points can be used for calculating the respective accuracy scores. For example, it may be checked whether one channel has more than two times the number of tracked points compared with the other channel. If the left channel only gets two points tracked in the validation and the right channel gets five points, the system will switch to using the right channel for its better robustness. So, to determine the dominant eye, the system will compare the quality of the three channels based on the corresponding accuracy scores and pick the best one. The general rule is to use the best channel, while the validation quality is the most natural and reliable quality measure.

[0062] So, if the left channel is determined as the best channel, the left eye is determined to be the dominant eye. If the left and the right channel have equal or similar accuracy scores or the accuracy score of the binocular channel is the best one, then it is determined that both eyes are equally dominant or, in other words, that there is no dominant eye. Based on these results in step S22 an output signal in step S24 can be provided indicating the dominant eye. The signal then can be used for outputting the results of this determination on a screen or it can be used for a subsequent eye tracking by using the best channel.

[0063] In another embodiment, the validation procedure does not have to be performed necessarily. Optionally, the calibration results in step S14 can be used directly for determining the dominant eye in step S22 as a kind of self validation. So, this self validation allows for a gaze channel selection without additional validation. Thereby, the left and/or right and/or binocular gaze channel quality is determined based on the calibration quality. Therefore, the calibration results, especially the residual errors according to (2), For the left channel, the corresponding one for the right channel and the binocular residual error according to (4) or the respective overall residual errors according to (6) and (7) can be used to calculate the corresponding accuracy scores, which define the calibration quality for each channel. Also, the number of usable calibration points can be considered when calculating the respective accuracy scores. So, additionally, when the two channels have a different number of usable calibration points, wherein the reason could be smart calibration, loss of eye tracking, and so on, the channel with the fewer calibration points will be given a very low quality or accuracy score, for its calibration is much less reliable, even when its residual is smaller.

[0064] To sum up, the self validation is done on the calibration set, wherein the preferred solution is to use the calibration quality, i.e. the mean residual, as the validation quality. Another option would be the “leave-one-out cross-validation”. Also, the calibration quality can include the information regarding the number of valid calibration points used in model fitting, since more calibration points generally result in a more accurate gaze correction. For providing the accuracy scores for each of the channels considering all or some of the above-mentioned quality influencing parameters, an accuracy function can be provided, wherein for example all these parameters can be summed up with corresponding weights. So, based on the so calculated accuracy scores for each channel, the dominant eye can be determined in the same way as described with regard to the results of the validation procedure and again a signal indicating the dominant eye can be outputted in step S24.

[0065] FIG. 2 shows a schematic view of five stimulus points shown in a stimulus plane during the calibration procedure together with the calibrated points of regard for the left and right eye each. The calibration points or stimulus points are denoted in FIG. 2 as pi.sup.G,C for i=1, 2, 3, 4, 5. Furthermore, the determined and calibrated points of regard for the left eye are denoted by P(M.sup.L(g.sub.i.sup.L,C)) for i=1, 2, 3, 4, 5 and the determined and calibrated points of regard for the right eye are denoted by P(M.sup.R(g.sub.i.sup.R,C)). In FIG. 2 five stimulus points p.sub.i.sup.G,C are shown for illustrative purposes all in one image, but during the calibration procedure these stimulus points p.sub.i.sup.G,C are shown one at a time. Also, the shown calibrated points of regard P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.R(g.sub.i.sup.R,C)) are just for illustrative purposes and not shown during the calibration procedure. Actually, FIG. 2 can be seen as the result of the calibration procedure with the shown stimulus points p.sub.i.sup.G,C and the corresponding determined arid calibrated points of regard P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.R(g.sub.i.sup.R,C)).

[0066] As can be seen, for each stimulus point p.sub.i.sup.G,C the point of regard P(M.sup.L(g.sub.i.sup.L,C)), P(M.sup.R(g.sub.i.sup.L,C)) of the left and the right eye can be determined. In this case, the number of validly tracked points of regard P(ML(g.sub.iL,c)).sub.s p(mR(g,R,c)) for the left and the right eye is five each. The residual error r.sup.L for the left channel can be determined as 0.5° in x-direction and 0.7° in y-direction, wherein the residual error r.sup.R for the right eye or right channel can be determined as 0.1° in x-direction and 0.5° in y-direction. These results could directly be used as self validation for determining the dominant eye, wherein in this example no dominant eye would be detected as the accuracies for the left and the right channel are quite similar.

[0067] Optionally, after this calibration procedure, a validation procedure can be performed using the calibration model M determined on the basis of the calibration procedure. Two examples of a validation procedure, especially their results, are shown in FIG. 3a and FIG. 3b.

[0068] In FIG. 3a four validation points are shown, which are denoted by p.sub.i.sup.G,V1 with i=1, 2, 3, 4, and which are different in their positions from the calibration points p.sub.i.sup.G,C. Furthermore, the points of regard relating to each of the validation points with respect to the right and the left eye each, which are determined during the validation procedure, are shown, too. The points of regard for the left eye are denoted by P(M.sup.L(g.sub.i.sup.L,V1)) with i=1, 2, 3, 4, and the points of regard for the right eye are denoted by P(M.sup.R(g.sub.i.sup.R,V1). In this case, the average residual error r.sup.L for the left channel is 0.3° in the x-direction and 0.6° in the y-direction and the average residual r.sup.R error for the right channel is 0.2° in the x-direction and 0.3° in the y-direction. As these points of regard P(M.sup.L(g.sub.i.sup.R,V1)) of the left and the right eye have comparable validation qualities in this example, the binocular channel is chosen for eye tracking or other purposes. Based on these validation results, it can be determined that none of the left and the right eye is a dominant eye.

[0069] In contrast thereto, FIG. 3b shows another example with different results of the validation procedure. Again, four validation points denoted by p.sub.i.sup.G,V2 with , 2, 3, 4, are shown, which are again different in their positions from the calibration points p.sub.i.sup.G,C. Also, the points of regard with respect to the right and the left eye are shown, which are denoted by P(M.sup.L(g.sub.i.sup.L,V2)) for the left eye and P(M.sup.R(g.sub.i.sup.R,V2)) for the right eye, with i=1, 2, 3, 4. The average residual error r.sup.R for the right channel in this case is 0.7° for the x-direction and 0.6° for the y-direction, whereas for the left eye no useful points could be determined. In this case, as can be seen, the validation quality for the right eye is much better than for the left eye and tracking can be switched to monocular tracking with the better eye, in this case the right eye, using the right channel.

[0070] So, in general, if the difference between the validation qualities, which can be provided by calculating the corresponding accuracy scores, especially based on the respective errors, is above a predefined threshold, the better channel can be chosen for better robustness.

[0071] FIG. 4 shows a schematic illustration of a method for operating an eye tracking system according to another embodiment of the invention. In step S30 the eye tracking system captures images of both eyes of a user. This can be done by using one or more cameras. On the basis of the captured images, data relating to the left eye are derived in step S32a, with regard to the right eye in step S32c, and with regard to both eyes in step S32b. These data can, for example, be eye properties such as the position of the eyeball, the position of the pupil, the positions of eventually detected glints, the gaze direction and/or points of regard. Based on these data in each channel, namely for the left eye, the right eye and both eyes, a respective accuracy score is calculated for the left eye in step S34a, for the right eye in step S34c and for both eyes in step S34b. This accuracy score is a measure for the tracking quality in each channel. Based on these accuracy scores, in step S36 it is determined whether one of the left or the right eye is dominant. For example, if the left accuracy score and the right accuracy score are equal or their difference is below a predefined threshold, it is determined that none of the left and the right eye is dominant. If the left accuracy score is higher than the right accuracy score and optionally additionally the difference between the left and right accuracy score exceeds a predefined threshold, the left eye is determined as being dominant, especially except the binocular accuracy score is still higher than the left and the right one. The same applies for the right eye, namely if the right accuracy score is higher than the left accuracy score and optionally additionally the difference between the left and right accuracy score exceeds a predefined threshold, the right eye is determined as being dominant, especially except the binocular accuracy score is still higher than the left and the right one. In all other cases it would be determined that there is no dominant eye.

[0072] So, if for example it is determined in step S36 that the left eye is the dominant eye, a signal is provided in step S38a indicating the left eye as the dominant eye and eye tracking is performed using the left channel. If otherwise it would have been determined that the right eye is the dominant eye, a signal is outputted in step S38c indicating the right eye as the dominant eye and eye tracking is performed using the right channel. If in step S36 it would have been determined that no eye is dominant, a signal is outputted in step S38b indicating that no eye is dominant and both eyes or rather the binocular channel is used for eye tracking.

[0073] Alternatively, if one of the eyes is determined as dominant eye by the signal, it also would be possible to use the binocular gaze for eye tracking but wherein the binocular gaze is calculated on the basis of the left and right eye data, wherein the eye data of the dominant eye are weighted more heavily, e.g. the ratio of the weights could be or correspond to the ratio of the accuracy scores.

[0074] FIG. 5 shows a schematic view of an eye tracking system 10 according to an embodiment of the invention. The eye tracking system 10 comprises a processing or control unit 12, a display unit 14 and a capturing unit 16. Optionally, the eye tracking system 10 can comprise an illumination unit 18. The display unit 14 can be a monitor with a display, e.g. a computer monitor, or the eye tracking system 10 can be implemented as a head mounted display with an eye tracker. The capturing unit 16 can comprise one or more cameras capturing the eyes of a user. During a calibration and/or validation procedure of the eye tracking system 10, calibration points and/or validation points can be shown by means of the display unit 14. During the calibration and/or validation and for example also during subsequent eye tracking, the capturing unit 16 captures images of the eyes of a user. These images are then processed by the processing unit 12, which determines according to a predefined accuracy function a left accuracy score for the left eye and a right accuracy score for the right eye and optionally a binocular accuracy score for both eyes. Based on these accuracy scores, the processing unit 12 determines if one of the left and the right eye of the user is dominant. This result can be outputted on the display unit 14 or to be used for subsequent eye tracking. The illumination unit 18 can comprise one or more light sources, for example infrared LEDs, for illuminating the eyes of the user for the purpose of producing glints or other Purkinje images, which facilitates determining eye properties like the positions and orientations of eye features, the gazes, and so on.

[0075] To sum up, a method operating an eye tracking system and an eye tracking system is provided, by means of which advantageously the dominant eye of a user can be easily determined and which can be used for diagnostic purposes and especially for achieving much higher accuracy in eye tracking.

EYE TRACKING SYSTEM AND METHOD TO DETECT THE DOMINANT EYE

Assignee

Inventors

Cpc classification

Classification Explorer

A61B3/032

HUMAN NECESSITIES

Classification Explorer

G06T7/0012

PHYSICS

Classification Explorer

G06T7/70

PHYSICS

Classification Explorer

G06T2207/30041

PHYSICS

Classification Explorer

A61B3/08

HUMAN NECESSITIES

Classification Explorer

A61B3/14

HUMAN NECESSITIES

Classification Explorer

A61B3/113

HUMAN NECESSITIES

International classification

Classification Explorer

A61B3/113

HUMAN NECESSITIES

Classification Explorer

A61B3/08

HUMAN NECESSITIES

Classification Explorer

G06T7/00

PHYSICS

Classification Explorer

G06T7/70

PHYSICS

Classification Explorer

A61B3/14

HUMAN NECESSITIES

Abstract

Claims

Description