EYE CENTER LOCALIZATION METHOD AND LOCALIZATION SYSTEM THEREOF

Abstract

An eye center localization method includes performing an image sketching step, a frontal face generating step, an eye center marking step and a geometric transforming step. The image sketching step is performed to drive a processing unit to sketch a face image from the image. The frontal face generating step is performed to drive the processing unit to transform the face image into a frontal face image according to a frontal face generating model. The eye center marking step is performed to drive the processing unit to mark a frontal eye center position information on the frontal face image. The geometric transforming step is performed to drive the processing unit to calculate two rotating variables between the face image and the frontal face image, and calculate the eye center position information according to the two rotating variables and the frontal eye center position information.

Claims

1. An eye center localization method, which is configured to locate an eye center position information from an image, the eye center localization method comprising: performing an image sketching step to drive a processing unit to sketch a face image from the image of a database; performing a frontal face generating step to drive the processing unit to transform the face image into a frontal face image according to a frontal face generating model; performing an eye center marking step to drive the processing unit to mark a frontal eye center position information on the frontal face image according to a gradient method; and performing a geometric transforming step to drive the processing unit to calculate two rotating variables between the face image and the frontal face image, and calculate the eye center position information according to the two rotating variables and the frontal eye center position information.

2. The eye center localization method of claim 1, wherein the image sketching step comprises: performing a facial feature marking step to mark a chin feature point, a right eye feature point and a left eye feature point on the image; and performing a facial area sketching step to sketch the face image according to the chin feature point, the right eye feature point and the left eye feature point.

3. The eye center localization method of claim 1, wherein the eye center marking step comprises: performing a weight adjusting step to adjust a weight value of the frontal face image according to an Iris-Ripple filter method.

4. The eye center localization method of claim 1, wherein the geometric transforming step comprises: performing a rotating variable calculating step to calculate the two rotating variables between the face image and the frontal face image according to a linear relation equation; and performing an eye center transforming step to predict a depth transforming coordinate of the face image with respect to the frontal face image according to the two rotating variables, and calculate the eye center position information according to the depth transforming coordinate.

5. The eye center localization method of claim 1, further comprising: performing a model training step to drive the processing unit to train the face image, the eye center position information, the frontal face image and the frontal eye center position information to generate an eye center locating model.

6. An eye center localization system, which is configured to locate an eye center position information from an image, the eye center localization system comprising: a database configured to access the image, a frontal face generating model and a gradient method; and a processing unit electrically connected to the database, wherein the processing unit receives the image, the frontal face generating model and the gradient method and is configured to implement an eye center localization method comprising: performing an image sketching step to sketch a face image from the image; performing a frontal face generating step to transform the face image into a frontal face image according to the frontal face generating model; performing an eye center marking step to mark a frontal eye center position information on the frontal face image according to the gradient method; and performing a geometric transforming step to calculate two rotating variables between the face image and the frontal face image, and calculate the eye center position information according to the two rotating variables and the frontal eye center position information.

7. The eye center localization system of claim 6, wherein the image sketching step comprises: performing a facial feature marking step to mark a chin feature point, a right eye feature point and a left eye feature point on the image; and performing a facial area sketching step to sketch the face image according to the chin feature point, the right eye feature point and the left eye feature point.

8. The eye center localization system of claim 6, wherein the eye center marking step comprises: performing a weight adjusting step to adjust a weight value of the frontal face image according to an Iris-Ripple filter method.

9. The eye center localization system of claim 6, wherein the geometric transforming step comprises: performing a rotating variable calculating step to calculate the two rotating variables between the face image and the frontal face image according to a linear relation equation; and performing an eye center transforming step to predict a depth transforming coordinate of the face image with respect to the frontal face image, and calculate the eye center position information according to the depth transforming coordinate.

10. The eye center localization system of claim 6, wherein the processing unit further comprises: performing a model training step to train the face image, the eye center position information, the frontal face image and the frontal eye center position information to generate an eye center locating model.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present disclosure can be more fully understood by reading the following detailed description of the embodiment, with reference made to the accompanying drawings as follows:

[0008] FIG. 1 shows a flow chart of an eye center localization method according to a first embodiment of the present disclosure.

[0009] FIG. 2 shows a flow chart of an eye center localization method according to a second embodiment of the present disclosure.

[0010] FIG. 3 shows a schematic view of an image of an image sketching step of the eye center localization method of FIG. 2.

[0011] FIG. 4 shows a schematic view of a face image of the image sketching step of the eye center localization method of FIG. 2.

[0012] FIG. 5 shows a schematic view of a frontal face image of a frontal face generating step of the eye center localization method of FIG. 2.

[0013] FIG. 6 shows a schematic view of a rotating variable of a geometric transforming step of the eye center localization method of FIG. 2.

[0014] FIG. 7 shows a schematic view of another rotating variable of the geometric transforming step of the eye center localization method of FIG. 2.

[0015] FIG. 8 shows a schematic view of a model training step of the eye center localization method of FIG. 2.

[0016] FIG. 9 shows a block diagram of an eye center localization system according to a third embodiment of the present disclosure.

DETAILED DESCRIPTION

[0017] The embodiment will be described with the drawings. For clarity, some practical details will be described below. However, it should be noted that the present disclosure should not be limited by the practical details, that is, in some embodiment, the practical details is unnecessary. In addition, for simplifying the drawings, some conventional structures and elements will be simply illustrated, and repeated elements may be represented by the same labels.

[0018] It will be understood that when an element (or device) is referred to as be “connected to” another element, it can be directly connected to other element, or it can be indirectly connected to the other element, that is, intervening elements may be present. In contrast, when an element is referred to as be “directly connected to” another element, there are no intervening elements present. In addition, the terms first, second, third, etc. are used herein to describe various elements or components, these elements or components should not be limited by these terms. Consequently, a first element or component discussed below could be termed a second element or component.

[0019] Please refer to FIG. 1. FIG. 1 shows a flow chart of an eye center localization method 100 according to a first embodiment of the present disclosure. The eye center localization method 100 is configured to locate an eye center position information from an image. The eye center localization method 100 includes performing an image sketching step S01, a frontal face generating step S02, an eye center marking step S03 and a geometric transforming step S04. The image sketching step S01 is performed to drive a processing unit to sketch a face image from the image of a database. The frontal face generating step S02 is performed to drive the processing unit to transform the face image into a frontal face image according to a frontal face generating model. The eye center marking step S03 is performed to drive the processing unit to mark a frontal eye center position information on the frontal face image according to a gradient method. The geometric transforming step S04 is performed to drive the processing unit to calculate two rotating variables between the face image and the frontal face image, and calculate the eye center position information according to the two rotating variables and the frontal eye center position information. In detail, the image can be a normal view face image, a non-normal view face image, a face image with shaded eye region or a face image with defect region. Thus, the eye center localization method 100 of the present disclosure locates the eye center position information from a non-normal view face image.

[0020] Please refer to FIG. 2 to FIG. 7. FIG. 2 shows a flow chart of an eye center localization method 100 a according to a second embodiment of the present disclosure. FIG. 3 shows a schematic view of an image I of an image sketching step S11 of the eye center localization method 100a of FIG. 2. FIG. 4 shows a schematic view of a face image I.sub.f of the image sketching step S11 of the eye center localization method 100a of FIG. 2. FIG. 5 shows a schematic view of a frontal face image IF.sub.f of a frontal face generating step S12 of the eye center localization method 100a of FIG. 2. FIG. 6 shows a schematic view of a rotating variable face.sub.θ1 of a geometric transforming step S14 of the eye center localization method 100a of FIG. 2. FIG. 7 shows a schematic view of another rotating variable face.sub.θ2 of the geometric transforming step S14 of the eye center localization method 100a of FIG. 2. The eye center localization method 100a includes performing an image sketching step S11, a frontal face generating step S12, an eye center marking step S13 and a geometric transforming step S14. The image sketching step S11 is performed to drive a processing unit to sketch the face image I.sub.f from the image I of a database. The image sketching step S11 includes a facial feature marking step S112 and a facial area sketching step S114. The facial feature marking step S112 is performed to mark a chin feature point p.sub.8, right eye feature points p.sub.36, p.sub.37, p.sub.38, p.sub.39, p.sub.40, p.sub.41 and left eye feature points p.sub.42, p.sub.43, p.sub.44, p.sub.45, p.sub.46, p.sub.47 on the image I. The facial area sketching step S114 is performed to sketch the face image I.sub.f according to the chin feature point p.sub.8, the right eye feature points p.sub.36-p.sub.41 and the left eye feature points p.sub.42-p.sub.47.

[0021] Please refer to FIG. 3 and FIG. 4. The facial feature marking step S112 is performed to mark the chin feature point p.sub.8, fetch a plurality of right eye feature points p.sub.36-p.sub.41 and a plurality of left eye feature points p.sub.42-p.sub.47 around the right eye and the left eye, respectively, and predict an estimate right eye center coordinate (AEC.sub.r_x, AEC.sub.r_y) and an estimate left eye center coordinate (AEC.sub.l_x, AEC.sub.l_y) according to the right eye feature points p.sub.36-p.sub.41 and the left eye feature points P.sub.42-P.sub.47. The calculating method of the estimate right eye center coordinate (AEC.sub.r_x, AEC.sub.r_y) is satisfied by a formula (1). The facial area sketching step S114 is performed to calculate a face height face.sub.h and a face width face.sub.w of a facial area of the image I according to the estimate right eye center coordinate (AEC.sub.r_x, AEC.sub.r_y) and the estimate left eye center coordinate (AEC.sub.I_x, AEC.sub.I_y), and calculate a range of the facial area to sketch the face image I.sub.f. The calculating method of the face height face.sub.h and the face width face.sub.w are satisfied by a formula (2) to a formula (5).

[00001]$\begin{array}{cc}\left({\mathrm{AEC}}_r\mathrm{\_x},{\mathrm{AEC}}_r\mathrm{\_y}\right)=\frac{\left(p_{36}-p_{39}\right)+\left(p_{37}-p_{40}\right)+\left(p_{38}-p_{41}\right)}{2}.& \left(1\right)\end{array}$$\begin{array}{cc}{D}^{\star }=\begin{array}{c}\arg \max \\ D\end{array}\{\begin{array}{c}{D}_r={.Math.{\mathrm{AEC}}_r-p_8.Math.}_2\\ D_I={.Math.{\mathrm{AEC}}_I-p_8.Math.}_2\end{array}.& \left(2\right)\end{array}$$\begin{array}{cc}{\mathrm{face}}_h={\mathrm{face}}_w=D^{*}+\left(\frac{D^{*}}{{\alpha }_1}\right).& \left(3\right)\end{array}$$\begin{array}{cc}(\mathrm{ULC\_x},\mathrm{ULC\_y}=\left\{{\mathrm{AEC}}_r\mathrm{\_x}-\frac{D^{*}}{{\alpha }_2},{\mathrm{AEC}}_r\mathrm{\_y}-\frac{D^{*}}{{\alpha }_2}\right\}.& \left(4\right)\end{array}$$\begin{array}{cc}{I}_f={.Math.}_{x=\mathrm{ULC\_x}}^{\mathrm{ULC\_x}+{\mathrm{face}}_w}{.Math.}_{y=\mathrm{ULC\_y}}^{\mathrm{ULC\_y}+{\mathrm{face}}_h}I\left(x,y\right).& \left(5\right)\end{array}$

D* is a maximum value of an Euclidean distance from the estimate right eye center coordinate (AEC.sub.r_x, AEC.sub.r_y) and the estimate left eye center coordinate (AEC.sub.I_x, AEC.sub.l_y) to the chin feature point p.sub.8. α.sub.1 and α.sub.2 are adjustable coefficients. (ULC_x, ULC_y) is a coordinate of a begin point of sketching the facial area.

[0022] Please refer to FIG. 5. The frontal face generating step S12 is performed to drive the processing unit to transform the face image I.sub.f into the frontal face image IF.sub.f according to a frontal face generating model. In detail, the frontal face generating model is trained from a complete representation-generative adversarial network (CR-GAN) and a supervised-learning. The CR-GAN and the supervised-learning are conventional and will not be described again.

[0023] The eye center marking step S13 is performed to drive the processing unit to mark a frontal eye center position information C on the frontal face image IF.sub.f according to a gradient method. The eye center marking step S13 includes a weight adjusting step S132. The weight adjusting step S132 is performed to adjust a weight value of the frontal face image IF.sub.f according to an Iris-Ripple filter method. More particularly, the frontal eye center position information C includes a frontal right eye center coordinate (C.sub.r_x, C.sub.r_y) and a frontal left eye center coordinate (C.sub.l_x, C.sub.l _y). During marking the frontal eye center position information C, the shadow of the specific area (such as an eyelid area, a canthus area and an eyebrow area) of the frontal face image IF.sub.f will interfere the gradient of the frontal face image IF.sub.f, and reduce the accuracy of marking the frontal eye center position information C by the gradient method. Thus, adjusting the weight value by the Iris-Ripple method can increase the locating accuracy. The Iris-Ripple filter method is satisfied by a formula (6) and a formula (7), and the Iris-Ripple method combines with the gradient method is satisfied by a formula (8).

[00002]$\begin{array}{cc}{R}_r^{*}=\begin{array}{c}\arg \max \\ R_r\end{array}\{\begin{array}{c}{.Math.p_{36}-{\mathrm{AEC}}_r.Math.}_2\\ {.Math.p_{39}-{\mathrm{AEC}}_r.Math.}_2\end{array}.& \left(6\right)\end{array}$$\begin{array}{cc}\mathrm{IR}\left(x,y\right)={.Math.}_{r=0}^{\frac{{\mathrm{Eye}}_m}{2}}\{\begin{array}{c}r\tau \left\{\mathrm{Lx},\mathrm{Ly}\right\}=\omega \left(1-\frac{r}{R_r^{\star }}\right),\mathrm{if}r\le R_r^{\star }\\ r\tau \left\{\mathrm{Lx},\mathrm{Ly}\right\}=\omega \left(0\right),\mathrm{if}r>R_r^{\star }\end{array}.& \left(7\right)\end{array}$$\begin{array}{cc}C=\begin{array}{c}\arg \max \\ C^{\prime }\end{array}\left\{\frac{1}{N}{.Math.}_{x=1}^{{\mathrm{Eye}}_m}{.Math.}_{y=1}^{{\mathrm{Eye}}_n}\mathrm{IR}\left(x,y\right).Math.\left[{\alpha }_3-{\mathrm{IF}}_e\left(\mathrm{AEC}\left(x,y\right)\right).Math.{\left(d^t\left(x,y\right).Math.g\left(x,y\right)\right)}^2\right]\right\}.& \left(8\right)\end{array}$

R.sub.r* represents the eye area, IR(x, y) represents the coordinate of the current adjusting pixel, Eye.sub.m represents a column number of the pixel of the eye area, Eye.sub.n represents a row number of the pixel of the eye area, r represents a radius of the eye area, τ=2π, {Lx, Ly} is a coordinate of a pixel which is calculated by a radius perimeter taking the estimate right eye center coordinate (AEC.sub.r_x, AEC-.sub.r_y) and the estimate left eye center coordinate (AEC.sub.l_x, AEC.sub.l_y) as centers, w(⋅)is a weight value before calculating, C′ represents a current eye center coordinate, N is a pixel number of the eye area, IF.sub.e(AEC(x, y)) is a strength of predicting the center of the eye area, d(x, y) is a displacement vector between c and p(x, y), g(x, y) is a gradient vector, and α.sub.3 is a maximum grayscale.

[0024] Please refer to FIG. 6 and FIG. 7, the geometric transforming step S14 is performed to drive the processing unit to calculate two rotating variables face.sub.θ1, face.sub.θ2 between the face image I.sub.f and the frontal face image IF.sub.f, and calculate the eye center position information I.sub.ec according to the two rotating variables face.sub.θ1 , face.sub.θ2 and the frontal eye center position information C. The geometric transforming step S14 includes a rotating variable calculating step S142 and an eye center transforming step S144. The rotating variable calculating step S142 is performed to calculate the two rotating variables face.sub.θ1 , face.sub.θ2 between the face image I.sub.f and the frontal face image IF.sub.f according to a linear relation equation, the linear relation equation is satisfied with a formula (9).

[00003]$\begin{array}{cc}\{\begin{array}{c}{\mathrm{face}}_{\mathrm{\theta 1}}={\tan }^{-1}\left(.Math.\frac{m_1\left(L_1\right)-m_2\left(L_2\right)}{1+m_1\left(L_1\right)\star m_2\left(L_2\right)}.Math.\right)\star \frac{180}{\pi }\\ {\mathrm{face}}_{\mathrm{\theta 2}}={\tan }^{-1}\left(.Math.\frac{m_1\left(L_1\right)-m_3\left(L_3\right)}{1+m_1\left(L_1\right)\star m_3\left(L_3\right)}.Math.\right)\star \frac{180}{\pi }\end{array}.& \left(9\right)\end{array}$

The rotating variable face.sub.θ1 is a rotating variable between the face image I.sub.f and the frontal face image IF.sub.f which is rotating along the x axis (i.e., yaw rot.), the rotating variable face.sub.θ2 is a rotating variable between the face image I.sub.f and the face transforming image I.sub.f′ which is rotating along the z axis (i.e., roll rot.). L1 is a linear relation equation between the estimate right eye center coordinate (AEC.sub.r _x, AEC.sub.r_y) and the estimate left eye center coordinate (AEC.sub.l_x, AEC.sub.l_y), L2 is a linear relation equation between the frontal right eye center coordinate (C.sub.r_x, C.sub.r_y) and the frontal left eye center coordinate (C.sub.l_x, C.sub.l_y), and L3 is a linear relation equation between the estimate right eye center coordinate (AEC.sub.r_x, AEC.sub.r_y) and the estimate left eye center coordinate (AEC.sub.l_x, AEC.sub.l_y) after transforming into the three-dimensional coordinate. m1 is a slope of the linear relation equation L1, m2 is a slope of the linear relation equation L2, and m3 is a slope of the linear relation equation L3.

[0025] The eye center transforming step S144 is performed to predict a depth transforming coordinate (I.sub.erc1_x, l.sub.erc1_y) of the face image I.sub.f with respect to the frontal face image IF.sub.f according to the two rotating variables face.sub.θ1, face.sub.θ2, and calculate the eye center position information l.sub.eC according to the depth transforming coordinate (l.sub.erc1_x, l.sub.erc1_y). The eye center transforming step S144 predicts the depth transforming coordinate (l.sub.erc1_x, l.sub.erc1_y) by a formula (10):

[00004]$\begin{array}{cc}\left(I_{\mathrm{erC}1}\mathrm{\_x},I_{\mathrm{erC}1}\mathrm{\_y}\right)=\left\{\frac{C_r\mathrm{\_x}-{\mathrm{IF}}_{{\mathrm{AEC}}_r}\mathrm{\_x}}{\cos \left({\mathrm{face}}_{\mathrm{\theta 1}}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 2}}\right)},\frac{C_r\mathrm{\_y}-{\mathrm{IF}}_{{\mathrm{AEC}}_r}\mathrm{\_y}}{\cos \left({\mathrm{face}}_{\mathrm{\theta 1}}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 2}}\right)}\right\}.& \left(10\right)\end{array}$

The eye center position information l.sub.eC includes a right eye center coordinate (l.sub.erC_x, l.sub.erC_y) and a left eye center coordinate (l.sub.elC_x, l.sub.elC_y), and (IF.sub.AECr_x, IF.sub.AECr_y) is a frontal face estimate right eye center coordinate.

[0026] In detail, after the formula (10) obtains the depth transforming coordinate (l.sub.erc1_x, l.sub.erc1_y), in order to avoid the difference between the frontal eye center position information C calculated by the frontal face image IF.sub.f which is generated from the frontal face generating model and the actual value. The eye center transforming step 144 can adjust the depth transforming coordinate (l.sub.erc1_x, l.sub.erc1_y) by a formula (11):

[00005]$\begin{array}{cc}\left(I_{\mathrm{erC}}\mathrm{\_x},I_{\mathrm{erC}}\mathrm{\_y}\right)=\left\{\left[\left(I_{\mathrm{erC}1}\mathrm{\_x}+\mathrm{\alpha 4}\left(\frac{I_{\mathrm{erC}2}\mathrm{\_x}-{\mathrm{AEC}}_r\mathrm{\_x}}{\cos \left({\mathrm{face}}_{\mathrm{\theta 1}}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 2}}\right)}\right)\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 1}}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 2}}\right)\right],\left[\left(I_{\mathrm{erC}1}\mathrm{\_y}+{\mathrm{AEC}}_r\mathrm{\_y}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 1}}\right)\star \cos \left({\mathrm{face}}_{\mathrm{\theta 2}}\right)\right]\right\}.& \left(11\right)\end{array}$

(l.sub.erc2_x, l.sub.erc2_y) is a frontal right eye center coordinate which has a big difference with the actual value, α.sub.4 is a correction coefficient. Thus, the eye center localization method 100a of the present disclosure adjusts the eye center position information l.sub.eC by the correction coefficient α.sub.4 to avoid the difference caused by the frontal face image IF.sub.f, thereby increasing the accuracy of the eye center position information l.sub.eC.

[0027] Please refer to FIG. 2 to FIG. 8. FIG. 8 shows a schematic view of a model training step S15 of the eye center localization method 100a of FIG. 2. The eye center localization method 100a of FIG. 2 can further include a model training step S15. The model training step S15 is performed to drive the processing unit to train the face image I.sub.f, the eye center position information l.sub.eC, the frontal face image IF.sub.f and the frontal eye center position information C to generate an eye center locating model 40. In other words, the model training step S15 sketches the eye region images I.sub.f _r, IF.sub.f_r from the face image If and the frontal face image IF.sub.f, respectively, takes the eye region before marking the eye center position information l.sub.eC and the frontal eye center position information C as a first training sample Tx, and takes the eye region after marking the eye center position information l.sub.eC and the frontal eye center position information C as a second training sample Ty. The first training sample Tx and the second training sample Ty are trained by an image translation learning to generate the eye center locating model 40. Thus, the eye center localization method 100a of the present disclosure can predict the eye center position information l.sub.eC from the image I directly by the eye center locating model 40.

[0028] Please refer to FIG. 9. FIG. 9 shows a block diagram of an eye center localization system 200 according to a third embodiment of the present disclosure. The eye center localization system 200 is configured to locate an eye center position information l.sub.ec from an image I. The eye center localization system 200 includes a database 210 and a processing unit 220.

[0029] The database 210 is configured to access the image I, a frontal face generating model 20 and a gradient method 30. In detail, the database 210 can be a memory or other data accessing element.

[0030] The processing unit 220 is electrically connected to the database 210, the processing unit 220 receives the image I, the frontal face generating model 20 and the gradient method 30, and the processing unit 220 is configured to implement the eye center localization methods 100, 100a. In detail, the processing unit 220 can be a microprocessor, a central processing unit (CPU) or other electronic processing unit, but the present disclosure is not limited thereto. Thus, the eye center localization system 200 locates the eye center position information l.sub.eC from an image I with non-frontal face.

[0031] According to the aforementioned embodiments and examples, the advantages of the present disclosure are described as follows.

[0032] 1. The eye center localization method and localization system thereof locate the eye center position information from image with non-frontal face.

[0033] 2. The eye center localization method of the present disclosure adjusts the eye center position information by the correction coefficient to avoid the difference caused by the frontal face image, thereby increasing the accuracy of the eye center position information.

[0034] 3. The eye center localization method of the present disclosure can predict the eye center position information from the image directly by the eye center locating model.

[0035] Although the present disclosure has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein.

[0036] It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present disclosure without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this disclosure provided they fall within the scope of the following claims.