Speech-driven facial animation generation method

Abstract

The present disclosure discloses a speech-driven facial animation generation method. The method is mainly divided into six steps: extracting speech features, collecting frequency information, summarizing time information, decoding action features, driving a facial model, and sliding a signal window. The present disclosure can drive, according to an input speech audio signal, any facial model in real time under a particular delay to generate animation. The quality of the animation reaches the currently most advanced speech animation technology level, and has the characteristics of light weight and good robustness. The present disclosure can be used to generate speech animation under different scenes, such as VR virtual social networking, and virtual speech assistants and games.

Claims

1. A speech-driven facial animation generation method, comprising steps of: (1) extracting speech features: extracting Mel spectrogram features from speech in a window; the features being a three-dimensional tensor composed of a feature map dimension, a frequency dimension, and a time dimension; (2) collecting frequency information: for the Mel spectrogram obtained in step (1), abstracting and collecting, by using a neural network, all frequency information along a direction of the frequency dimension, to obtain frequency abstract information; (3) summarizing time information; for the frequency abstract information obtained in step (2), determining, by using the neural network, a degree of importance of each frame of information in a time context along a direction of the time dimension, and performing summarization according to the degree of importance, to obtain time summary information; (4) decoding action features: connecting the time summary information obtained in step (3) to a one-hot vector of a control style inputted by a user; outputting scaling/shearing coefficients and rotation coefficients respectively through two branches of the neural network, and combining the coefficients outputted by the two branches to obtain deformation gradients representing facial actions; (5) driving a facial model: for any given expressionless and mouth-closed facial model, driving the facial model by using the deformation gradients obtained in step (4), to perform a corresponding facial action; and (6) sliding a signal window: repeating step (1) to step (5), and processing all speech signal windows, to generate complete facial animation.

2. The speech-driven facial animation generation method according to claim 1, wherein step (1) comprises steps of: (1.1) performing short-term Fourier transform on a speech signal with an input audio window length L.sub.audio, a frame length being L.sub.fft, and a frame interval being L.sub.hop; and converting, by using F.sub.mel Mel filters, Fourier transform results to a Mel frequency to obtain Mel spectrogram with a frame length L.sub.frame; and (1.2) taking first and second derivatives of the Mel spectrogram with respect to time as auxiliary features, and stacking the auxiliary features with original features to form a tensor in a shape of 3×F.sub.mel×L.sub.frame, wherein 3 in a first dimension denotes the number of feature maps, F.sub.mel in a second dimension denotes a length of the frequency dimension, and L.sub.frame in a third dimension denotes a length of the time dimension.

3. The speech-driven facial animation generation method according to claim 1, wherein step (2) comprises steps of: (2.1) for the Mel spectrogram obtained in step (1), extracting a local frequency feature of the Mel spectrogram by using a two-dimensional convolutional network; the two-dimensional convolutional network sequentially comprising: a first two-dimensional convolutional layer, a first two-dimensional maximum pooling layer, a second two-dimensional convolutional layer, and a second two-dimensional maximum pooling layer; the two two-dimensional convolutional layers obtaining a number of local feature maps by performing convolution calculation on inputs through C.sub.freq_conv0 and C.sub.freq_conv1 convolution kernels whose sizes are both K.sub.freq×1 along the direction of the frequency dimension respectively, wherein the number of the local feature maps is equal to the number of the convolution kernels, K.sub.freq denotes a magnitude of the direction of the frequency dimension, and l denotes a magnitude of the direction of the time dimension; the two two-dimensional convolutional layers use a leaky rectified linear unit with a negative tilt rate of 0.2 as an activation function; the two two-dimensional maximum pooling layers select local feature maximum values in a region of the size S.sub.freq×1 along the direction of the frequency dimension, to complete a down-sampling pooling operation; and the local frequency feature obtained is a tensor in a shape of $C_{\mathrm{freq\_conv}1}\times \frac{F_{\mathrm{mel}}}{{S_{\mathrm{freq}}}^2}\times L_{\mathrm{frame}},$ wherein C.sub.freq_conv1 in a first dimension denotes the number of feature maps, $\frac{F_{\mathrm{mel}}}{{S_{\mathrm{freq}}}^2}$ in a second dimension denotes a length of the frequency dimension, and L.sub.frame in a third dimension denotes a length of the time dimension; (2.2) for the local frequency feature obtained in step (2.1), projecting the local frequency feature by using C.sub.freq_conv2 convolution kernels whose size are 1×1; taking the leaky rectified linear unit with a negative tilt rate of 0.2 as the activation function; and outputting a tensor in a shape of $C_{\mathrm{freq\_conv}2}\frac{F_{\mathrm{mel}}}{{S_{\mathrm{freq}}}^2}\times L_{\mathrm{frame}},$ wherein C.sub.freq_conv2 in the first dimension denotes the number of feature maps, $\frac{F_{\mathrm{mel}}}{{S_{\mathrm{freq}}}^2}$ in the second dimension denotes the length of the frequency dimension, L.sub.frame in the third dimension denotes the length of the time dimension, and the size being 1×1 indicates that both the magnitude of the direction of the frequency dimension and the magnitude of the direction of the time dimension are equal to 1; (2.3) for the local frequency feature after projection obtained in step (2.2), along positive and negative directions of the frequency dimension, recurrently processing each feature of the frequency dimension with a long short-term memory on the frequency dimension; and (2.4) concatenating all outputs of the long short-term memory along the positive and negative directions of the frequency dimension in step (2.3) to form a vector to obtain a tensor in a shape of $\frac{C_{\mathrm{freq\_LSTM}}\times F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}\times L_{\mathrm{frame}},$ wherein $\frac{C_{\mathrm{freq\_LSTM}}\times F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}$ in the first dimension denotes the number of feature maps, and L.sub.frame in the second dimension denotes the length of the time dimension; and using a fully connected layer with the number of feature maps of C.sub.freq for projection, and collecting information of all frequencies, to obtain that the frequency abstract information z.sub.freq is a tensor in a shape of C.sub.freq×L.sub.frame, wherein C.sub.freq in the first dimension denotes the number of feature maps, and L.sub.frame in the second dimension denotes the length of the time dimension; such that the frequency dimension has been fully collected and abstracted into the feature map dimension.

4. The speech-driven facial animation generation method according to claim 1, wherein step (3) comprises steps of: (3.1) for the frequency abstract information obtained in step (2), using two hidden layers to transmit time dimension information to obtain time context information m.sub.freq; in each of the hidden layers, along positive and negative directions of the time dimension, recurrently processing each frame on the time dimension by using a long short-term memory to transmit time information; the number of feature maps of the long short-term memory in each direction being $\frac{C_{\mathrm{time}}}{2},$ a sum of the numbers of the feature maps in the two directions being C.sub.time, and the time context information m.sub.freq being a tensor in a shape of C.sub.time×L.sub.frame, wherein C.sub.time in a first dimension denotes the number of feature maps, and L.sub.frame in a second dimension denotes the length of the time dimension; and (3.2) for the time context information obtained in step (3.1), using a hidden layer to determine a weight of importance of each frame of information in a context and to perform weighting and summarization; in the hidden layer, selecting K.sub.qry frames in a middle of the time context information m.sub.freq to be projected by Catt one-dimension convolution kernels of the size of K.sub.qry as a query item q.sub.att; linearly projecting the entire time context information m.sub.freq as a key-value item k.sub.att; normalizing a sum of the query item q.sub.att and the key-value item k.sub.att with a tan h activation function, linear projection and softmax, to obtain a weight of each frame; and using the weight to perform weighting and summarization on the time context information m.sub.freq, to obtain time summary information z.sub.att; a shape of the query item q.sub.att being C.sub.att×1, wherein C.sub.att is the number of feature maps, which is the same as the number of the convolution kernels; and l is the length of the time dimension; a shape of the key-value item k.sub.att being C.sub.att×L.sub.frame, wherein C.sub.att is the number of feature maps, and L.sub.frame, is the length of the time dimension; the number of feature maps for the linear projection being projected from C.sub.att into 1; a shape of the weight being 1×L.sub.frame; and a shape of the time summary information z.sub.att being C.sub.time, wherein C.sub.time is the number of feature maps.

5. The speech-driven facial animation generation method according to claim 1, wherein in step (4), the facial action is represented with the deformation gradients, the deformation gradients are defined on a template facial model, and the template facial model is expressionless and mouth-closed and is composed of N triangles.

6. The speech-driven facial animation generation method according to claim 1, wherein step (5) comprises the steps of: (5.1) obtaining a triangle correspondence between a given facial model and a template facial model, the facial model being composed of M triangles, and the template facial model being composed of N triangles; (5.2) migrating deformation gradients corresponding to the template facial model to the given facial model; and (5.3) solving positions of vertices of the given facial model according to the migrated deformation gradients.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a schematic flowchart of a method according to the present disclosure;

(2) FIG. 2 is a schematic workflow of a long short-term memory in sub-step (2.3) of step (2) in the method according to the present disclosure;

(3) FIG. 3 is a schematic flowchart of step (3) in the method according to the present disclosure;

(4) FIG. 4 is an implementation example of driving facial model animation by speech signals according to the present disclosure, which is a sequence excerpt of an animation frame for driving a human face model to say an English word “smash”; and

(5) FIG. 5 is an implementation example of driving facial model animation by speech signals according to the present disclosure, which is a sequence excerpt of an animation frame for driving a cartoon animal face model to say an English word “smash”.

DESCRIPTION OF EMBODIMENTS

(6) The core technology of the present disclosure uses frequency-dimension convolution and bidirectional long short-term memory to abstract speech features, uses time-dimension bidirectional long short-term memory and attention module to summarize time context information within a window, and uses deformation gradients to represent facial motion. As shown in FIG. 1, the method mainly includes six steps: extracting speech features, collecting frequency information, summarizing time information, decoding action features, driving a facial model, and finally repeating the previous five steps by sliding a signal window over a speech sequence to obtain a complete animation sequence.

(7) 1. Extract speech features: extract Mel spectrogram features from speech within one window; the features being a three-dimensional tensor composed of a feature map dimension, a frequency dimension, and a time dimension.

(8) 1.1 Perform short-term Fourier transform on a speech signal with an input audio window length L.sub.audio (a frame length is L.sub.fft, and a frame interval is L.sub.hop); and convert, by using F.sub.mel Mel filters, Fourier transform results to a Mel frequency to obtain Mel spectrogram with a frame length L.sub.frame.

(9) 1.2 Take first and second derivatives of the Mel spectrogram with respect to time as auxiliary features, and stack the auxiliary features with original features to form a tensor in the shape of 3×F.sub.mel×L.sub.frame, wherein 3 in the first dimension denotes the number of feature maps, F.sub.mel in the second dimension denotes the length of the frequency dimension, and L.sub.frame in the third dimension denotes the length of the time dimension.

(10) 2. Collect frequency information: for the Mel spectrogram obtained in step (1), abstract and collect, by using a neural network, all frequency information along a direction of the frequency dimension, to obtain frequency abstract information.

(11) 2.1 For the Mel spectrogram obtained in step (1), extract a local frequency feature of the Mel spectrogram by using a two-dimensional convolutional network; the two-dimensional convolutional network sequentially includes: a first two-dimensional convolutional layer, a first two-dimensional maximum pooling layer, a second two-dimensional convolutional layer, and a second two-dimensional maximum pooling layer; the two two-dimensional convolutional layers perform convolution calculation on inputs respectively through C.sub.freq_conv0 and C.sub.freq_conv1 convolution kernels (whose sizes are both K.sub.freq×1, wherein K.sub.freq denotes the magnitude of the direction of the frequency dimension, and 1 denotes the magnitude of the direction of the time dimension) along the direction of the frequency dimension to acquire a number of local feature maps (the number is equal to the number of the convolution kernels), and the two two-dimensional convolutional layers use a Leaky ReLU with a negative tilt rate of 0.2 as an activation function; the two two-dimensional maximum pooling layers select local feature maximum values in a region (of the size S.sub.freq×1) along the direction of the frequency dimension, to complete a down-sampling pooling operation; and the local frequency feature obtained is a tensor in the shape of

(12) $C_{\mathrm{freq\_conv}1}\times \frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}\times L_{\mathrm{frame}},$
wherein C.sub.freq_conv1 in the first dimension denotes the number of feature maps,

(13) $\frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}$
in the second dimension denotes the length of the frequency dimension, and L.sub.frame in the third dimension denotes the length of the time dimension.

(14) 2.2 For the local frequency feature obtained in step (2.1), project the local frequency feature by using C.sub.freq_conv2 convolution kernels whose size are 1×1 (both the magnitude of the direction of the frequency dimension and the magnitude of the direction of the time dimension are 1); take the Leaky ReLU with a negative tilt rate of 0.2 as the activation function; and output a tensor in the shape of

(15) $C_{\mathrm{freq\_conv}2}\times \frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}\times L_{\mathrm{frame}},$
wherein C.sub.freq_conv2 in the first dimension denotes the number of feature maps,

(16) $\frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}$
in the second dimension denotes the length of the frequency dimension, and L.sub.frame in the third dimension denotes the length of the time dimension.

(17) 2.3 For the local frequency feature after projection which is obtained in step (2.2), along positive and negative directions of the frequency dimension, recurrently process each feature of the frequency dimension with a long short-term memory on the frequency dimension. As shown in FIG. 2, the long short-term memory has a state machine (used to store historical information of the memory) and three gates. An input gate i.sub.t acts on each frequency feature x.sub.t (x denotes an input, the subscript t denotes the t.sup.th input moment) and an output h.sub.t-1 in a previous step (h denotes an output, and the subscript t−1 denotes the (t−1).sup.th input moment, that is, the previous step) of the memory, and indicates whether to allow addition of new frequency information to the state machine of the memory, and the value ranges from 0 to 1 (including two ends). If the value of the input gate is 1 (that is, open the gate), new information is added. If the value is 0 (that is, close the gate), a zero vector is added. If the value is a value between 0 and 1, the new information multiplied by the gate value is added. A forget gate f.sub.t acts on the state machine of the memory, and indicates whether to retain historical frequency information S.sub.t-1 stored in the state machine in the previous step (S denotes a state of the state machine, and the subscript t−1 denotes the (t−1).sup.th input moment, that is, the previous step), and the value ranges from 0 to 1 (including two ends). If the value of the forget gate is 1 (that is, open the gate), stored information is retained. If the value is 0 (that is, close the gate), the stored information is reset to a zero vector. If the value is a value between 0 and 1, the stored information multiplied by the gate value is retained. An output gate o.sub.t acts on the state machine of the memory, and indicates whether to take a current state S.sub.t (S denotes a state of the state machine, and the subscript t denotes the t.sup.th input moment) of the memory as an output, and the value ranges from 0 to 1 (including two ends). If the value is 1 (that is, open the gate), the current state of the memory is taken as an output. If the value is 0 (that is, close the gate), a zero vector is taken as an output. If the value is a value between 0 and 1, the current state of the memory multiplied by the gate value is taken as an output. Specific values of the three gates are obtained by connecting and projecting the current input x.sub.t and the output h.sub.t-1 in the previous step of the memory, of which the specific formula is as follows:
i.sub.t=σ(W.sub.i[x.sub.t,h.sub.t-1]+b.sub.i)
f.sub.t=σ(W.sub.f[x.sub.t,h.sub.t-1]+b.sub.f)
o.sub.t=σ(W.sub.o[x.sub.t,h.sub.t-1]+b.sub.o)
{tilde over (x)}.sub.t=tan h(W.sub.x[x.sub.t,h.sub.t−1]+b.sub.x)
S.sub.t=f.sub.t*S.sub.t−1+i.sub.t*{tilde over (x)}.sub.t
h.sub.t=o.sub.t*tan h(S.sub.t)  (1)

(18) where x.sub.t is the current input, and h.sub.t−1 is the output in the previous step of the memory; i.sub.t is the value of the input gate, and W.sub.i and b.sub.i are a weight and a bias parameter of the input gate respectively; f.sub.t is the value of the forget gate, and W.sub.f and b.sub.f are a weight and a bias parameter of the forget gate respectively; o.sub.f is the value of the output gate, and W.sub.o and b.sub.o are a weight and a bias parameter of the output gate respectively; it is a projection for the current input and the output in the previous step, and W.sub.f and b.sub.f are a weight and a bias parameter of the projection respectively; S.sub.t−1 and S.sub.t are a state in the previous step and a current state of the state machine of the memory respectively; and h.sub.t is a current output of the memory.

(19) The number of feature maps of the long short-term memory in each direction is

(20) $\frac{C_{\mathrm{freq\_LSTM}}}{2},$
the sum of the numbers of feature maps in two directions is C.sub.freq_LSTM, and thus an output of the bidirectional long short-term memory in this step is a tensor in the shape of

(21) $C_{\mathrm{freq\_LSTM}}\times \frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}\times L_{\mathrm{frame}},$
wherein C.sub.freq_LSTM in the first dimension denotes the number of feature maps,

(22) $\frac{F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}$
in the second dimension denotes the length of the frequency dimension, and L.sub.frame in the third dimension denotes the length of the time dimension.

(23) The state machine of the long short-term memory and three gates operating around the state machine make it possible to fully consider features of other frequencies when analyzing features of a frequency, and conform to the natural phenomenon that formants occur in human articulation.

(24) 2.4 Concatenate all outputs of the long short-term memory along the positive and negative directions of the frequency dimension in step (2.3) to form a vector to obtain a tensor in the shape of

(25) $\frac{C_{\mathrm{freq\_LSTM}}\times F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}\times L_{\mathrm{frame}},$
wherein

(26) $\frac{C_{\mathrm{freq\_LSTM}}\times F_{\mathrm{mel}}}{S_{{\mathrm{freq}}^2}}$
in the first dimension denotes the number of feature maps, and L.sub.frame in the second dimension denotes the length of the time dimension; and use a fully connected layer with the number of feature maps of C.sub.freq for projection, and collect information of all frequencies, to obtain that the frequency abstract information z.sub.freq being a tensor in the shape of C.sub.freq×L.sub.frame, wherein C.sub.freq in the first dimension denotes the number of feature maps, and L.sub.frame in the second dimension denotes the length of the time dimension. So far, the frequency dimension has been fully collected and abstracted into the feature map dimension.

(27) 3. Summarize time information: for the frequency abstract information obtained in step (2), determine, by using the neural network, a degree of importance of each frame of information in a time context along a direction of the time dimension, and perform summarization according to the degrees of importance, to obtain time summary information. The specific process is shown in FIG. 3.

(28) 3.1 For the frequency abstract information obtained in step (2), use two hidden lavers to transmit time dimension information to obtain time context information; in each of the hidden layers, along positive and negative directions of the time dimension, recurrently process each frame in the time dimension by using a long short-term memory to transmit time information. The long short-term memory has the same structural principle as the long short-term memory described in step (2.3), but acts on the direction of the time dimension, and has a state machine (used to store historical information of the memory) and three gates. An input gate acts on each frame of time feature and an output in a previous step of the memory, and indicates whether to allow addition of new time frame information to the state machine of the memory, and the value ranges from 0 to 1 (including two ends). If the value of the input gate is 1 (that is, open the gate), new information is added. If the value is 0 (that is, close the gate), a zero vector is added. If the value is a value between 0 and 1, the new information multiplied by the gate value is added. A forget gate acts on the state machine of the memory, and indicates whether to retain historical time information stored in the state machine in the previous step, and the value ranges from 0 to 1 (including two ends). If the value of the forget gate is 1 (that is, open the gate), stored information is retained. If the value is 0 (that is, close the gate), the stored information is reset to a zero vector. If the value is a value between 0 and 1, the stored information multiplied by the gate value is retained. An output gate acts on the state machine of the memory, and indicates whether to take a current state of the memory as an output, and the value ranges from 0 to 1 (including two ends). If the value is 1 (that is, open the gate), the current state of the memory is taken as an output. If the value is 0 (that is, close the gate), a zero vector is taken as an output. If the value is a value between 0 and 1, the current state of the memory multiplied by the gate value is taken as an output. Specific values of the three gates are obtained by connecting and projecting the current input time frame (or a previous output of the hidden layer) and the output in the previous step of the memory.

(29) The number of feature maps of the long short-term memory in each direction is

(30) 0$\frac{C_{\mathrm{time}}}{2},$
the sum of the numbers of feature maps in two directions is C.sub.time, and thus time context information m.sub.freq obtained in this step is a tensor in the shape of C.sub.time×L.sub.frame, wherein C.sub.time in the first dimension denotes the number of feature maps, and L.sub.frame in the second dimension denotes the length of the time dimension

(31) 3.2 For the time context information obtained in step (3.1), use a hidden layer to determine a weight of importance of each frame of information in a context and to perform weighting and summarization; in the hidden layer, select K.sub.qry frames in the middle of the time context information m.sub.freq to be projected by C.sub.att one-dimension convolution kernels (also of the size of K.sub.qry) as a query item q.sub.att (the shape is C.sub.att×1, wherein C.sub.att is the number of feature maps which is the same as the number of the convolution kernels, and 1 is the length of the time dimension), linearly project the entire time context information m.sub.freq as a key-value item k.sub.att (the shape is C.sub.att×L.sub.frame, wherein C.sub.att is the number of feature maps, and L.sub.frame is the length of the time dimension), normalize the sum of the query item q.sub.att and the key-value item k.sub.att with a tan h activation function, linear projection and softmax (the number of feature maps is projected from C.sub.att into 1), to obtain a weight of each frame (the shape is 1×L.sub.frame), and use the weight to weight and summarize the time context information m.sub.freq, to obtain time summary information z.sub.att (the shape is C.sub.time, wherein C.sub.time is the number of feature maps). The hidden layer imitates the pattern of human natural pronunciation through the weight of time dimension. For example, pronunciation of a vowel has a long time span, and pronunciation of a consonant is manifested as transient pronunciation and is related to transition vowels before and after the consonant.

(32) 4. Decode action features: connect the time summary information obtained in step (3) to a one-hot vector of a control style inputted by a user; output scaling/shearing coefficients and rotation coefficients respectively through two branches of the neural network, and combine the coefficients outputted by the two branches to obtain deformation gradients representing facial actions.

(33) The present disclosure uses deformation gradients to represent facial actions in speech-driven speech animation for the first time, and can describe local changes of facial motion more accurately than the previous technologies.

(34) 4.1 Deformation Gradient

(35) The present disclosure uses a method described in “(Robert W Sumner and Jovan Popovic. Deformation transfer for triangle meshes. ACM Transactions on graphics (TOG), 23(3):399-405, 2004.)” to calculate deformation gradients of a facial model. The facial model is composed of three triangular face segments. v.sub.i.sup.(k), {tilde over (v)}.sub.i.sup.(k), k∈{1,2,3} are used to represent three vertices of the i.sup.th triangle in the facial model and the deformed facial model respectively. In order to deal with the deformation perpendicular to the direction of the triangle, the fourth vertex of the triangle is calculated according to the following formula:

(36) $\begin{array}{cc}{n}_i=\left(v_i^{\left(2\right)}-v_i^{\left(1\right)}\right)\times \left(v_i^{\left(3\right)}-v_i^{\left(1\right)}\right)v_i^{\left(4\right)}=v_i^{\left(i\right)}+\frac{n_i}{\sqrt{.Math.n_i.Math.}}& \left(2\right)\end{array}$

(37) The deformation gradient of the i.sup.th triangle is a transformation matrix T.sub.i satisfying the following formula:
T.sub.iV.sub.i={tilde over (V)}.sub.i  (3)

(38) where V.sub.i and {tilde over (V)}.sub.i are formed by stacking three vectors of reference and deformed triangles respectively:
V.sub.i=[v.sub.i.sup.(2)−v.sub.i.sup.(1)v.sub.i.sup.(3)−v.sub.i.sup.(1)v.sub.i.sup.(4)−v.sub.i.sup.(1)]
{tilde over (V)}.sub.i=[{tilde over (v)}.sub.i.sup.(2)−{tilde over (v)}.sub.i.sup.(1){tilde over (v)}.sub.i.sup.(3)−{tilde over (v)}.sub.i.sup.(1){tilde over (v)}.sub.i.sup.(4)−{tilde over (v)}.sub.i.sup.(1)]  (4)

(39) Therefore, T.sub.i={tilde over (V)}.sub.iV.sub.i.sup.−1.

(40) The present disclosure further uses a method described in “(Qianyi Wu, Juyong Zhang, Yu-Kun Lai, Jianmin Zheng, and Jianfei Cai. Alive caricature from 2d to 3d. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7336-7345, 2018)” to perform polar decomposition on the transformation matrix, T.sub.i=R.sub.iS.sub.i. Here, S.sub.i denotes scalinglshearing transformation, and is a symmetric matrix, which may be represented with 6 parameters; R.sub.i denotes rotation transformation, and may be represented with 3 parameters by using Rodrigues' formula. Therefore, a deformation gradient of each triangle is represented with 9 parameters.

(41) The deformation gradient in the present disclosure is defined on one template facial model. The template facial model is expressionless and mouth-closed and is composed of N triangles. Therefore, the corresponding deformation gradient includes 9N=6N+3N parameters.

(42) 4.2 Decode Actions

(43) C.sub.speaker-dimension one-hot vector for controlling a speaker style is concatenated to the time summary information z.sub.att obtained in step (3). z.sub.dec (in the shape of C.sub.dec0) is obtained through a fully connected layer with the number of feature maps of C.sub.dec0 and a Leaky ReLU (LReLU) activation function with a negative tilt rate of 0.2. Then, z.sub.dec goes through two neural network branches that are similar in structure and parallel to decode scaling/shearing and rotation parameters.

(44) In each branch, z.sub.dec is connected to a C.sub.speaker-dimension one-hot control vector again and projected by three fully connected layers. The number of feature maps of the first fully connected layer is C.sub.dec1, and the activation function is LReLU with a negative tilt rate of 0.2. The number of feature maps of the second fully connected layer is C.sub.dec2, and the activation function is tan h. The third fully connected layer has no activation function (in scaling/shearing and rotation branches, the numbers of feature maps are C.sub.pca_s and C.sub.pca_r respectively). At the end of the branch is a fixed linear fully connected layer (in scaling/shearing, and rotation branches, the numbers of feature maps are 6N and 3N respectively), and parameters thereof are initiated by a principal component analysis base and a mean of training data corresponding to the branch. 97% of the energy is retained during principal component analysis, and the numbers of bases retained in the scaling/shearing and rotation branches are C.sub.pca_s and C.sub.pca_r respectively (which are the same as the numbers of feature maps in the third fully connected layer of the branch). The two branches are decoded respectively to obtain a parameter s (the size of which is 6N) representing scaling/shearing and a parameter r (the size of which is 3N) representing rotation.

(45) 5. Drive a facial model: for any given facial model (expressionless and mouth-closed), drive the facial model by using the deformation gradients obtained in step (4), to perform a corresponding facial action.

(46) 5.1 Obtain a triangle correspondence between a given facial model (composed of M triangles) and a template facial model (composed of N triangles) (each given model only needs to perform this sub-step once).

(47) If the topologies of the given facial model and the template facial model are different from each other, the deformation gradient obtained in step (4) cannot be used directly. A triangular correspondence between the two models needs to be obtained first. if the topologies are the same, the deformation gradient can be used directly.

(48) The present disclosure uses a method described in “(Robert W Sumner and Jovan Popovic. Deformation transfer for triangle meshes. ACM Transactions on graphics (TOG), 23(3):399-405, 2004.)” to automatically solve a triangular correspondence between two facial models with different topologies under the condition that the user is given correspondences of a number of vertices.

(49) The automatic solving method first needs to find a series of transformation matrices (including scaling/shearing and rotation transformation, but not including translation transformation) O.sub.i, i∈{1, . . . , M} to deform the given facial model to a state closest to the template facial model. The sum E of the following three energy equations E.sub.S, E.sub.I, E.sub.C and an energy equation under limited conditions is defined, and E is minimized to deform the given facial model to a target state:

(50) $\begin{array}{cc}{E}_S=\underset{i=1}{\overset{M}{.Math.}}\underset{j\in \mathrm{adj}\left(i\right)}{.Math.}{.Math.O_i-O_j.Math.}^2{E}_I=\underset{i=1}{\overset{M}{.Math.}}{.Math.O_i-I.Math.}^2{E}_C=\underset{i=1}{\overset{n}{.Math.}}{.Math.{\hat{v}}_i-c_i.Math.}^2\underset{{\hat{v}}_1\mathrm{.Math.}{\hat{v}}_n}{\min }E=w_S{E}_S+w_I{E}_I+w_C{E}_C,\mathrm{subject}\mathrm{to}{\hat{v}}_{s_k}=m_k,k\in \left\{1\mathrm{.Math.}m\right\}& \left(3\right)\end{array}$

(51) where E.sub.S denotes the energy that constrains smoothness of the deformation, M is the number of triangles in the given facial model, and adj(i) denotes a set of neighboring triangles around the i.sup.th triangle; E.sub.I denotes the energy that constrains the degree of the deformation, and I denotes a unit matrix; E.sub.C denotes the energy of a distance between vertices of the two models after deformation, n is the number of vertices in the given facial model, {circumflex over (v)}.sub.i is the position of the vertex of the given facial model after deformation, and c.sub.i is the position of the vertex, closest to v′_i, in the template facial model; E is the sum of the first three energy items, {circumflex over (v)}.sub.1 . . . {circumflex over (v)}.sub.n denotes the positions of n vertices in the given facial model after deformation, w.sub.S, w.sub.I, and w.sub.C are weights corresponding to E.sub.S, E.sub.I, and E.sub.C respectively, the energy equation is subject to m vertex correspondences given by the user, {circumflex over (v)}.sub.s.sub.k is the position of the vertex of the given facial model after deformation in the k.sup.th vertex correspondence, and m.sub.k is the target position of the vertex in the k.sup.th vertex correspondence.

(52) In the process of minimizing the above energy equation E, {{circumflex over (v)}.sub.i, c.sub.i} (i.e. the nearest vertex in the template facial model for each vertex of the given facial model after deformation) needs to be found. During optimization, the vertex position changes, and the relationship of the nearest vertex also changes. Therefore, it is necessary to iterate several steps in the process of finding the nearest vertex and minimizing the above energy equation E.

(53) After the given facial model is deformed to the state closest to the template facial model, the centroid of all the triangles in the template facial model and the given facial model after deformation is calculated. For each triangle in the given facial model after deformation, to find a reasonable corresponding triangle in the template facial model, a centroid distance needs to be less than a particular threshold (manually adjusted) and an angle between two normal vectors needs to be less than 90°. Similarly, for each triangle in the template facial model, a reasonable corresponding triangle is found in the given facial model after deformation. All reasonable correspondences constitute a triangle correspondence between the two models.

(54) 5.2 Migrate deformation gradients corresponding to the template facial model to the given facial model:

(55) The scaling/shearing parameter s and the rotation parameter r obtained in step (4) are restored to a transformation matrix set {T.sub.i}.sub.i=1.sup.N of all triangles of the template facial model (wherein N is the number of transformation matrices of the template facial model, which is equal to the number of the triangles). A transformation matrix set {T.sub.i}.sub.i=1.sup.M′ of the given facial model is constructed according to the triangle correspondence obtained in step (5.1) (wherein M′ is the number of transformation matrices of the given facial model; if a triangle k in the given facial model does not have a corresponding triangle in the template facial model, a unit matrix is taken as a transformation matrix of k; if it has a corresponding triangle, a transformation matrix of the corresponding triangle is taken as a transformation matrix of k; if it has multiple corresponding triangles, a number of copies are made for k, and each copy corresponds to one of them; since there is a case of having multiple corresponding triangles, the number of transformation matrices finally obtained is M′≥M).

(56) 5.3 Solve positions of vertices of the given facial model according to the migrated deformation gradients:

(57) By minimizing the following energy equation, the positions of vertices {tilde over (x)} of the given facial model is calculated under the deformation gradient corresponding to the migration:
E(c)=∥c−A{tilde over (x)}∥.sup.2  (6)

(58) where c is formed by stacking of {T.sub.i}.sub.i=1.sup.M′, and A is a large sparse matrix that associates c with {tilde over (x)}. By setting the gradient of the energy equation to 0, {tilde over (x)} can be solved by the following formula:
A.sup.TA{tilde over (x)}=A.sup.Tc  (7)

(59) Since A is only related to the given facial model, A and A.sup.TA can be pre-calculated, and each model requires only one precalculation.

(60) 6. Slide a signal window: repeating step (1) to step (5), and process all speech signal windows, to generate complete facial animation.

(61) On the whole input speech signal, a series of audio windows are acquired at an interval of

(62) $\frac{1}{\mathrm{fps}}$
seconds. Steps (1) to (5) are repeated for each window to generate complete animation at a frame rate of fps frames per second. The generation speed may reach real-time and the delay is

(63) $\frac{L_{\mathrm{audio}}}{2}$
(wherein L.sub.audio is the input audio window length described in step (1)).

Implementation Example

(64) Loss function: the inventor trains neural network parameters involved in steps (2) to (4) by using a supervised learning method. Speech and animation data are organized into a number of data pairs (x.sub.t, y.sub.t), wherein x.sub.t represents a speech signal window corresponding to data in the i.sup.th frame, and y.sub.t represents a corresponding deformation gradient parameter. According to the description in step (4), y.sub.t can be further divided into a scaling/shearing part s.sub.t∈.sup.6N and a rotation part r.sub.t∈.sup.3N. In the training process, the outputs in step (4) are marked as {tilde over (s)}.sub.t and {tilde over (r)}.sub.t. For the two parts of the parameters, the present disclosure uses similar energy items for constraints. Taking the scaling/shearing part as an example, the energy items include L.sub.d.sup.s considering an absolute value and L.sub.v.sup.s considering a numerical time derivative:
L.sub.d.sup.s=∥s.sub.t−{tilde over (s)}.sub.t∥.sup.2
L.sub.v.sup.s=∥(s.sub.t−s.sub.t−1)−({tilde over (s)}.sub.t−{tilde over (s)}.sub.t−1)∥.sup.2  (8)

(65) For the rotation part, the manner of defining L.sub.d.sup.r and L.sub.v.sup.r is similar to that of the above formula. The final loss function is the weighted sum of four energy terms, and weights are automatically dynamically balanced using the technologies proposed by Karras et al. (Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial animation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):94, 2017.).

(66) Training example: the inventor implements the example of the present disclosure on a computer equipped with an Intel Core i7-8700K central processing unit (3.70 GHz) and a NVIDIA GTX1080Ti graphics processor (11 GB). A database VOCASET (Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. Capture, learning, and synthesis of 3D speaking styles. Computer Vision and Pattern Recognition (CVPR), pages 10101-10111, 2019.) is used to train the model during the implementation.

(67) Model parameters: when the inventor implements the example of the present disclosure, the parameters involved in steps (1) to (6) are as follows:

(68) (1) Extract speech features: an audio window length L.sub.audio=0.568 seconds; a short-term Fourier transform frame length L.sub.fft=0.064 seconds, and a frame interval L.sub.hop=0.008 seconds; the number of Mel filters F.sub.mel=128; and the number of Mel spectrogram frames obtained L.sub.frame=64.

(69) (2) Collect frequency information: the numbers of convolution kernels (also the number of feature maps after convolution) are C.sub.freq_conv0=32, C.sub.freq_conv1=64, and C.sub.freq_conv2=64 respectively; the size of the convolution kernels of the first two layers is K.sub.freq=3, and the size of a pooling region is S.sub.freq=2; the sum of the numbers of feature maps of the bidirectional long short-term memory of the frequency dimension is C.sub.freq_LSTM=64 (i.e. the number of feature maps in a single direction is 32); and the number of feature maps for fully connected projection is C.sub.freq=256.

(70) (3) Summarize time information: the sum of the numbers of feature maps of the bidirectional long short-term memory of the time dimension is C.sub.time=512 (i.e. the number of feature maps in a single direction is 256); and in the attention module, K.sub.qry=3 and C.sub.att=128.

(71) (4) Decode action features: the number of triangles of the template facial model is N=9976; a speaker style control vector dimension is C.sub.speaker=8; the number of feature maps of the first fully connected layer is C.sub.dec0=512; the numbers of feature maps of the first two fully connected layers in each branch are C.sub.dec1=512, C.sub.dec2=256; the number of principal component analysis bases retained by the scaling/shearing parameter s (also the number of feature maps of the third fully connected layer in the scaling/shearing branch) is C.sub.pca_s=85, and the number of principal component analysis bases retained by the rotation parameter r (also the number of feature maps of the third fully connected layer in the rotation branch) is C.sub.pca_r=180.

(72) (5) Drive a facial model: M is determined specifically by given model parameters; in the process of iterating and optimizing the formula (5) in step (5.1), the first step is w.sub.S=1.0, w.sub.I=0.001, and w.sub.C=0, then the fourth step is iterated, and W.sub.C changes from 1 to 5000.

(73) (6) Slide a signal window: steps (1) to (5) are repeated: an animation frame rate is fps=60.

(74) Example time consumption: VOCASET's facial model is taken as a template facial model (composed of 9976 triangles), and the model is trained on VOCASET's data for 50 iterations, which takes about 5 hours. For the input speech signal, each window generates one frame of animation (from step (1) to (5), the template facial model is directly driven in step (5)), which takes about 10 milliseconds to achieve a real-time rate. For other given facial models whose topology is different from that of the template facial model, correspondences of model triangles should be set in advance according to step (5.1). It takes about 15 to 40 minutes according to the complexity of the model and the proficiency of the implementer. This work only needs to be done once for any model.

(75) Animation excerpt: the inventor implements the example of the present disclosure by using speech signals to drive facial animation. When VOCASET's facial model is used to generate speech animation, sequence excerpt frames thereof are as shown in FIG. 4 (in which a person is saying an English word “smash”). When a cartoon animal facial model whose topology is different from that of the template facial model is used to speech animation, sequence excerpt frames thereof are as shown in FIG. 5 (in which a cartoon animal is saying an English word “smash”).