AN EXAMPLE-BASED REAL-TIME CLOTHING SYNTHESIS METHOD

Abstract

The invention presents an example-based real-time clothing synthesis method, which includes the following steps: creating the 3D human model according to the input body pose parameters and the body shape parameters; generating 3D clothing deformation on the standard shape and the input pose through a clothing pose model; searching nearby clothing shape models according to the body pose parameters; for each nearby clothing shape model, generating corresponding 3D clothing mesh based on the body shape parameters, and then deforming the clothing mesh toward the input pose using Linear Blend Skinning (LBS) method; synthesizing the clothing deformation for the input body pose and shape using the clothing data obtained in above steps; resolving the interpenetrations between the clothing data and the human model. The invention can generate real-time clothing animation results with detailed wrinkles for various body poses and shapes with different clothing types on a commodity CPU, without resizing the cloth model. It has potential use in assessment of fitness in virtual try-on, as well as in the field of computer animation and video games.

Claims

1. An example-based real-time clothing synthesis method comprising the following steps: (1) creating a 3D human model according to input body pose parameters and body shape parameters; (2) according to the body pose parameters, generating 3D clothing deformation on a standard shape and an input pose through a clothing pose model; (3) according to the body pose parameters, searching nearby clothing shape models; (4) for each nearby clothing shape model in step (3), generating corresponding 3D clothing mesh based on the body shape parameters, and then deforming the clothing mesh toward the input pose using Linear Blend Skinning (LBS) method; (5) synthesizing a 3D clothing deformation for the input body pose and the body shape parameters using the clothing data obtained in step (2) and (4); and (6) resolving interpenetrations between the clothing data in step (5) and the 3D human model in step (1).

2. The example-based real-time clothing synthesis method according to claim 1, wherein the 3D human model for the input body pose and shape in step (1) is generated by using SMPL model.

3. The example-based real-time clothing synthesis method according to claim 1, wherein the clothing pose model in step (2) refers to a model that is able to generate 3D clothing data according to the input pose parameters.

4. The example-based real-time clothing synthesis method according to claim 1, wherein the clothing shape model in step (3) refers to a model that is able to generate 3D clothing data according to the input shape parameters.

5. The example-based real-time clothing synthesis method according to claim 4, wherein the clothing data for the input shape is generated as follows: (3-1) for a certain pose, generating several 3D human models for different shape parameters β using SMPL model, and then simulating clothing for these bodies; (3-2) converting the clothing coordinate data for the k-th body into a column vector, denoted as {right arrow over (d)}.sup.k; wherein the coordinate data for all clothing meshes are denoted as a matrix S=[{right arrow over (d)}.sup.1, {right arrow over (d)}.sup.2, . . . , {right arrow over (d)}.sup.k, . . . ]; (3-3) performing Principal Component Analysis (PCA) on matrix S, then {right arrow over (d)}.sup.k can be approximated as:
{right arrow over (d)}.sup.k=U{right arrow over (ϕ)}.sup.k+{right arrow over (u)}  (1), wherein {right arrow over (u)} is the mean coordinates of the clothing meshes, matrix U represents the first few principal components of the shape deformation space, and {right arrow over (ϕ)}.sup.k is the clothing shape parameters; (3-4) after step (3-3), each clothing {right arrow over (d)}.sup.k corresponding to a set of clothing shape parameters {right arrow over (ϕ)}.sup.k and a set of body shape parameters β.sup.k, wherein a mapping between β and ϕ is learned by using L2-regularized least squares with the weight of the regularized term being 0.2:
{right arrow over (ϕ)}′=W.Math.({right arrow over (β)},{right arrow over (β)}.sup.2,1).sup.T  (2) wherein {right arrow over (ϕ)}′ is the clothing shape parameters, {right arrow over (β)} is the body shape parameters, {right arrow over (β)}.sup.2 is the square of {right arrow over (β)}, T is the transpose matrix, and W is the unknown mapping matrix; using body shape parameters, calculating the corresponding clothing shape parameters by using equation (2), and then computing final clothing coordinate data using equation (1), which is the clothing shape model.

6. The example-based real-time clothing synthesis method according to claim 1, wherein the clothing data in step (5) are synthesized using the following equation: $\begin{array}{cc}f\left(\beta ,\theta \right)=f\left({\beta }_0,\theta \right)+\underset{s=1}{\overset{N_s}{.Math.}}{w}^s{\mathrm{LBS}}_{{\theta }_s}^{\theta }\left(f\left(\beta ,{\theta }_s\right)\right)-\underset{s=1}{\overset{N_s}{.Math.}}{w}^s{\mathrm{LBS}}_{{\theta }_s}^{\theta }\left(f\left({\beta }_0,{\theta }_s\right)\right)& \left(8\right)\end{array}$ wherein f(β, θ) is the clothing data for input shape β and input pose θ, which is a target clothing deformation, f(β.sub.0, θ) is the clothing data for the standard shape β.sub.0, and the input pose θ, which is generated by the clothing pose model, θ.sub.s is the pose of the s-th nearby clothing shape model (also known as the s-th example), f(β, θ.sub.s), f(β.sub.0, θ.sub.s) are the clothing deformations for the input shape β and the standard shape β.sub.0 respectively, predicted through the s-th clothing shape model, N.sub.s indicates the number of the nearby clothing shape models, LBS.sub.θ.sub.s.sup.θ deforms clothing mesh from pose θ.sub.s towards pose θ using Linear Blend Skinning method (LBS), and w.sup.s is the weight for the s-th example.

7. The example-based real-time clothing synthesis according to claim 1, wherein step (6) uses re-projection technique to resolve the interpenetrations in step (5).

8. The example-based real-time clothing synthesis according to claim 7, wherein the re-projection technique comprises the following steps: (6-1) when body shape is changed, each clothing shape model generating new clothing mesh for new body shape and recalculating the initial Euclidian distance (referred to as the initial clearance) between each clothing vertex and its closest body vertex; (6-2) in step (5), if the clearance between a clothing vertex and its closest body vertex is less than its initial clearance, for each nearby clothing shape model, re-projecting the clothing vertex towards the direction of the normal of its closest body vertex as:
ŷ.sup.s=y+d.sup.s.Math.{right arrow over (n)}
d.sup.s=max(0,d.sub.0.sup.s−h.sup.s)  (12) wherein ŷ.sup.s is the resulting clothing vertex, y is a clothing vertex obtained in step (5), {right arrow over (n)} is the normal of the closest body vertex of y; h.sup.s is the current clearance; setting d.sub.0.sup.s=min{h.sub.0.sup.s, ϵ.sub.p}, wherein 14 is the initial clearance for the s-th example, and E.sub.l, is to mimic the penetration depth margin in cloth simulation; (6-3) blending the re-projection results for all nearby examples using the following equation:
y=Σ.sub.s=1.sup.N.sup.sw.sup.sŷ.sup.s  (13) wherein y is the final position for clothing vertex y; N.sub.S indicates the number of nearby examples; w.sup.s is the weight for the s-th example; ŷ.sup.s and is the result obtained through equation (12).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] FIG. 1 is the overview of the example-based real-time clothing synthesis method of the present invention;

[0039] FIG. 2 is the illustration of the computation of the LBS synthesis process of the flow chart of the example-based virtual fitting method provided by the present invention..

SPECIFIC IMPLEMENTATION EMBODIMENTS OF THE INVENTION

[0040] The following describes the present invention in details with reference to the accompanying figures.

[0041] As shown in FIG. 1, the example-based real-time clothing synthesis method the present invention comprises the following steps:

[0042] (1) creating the 3D human model according to the input body pose parameters and the body shape parameters.

[0043] The 3D human model is generated using SMPL model (see Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. 2015. SMPL: A skinned multi-person linear model. ACM Trans. Graph.) 34, 6 (2015), 248.). The pose parameters for SMPL is Euler angles of joints and the shape parameters is a twelve-dimension vector.

[0044] (2) according to the body pose parameters, generating 3D clothing deformation on the standard shape and the input pose through a clothing pose model.

[0045] The clothing pose model refers to the model that is able to generate 3D clothing data according to the input pose parameters, e.g. SOR (see Weiwei Xu, Nobuyuki Umentani, Qianwen Chao, Jie Mao, Xiaogang Jin, and Xin Tong. 2014. Sensitivity-optimized rigging for example-based real-time clothing synthesis. ACM Trans. Graph. 33, 4 (2014), 107.). Standard shape refers to the body shape used by the clothing pose model, generated using SMPL model with shape parameters being zero vector. The clothing pose model only handle various body poses, and the body shape remains unchanged.

[0046] (3) according to the body pose parameters, searching nearby clothing shape models, as shown in FIG. 2.

[0047] The clothing shape model refers to the model that is able to generate 3D clothing data according to the input shape parameters while the pose remains fixed. A clothing shape model is built as follows:

[0048] (3-1) In offline database construction, for a certain pose, generating N.sub.k=17 3D human models for different shape parameters β using SMPL model, and then simulating clothing for these bodies. For each of the first 4 principal components of the body shape parameters b in the SMPL model, we generated 4 body shapes (β.sub.k=—2,—1,1,2), keeping the remaining parameters in β as 0. To these 16 body shapes, we added the nominal shape with β=0. For each generated body shape, we performed a one-second simulation to completely drape the clothing on the avatar.

[0049] (3-2) converting the clothing coordinate data for the k-th body into a column vector, denoted as {right arrow over (d)}.sup.k; the coordinate data for all clothing meshes are denoted as a matrix S=[{right arrow over (d)}.sup.1, {right arrow over (d)}.sup.2, . . . , {right arrow over (d)}.sup.k, . . . ];

[0050] (3-3) performing Principal Component Analysis (PCA) on matrix S, then {right arrow over (d)}.sup.k can be approximated as:

{right arrow over (d)}.sup.k=U{right arrow over (ϕ)}.sup.k+{right arrow over (u)}  (1)

wherein {right arrow over (u)} is the mean coordinates of the clothing meshes, matrix U represents the first few principal components of the shape deformation space, and {right arrow over (ϕ)}.sup.k is the clothing shape parameters.

[0051] (3-4) after step (3-3), each clothing {right arrow over (d)}.sup.k corresponded to a set of clothing shape parameters {right arrow over (ϕ)}.sup.k and a set of body shape parameters β.sup.k. A mapping between β and ϕ was learned using L2-regularized least squares with the weight of the regularized term being 0.2:

{right arrow over (ϕ)}′=W.Math.({right arrow over (β)},{right arrow over (β)}.sup.2,1).sup.T  (2)

wherein {right arrow over (ϕ)}′ is the clothing shape parameters, {right arrow over (β)} is the body shape parameters, {right arrow over (β)}.sup.2 is the square of {right arrow over (β)}, T is the transpose matrix, and W is the unknown mapping matrix.

[0052] Given body shape parameters, we calculated the corresponding clothing shape parameters using equation (2), and then we computed the final clothing coordinate data using equation (1). This is the so-called clothing shape model.

[0053] To generate our database (clothing shape models), we first sampled a sufficient number of poses from motion sequences representing the whole pose space. In our implementation, we selected a sufficient number of motion sequences from CMU motion capture library (CMU. 2003. CMU graphics lab motion capture database. http://mocap.cs.cmu.edu. (2003).). Then, we used weighted K-means to classify these poses into a certain number of clusters, which will be used to generate our database: clothing shape models. We used the sum of the norm of the sensitivity of a joint, s.sup.L, as its weight in the clustering process, calculated as:

s.sup.L=Σ.sub.y∈YΣ.sub.m=1.sup.3=∥s.sub.y,m∥  (3)

wherein s.sup.L denotes the sensitivity of joint L, y indicates clothing vertex, and m is the degree of freedom of the joint, and s.sub.y,m is calculated as:

[00008]$\begin{array}{cc}{\stackrel{\_}{s}}_{y,m}=\frac{1}{17}{.Math.}_{k=1}^{17}.Math.s_{y,m}^k.Math.,& \left(4\right)\end{array}$

wherein s.sub.y,m.sup.k indicates the coordinate differences of a clothing vertex y under a small joint rotation of m-th joint angle, calculated under the standard pose (T-pose: standing upright with your feet naturally open and your arms flat) and the k-th training shape, and 17 different body shapes is the same as in step (3-1).

[0054] To reduce the required clothing shape models in the database, we divided the clothing mesh into several regions in step (3). First, for each clothing vertex, we found its closest body vertex under the standard pose. The skinning weight w.sub.b.sup.0 for a clothing vertex is equal to that of its closest body vertex. Second, we partitioned the bones of the human model into N.sub.g=7 regions: calf and foot (left and right), femur (left and right), radius and hand (left and right), humerus (left and right), and the remaining part. In this way, our method synthesized the result for each region separately. For each region g, we computed the sensitivity-based distance D.sub.g.sup.s(θ) between the input pose θ and the pose of the s-th data point as the weighted sum of differences of joint angles:

[00009]$\begin{array}{cc}{D}_g^s\left(\theta \right)=\underset{yϵY}{.Math.}{w}_{g,y}\underset{m=1}{\overset{3N_L}{.Math.}}{.Math.{\stackrel{\_}{s}}_{y,m}.Math.{\Theta }_m\left(\theta ,{\theta }_s\right).Math.}^2=\underset{m=1}{\overset{3N_L}{.Math.}}{Q}_{g,m}{.Math.{\Theta }_m\left(\theta ,{\theta }_s\right).Math.}^2,& \left(5\right)\end{array}$

wherein w.sub.g,y is the region weight for a clothing vertex y which is computed by summing the bone weights w.sub.b.sup.0 for the bones of the current region g, N.sub.L indicates the number of joints, s.sub.y,m is the same as equation (3), and Θ.sub.m(θ,θ.sub.s) calculates the m-th joint angle difference, and Q.sub.g,m reflects the influence of the m-th joint angle on region g which is calculated as:

[00010]$\begin{array}{cc}{Q}_{g,m}=\underset{yϵY}{.Math.}{w}_{g,y}{.Math.{\stackrel{\_}{s}}_{y,m}.Math.}^2.Math.& \left(6\right)\end{array}$

Each time we changed the body shape, we updated Q.sub.g,m for each region once according to the new region weights of clothing vertices. In runtime, our method found several nearby clothing shape models for each region.

[0055] (4) for each nearby clothing shape model in step (3), generating corresponding 3D clothing mesh based on the body shape parameters, and then deforming the clothing mesh toward the input pose using Linear Blend Skinning (LBS) method.

[0056] For each clothing shape model, we first calculated the clothing instances for the input shape and the standard shape, denoted as f(β,θ.sub.s) and f(β.sub.0,θ.sub.s) (the clothing deformation is regarded as a function of body shape parameters β and body pose parameters θ). Then we deformed f(β,θ.sub.s) and f(β.sub.0, θ.sub.s) towards the input pose θ using Linear Blend Skinning (LBS) method. Taking the clothing instance f(β, θ.sub.s) for the s-th clothing shape model as an example, we first found the closest body vertex for each clothing vertex of f(β, θ.sub.s), and set the bone weight w.sub.b.sup.s of a clothing vertex to that of its closest body vertex. Then, we deformed f(β, θ.sub.s) towards the input pose θ as:

[00011]$\begin{array}{cc}{\stackrel{\_}{y}}^s=\underset{b=1}{\overset{N_b}{.Math.}}{w}_b^s\left(R_b^{{\theta }_{s,\theta }}{y}^s+T_b^{{\theta }_{s,\theta }}\right),& \left(7\right)\end{array}$

wherein R.sub.b.sup.θ.sup.s.sup.,θ and T.sub.b.sup.θ.sup.s.sup.,θ are the relative rotation and translation of bone b from example pose θ.sub.s to input pose θ, respectively, and w.sub.b.sup.s is the bone weight defined on the clothing vertex y.sup.s of f(β, θ.sub.s).

[0057] We denoted equation (7) as LBS.sub.θ.sub.s.sup.θ(f(β,θ.sub.s)), and the LBS deformation for clothing instance under the standard shape f(β.sub.0, θ.sub.s) is basically the same as the above process except for the body shape. We denoted its result as LBS.sub.θ.sub.s.sup.θ(f(β.sub.0,θ.sub.s)).

[0058] (5) synthesizing the clothing deformation for the input body pose and shape using the clothing data obtained in step (2) and (4);

[0059] The synthesizing equation is:

[00012]$\begin{array}{cc}f\left(\beta ,\theta \right)=f\left({\beta }_0,\theta \right)+\underset{s=1}{\overset{N_s}{.Math.}}{w}^s{\mathrm{LBS}}_{{\theta }_s}^{\theta }\left(f\left(\beta ,{\theta }_s\right)\right)-\underset{s=1}{\overset{N_s}{.Math.}}{w}^s{\mathrm{LBS}}_{{\theta }_s}^{\theta }\left(f\left({\beta }_0,{\theta }_s\right)\right),& \left(8\right)\end{array}$

wherein f(β, θ) is the clothing data for the input shape β and the input pose θ, i.e. the target clothing deformation, f(β.sub.0, θ) is the clothing data for the standard shape β.sub.0, and the input pose θ, which is generated by the clothing pose model, θ.sub.s is the pose of the s-th nearby clothing shape model (also known as the s-th example), f(β, θ.sub.s), f(β.sub.0,θ.sub.s) are the clothing deformations for the input shape β and the standard shape β.sub.0, respectively, predicted through the s-th clothing shape model, N.sub.s indicates the number of the nearby clothing shape models, LBS.sub.θ.sub.s.sup.θ deforms clothing mesh from pose θ.sub.s towards pose θ using Linear Blend Skinning method (LBS), Σ.sub.s=1.sup.N.sup.s w.sup.sLBS.sub.θ.sub.s.sup.θ(f(β,θ.sub.s)) and Σ.sub.s=1.sup.N.sup.sw.sup.sLBS.sub.θ.sub.s.sup.θ(f(β.sub.0, θ.sub.s)) are the LBS synthesis, and w.sup.s is the weight for the s-th example, which is calculated as (for simplicity, we drop the vertex index for all variables related to clothing mesh vertices):

[00013]$\begin{array}{cc}{w}^s=\underset{g=1}{\overset{N_G}{.Math.}}\left(w_g{W}_g^s\left(\theta \right)/\underset{s=1}{\overset{N_s}{.Math.}}{W}_g^s\left(\theta \right)\right),& \left(9\right)\end{array}$

wherein w.sub.g is the region weight of a clothing vertex for region g, and W.sub.g.sup.s(θ) is calculated as:

W.sub.g.sup.s(θ)=1/(D.sub.g.sup.s(θ)+ϵ).sup.k  (10)

wherein ϵ is a small number in case of zero division, and k regulates the influence of closer examples. Small k tends to smooth the animation and lose fine wrinkles, while large k tends to preserve fine details but results in discontinuity.

[0060] Equation (8) is the blend of results of nearby examples, which can also be taken as the blend of approximation results of multiple Taylor expansions.

[0061] To prevent sudden changes of the synthesized clothing for sudden input pose changes, we blended the distance at the current time step D; with that of the previous time step D′.sub.g.sup.s as:

D.sub.g.sup.s=ηD′.sub.g.sup.s+(1−η)D.sub.g.sup.s  (11)

wherein η is the damping ratio. D.sub.g.sup.s and D′.sub.g.sup.s are the distances of the current and the previous moment, respectively.

[0062] (6) resolving the interpenetrations between the clothing data in step (5) and the human model in step (1) using the re-projection technique which includes the following steps:

[0063] First, when change body shape, each clothing shape model will generate new clothing mesh for new body shape and recalculate the initial Euclidian distance (referred to as the initial clearance) between each clothing vertex and its closest body vertex.

[0064] Second, in step (5) if the clearance between a clothing vertex and its closest body vertex is less than its initial clearance, for each nearby clothing shape model, we re-project the clothing vertex towards the direction of the normal of its closest body vertex as:

ŷ.sup.s=y+d.sup.s.Math.{right arrow over (n)}

d.sup.s=max(0,d.sub.0.sup.s−h.sup.s)  (12)

wherein ŷ.sup.s is the resulting clothing vertex, y is a clothing vertex obtained in step (5), {right arrow over (n)} is the normal of the closest body vertex of y; h.sup.s is the current clearance. We set d.sub.0.sup.s=min{h.sub.0.sup.s, ϵ.sub.p}, wherein h.sub.0.sup.s is the initial clearance for the s-th example, and ϵ.sub.p is to mimic the penetration depth margin in cloth simulation (in our implementation, we empirically set ϵ.sub.p=5 mm).

[0065] Finally, blending the re-projection results for all nearby examples using the following equation:

[00014]$\begin{array}{cc}\stackrel{\_}{y}=\underset{s=1}{\overset{N_s}{.Math.}}{w}^s{\hat{y}}^s,& \left(13\right)\end{array}$

wherein y is the final position for clothing vertex y; N.sub.s indicates the number of nearby examples; w.sup.s is the weight for the s-th example; ŷ.sup.s and is the result obtained through equation (12).