METHOD FOR GENERATING HUMAN-COMPUTER INTERACTIVE ABSTRACT IMAGE

Abstract

A method for generating a human-computer interactive abstract image includes: S1: obtaining and preprocessing the original abstract images used as a training dataset B to obtain edge shape feature maps used as a training dataset A; S2: using the training dataset A and the training dataset B as cycle generative objects of a Cycle-GAN model, and training the Cycle-GAN model to capture a mapping relationship between the edge shape feature maps and the original abstract images; S3: obtaining a line shape image drawn by a user; and S4: according to the mapping relationship, intercepting a generative part in the Cycle-GAN model that the dataset B is generated from the dataset A, discarding a cycle generative part and a discrimination part in the Cycle-GAN model, and generating a complete abstract image based on the line shape image to generate the human-computer interactive abstract image.

Claims

1. A method for generating a human-computer interactive abstract image, comprising: S1: obtaining original abstract images, and preprocessing the original abstract images to obtain edge shape feature maps in one-to-one correspondence with the original abstract images; wherein the edge shape feature maps are used as a training dataset A, and the original abstract images are used as a training dataset B; S2: using the training dataset A and the training dataset B as cycle generative objects of a Cycle-generative adversarial network (GAN) model, and training the Cycle-GAN model to capture a mapping relationship between the edge shape feature maps and the original abstract images; S3: obtaining a line shape image drawn by a user; and S4: according to the mapping relationship, intercepting a generative part in the Cycle-GAN model that the dataset B is generated from the dataset A, discarding a cycle generative part and a discrimination part in the Cycle-GAN model, and generating a complete abstract image based on the line shape image to generate the human-computer interactive abstract image.

2. The method according to claim 1, wherein step S1 comprises: S101: obtaining the original abstract images, and using the original abstract images to construct the training dataset B; S102: performing a binarization processing on the original abstract images in the training dataset B to obtain binarized images, and extracting color edge information in the binarized images to obtain the edge shape feature maps in one-to-one correspondence with the original abstract images; and S103: calculating lengths of edge lines of the edge shape feature maps, and discarding edge lines with a length being greater than 150 pixels to obtain the training dataset A.

3. The method according to claim 1, wherein the Cycle-GAN model in step S2 comprises a first generator G, a second generator F, a first discriminator D.sub.G and a second discriminator D.sub.F; wherein the first generator G and the second generator F are identical structurally, and the first discriminator D.sub.G and the second discriminator D.sub.F are identical structurally; the first generator G is configured to capture the mapping relationship between the edge shape feature maps and the original abstract images; the second generator F is configured to capture an inverse mapping relationship between the edge shape feature maps and the original abstract images; the first discriminator D.sub.G is configured to discriminate a generative quality of the first generator G; and the second discriminator D.sub.F is configured to discriminate a generative quality of the second generator F.

4. The method according to claim 3, wherein each of the first discriminator D.sub.G and the second discriminator D.sub.F comprises a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer, wherein the first convolutional layer, the second convolutional layer, the third convolutional layer, the fourth convolutional layer and the fifth convolutional layer are successively connected; each of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer is provided with a first normalization operation and a rectified linear unit (ReLU) activation function; the fifth convolutional layer is provided with a Sigmoid function; and each of the first generator G and the second generator F comprises an encoding module, a residual module and a decoding module, wherein the encoding module, the residual module and the decoding module are successively connected.

5. The method according to claim 4, wherein a number of convolutional kernels of the first convolutional layer is 64, a size of the convolutional kernels of the first convolutional layer is 4×4, and a stride of the first convolutional layer is 2; a number of convolutional kernels of the second convolutional layer is 128, a size of the convolutional kernels of the second convolutional layer is 4×4, and a stride of the second convolutional layer is 2; a number of convolutional kernels of the third convolutional layer is 256, a size of the convolutional kernels of the third convolutional layer is 4×4, and a stride of the third convolutional layer is 2; a number of convolutional kernels of the fourth convolutional layer is 512, a size of the convolutional kernels of the fourth convolutional layer is 4×4, and a stride of the fourth convolutional layer is 2; and a number of convolutional kernel of the fifth convolutional layer is 1, a size of the convolutional kernel of the fifth convolutional layer is 4×4, and a stride of the fifth convolutional layer is 1.

6. The method according to claim 5, wherein the encoding module comprises a sixth convolutional layer, a seventh convolutional layer and an eighth convolutional layer, wherein the sixth convolutional layer, the seventh convolutional layer and the eighth convolutional layer are successively connected; each of the sixth convolutional layer, the seventh convolutional layer and the eighth convolutional layer is provided with a second normalization operation and the ReLU activation function; the residual module comprises a first residual layer, a second residual layer, a third residual layer, a fourth residual layer, a fifth residual layer and a sixth residual layer, wherein the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer and the sixth residual layer are successively connected; each of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer and the sixth residual layer is provided with the second normalization operation and the ReLU activation function; the decoding module comprises a first decoding layer, a second decoding layer and a third decoding layer, wherein the first decoding layer, the second decoding layer and the third decoding layer are successively connected; each of the first decoding layer and the second decoding layer is provided with the second normalization operation and the ReLU activation function; the third decoding layer is provided with a Tanh function; and the eighth convolutional layer is connected to the first residual layer, and the sixth residual layer is connected to the first decoding layer.

7. The method according to claim 6, wherein a number of convolutional kernels of the sixth convolutional layer is 32, a size of the convolutional kernels of the sixth convolutional layer is 7×7, and a stride of the sixth convolutional layer is 1; a number of convolutional kernels of the seventh convolutional layer is 64, a size of the convolutional kernels of the seventh convolutional layer is 3×3, and a stride of the seventh convolutional layer is 2; a number of convolutional kernels of the eighth convolutional layer is 128, a size of the convolutional kernels of the eighth convolutional layer is 3×3, and a stride of the eighth convolutional layer is 2; each of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer and the sixth residual layer comprises two convolutional layers; a number of convolutional kernels of each of the two convolutional layers is 128, a size of the convolutional kernels of each of the two convolutional layers is 3×3, and a stride of each of the two convolutional layers is 1; a number of convolutional kernels of the first decoding layer is 64, a size of the convolutional kernels of the first decoding layer is 3×3, and a stride of the first decoding layer is 2; a number of convolutional kernels of the second decoding layer is 32, a size of the convolutional kernels of the second decoding layer is 3×3, and a stride of the second decoding layer is 2; and a number of convolutional kernels of the third decoding layer is 3, a size of the convolutional kernels of the third decoding layer is 7×7, and a stride of the third decoding layer is 1.

8. The method according to claim 7, wherein step S2 comprises: S201: randomly selecting a first shape line image x from the training dataset A as a first input of the first generator G, and obtaining a first complete abstract image ŷ corresponding to the first shape line image x by the first generator G; S202: using a real abstract image y in the training dataset B as a first positive sample, using the first complete abstract image ŷ as a first negative sample, and inputting the first positive sample and the first negative sample into the first discriminator D.sub.G to obtain an adversarial loss value of the first discriminator D.sub.G; S203: using the first complete abstract image ŷ as a first input of the second generator F, obtaining a line shape image {circumflex over (x)} corresponding to the first complete abstract image ŷ by the second generator F, and calculating a first cycle loss value according to the line shape image {circumflex over (x)} and the first shape line image x; S204: randomly selecting the real abstract image y from the training dataset B as a second input of the second generator F, and obtaining a second shape line image {circumflex over (x)} corresponding to the real abstract image y by the second generator F; S205: using the first shape line image x in the training dataset A as a second positive sample, using the second shape line image {circumflex over (x)} obtained in step S204 as a second negative sample, and inputting the second positive sample and the second negative sample into the second discriminator D.sub.F to obtain an adversarial loss value of the second discriminator D.sub.F; S206: using the second shape line image {circumflex over (x)} obtained in step S204 as a second input of the first generator G, obtaining a second complete abstract image ŷ by the first generator G, and calculating a second cycle loss value according to the second complete abstract image ŷ and the real abstract image y; and S207: minimizing the adversarial loss value of the first discriminator D.sub.G, the adversarial loss value of the second discriminator D.sub.F, the first cycle loss value and the second cycle loss value by using an optimizer to complete training the Cycle-GAN model to capture the mapping relationship between the edge shape feature maps and the original abstract images.

9. The method according to claim 8, wherein the adversarial loss value of the first discriminator D.sub.G is expressed as follows: $L_{GAN}\left(G,D_G\right)=\underset{\Theta G}{\min }\underset{\Theta D_G}{\max }\left\{E_y\left[(\log D_G\left(y\right)\right]+E_x\left[\log (1-D_G\left(G\left(x\right)\right)\right]\right\},$ wherein, L.sub.GAN(G,D.sub.G) represents the adversarial loss value of the first discriminator D.sub.G; ΘG represents a parameter of the first generator G; ΘD.sub.G represents a parameter of the first discriminator D.sub.G; D.sub.G(y) represents a first output obtained by the first discriminator D.sub.G on the real abstract image y; E.sub.y represents an average over all real abstract images y; G(x) represents an output of the first generator G for the first shape line image x; D.sub.G(G(x)) represents a second output obtained by the first discriminator D.sub.G on a generated sample G(x); and E.sub.x represents an average over all first shape line images x; the adversarial loss value of the second discriminator D.sub.F is expressed as follows: $L_{GAN}\left(F,D_F\right)=\underset{\Theta F}{\min }\underset{\Theta D_F}{\max }\left\{E_x\left[(\log D_F\left(x\right)\right]+E_y\left[\log (1-D_F\left(F\left(y\right)\right)\right]\right\},$ wherein, L.sub.GAN(F,D.sub.F) represents the adversarial loss value of the second discriminator D.sub.F; ΘF represents a parameter of the second generator F; ΘD.sub.F represents a parameter of the second discriminator D.sub.F; D.sub.F(y) represents a third output obtained by the second discriminator D.sub.F on the real abstract image y; E.sub.y represents the average over all real abstract images y; F(x) represents an output of the second generator F for the first shape line image x; D.sub.F(F(x)) represents a fourth output obtained by the second discriminator D.sub.F on the generated sample G(x); and E represents the average over all first shape line images x.

10. The method according to claim 8, wherein a cycle loss function of the first generator G and the second generator F in step S205 is expressed as follows:
L.sub.cyc(D,F)=∥F(G(x))−x∥.sub.1+G(F(y))−y∥.sub.1, wherein, L.sub.cyc(D,F) represents the cycle loss function of the first generator G and the second generator F; F(G(x)) represents the line shape image corresponding to the complete abstract image obtained by the second generator F; x represents the first shape line image in the training dataset A; G(F(y)) represents the second complete abstract image obtained by the first generator G; y represents the real abstract image in the training dataset B.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0058] FIG. 1 is a schematic diagram of a traditional approach in the prior art.

[0059] FIG. 2 is a schematic diagram of edge features of the abstract image shape component extraction in an embodiment.

[0060] FIG. 3 is a flow chart of the method of the present invention.

[0061] FIG. 4 is a schematic diagram of an overall structure of a Cycle-GAN according to the present invention.

[0062] FIG. 5 is a schematic diagram of examples generated according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] The specific embodiments of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be noted that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention defined and discriminated by the appended claims, these changes are obvious, and all solutions that are made taking advantage of the present invention shall fall within the scope of protection of the present invention.

Embodiment

[0064] The present invention provides a method for generating a human-computer interactive abstract image based on an edge feature extracting operator and a Cycle-GAN. The present invention allows users to first observe an objective thing and draw a simple line shape that is related to the objective thing but is created subjectively. The present invention receives the shape lines drawn by users, and, on this basis, performs recreation on the shape and complete AI creation on colors, and finally outputs an artwork of an abstract image that contains a shape, a color and an idea capable of reflecting the users' subjective thinking. As shown in FIG. 2, different from the traditional approach, the present invention realizes the separation of the shape and color components of the abstract images. Besides, the key point enabling the generated artwork to reflect the users' subjective thinking in the human-computer interactive creation is to extract shape features of the artwork from the data of the abstract image by the edge feature extracting operator, thereby establishing a matching relationship between the shape and the color of the abstract image. As shown in FIG. 3, in an embodiment, the method provided in the present invention includes the following steps.

[0065] S1: original abstract images are obtained, and the original abstract images are preprocessed to obtain edge shape feature maps in one-to-one correspondence with the original abstract images. The edge shape feature maps are used as the training dataset A, and the original abstract images are used as the training dataset B.

[0066] In an embodiment, step S1 includes:

[0067] S101: the original abstract images are obtained, and the original abstract images are used to construct the dataset B;

[0068] S102: binarization processing is performed on the original abstract images in the dataset B, and color edge information in binarized images is extracted to obtain the edge shape feature maps in one-to-one correspondence with the original abstract images; and

[0069] S103: lengths of the edge shape feature maps are calculated, and edge lines with a length being greater than 150 pixels are discarded to obtain the dataset A.

[0070] In an embodiment, the present invention selects abstract images from WiKiArt and other related websites. A total of 4415 images are used to construct the dataset B of the original abstract images for training. The OpenCV2 image processing tool is used to process each of the original abstract images in the dataset B. First, binarization processing is performed on the images using a threshold function, where a binarization threshold is an average of colors of a current image. Then, the color edge information in the binarized images is extracted by using a findContours function. However, since not all changes in the colors indicate meaningful shape structures, there is a need to limit the length of extracted edge information. The present invention calculates the length of each edge line by using an arcLength function, and sets a length discarding threshold to be 150 pixels, so as to obtain the training dataset A with the same amount of elements as the dataset B.

[0071] S2: the training dataset A and the training dataset B are used as cycle generative objects of a Cycle-GAN model, and the Cycle-GAN model is trained to capture a mapping relationship between the edge shape feature maps and the original abstract images.

[0072] In an embodiment, step S2 includes:

[0073] S201: a shape line image x is randomly selected from the dataset A as an input of the first generator G, and a complete abstract image ŷ corresponding to the shape line image is obtained by the first generator G;

[0074] S202: a real abstract image y in the dataset B is used as a positive sample, the complete abstract image ŷ is used as a negative sample, and the positive sample and the negative sample are input into the first discriminator D.sub.G to obtain an adversarial loss value of the first discriminator D.sub.G;

[0075] S203: the complete abstract image ŷ is used as an input of the second generator F, a line shape image {circumflex over (x)} corresponding to the complete abstract image is obtained by the second generator F, and a first cycle loss value is calculated according to the line shape image {circumflex over (x)} and the shape line image x;

[0076] S204: the real abstract image y is randomly selected from the dataset B as an input of the second generator F, and a shape line image {circumflex over (x)} corresponding to the real abstract image y is obtained by the second generator F;

[0077] S205: the shape line image x in the dataset A is used as a positive sample, the shape line image {circumflex over (x)} obtained in step S204 is used as a negative sample, and the positive sample and the negative sample are input into the second discriminator D.sub.F to obtain an adversarial loss value of the second discriminator D.sub.F;

[0078] S206: the shape line image {circumflex over (x)} obtained in step S204 is used as an input of the first generator G, a complete abstract image ŷ is obtained by the first generator G, and a second cycle loss value is calculated according to the complete abstract image ŷ and the real abstract image y; and

[0079] S207: the adversarial loss value of the first discriminator D.sub.G, the adversarial loss value of the second discriminator D.sub.F, the first cycle loss value and the second cycle loss value are minimized by using an optimizer to complete training the Cycle-GAN model to capture the mapping relationship between the edge shape feature maps and the original abstract images.

[0080] In an embodiment, the adversarial loss value of the first discriminator D.sub.G is expressed as follows:

[00003]$L_{GAN}\left(G,D_G\right)=\underset{\Theta G}{\min }\underset{\Theta D_G}{\max }\left\{E_y\left[(\log D_G\left(y\right)\right]+E_x\left[\log (1-D_G\left(G\left(x\right)\right)\right]\right\},$

[0081] where, L.sub.GAN (G,D.sub.G) represents the adversarial loss value of the first discriminator D.sub.G; ΘG represents a parameter of the first generator G; ΘD.sub.G represents a parameter of the first discriminator D.sub.G; D.sub.G(y) represents an output obtained by the first discriminator D.sub.G on the real abstract image y; E.sub.y represents an average over all real abstract images y; G(x) represents an output of the first generator G for the shape line image x; D.sub.G(G(x)) represents an output obtained by the first discriminator D.sub.G on a generated sample G(x); and E.sub.x represents an average over all shape line images x.

[0082] The adversarial loss value of the second discriminator D.sub.F is expressed as follows:

[00004]$L_{GAN}\left(F,D_F\right)=\underset{\Theta F}{\min }\underset{\Theta D_F}{\max }\left\{E_x\left[(\log D_F\left(x\right)\right]+E_y\left[\log (1-D_F\left(F\left(y\right)\right)\right]\right\},$

[0083] where, L.sub.GAN(F,D.sub.F) represents the adversarial loss value of the second discriminator D.sub.F; ΘF represents a parameter of the second generator F; ΘD.sub.F represents a parameter of the second discriminator D.sub.F; D.sub.F(y) represents an output obtained by the second discriminator D.sub.F on the real abstract image y; E.sub.y represents the average over all real abstract images y; F(x) represents an output of the second generator F for the shape line image x; D.sub.F(F(x)) represents an output obtained by the second discriminator D.sub.F on the generated sample G(x); and E.sub.x represents the average over all shape line images x.

[0084] In an embodiment, with respect to a traditional adversarial loss function, the Cycle-GAN model further introduces a cycle loss function to ensure the cycle consistency of generated images. The cycle loss function of the first generator G and the second generator F is expressed as follows:

L.sub.cyc(D,F)=∥F(G(x))−x∥.sub.1+∥G(F(y))−y∥.sub.1,

[0085] where, L.sub.cyc(D,F) represents the cycle loss function of the first generator G and the second generator F; F(G(x)) represents the line shape image corresponding to the complete abstract image obtained by the second generator F; x represents the shape line image in the dataset A; G(F(y)) represents the complete abstract image obtained by the first generator G; y represents the real abstract image in the dataset B.

[0086] In an embodiment, the cycle loss function ensures that the inverse mapping F can map the complete abstract image generated by the mapping G back to the structural information, and also ensures that the mapping G can map the structural information generated by the inverse mapping F back to the real abstract image, so as to further ensure the reality of the complete abstract image generated by the mapping G.

[0087] A loss function of the entire training process is:

L=L.sub.GAN(G,D.sub.G)+L.sub.GAN(F,D.sub.F)+L.sub.cyc(G,F)

[0088] In an embodiment, the present invention uses the Cycle-GAN model to learn the mutual mapping relationship between the dataset A and the dataset B, namely learning the mapping G: A.fwdarw.B from a simple structure to a complete abstract image and the inverse mapping F: B.fwdarw.A from the complete abstract image to the simple structure. Although the present invention does not use the inverse mapping F to generate an abstract image, the inverse mapping F provides a cycle consistency for the Cycle-GAN.

[0089] In an embodiment, the mapping G and the inverse mapping F are generators in the Cycle-GAN. The generators (namely the first generator G and the second generator F) are provided with discriminators to supervise the generative qualities of the generators, which are the first discriminator D.sub.G and the second discriminator D.sub.F, respectively. Each of the discriminators is a 70×70 PatchGAN discriminator. The overall structure of the Cycle-GAN is shown in FIG. 4. In the Cycle-GAN, the first generator G and the second generator F are codecs and have an identical structure. The details are as follows:

[0090] The Cycle-GAN model includes a first generator G and a second generator F having an identical structure, and a first discriminator D.sub.G and a second discriminator D.sub.F having an identical structure. The first generator G is configured to capture a mapping relationship between the edge shape feature maps and the original abstract images. The second generator F is configured to capture an inverse mapping relationship between the edge shape feature maps and the original abstract images. The first discriminator D.sub.G is configured to discriminate a generative quality of the first generator G. The second discriminator D.sub.F is configured to discriminate a generative quality of the second generator F. Each of the first discriminator D.sub.G and the second discriminator D.sub.F includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer and a fifth convolutional layer, which are successively connected. Each of the first convolutional layer, the second convolutional layer, the third convolutional layer and the fourth convolutional layer is provided with a normalization operation and a rectified linear unit (ReLU) function. The fifth convolutional layer is provided with a Sigmoid function. Each of the first generator G and the second generator F includes an encoding module, a residual module and a decoding module, which are successively connected. As shown in Table 1, Table 1 is a table of the structure of the first discriminator D.sub.G and the second discriminator D.sub.F. The number of convolutional kernels of the first convolutional layer is 64, the size of the convolutional kernels of the first convolutional layer is 4×4, and the stride of the first convolutional layer is 2. The number of convolutional kernels of the second convolutional layer is 128, the size of the convolutional kernels of the second convolutional layer is 4×4, and the stride of the second convolutional layer is 2. The number of convolutional kernels of the third convolutional layer is 256, the size of the convolutional kernels of the third convolutional layer is 4×4, and the stride of the third convolutional layer is 2. The number of convolutional kernels of the fourth convolutional layer is 512, the size of the convolutional kernels of the fourth convolutional layer is 4×4, and the stride of the fourth convolutional layer is 2. The number of convolutional kernel of the fifth convolutional layer is 1, the size of the convolutional kernel of the fifth convolutional layer is 4×4, and the stride of the fifth convolutional layer is 1. As shown in Table 2, Table 2 is a table of the structure of the first generator G and the second generator F. The encoding module includes a sixth convolutional layer, a seventh convolutional layer and an eighth convolutional layer, which are successively connected. Each of the sixth convolutional layer, the seventh convolutional layer and the eighth convolutional layer is provided with a normalization operation and a ReLU activation function. The residual module includes a first residual layer, a second residual layer, a third residual layer, a fourth residual layer, a fifth residual layer and a sixth residual layer, which are successively connected. Each of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer and the sixth residual layer is provided with a normalization operation and a ReLU activation function. The decoding module includes a first decoding layer, a second decoding layer and a third decoding layer, which are successively connected. Each of the first decoding layer and the second decoding layer is provided with a normalization layer and a ReLU activation function. The third decoding layer is provided with a Tanh function. The eighth convolutional layer is connected to the first residual layer, and the sixth residual layer is connected to the first decoding layer. The number of convolutional kernels of the sixth convolutional layer is 32, the size of the convolutional kernels of the sixth convolutional layer is 7×7, and the stride of the sixth convolutional layer is 1. The number of convolutional kernels of the seventh convolutional layer is 64, the size of the convolutional kernels of the seventh convolutional layer is 3×3, and the stride of the seventh convolutional layer is 2. The number of convolutional kernels of the eighth convolutional layer is 128, the size of the convolutional kernels of the eighth convolutional layer is 3×3, and the stride of the eighth convolutional layer is 2. Each of the first residual layer, the second residual layer, the third residual layer, the fourth residual layer, the fifth residual layer and the sixth residual layer includes two convolutional layers. The number of convolutional kernels of each of the two convolutional layers is 128, the size of the convolutional kernels of each of the two convolutional layers is 3×3, and the stride of each of the two convolutional layers is 1. The number of convolutional kernels of the first decoding layer is 64, the size of the convolutional kernels of the first decoding layer is 3×3, and the stride of the first decoding layer is 2. The number of convolutional kernels of the second decoding layer is 32, the size of the convolutional kernels of the second decoding layer is 3×3, and the stride of the second decoding layer is 2. The number of convolutional kernels of the third decoding layer is 3, the size of the convolutional kernels of the third decoding layer is 7×7, and the stride of the third decoding layer is 1.

TABLE-US-00001 TABLE 1 Number of Size of Name of Convo- Convo- Output Convolutional lutional lutional Normalization Activation Layer Kernels Kernels Stride Operation Function first 64 4 × 4 2 batch ReLU convolutional normalization layer second 128 4 × 4 2 batch ReLU convolutional normalization layer third 256 4 × 4 2 batch ReLU convolutional normalization layer fourth 512 4 × 4 2 batch ReLU convolutional normalization layer fifth 1 4 × 4 1 Sigmoid convolutional layer

TABLE-US-00002 TABLE 2 Number Size of Output Activ- Compo- of Convo- Convo- Normal- ation Module nent lutional lutional ization Func- Name Name Kernels Kernels Stride Operation tion encoding sixth 32 7 × 7 1 single ReLU layer convo- instance lutional normal- layer ization seventh 64 3 × 3 2 single ReLU convo- instance lutional normal- layer ization eighth 128 3 × 3 2 single ReLU convo- instance lutional normal- layer ization residual first 128 3 × 3 1 single ReLU module residual instance layer normal- ization 128 3 × 3 1 single ReLU instance standard- ization second 128 3 × 3 1 single ReLU residual instance layer normal- ization 128 3 × 3 1 single ReLU instance normal- ization third 128 3 × 3 1 single ReLU residual instance layer normal- ization 128 3 × 3 1 single ReLU instance normal- ization fourth 128 3 × 3 1 single ReLU residual instance layer normal- ization 128 3 × 3 1 single ReLU instance normal- ization fifth 128 3 × 3 1 single ReLU residual instance layer normal- ization 128 3 × 3 1 single ReLU instance normal- ization sixth 128 3 × 3 1 single ReLU residual instance layer normal- ization 128 3 × 3 1 single ReLU instance normal- ization decoding first 64 3 × 3 2 single ReLU module decoding instance layer normal- ization second 32 3 × 3 2 single ReLU decoding instance layer normal- ization third 3 7 × 7 1 Tanh decoding layer

[0091] S3: a line shape image drawn by a user is obtained.

[0092] In an embodiment, drawing panels are provided for users to receive users' actions of drawing lines to then form line shape images. There are various types of such drawing panels. A simple manner is that users use drawing tools of tablet computers to draw image files.

[0093] S4: according to the mapping relationship, a generative part in the Cycle-GAN model that the dataset B is generated from the dataset A is intercepted, a cycle generative part and a discrimination part in the Cycle-GAN model are discarded, and a complete abstract image is generated based on the line shape image to generate the human-computer interactive abstract image.

[0094] In an embodiment, assuming a generation process a.fwdarw.b.fwdarw.a′, where a represents a real shape line image, b represents a generated complete abstract image, and a′ represents a generated shape line image, the generative part refers to the part a.fwdarw.b, the cycle generative part refers to the part b.fwdarw.a′, and the discrimination part refers to discriminating, by a discriminator, whether b′ is generated. During the training process, all the three parts are needed; while after the training, only the part a.fwdarw.b is needed.

[0095] In an embodiment, the first generator G and the second generator F are obtained. The present invention involves a process of generating complete abstract images from shape line images, thus the first generator G is selected as the final generative model, receive users' input images at the same time, and outputs corresponding generated abstract images.

[0096] In an embodiment, since the present invention is based on line shape images subjectively drawn by users, in addition to relying on the completely subjective criterion of “looks like abstract images”, users can also determine the quality of the generated abstract images from the following aspects:

[0097] (1) Whether the generated abstract images reflect line shape features input by users. Users' inputs are not only shape features, but also an important basis for abstract images generated by the method according to the present invention to reflect users' subjective thinking. Thus, the generated abstract images need to clearly reflect users' input shapes.

[0098] (2) Whether the generated abstract images are recreated on line shapes input by users. Since users are non-professional painters, line shapes input by them may be excessively simple or have insufficient structures. The adversarial loss function of the Cycle-GAN model ensures that the generated abstract images are as consistent as possible with abstract images drawn by professional painters. This means that the present invention needs to perform recreation on the non-professional shape structures input by users to obtain more complete abstract structures.

[0099] (3) Whether complete AI color creation is performed on the generated abstract images. The present invention separates a shape and a color of abstract images by the Cycle-GAN model. After users input a basic shape image, the present invention should return a complete abstract image containing both the shape and the color. This means that the computer needs to perform independent and complete AI color creation.

[0100] As shown in FIG. 5, three examples given in FIG. 5 show generated results according to the present invention. In the first group of sunflowers, firstly, the generated result clearly reflects the input shape. Secondly, a shape structure that wraps the lines is formed in addition to the main part of the sunflowers, which performs the recreation on the basis of a given shape. Finally, independent and complete AI creation is made for the colors of the flower petals and flower cores of the sunflowers, as well as the colors of different backgrounds. Besides, the flower cores of the sunflowers using bright red instead of single brown can reflect the abstract artistic expression form.