Method for generating web code for UI based on a generative adversarial network and a convolutional neural network

Abstract

Provided is a method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN). The method includes steps described below. A mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element is constructed. A location of an HTML element in an image I is recognized. Complete HTML codes of the image I are generated. The similarity between manually-written HTML codes and the generated complete HTML codes and the similarity between the image I and an image I.sub.1 generated by the generated complete HTML codes are obtained. After training, an image-to-HTML-code generation model M is obtained. A to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes. According to the method of the present disclosure, an image-to-HTML-code generation model M can be obtained.

Claims

1. A method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN), comprising: step 1: constructing a mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element; step 2: performing object recognition on an image I of the UI to recognize a location of an HTML element in the image I; step 3: generating complete HTML codes of the image I based on the mapping relationship in step 1 and the location of the HTML element in step 2; step 4: obtaining, for the image I of the UI, a similarity Sim.sub.1 between manually-written HTML codes and the generated complete HTML codes; step 5: obtaining a similarity Sim.sub.2 between the image I of the UI and an image I.sub.1 generated by the generated complete HTML codes; step 6: performing minimax game on generative networks G in the GAN and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and discriminative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, wherein the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; balancing, by the equilibrium solution, the Sim.sub.1 in step 4 and the Sim.sub.2 in step 5 as a Sim.sub.0; determining whether the Sim.sub.0 is greater than a set threshold t, and in a case where the Sim.sub.0 is less than t, repeating steps 2 to 5, in a case where the Sim.sub.0 is not less than t, executing step 7; and step 7: obtaining an image-to-HTML-code generation model M according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and inputting a to-be-processed UI image into the model M so as to obtain corresponding HTML codes.

2. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 1, wherein step 1 of constructing the mapping relationship between the display effects of the HTML element and the source codes of the HTML element comprises: extracting feature maps of HTML elements by using the CNN, and establishing a one-to-one correspondence between the feature maps of the HTML element and HTML codes.

3. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 1, wherein step 2 of performing the object recognition on the image I of the UI to recognize the location of the HTML element in the image I comprises: step 2-1: extracting a feature map of the image I of the UI by using the CNN, comprising: extracting the feature map by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and performing edge extension during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution; step 2-2: generating an object proposal region by using a region proposal network (RPN); step 2-3: extracting, by a region of interest (ROI) pooling layer in the pooling layer, a proposal feature map corresponding to the object proposal region; and step 2-4: detecting, by a classification layer and based on the proposal feature map corresponding to the object proposal region, a final location of the object proposal region, that is, the location of the HTML element in the image I of the UI.

4. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 3, wherein step 2-2 of generating the object proposal region by using the RPN comprises: step 2-2-1: setting $\mathrm{ceil}\left(\frac{a\mathrm{width}\mathrm{of}\mathrm{an}\mathrm{original}\mathrm{image}}{\mathrm{SW}}\right)\times \mathrm{ceil}\left(\frac{a\mathrm{height}\mathrm{of}\mathrm{the}\mathrm{original}\mathrm{image}}{\mathrm{SH}}\right)\times Z$  proposal anchors in the image I, wherein SW represents a scale of the width of the original image, SH represents a scale of the height of the original image, and Z is an integer; step 2-2-2: determining, by using the CNN, which anchors are foreground points having an object and which anchors are background points having no object; and step 2-2-3: performing bounding box regression processing on the foreground points having the object so as to obtain an object proposal region corresponding to the foreground points having the object.

5. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 4, wherein step 2-3 of extracting, by the ROI pooling layer in the pooling layer, the proposal feature map corresponding to the object proposal region comprises: it is assumed that a size of the object proposal region is M×N, and a scale of the object proposal region is SW×SH; step 2-3-1: mapping a dimension of the object proposal region to $\left(\frac{M}{\mathrm{SW}}\right)\times \left(\frac{N}{\mathrm{SH}}\right);$ step 2-3-2: horizontally dividing a feature map region corresponding to the dimension-mapped object proposal region into grid cells each of which has a size of pooled.sub.w×pooled.sub.h; and step 2-3-3: performing max pooling processing on a feature map in each of the grid cells so as to obtain the proposal feature map corresponding to the object proposal region.

6. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 5, wherein step 2-4 of detecting, by the classification layer and based on the proposal feature map corresponding to the object proposal region, the final accurate location of the object proposal region, that is, the location of the HTML element in the image I of the UI comprises: step 2-4-1: calculating, for each of the proposal feature map, which HTML element a respective object proposal region belongs to through a fully connected layer and a softmax classifier to obtain a rough location of the respective object proposal region, and outputting a probability vector of the rough location; and step 2-4-2: obtaining, based on the probability vector, a location offset of the respective object proposal region by using the bounding box regression processing to regress to a more accurate object region.

7. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 1, wherein step 3 of generating the complete HTML codes of the image I based on the mapping relationship in step 1 and the location of the HTML element in step 2 comprises: step 3-1: extracting, by using the CNN, a feature map corresponding to an object proposal region whose accurate location is obtained in step 2; step 3-2: matching, based on the mapping relationship in step 1, HTML codes corresponding to the feature map in step 3-1 according to a value of the feature map in step 3-1 so as to obtain HTML codes corresponding to all of HTML elements in the image I; and step 3-3: combining the HTML codes corresponding to all of the HTML elements to generate the final complete HTML codes.

8. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 1, wherein step 4 of obtaining the similarity Sim.sub.1 between the manually-written HTML codes and the generated complete HTML codes for the image I of the UI comprises: step 4-1: extracting a set S.sub.1 of attribute class values in the manually-written HTML codes and a set S.sub.2 of attribute class values in the generated complete HTML codes respectively; and step 4-2: obtaining the similarity Sim.sub.1 by using a Jaccard coefficient through the following formula: ${\mathrm{Sim}}_1=J\left(S_1,S_2\right)\frac{s_1.Math.s_2}{s_1.Math.s_2},$ wherein S.sub.1 represents the set of the attribute class values in the manually-written HTML codes, and S.sub.2 represents the set of the attribute class values in the generated HTML codes.

9. The method for generating the web codes for the UI based on the GAN and the CNN according to claim 1, wherein generating the image I.sub.1 by the generated complete HTML codes in step 5 comprises: running the generated complete HTML codes by using a modern standard browser, and taking a screenshot of display results to obtain the I.sub.1; or obtaining the I.sub.1 by rendering the generated complete HTML codes by using an imgkit package of Python.

10. The method for generating the web codes for the UT based on the GAN and the CNN according to claim 1, wherein step 5 of obtaining the similarity Sim.sub.2 between the image I of the UT and the image I.sub.1 generated by the generated complete HTML codes comprises: obtaining the similarity Sim.sub.2 based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms.

11. The method for generating the web codes for the UT based on the GAN and the CNN according to claim 2, wherein step 2 of performing the object recognition on the image I of the UT to recognize the location of the HTML element in the image I comprises: step 2-1: extracting a feature map of the image I of the UT by using the CNN, comprising: extracting the feature map by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and performing edge extension during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution; step 2-2: generating an object proposal region by using a region proposal network (RPN); step 2-3: extracting, by a region of interest (ROI) pooling layer in the pooling layer, a proposal feature map corresponding to the object proposal region; and step 2-4: detecting, by a classification layer and based on the proposal feature map corresponding to the object proposal region, a final location of the object proposal region, that is, the location of the HTML element in the image I of the UI.

12. The method for generating the web codes for the UT based on the GAN and the CNN according to claim 9, wherein step 5 of obtaining the similarity Sim.sub.2 between the image I of the UI and the image I.sub.1 generated by the generated complete HTML, codes comprises: obtaining the similarity Sime based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms.

Description

BRIEF DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a flowchart of a method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN) according to the present disclosure;

(2) FIG. 2 is an input web design diagram demo1.png according to an embodiment of the present disclosure;

(3) FIG. 3 is a Button object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(4) FIG. 4 is a Divider object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(5) FIG. 5 is a Flag object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(6) FIG. 6 is an Icon object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(7) FIG. 7 is a Breadcrumb object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(8) FIG. 8 is a Message object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure;

(9) FIG. 9 is a Checkbox object diagram after object recognition is performed on demo1.png according to an embodiment of the present disclosure; and

(10) FIG. 10 is a screenshot after demo2.html is run according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

(11) Referring to FIG. 1, the method for generating web codes for a user interface (UI) based on a generative adversarial network (GAN) and a convolutional neural network (CNN) of the present disclosure includes steps described below.

(12) In step 1, a mapping relationship between display effects of a HyperText Markup Language (HTML) element and source codes of the HTML element is constructed.

(13) In step 2, object recognition is performed on an image I of the UI to recognize a location of an HTML element in the image I.

(14) In step 3, complete HTML codes of the image I are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2.

(15) In step 4, a similarity Sim.sub.1 between manually-written HTML codes and the generated complete HTML codes is obtained for the image I of the UI.

(16) In step 5, a similarity Sime between the image I of the UI and an image I.sub.1 generated by the generated complete HTML codes is obtained.

(17) In step 6, minimax game is performed on generative networks G in the generative adversarial network (GAN) and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and discriminative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, where the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; the Sim.sub.1 in step 4 and the Sime in step 5 are balanced as a Sim.sub.0 by the equilibrium solution; and whether the Sim.sub.0 is greater than a set threshold t is determined; in a case where the Sim.sub.0 is less than t, steps 2 to 5 are repeated; in a case where the Sim.sub.0 is not less than t, step 7 is executed.

(18) In step 7, an image-to-HTML-code generation model M is obtained according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and a to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes.

(19) Compared with the related art, the present embodiment has significant advantages. 1) Computer vision technologies are fully used for preforming object recognition processing on a UI image in advance, so as to improve a mapping relationship between display effects of an HTML element and source codes of the HTML element and make the mapping relationship more reliable and accurate. 2) The mapping relationship between display effects of the HTML element and the codes is acquired, and the mapping relationship may be continuously extended, so that the effect of the model of the present disclosure is finally improved, the accuracy of the codes generated for the image is improved, and the application range becomes wider. 3) An image-to-HTML-code conversion model is acquired. When the conversion model is used in practice, as long as a corresponding image is input, the final code results can be obtained. Therefore, some parts in the actual development can be replaced, the actual use cost becomes lower, and the application field becomes wider.

(20) In an embodiment, step 1 specifically includes the step described below. Feature maps of HTML elements are extracted by using the convolutional neural network (CNN), and a one-to-one correspondence is established between the feature maps of the HTML element and HTML codes. Common HTML elements are: Elements (Button, Container, Divider, Flag, Header, Icon, Image, Input, Label, List, Loader, Placeholder, Rail, Reveal, Segment and Step), Collections (Breadcrumb, Form, Grid, Menu, Message and Table), Views (Advertisement, Card, Comment, Feed, Item and Statistic) and Modules (Accordion, Checkbox, Dimmer, Dropdown, Embed, Modal, Popup, Progress, Rating, Search, Shape, Sidebar, Sticky, Tab and Transition).

(21) In an embodiment, step 2 in which the object recognition is performed on the image I of the UI to recognize the location of the HTML element in the image I specifically includes steps described below.

(22) In step 2-1, a feature map of the image I of the UI is extracted by using the convolutional neural network (CNN). Step 2-1 specifically includes the step described below. The feature map is extracted by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and edge extension is performed during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution.

(23) In step 2-2, an object proposal region is generated by using a region proposal network (RPN).

(24) In step 2-3, a proposal feature map corresponding to the object proposal region is extracted by a region of interest (ROI) pooling layer of the pooling layer.

(25) In step 2-4, a final accurate location of the object proposal region, that is, the location of the HTML element in the image I of the UI, is detected by a classification layer and based on the proposal feature map corresponding to the object proposal region.

(26) In an embodiment, step 2-2 in which the object proposal region is generated by using the region proposal network (RPN) specifically includes steps described below.

(27) In step 2-2-1,

(28) $\mathrm{ceil}\left(\frac{a\mathrm{width}\mathrm{of}\mathrm{an}\mathrm{original}\mathrm{image}}{\mathrm{SW}}\right)\times \mathrm{ceil}\left(\frac{a\mathrm{height}\mathrm{of}\mathrm{the}\mathrm{original}\mathrm{image}}{\mathrm{SH}}\right)\times Z$
proposal anchors are set in the original image I, where SW represents a scale of the width of the original image, SH represents a scale of the height of the original image, and Z is an integer.

(29) In step 2-2-2, which anchors are foreground points having an object and which anchors are background points having no object are determined by using the CNN.

(30) In step 2-2-3, bounding box regression processing is performed on the foreground points having the object so as to obtain an object proposal region corresponding to the foreground points having the object. Exemplarily, SW=16, SH=16, and Z=9.

(31) In an embodiment, step 2-3 in which the proposal feature map corresponding to the object proposal region is extracted by the ROI pooling layer in the pooling layer specifically includes steps described below.

(32) It is assumed that a size of the object proposal region is M×N, and a scale of the object proposal region is SW×SH.

(33) $\left(\frac{M}{\mathrm{SW}}\right)\times \left(\frac{N}{\mathrm{SH}}\right).$

(34) In step 2-3-1, a dimension of the object proposal region is mapped to

(35) In step 2-3-2, a feature map region corresponding to the dimension-mapped object proposal region is horizontally divided into grid cells each of which has a size of pooled.sub.w×pooled.sub.h.

(36) In step 2-3-3, max pooling processing is performed on a feature map in each of the grid cells so as to obtain the proposal feature map corresponding to the object proposal region.

(37) In an embodiment, step 2-4 in which the final accurate location of the object proposal region, that is, the location of the HTML element in the image I of the UI, is detected by the classification layer and based on the proposal feature map corresponding to the object proposal region includes steps described below.

(38) In step 2-4-1, for each of the proposal feature map, which HTML element a respective object proposal region belongs to is calculated through a fully connected layer and a softmax classifier to obtain a rough location of the respective object proposal region, and a probability vector of the rough location is output.

(39) In step 2-4-2, based on the probability vector, a location offset of the respective object proposal region is obtained by using the bounding box regression method to regress to a more accurate object region.

(40) Further, step 3 in which the complete HTML codes of the image I are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2 specifically includes steps described below.

(41) In step 3-1, a feature map corresponding to an object proposal region whose accurate location is obtained in step 2 is extracted by using the CNN.

(42) In step 3-2, based on the mapping relationship in step 1, HTML codes corresponding to the feature map in step 3-1 are matched according to a value of the feature map in step 3-1 so as to obtain HTML codes corresponding to all of HTML elements in the image I.

(43) In step 3-3, all HTML code blocks are combined to generate the final complete HTML codes.

(44) Further, step 4 in which the similarity Sim.sub.1 between the manually-written HTML codes and the generated complete HTML codes is calculated for the image I of the UI specifically includes steps described below.

(45) In step 4-1, a set S.sub.1 of attribute class values in the manually-written HTML codes and a set S.sub.2 of attribute class values in the generated complete HTML codes are respectively extracted.

(46) In step 4-2, the similarity Sim.sub.1 is obtained by using a Jaccard coefficient through the following formula:

(47) ${\mathrm{Sim}}_1=J\left(S_1,S_2\right)\frac{s_1.Math.s_2}{s_1.Math.s_2}.$

(48) S.sub.1 represents the set of the attribute class values in the manually-written HTML codes, and S.sub.2 represents the set of the attribute class values in the generated HTML codes.

(49) Exemplarily, the image I.sub.1 is generated by the generated complete HTML codes in step 5 includes two manners described below.

(50) The generated complete HTML codes are run by using a modern standard browser, and a screenshot of display results is taken to obtain the I.sub.1.

(51) Alternatively, the I.sub.1 is obtained by rendering the generated complete HTML codes by using an imgkit package of Python.

(52) Exemplarily, step 5 in which the similarity Sim.sub.2 between the image I of the UI and the image I.sub.1 generated by the generated complete HTML codes is obtained in step 5 specifically includes the step described below. The similarity Sim.sub.2 is obtained based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms.

Embodiment

(53) The method for generating web codes for a UI based on a generative adversarial network (GAN) and a convolutional neural network (CNN) of the present disclosure includes contents described below.

(54) In step 1, a mapping relationship between display effects of an HTML element and source codes of the HTML element is constructed. Step 1 specifically includes the step described below.

(55) Feature maps of HTML elements are extracted by using the CNN, and a one-to-one correspondence is established between the feature maps of the HTML element and HTML codes.

(56) Common HTML elements include Elements (Button, Container, Divider, Flag, Header, Icon, Image, Input, Label, List, Loader, Placeholder, Rail, Reveal, Segment and Step), Collections (Breadcrumb, Form, Grid, Menu, Message and Table), views (Advertisement, Card, Comment, Feed, Item and Statistic) and Modules (Accordion, Checkbox, Dimmer, Dropdown, Embed, Modal, Popup, Progress, Rating, Search, Shape, Sidebar, Sticky, Tab and Transition).

(57) In the embodiment, an input web design diagram demo1.png is shown in FIG. 2. In the embodiment, the one-to-one mapping relationship of Button, Divider, Flag, Icon, Breadcrumb, Message and Checkbox is defined as shown in Table 1 below.

(58) TABLE-US-00001 TABLE 1 Mapping relationship between HTML element codes and display effects of the HTML element Number HTML elements Images Codes 1 Button Number 1 <button class=″ui button″>Follow</button> in FIG. 2 2 Divider Number 2 <div class=″ui divider″></div> in FIG. 2 3 Flag Number 3 <i class=″ae flag″></i> in FIG. 2 <i class=″france flag″></i> <i class=″myanmar flag″></i> 4 Icon Number 4 <i class=″question circle icon″></i> in FIG. 2 5 Breadcrumb Number 5 <div class=″ui breadcrumb″> in FIG. 2 <a class=″section″>Home</a> <i class=″right angle icon divider″></i> <a class=″section″>Store</a> <i class=″right angle icon divider″></i> <div class=″active section″>T-shirt</div> </div> 6 Message Number 6 <div class=″ui compact message″> in FIG. 2 <p>Get all the best inventions in your e-mail everday. Sign up now!</p> </div> 7 Checkbox Number 7 <div class=″ui checkbox″> in FIG. 2 <input type=″checkbox″ name=″example″> <label>Make my profile visible</label> </div>

(59) In step 2, object recognition is performed on an image 1 to recognize a location of an HTML element in the image 1. Step 2 specifically includes steps described below.

(60) In step 2-1, a feature map of the image 1 of the UI is extracted by using the CNN. Step 2-1 specifically includes the step described below. The feature map is extracted by using a convolution layer, a pooling layer and a Rectified Linear Unit (ReLU) layer, and edge extension is performed during multiple times of convolution to ensure that a size of an input matrix and a size of an output matrix do not change after the multiple times of convolution.

(61) In step 2-2, an object proposal region is generated by using a region proposal network (RPN).

(62) In step 2-2-1,

(63) $\mathrm{ceil}\left(\frac{a\mathrm{width}\mathrm{of}\mathrm{an}\mathrm{original}\mathrm{image}}{\mathrm{SW}}\right)\times \mathrm{ceil}\left(\frac{a\mathrm{height}\mathrm{of}\mathrm{the}\mathrm{original}\mathrm{image}}{\mathrm{SH}}\right)\times 9$
proposal anchors are set in the image 1.

(64) In step 2-2-2, which anchors are foreground points having an object and which anchors are background points having no object are determined by using the CNN.

(65) In step 2-2-3, bounding box regression processing is performed on the foreground points having the object so as to obtain an object proposal region corresponding to the foreground points having the object.

(66) In step 2-3, a proposal feature map corresponding to the object proposal region is extracted by an ROI pooling layer in the pooling layer.

(67) It is assumed that a size of the object proposal region is M×N.

(68) In step 2-3-1, a dimension of the object proposal region is mapped to

(69) $\left(\frac{M}{16}\right)\times \left(\frac{N}{16}\right).$

(70) In step 2-3-2, a feature map region corresponding to the dimension-mapped object proposal region is horizontally divided into grid cells each of which has a size of pooled.sub.w×pooled.sub.h.

(71) In step 2-3-3, max pooling processing is performed on a feature map in each of the grid cells so as to obtain the proposal feature map corresponding to the object proposal region.

(72) In step 2-4, a final accurate location of the object proposal region is detected by a classification layer, that is, which HTML element a respective object proposal region belongs to is calculated through a fully connected layer and a softmax classifier by using the obtained feature map, and a probability vector cls_prob is output. At the same time, a location offset bbox_pred of the respective object proposal region is obtained by using the bounding box regression method again for regression of a more accurate object region.

(73) In the embodiment, the finally obtained object images are shown in Table 2 below.

(74) TABLE-US-00002 TABLE 2 Recognized object images Number Object images 1 Image 3 2 Image 4 3 Image 5 4 Image 6 5 Image 7 6 Image 8 7 Image 9

(75) In step 3, complete HTML codes of the image 1 are generated based on the mapping relationship in step 1 and the location of the HTML element in step 2.

(76) According to the object images in Table 2 above, codes are searched for in the mapping relationship in Table 1, and the codes are obtained as shown in Table 3 below.

(77) TABLE-US-00003 TABLE 3 Codes corresponding to object images Object images Codes FIG. 3 <button class=″ui button″>Follow</button> FIG. 4 <div class=″ui divider″></div> FIG. 5 <i class=″ae flag″></i> <i class=″france flag″></i> <i class=″myanmar flag″></i> FIG. 6 <i class=″question circle icon″></i> FIG. 7 <div class=″ui breadcrumb″> <a class=″section″>Home</a> <i class=″right angle icon divider″></i> <a class=″section″>Store</a> <i class=″right angle icon divider″></i> <div class=″active section″>T-shirt</div> </div> FIG. 8 <div class=″ui compact message″> <p>Get all the best inventions in your e-mail every day. Sign up now!</p> </div> FIG. 9 <div class=″ui checkbox″> <input type=″checkbox″ name=″example″ /> <label>Make my profile visible</label> </div>

(78) The above codes are stitched to obtain following complete codes demo2.html:

(79) TABLE-US-00004 <body>  <button class=“ui button”>Follow</button>  <div class=“ui divider”></div>  <div>   <i class=“ae flag”></i>   <i class=“france flag”></i>   <i class=“myanmar flag”></i>  </div>  <div>   <i class=“question circle icon”></i>  </div>  <div class=“ui breadcrumb”>   <a class=“section”>Home</a>   <i class=“right angle icon divider”></i>   <a class=“section”>Store</a>   <div class=“active section”>T-Shirt</div>  </div>  <div>   <div class=“ui compact message”>    <p>     Get all the best inventions in your e-mail every day. Sign up     now!    </p>   </div>  </div>  <div>   <div class=“ui checkbox”>    <input type=“checkbox” name=“example” />    <label>Make my profile visible</label>   </div>  </div> </body>.

(80) In step 4, a similarity Sim.sub.1 between manually-written HTML codes demo1.html and the generated complete HTML codes demo2.html is obtained for the image 1. Step 4 specifically includes steps described below.

(81) In step 4-1, a set S.sub.1 of attribute class values in the manually-written HTML codes and a set S.sub.2 of attribute class values in the generated complete HTML codes are respectively extracted.

(82) In step 4-2, the similarity Sim.sub.1 is obtained by using a Jaccard coefficient through the following formula:

(83) ${\mathrm{Sim}}_1=J\left(S_1,S_2\right)\frac{s_1.Math.s_2}{s_1.Math.s_2}.$

(84) S.sub.1 represents the set of the attribute class values in the manually-written HTML codes, and S.sub.2 represents the set of the attribute class values in the generated HTML codes.

(85) In the embodiment, the manually-written HTML codes demo1.html are:

(86) TABLE-US-00005   <!DOCTYPE html>   <html lang=“en”>    <head>     <meta charset=“UTF-8” />     <meta name=“viewport” content=“width=device-width, initial-scale=1.0” />     <meta http-equiv=“X-UA-Compatible” content=“ie=edge” />     <title>Document</title>     <link      rel=“stylesheet”      href=“https://cdn.jsdelivr.net/npm/semantic-ui @2.4.2/dist/semantic.min.css”     />     <script  src=“https://cdn.jsdelivr.net/npm/semantic-ui @2.4.2/dist/semantic.min.js”></script>    </head>    <body>     <button class=“ui button”>Follow</button>     <div class=“ui divider”></div>     <div>      <i class=“ae flag”></i>      <i class=“france flag”></i>      <i class=“myanmar flag”></i>     </div>     <div>      <i class=“question circle icon”></i>     </div>     <div class=“ui breadcrumb”>      <a class=“section”>Home</a>      <i class=“right angle icon divider”></i>      <a class=“section”>Store</a>      <i class=“right angle icon divider”></i>      <div class=“active section”>T-Shirt</div>     </div>     <div>      <div class=“ui compact message”>       <p>        Get all the best inventions in your e-mail every day. Sign up now!       </p>      </div>     </div>     <div>      <div class=“ui checkbox”>       <input type=“checkbox” name=“example” />       <label>Make my profile visible</label>      </div>     </div>    </body> </html>.

(87) From the above, it can be obtained that:

(88) S1={“ui button”, “ui divider”, “ae flag”, “france flag”, “myanmar flag”, “question circle icon”, “ui breadcrumb”, “section”, “right angle icon divider”, “section”, “right angle icon divider”, “active section”, “ui compact message”, “ui checkbox”, };

(89) S.sub.2={“ui button”, “ui divider”, “ae flag”, “france flag”, “myanmar flag”, “question circle icon”, “ui breadcrumb”, “section”, “right angle icon divider”, “section”, “right angle icon divider”, “active section”, “ui compact message”, “ui checkbox”, }.

(90) In this way, it can be obtained that the similarity Sim.sub.1 satisfies that

(91) $Si{m}_1=J\left(S_1,S_2\right)\frac{s_1.Math.s_2}{s_1.Math.s_2}=100\%.$

(92) In step 5, a similarity Sim.sub.2 between the image 1 of the UI and an image generated by the generated complete HTML codes is obtained. Step 5 specifically includes steps described below.

(93) The method for generating the image by the generated complete HTML codes is described below. The generated complete HTML codes are run by using a modern standard browser, and a screenshot of display results is taken; or, the generated complete HTML codes are rendered by using an imgkit package of Python.

(94) The similarity Sim.sub.2 is obtained based on a histogram, image template matching and perceptual hashing algorithm of image similarity algorithms. In the embodiment, the codes demo1.html are run, and the display effect of the codes are saved as shown in FIG. 10, and the Sim.sub.2 between the image 1 and FIG. 10 is obtained as 100%.

(95) In step 6, minimax game is performed on generative networks G in the generative adversarial network (GAN) and discriminative networks D in the GAN to obtain an equilibrium solution from all possible generative networks G and generative networks D, so that the generative networks G generate HTML codes having a same distribution as the manually-written HTML codes, where the generative networks G in the GAN are formed by step 2 and step 3, and the discriminative networks D in the GAN are formed by step 4 and step 5; the Sim.sub.1 and the Sim.sub.2 are balanced as a Sim.sub.0 by the equilibrium solution; whether the Sim.sub.0 is greater than a set threshold t is determined; and in a case where the Sim.sub.0 is less than t, steps 2 to 5 are repeated; in a case where the Sim.sub.0 is not less than t, step 7 is executed.

(96) In the embodiment, t is set to 95%; and Sim.sub.1=100% and Sim.sub.2=100%, so that the balanced Sim.sub.0 is obtained as 100%. At this time, Sim.sub.0=100% and t=95%, so that Sim.sub.0 is greater than t, and the next step is executed.

(97) In step 7, an image-to-HTML-code generation model M is obtained according to a finally obtained generative G and discriminative network D of the GAN after the training in step 6, and a to-be-processed UI image is input into the model M so as to obtain corresponding HTML codes.

(98) According to the method of the present disclosure, an image-to-HTML-code generation model M can be obtained. A to-be-processed UI image is input into the model, and then corresponding HTML codes can be generated. Therefore, the model has stronger universality and generality and can replace some parts in the actual development, and thus the actual use cost is lower and the application field is wider.