SYSTEMS, METHODS, AND STORAGE MEDIA FOR CREATING IMAGE DATA EMBEDDINGS TO BE USED FOR IMAGE RECOGNITION

Abstract

Disclosed implementations include a method, apparatus and computer media for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition. Image data representing the image of an object is received and N landmarks are detected on the image using a deep regression algorithm, wherein N is an integer. A weighted, fully connected, graph is constructed from the landmarks by assigning initial weights for the landmarks randomly. An optimized tree structure is determined based on the initial weights. A sequence is generated by traversing nodes of the tree structure and a series of embeddings representing the object image are generated based on the sequence. The embeddings can be processed by a neural network to generate an image recognition signal based on the embeddings.

Claims

1. An object recognition method for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition, the method comprising: receiving image data representing the image of an object; detecting N landmarks on the image using a deep regression algorithm, wherein N is an integer; constructing a weighted, fully connected, graph k.sub.n from the landmarks by assigning initial weights for the landmarks randomly; determining an optimized tree structure based on the initial weights; generating a sequence by traversing the tree structure; forming a series of embeddings based on the sequence, the embeddings representing the object image; and inputting the embeddings into a neural network to thereby generate an image recognition signal based on the embeddings.

2. The method of claim 1, wherein the image data is facial image data representing an image of a person's face and the image recognition signal is a facial recognition signal.

3. The method of claim 2, further comprising, frontalizing the facial image data with a dual attention generative adversarial network (DA-GAN) before the constructing.

4. The method of claim 1, further comprising inputting the sequence into the neural network to thereby generate the image recognition signal based on the sequence and the embeddings.

5. The method of claim 4, wherein an output of the neural network is fed back for use in the constructing, generating, and forming steps.

6. The method of claim 1, wherein the neural network is a Long Short-Term Memory (LSTM) neural network.

7. The method of claim 1, wherein the sequence represents a structure of the object and generating the sequence comprises finding an optimum minimum-cost spanning tree G(A) by using the Prim's Algorithm to solve; $a_{\mathrm{ij}}\underset{i=1}{\overset{n}{.Math.}}\underset{j=1}{\overset{n}{.Math.}}{a}_{\mathrm{ij}}.Math.G\left(A\right)\mathrm{is}\mathrm{tree},$ where; A is an adjacency matrix of graph G(A), and α.sub.ij is an element of matrix A.

8. The method of claim 7, wherein generating the sequence further comprises traversing the tree structure using a preorder depth-first order traversal algorithm starting point of a geometrically most centered landmark of the N facial landmarks.

9. The method of claim 1, wherein N=50.

10. The method of claim 1, wherein forming the embeddings comprises determining the embeddings based on;
O(X,W.sub.graph)=x.sub.αx.sub.β . . . x.sub.γ, where; X is the set of the landmarks' coordinates, W.sub.graph is the weights of the complete graph, and each of x.sub.αx.sub.β . . . x.sub.γ belongs to the set X.

11. The method of claim 1, wherein the embeddings represent a texture of the image.

12. The method of claim 11, where generating the embeddings comprises: forming n×n patches respectively centered around each of the landmarks; and cropping the patches from the image data and feeding the corresponding data to a pretrained convolutional neural network.

13. An object recognition system for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition, the method comprising: at least one computer processor configured to execute computer-readable instructions; and at least one memory device having computer-readable instructions recorded thereon which, when executed by the at least one computer processor, cause the at least one computer processor to conduct the method of; receiving image data representing the image of an object; detecting N landmarks on the image using a deep regression algorithm, wherein N is an integer; constructing a weighted, fully connected, graph k.sub.n from the landmarks by assigning initial weights for the landmarks randomly; determining an optimized tree structure based on the initial weights; generating a sequence by traversing the tree structure; forming a series of embeddings based on the sequence, the embeddings representing the object image; and inputting the embeddings into a neural network to thereby generate an image recognition signal based on the embeddings.

14. The system of claim 13, wherein the image data is facial image data representing an image of a person's face and the image recognition signal is a facial recognition signal.

15. The system of claim 14, wherein the method further comprises, frontalizing the facial image data with a dual attention generative adversarial network (DA-GAN) before the constructing.

16. The system of claim 13, wherein the method further comprises inputting the sequence into the neural network to thereby generate the image recognition signal based on the sequence and the embeddings.

17. The system of claim 16, wherein an output of the neural network is fed back for use in the constructing, generating, and forming steps.

18. The system of claim 13, wherein the neural network is a Long Short-Term Memory (LSTM) neural network.

19. The system of claim 13, wherein the sequence represents a structure of the object and generating the sequence comprises finding an optimum minimum-cost spanning tree G(A) by using the Prim's Algorithm to solve; $a_{\mathrm{ij}}\underset{i=1}{\overset{n}{.Math.}}\underset{j=1}{\overset{n}{.Math.}}{a}_{\mathrm{ij}}.Math.G\left(A\right)\mathrm{is}\mathrm{tree},$ where; A is an adjacency matrix of graph G(A), and a.sub.ij is an element of matrix A.

20. The system of claim 19, wherein generating the sequence further comprises traversing the tree structure using a preorder depth-first order traversal algorithm starting point of a geometrically most centered landmark of the N facial landmarks.

21. The system of claim 13, wherein N=50.

22. The system of claim 13, wherein forming the embeddings comprises determining the embeddings based on;
O(X,W.sub.graph)=x.sub.αx.sub.β . . . x.sub.γ, where; X is the set of the landmarks' coordinates, W.sub.graph is the weights of the complete graph, and each of x.sub.αx.sub.β . . . x.sub.γ belongs to the set X.

23. The system of claim 13, wherein the embeddings represent a texture of the image.

24. The system of claim 23, where generating the embeddings comprises: forming n×n patches respectively centered around each of the landmarks; and cropping the patches from the image data and feeding the corresponding data to a pretrained convolutional neural network.

25. Non-transient computer readable media having computer-readable instructions recorded thereon which, when executed by at least one computer processor, cause the at least one computer processor to conduct the method of; receiving image data representing the image of an object; detecting N landmarks on the image using a deep regression algorithm, wherein N is an integer; constructing a weighted, fully connected, graph k.sub.n from the landmarks by assigning initial weights for the landmarks randomly; determining an optimized tree structure based on the initial weights; generating a sequence by traversing the tree structure; forming a series of embeddings based on the sequence, the embeddings representing the object image; and inputting the embeddings into a neural network to thereby generate an image recognition signal based on the embeddings.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1 shows a face image along with 4 random trees that were built on the extracted facial landmarks.

[0016] FIG. 2 is a schematic block diagram of a pipeline architecture in accordance with one or more implementations.

[0017] FIG. 3 is a schematic illustration of a tree traversal process and building of the LSTM input embedding in accordance with one or more implementations.

[0018] FIG. 4 is a schematic block diagram of a fusion methodology in accordance with disclosed implementations.

[0019] FIG. 5 a flow chart of a method for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition

[0020] FIG. 6 illustrates the probability mass function (pmf) of an example of the disclosed implementations.

[0021] FIG. 7. illustrates F1 scores of each expressions achieved by an example of the disclosed implementations.

[0022] FIG. 8 shows some of the classified samples achieved by an example of the disclosed implementations.

[0023] FIG. 9 illustrates an example of minimum-cost spanning trees, obtained by an example of the disclosed implementations.

DETAILED DESCRIPTION

[0024] Disclosed implementations learn an optimal face tree topology to be used in subsequent components of an image recognition network. However, the potential for differences, such as different poses present in facial images, or different angles of view in images of other objects, can cause erroneous variations in the tree topology, even when the same information is contained in the image. To remedy this, a Deep Attention Generative Adversarial Network (DA-GAN) can be used for face frontalization or other normalization techniques, thus making the disclosed implementations more robust against variations, such as pose variations. Image landmarks, such as facial landmarks in the example of a facial image, are determined from the frontalized images using a deep regression architecture and an integer number of landmarks from each input image.

[0025] FIG. 1 shows a face image along with 4 random trees that were built on the extracted facial landmarks. Traversing these trees results in 4 different sequence-based embeddings that can be fed, for example, to 4 independent LSTM networks for a 7-class FER classification task. The resulting recognition rates are shown below each image in FIG. 1. An obtained performance range of 3.7% confirms the importance of finding the optimal tree for further image recognition processing.

[0026] FIG. 2 illustrates a pipeline architecture 200 in accordance with disclosed implementations, which can be implemented by a programmed computing system. Pipeline architecture 200 can be divided into 3 functional components: tree topology component 210; structure stream component 220; and texture stream component 230, as shown in FIG. 2. In tree topology component 210, the tree learning is performed by extraction of facial landmarks by Pose-Weighted Generative Adversarial Network (PW-GAN) 212 and optimization component 214 to obtain the optimal tree. The optimized tree can be processed by structure stream component 220 and/or texture stream component 230 to learn separate structure and texture representations. The optimized tree is traversed to form a sequence, whose resultant embedding is then fed to structure stream component 220. Specifically, Long Short Term Memory network (LSTM) 222 receives the embedding as an input and the output of LSTM 220 can be fed to soft attention module 224 to focus on the salient cells of the LSTM network. LSTMs are a special kind of neural network, capable of learning long-term dependencies. “Soft attention” is when a context vector is calculated as a weighted sum of the encoder hidden states. In texture stream component 230, ResNet50 features are extracted from the local patches around extracted image landmarks determined by bounding box process module 232, prior to training separate LSTM 234 and fed to softs attention module 236, similar to the structure stream. The resulting embedding is fused with that of the structure stream to yield an expression class prediction. The operation of pipeline architecture 200 is described in more detail below.

[0027] In tree topology component 210, a weighted fully connected graph (k.sub.n) is constructed from the extracted facial landmarks. The initial weights of the graph can be chosen at random. The optimum minimum-cost spanning tree with regards to these weights is determined, whose traversal generates a sequence specifying an embedding that can be input into an LSTM network for recurrently learning.

[0028] The minimum-cost spanning tree can be defined as:

[00001]$\begin{array}{cc}{a}_{\mathrm{ij}}\underset{i=1}{\overset{n}{.Math.}}\underset{j=1}{\overset{n}{.Math.}}{a}_{\mathrm{ij}}.Math.G\left(A\right)\mathrm{is}\mathrm{tree},& \left(1\right)\end{array}$

where A is an adjacency matrix of graph G(A) and a.sub.ij is an element of matrix A. The resulting G(A) is a minimum-cost spanning tree. The Prim's algorithm can be used to solve Eq. 1. The Prim's algorithm is a known algorithm that only visits a vertex once in each step. During this visit, an edge with a minimum weight, among all edges that are connected to that vertex, is considered as the selected edge for the minimum-cost spanning tree. By visiting all vertices, Prim's algorithm efficiently builds a “minimum-cost”, e.g. most efficient, spanning tree. The built tree is then traversed using a traversal algorithm such as a preorder depth-first order algorithm, with the starting point of the most centered landmark, e.g., the nose tip. In the traversal, when reaching a leaf node, the traversal backtracks in the tree, and returns to the starting point. The traversal forms a sequence, which in turn is used to form an embedding that is input to structure stream component 220 and texture stream component 235.

[0029] FIG. 3 depicts the tree traversal process and building of the embedding. In FIG. 3, the process is illustrated for a tree example with only 10 nodes for clarity. The weighted fully connected graph 310 can be traversed using a preorder depth-first order traversal technique and the traversal is backtracked when it reaches a leaf node, as noted above. The resulting tree optimization 320 is a data structure in which the most efficient traversals from each leaf to the root node are identified. The optimized tree in the example of FIG. 3 yields a node sequence 330 of A-B-C-B-A-D-E-F-E-D-A-G-H-I-H-J-H-G-A that is used as embeddings to be input into LSTM network 222 (see also FIG. 2).

[0030] Tree topology learning module 210 (FIG. 20 updates the learnable weights associated with k.sub.n. To this end, a combinatorial optimization algorithm as used. For the combinatorial optimizer's objective function, the loss function of the end-to-end pipeline is expressed as follows:

[00002]$\begin{array}{cc}J\left(w\right)=\frac{1}{m}\underset{i=1}{\overset{m}{.Math.}}L\left({\hat{y}}^{\left(i\right)},y^i\right),& \left(2\right)\end{array}$

where w is the weights of the complete graph k.sub.n, m is the size of the training set, and L(x) is the loss function selected to maximize performance for the specific problem.

[0031] At each training epoch, a combinatorial algorithm generates a set of weights associated with k.sub.n, from which the minimum-cost spanning tree, whose traversal results in a particular sequence, is generated. Next, this sequence is frozen, and the subsequent structure and texture streams are fully trained. Once training of the two streams is complete, the final loss is fed to the combinatorial optimizer as the objective of an optimization, which then generates a new set of weights for k.sub.n and this process can continue repeatedly until convergence. It should be noted that the obtained graph is converted to a tree for determining the optimal sequence because the acyclic nature of trees is better suited for traversal, and thus better for sequence generation.

[0032] The coordinates of the facial landmarks extracted in tree topology learning component 210, along with the obtained sequence, are the inputs to structure stream component, 220, as shown in FIG. 2. By using this sequence, an embedding is generated and can be fed to an LSTM network with peephole connections. The embedding can be expressed as:

O(X,W.sub.graph)=x.sub.αx.sub.β . . . x.sub.γ,  (3)

where, X is the set of the facial landmarks' coordinates, W.sub.graph is the weights of the complete graph, and each of xαx.sub.β. . . x.sub.γ belongs to the set X. Note that, since the tree traversal algorithm may visit a node more than once, as shown in FIG. 3, the output chain may include repeated nodes. Soft attention modules 224 and 236 are included to better focus on more important parts of the embeddings by applying learned weights. To this end, the hidden state vectors h.sub.i of the LSTM cells are multiplied by a set of coefficients α.sub.i which determines the level of the attention that it receives and can be calculated using:

[00003]$\begin{array}{cc}{\alpha }_i=\frac{e^{u_i}}{\underset{k=1}{\overset{n}{.Math.}}{e}^{u_k}},& \left(4\right)\end{array}$

where, n is the number of LSTM units, and u.sub.i is calculated as:

u.sub.i=tan h(W.sub.hh.sub.i+b.sub.h).  (5)

Where W.sub.h and bh are the trainable weights and biases. The attentive output can be calculated using:

[00004]$\begin{array}{cc}H=\underset{i}{.Math.}{\alpha }_i{h}_i,& \left(6\right)\end{array}$

where H is the final embedding of the structure stream.

[0033] The inputs of texture stream component 230 are the facial landmarks and the sequence generated in the face tree topology learning step. However, unlike structure stream component 220, where the coordinates of facial landmarks are used to form an embedding for subsequent use in an LSTM network, in texture stream component 230, the focus is on the texture of the facial images. To this end, first n×n patches centered around each facial landmark are formed. The patches are then cropped from the images and fed to a ResNet50 pretrained on, for example, the VGG-Face2 dataset. Through empirical experiments, it was found that a patch size of 17×17 pixels yields good results. These embeddings are stacked in accordance with the sequence obtained from the tree topology learning component 210. The resulting embedding is then passed onto LSTM network 234. This is followed by soft attention process component 236 in a manner similar to the structure stream (see Equations 4-6 above).

[0034] In order to fuse the outputs of the structure and texture streams, a two-stream fusion strategy, such as that disclosed in Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic, Hybrid attention based multimodal network for spoken language classification, Proceedings of the conference. Association for Computational Linguistics, Meeting, volume 2018, page 2379 can be used. FIG. 4 illustrates a fusion methodology 400 in accordance with disclosed implementations. This methodology has been designed to preserve the original characteristics of its input embeddings and has yielded state-of-the-art results in a number of areas where preserving and incorporating these characteristics are of particular importance.

[0035] Two dense, i.e. non-linear, layers of classifiers are used as encoders 410 and 420, respectively, for each of the input streams (structure and texture). Each of encoders 410 and 420 generate stream-specific features. Then, a soft attention component 430 is applied to learn the weight by using the following equation:

α=softmax(tan h(W.sub.f[T*,S*]+b.sub.f)),  (7)

where W.sub.f and b.sub.f are trainable weights and biases respectively. T* and S* are the stream-specific features of texture and structure streams, respectively. Next, a dense layer classifier 440 is utilized to learn the optimal association across the embedding-specific features as:

y=tan h(W.sub.y[(1+α.sub.T)T,(1+α.sub.S)S]+b.sub.y),  (8)

where y is the final embedding, W.sub.y and b.sub.y are trainable weights and biases, and T and S are the output embeddings of texture and structure streams, respectively. The final decision is made by using a softmax classifier with y as its input.

[0036] FIG. 5 illustrates a high level method 500 for learning an optimal graph in the form of a tree topology defining a sequence that can be used by a learning network for image recognition in accordance with disclosed implementations. At 502 image data representing the image of an object is received. At 504, N landmarks are detected on the image using a deep regression algorithm, wherein N is an integer. At 506, a weighted, fully connected, graph is constructed from the landmarks by assigning initial weights for the landmarks randomly and determining the optimized tree structure based on the initial weights. At 508, a sequence is generated by traversing the tree structure. At 510, a series of embeddings representing the object image is formed based on the sequence. At 512, the embeddings can be inputted into a neural network to thereby generate an image recognition signal based on the embeddings.

[0037] Details of a specific example of the disclosed implementation applied to FRE will be discussed below. The entire architecture can be been implemented using TensorFlow and can be trained using a pair of Nvidia RTX 2080 Ti GPUs. For optimizing the weights of the two streams, an ADAM optimizer was used, with the learning rate, first-momentum decay, and second-momentum decay of 0.001, 0.9, 0.99 respectively. All of the reported results were obtained on the validation sets of the respective datasets.

[0038] Two popular in-the-wild facial expression datasets were used to evaluate the disclosed implementations. The first dataset, AffectNet includes more than 1 million images, all of which have been obtained from the internet by leveraging 3 search engines. About half of the dataset has been manually annotated in both categorical model for the presence of 7 facial expressions (anger, disgust, fear, happy, sad, surprise, and contempt) plus neutral, and in dimensional model for valence and arousal intensities. The validation set of this dataset includes 500 images for each expression class. The FER2013 dataset includes 28709, 3589, and 3589 images for training, validation, and testing. All the images have been resized to 48×48 pixels. Face images in this dataset are labeled as having one of the 6 basic expressions (anger, disgust, fear, happy, sad, and surprise) along with neutral.

[0039] FIG. 6 illustrates the probability mass function (pmf) of AffectNet and FER2013 datasets, showing that these distributions are highly skewed towards the ‘happiness’ expression. This matter can cause the FER methods to perform poorly on minority classes. To tackle this problem, focal loss was utilized as the loss function L(x) in Eq. 2. Focal

focL=−α.sub.b(1−p.sub.b).sup.γ log(p.sub.b),  (9)

where γ is the focusing parameter and p.sub.b is defined as:

[00005]$\begin{array}{cc}{p}_b=\{\begin{array}{cc}p& y=1\\ 1-p& \mathrm{otherwise}\end{array},& \left(10\right)\end{array}$

where p is the probability of y=1, which is estimated by
the model. Focal loss causes the learning algorithm to be less focused on the classes with higher number of samples.

[0040] The obtained recognition rates by the example (Face Tree Topology) and other state-of-the-art benchmarking methods are presented in Table 1 below for both AffectNet and FER2013 datasets. With respect to the AffectNet dataset, the disclosed implementations show an improvement of 1.52% and 3.28% over the BreG-NeXt50 and BreG-NeXt 32, which are currently the state-of-the-art FER methods. Note that BreG-NeXt 50 and BreG-Next 32 have also utilized focal loss as their loss function. Given the FER2013 dataset, the disclosed implementations outperform the current state-of-the-art FER method, i.e., BreG-Next 50, showing a performance improvement of 1.13%.

TABLE-US-00001 Dataset Year Method Rec. Rate AffectNet 2018 CNN for inconsistent label handling 57.31 2018 Light CNN 58 2019 Ensemble deep learning 62.11 2019 Facial mask learning 61.50 2020 Expression synthesis from neutral 60 2020 BreG-NeXt 32 66.74 2020 BreG-NeXt 50 68.50 2021 Face tree topology 70.02 FER2013 2013 CNN + SVM 69.3 2016 CNN 66.4 2017 New CNN architecture 71.2 2020 BreG-NeXt 32 69.11 2020 BreG-NeXt 50 71.53 2021 Face tree topology 72.66

[0041] For a side-by-side comparison between the disclosed implementations and BreG-NeXt 50, F1 scores of each expression were outlined by using the one-vs-all method. The results are shown in FIG. 7. With respect to the AffectNet dataset, the disclosed implementations (FTNet) have higher F1 scores than BreG-NeXt 50 in all the 8 expression classes except happiness and neutral. As mentioned before, AffectNet is skewed toward happiness and neutral expressions. However, the disclosed implementations have shown better results in the minority expressions which are more difficult to be recognized. With regards to FER2013, FTNet has achieved better F1 scores in 4 out of 7 expressions. The exceptions are surprise, happiness, and disgust expressions, in which BreG-NeXt 50 perform slightly better.

[0042] FIG. 8 shows some of the well-classified and misclassified samples, along with the corresponding input images from the validation sets of AffectNet and FER2013 datasets along with the corresponding predicted and ground truth expressions. It can be seen from FIG. 8 that, in the pmf of miss-classified cases, ground truth expression has the 2nd highest probability. This indicates that even in miss-classified cases, the disclosed implementation have a probabilistic sense to the ground truths, and do not generate pmfs which are far away from the ground truth.

[0043] FIG. 9 illustrates an example of the minimum-cost spanning trees, obtained by disclosed implementations for AffectNet and FER2013 datasets. Traversal of these trees provides the optimal sequence to feed the sequence learner, e.g., an LSTM. It is clear from FIG. 9 that larger variations between the two trees occurred near the mouth and eyes regions. However, the trees are almost identical around the jaw line. This is not surprising, because the regions around the eyes and mouth are very representative in FER, so facial expressions are more reflected in these regions. As a result, these regions are more sensitive, because they are designed based on the properties of each dataset.

[0044] The disclosed implementations can be implemented by various computing devices programmed with software and/or firmware to provide the disclosed functions and modules of executable code implemented by hardware. The software and/or firmware can be stored as executable code on one or more non-transient computer-readable media. The computing devices may be operatively linked via one or more electronic communication links. For example, such electronic communication links may be established, at least in part, via a network such as the Internet and/or other networks.

[0045] A given computing device may include one or more processors configured to execute computer program modules. The computer program modules may be configured to enable an expert or user associated with the given computing platform to interface with the system and/or external resources. By way of non-limiting example, the given computing platform may include one or more of a desktop computer, a laptop computer, a handheld computer, a tablet computing platform, a Smartphone, a gaming console, and/or other computing platforms.

[0046] The various data and code can be stored in electronic storage devices which may comprise non-transitory storage media that electronically stores information. The electronic storage media of the electronic storage may include one or both of system storage that is provided integrally (i.e., substantially non-removable) with the computing devices and/or removable storage that is removably connectable to the computing devices via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storage may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media.

[0047] Processor(s) of the computing devices may be configured to provide information processing capabilities and may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. As used herein, the term “module” may refer to any component or set of components that perform the functionality attributed to the module. This may include one or more physical processors during execution of processor readable instructions, the processor readable instructions, circuitry, hardware, storage media, or any other components.

[0048] Although the present technology has been described in detail for the purpose of illustration based on what is currently considered to be the most practical and preferred implementations, it is to be understood that such detail is solely for that purpose and that the technology is not limited to the disclosed implementations, but, on the contrary, is intended to cover modifications and equivalent arrangements that are within the spirit and scope of the appended claims. For example, it is to be understood that the present technology contemplates that, to the extent possible, one or more features of any implementation can be combined with one or more features of any other implementation.