DEVICE AND METHOD FOR IMPLEMENTING A TENSOR-TRAIN DECOMPOSITION OPERATION

Abstract

A device for implementing a tensor-train decomposition operation for a respective convolutional layer of a convolutional neural network (CNN) is provided. The device is configured to receive input data comprising a first number of channels, and perform a 1×1 convolution on the input data to obtain a plurality of data groups. The plurality of data groups comprises a second number of channels. The device is further configured to perform a group convolution on the plurality of data groups to obtain intermediate data comprising a third number of channels, and perform a 1×1 convolution on the intermediate data to obtain output data comprising a fourth number of channels.

Claims

1. A device for implementing a tensor-train decomposition operation for a respective convolutional layer of a convolutional neural network (CNN), the device being configured to: receive input data comprising a first number of channels; perform a 1×1 convolution on the input data, to obtain a plurality of data groups, the plurality of data groups comprising a second number of channels; perform a group convolution on the plurality of data groups, to obtain intermediate data comprising a third number of channels; and perform a 1×1 convolution on the intermediate data, to obtain output data comprising a fourth number of channels.

2. The device according to claim 1, wherein: the group convolution is performed based on a kernel shared between the plurality of data groups.

3. The device according to claim 1, wherein: the third number of channels is determined based on a number of data groups in the plurality of data groups.

4. The device according to claim 3, wherein: the third number of channels is further determined based on one or more hardware characteristics of the device.

5. The device according to claim 1, wherein: each data group comprises a fifth number of channels, and wherein the second number of channels is determined based on the third number of channels and the fifth number of channels.

6. The device according to claim 1, further configured to: obtain the CNN comprising a first number of convolutional layers, wherein each convolutional layer is associated with a respective first ranking number; and provide a decomposed CNN comprising a second number of convolutional layers and a third number of decomposed convolutional layers based on a training of the CNN, wherein the first number of convolutional layers equals a sum of the second number of convolutional layers and the third number of decomposed convolutional layers, and wherein each decomposed convolutional layer is associated with a respective second ranking number.

7. The device according to claim 6, further configured to determine, for a respective convolutional layer of the CNN, a weighting pair based on: a weighted convolutional layer obtained by allocating a first weighting trainable parameter to the respective convolutional layer; and a weighted decomposed convolution layer obtained by allocating a second weighting trainable parameter to a decomposed convolution layer determined for the respective convolutional layer.

8. The device according to claim 7, further configured to: perform an initial training iteration of the CNN based on at least one the weighting pair.

9. The device according to claim 8, further configured to: determine, after performing the initial training iteration, at least one convolutional layer having a minimal first weighting trainable parameter.

10. The device according to claim 9, further configured to: perform an additional training iteration of the CNN, based on substituting a weighting pair of the at least one convolutional layer having the minimal first weighting trainable parameter with a corresponding decomposed convolution layer, and a remaining of the at least one weighting pair from a previous iteration.

11. The device according to claim 8, further configured to: iteratively perform, determining a respective convolutional layer having a minimal first weighting trainable parameter, substituting the weighting pair of the respective convolutional layer having the minimal first weighting trainable parameter with a corresponding decomposed convolution layer, and performing a next training iteration, until a predetermined number of convolutional layers are substituted with corresponding decomposed convolution layers.

12. The device according to claim 11, comprising an artificial intelligence accelerator adapted for tensor processing operation of the CNN.

13. A method for implementing a tensor-train decomposition operation for a convolutional layer of a convolutional neural network (CNN), the method comprising: receiving input data comprising a first number of channels; performing a 1×1 convolution on the input data to obtain a plurality of data groups, the plurality of data groups comprising a second number of channels; performing a group convolution on the plurality of data groups, to obtain intermediate data comprising a third number of channels; and performing a 1×1 convolution on the intermediate data, to obtain output data comprising a fourth number of channels.

14. A tangible, non-transitory computer-readable medium having instructions thereon, which, upon being executed by a computer, cause the steps of the method of claim 13 to be performed.

15. The method according to claim 13, wherein the group convolution is performed based on a kernel shared between the plurality of data groups.

16. The method according to claim 13, wherein the third number of channels is determined based on a number of data groups in the plurality of data groups.

17. The method according to claim 16, wherein the third number of channels is further determined based on one or more hardware characteristics of the device.

18. The method according to claim 13, wherein each data group comprises a fifth number of channels, and wherein the second number of channels is determined based on the third number of channels and the fifth number of channels.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0057] The above described aspects and implementation forms will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which

[0058] FIG. 1 illustrates a device for implementing a tensor-train decomposition operation for a convolutional layer of a CNN, according to an embodiment;

[0059] FIG. 2 illustrates a tensor-train decomposition for a three dimensional convolutional tensor according an embodiment;

[0060] FIG. 3 illustrates performing a 1×1 convolution according to an embodiment;

[0061] FIG. 4 illustrates a flowchart of a method for a tensor train decomposition operation according to an embodiment;

[0062] FIG. 5 illustrates a flowchart of a method for obtaining a decomposed CNN based on a training of a CNN according to an embodiment;

[0063] FIG. 6 illustrates replacing convolutional layers to weighted convolutions according to an embodiment;

[0064] FIG. 7 illustrates substituting a weighting pair of a convolutional layer with its decomposed convolution layer according to an embodiment;

[0065] FIG. 8 illustrates changing a set of weighting pairs with their corresponding convolutional layers according to an embodiment; and

[0066] FIG. 9 illustrates a flowchart of a method for implementing a tensor-train decomposition operation for a convolutional layer of a convolutional neural network, according to an embodiment.

DETAILED DESCRIPTION

[0067] FIG. 1 shows a device 100 for implementing a tensor-train decomposition operation for a convolutional layer of a CNN, according to an embodiment of the disclosure.

[0068] The device 100 may be an electronic device such as a computer, a personal computer, a smartphone, surveillance camera, etc.

[0069] The device 100 is configured to receive input data 110 comprising a first number of channels.

[0070] The device 100 is further configured to perform a 1×1 convolution on the input data 110, to obtain a plurality of data groups 120. The plurality of data groups 120 comprise a second number of channels.

[0071] The device 100 is further configured to perform a group convolution on the plurality of data groups 120, to obtain intermediate data 130. The intermediate data 130 comprises a third number of channels.

[0072] The device 100 is further configured to perform a 1×1 convolution on the intermediate data 130, to obtain output data 140. The output data 140 comprises a fourth number of channels.

[0073] The device 100 may implement the tensor train convolution operation for the convolutional layer of the CNN.

[0074] The device 100 may increase the accurate tuning and may enable additional acceleration on real hardware, for example, by not using different ranks for tensor-train cores such acceleration may be achieved.

[0075] For example, the device 100 may perform a sequence of a 1×1 convolution, a group convolution with shared weights and another 1×1 convolution, for a hardware-friendly Tensor-train decomposition implementation. Moreover, by using weight sharing in the group convolution, the device 100 may enable an additional acceleration on real hardware due to weights reuse and reduced data transfer, and may avoid time-consuming permute and reshape operations, etc.

[0076] The device 100 may comprise processing circuitry (not shown in FIG. 1) configured to perform, conduct or initiate the various operations of the device 100 described herein. The processing circuitry may comprise hardware and software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors. In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 100 to perform, conduct or initiate the operations or methods described herein.

[0077] FIG. 2 shows a schematically a procedure of performing of a tensor-train decomposition for a three dimensional convolutional tensor. For example, the device 100 may perform the illustrated tensor-train decomposition for the three dimensional convolutional tensor.

[0078] The device 100 may, in particular, receive the input data 110 comprising C channels (first number of channels).

[0079] The device 100 may further perform a 1×1 convolution from the C channels to R.sub.1R.sub.2 channels. For example, the device 100 may perform a 1×1 convolution on the input data 110, to obtain a plurality of data groups 120 comprising a second number of channels. In the diagram of FIG. 2, the second number of channels is R.sub.1R.sub.2.

[0080] The device 100 may further perform a l×l group convolution on the plurality of data groups 120, having R.sub.1R.sub.2 channels, to obtain the intermediate data 130 having R.sub.2 channels (the third number of channels). For example, the device 100 may perform the group convolution with a shared kernel weight. In the diagram 200 of FIG. 2, the plurality of data groups 120 comprise three data group 221, 222, 223, and the group convolution is performed based on the shared kernel shared between the data groups 221, 222, 223.

[0081] The device 100 may further perform the 1×1 convolution from the R.sub.2 channels to S channels. For example, the device 100 may perform the 1×1 convolution on the intermediate data 130, to obtain output data 140 comprising S channels (the fourth number of channels).

[0082] In the diagram 200 of FIG. 2, the tensor train decomposition operation is represented as a three convolutions, wherein the second convolution is a group convolution with shared kernel weights.

[0083] FIG. 3 shows schematically a procedure of performing of a 1×1 convolution.

[0084] The diagram 300 of FIG. 3 is an exemplary illustration, in which the device 100 may perform a first 1×1 convolution on input data 110 comprising the C number of channels, to obtain data group 320 comprising R channels (a second number of channels).

[0085] The device 100 may further perform a second 1×1 convolution on the data group 320, to obtain output data comprising S channel (the fourth number of channels). An example of the tensor train decomposition operation may be as follows:

[00005]$Y\left[h,w,s\right]=\underset{c=1}{\overset{C}{.Math.}}\underset{r=1}{\overset{R}{.Math.}}{G}^1\left[c,r\right]G^2\left[r,s\right]X\left[h,w,c\right]$

[0086] FIG. 4 shows a flowchart of a method 400 for a tensor-train decomposition operation. The method 400 may be performed by the device 100, as it is described above.

[0087] At step 401, the device 100 may obtain the input data 110. The input data 110 may comprise a batch of image filters X∈.sup.n×C×H×W.

[0088] At step 402, the device 100 may perform a 1×1 convolution on the input data 110. For example, the device 100 may convolve X with a kernel G.sub.0, wherein G.sub.0∈.sup.1×1×C×R.sup.1*.sup.R.sup.2, and the device 100 may further may obtain X.sub.0=Conv(X, G.sub.0), wherein X.sub.0∈.sup.n×R.sup.1.sup.R.sup.2.sup.×H×W.

[0089] At step 403, the device 100 may perform a group convolution. For example, the device 100 may group-convolve X.sub.0 with a kernel G.sub.1, wherein G.sub.1∈.sup.l×l×R.sup.1.sup.×1, and G.sub.1 is shared over R.sub.2 groups. The device 100 may further obtain X.sub.1 as follows:

X.sub.1=SharedGroupConv(X.sub.0,G.sub.1,R.sub.2), where X.sub.1∈.sup.n×R.sup.2.sup.×H′×W′.

[0090] At step 404, the device 100 may convolve X.sub.1 with a kernel G.sub.2, wherein G.sub.2∈.sup.1×1×R.sup.2.sup.×S. The device 100 may further obtain Y=Conv(X.sub.1, G.sub.2), where Y∈.sup.n×S×H′×W′.

[0091] At step 405, the device 100 may obtain the output data 140. The output data 140 may be a batch of output filters, wherein Y∈.sup.n×S×H′×W′.

[0092] Reference is now made to FIG. 5, which shows a flowchart of a method 500 for obtaining decomposed convolutional layers of a CNN. The method 500 may be performed by the device 100, as it is described above.

[0093] At step 501, the device may obtain a CNN comprising a first number (L) of convolutional layers. For example, the device 100 may receive the input architecture A with L convolutional layers in the data set D.

[0094] At step 502, the device 100 may replace each convolution layer Conv.sub.l(x.sub.l) with a weighted pair Op.sub.l(x.sub.l, α.sub.l). The device 100 may further initialize each α.sub.l with the value of 0.5.

[0095] An exemplarily illustration of replacing convolutional layers with weighted convolutions is shown in the diagram 600 of FIG. 6. The diagram 600 of FIG. 6 illustrates, for example, that the device 100 may replace all L convolutional layers with weighted convolutions.

[0096] At step 503, the device 100 may cycle C, for k=1 to k=K.

[0097] At step 504, the device 100 may train the CNN with this op(x) instead of usual convolution over m epochs. For example, the device 100 may perform an initial training iteration of the CNN A based on at least one weighting pair op(x, α) and at least one weighted convolutional layer α*Conv(x).

[0098] At step 505, the device 100 may determine, after performing the initial training iteration, a convolutional layer Conv(x) having a minimal weighting parameter α. For example, the device 100 may find a convolutional layer with minimal weight α.sub.l according to:

[00006]$l_k=\arg \underset{l\in L}{\min }{\alpha }_l$

[0099] At step 506, the device 100 may determine, whether α.sub.l.sub.k<0.5. Moreover, when the device 100 determines “Yes”, the device 100 goes to step 507, and when it determines “No”, the device 100 returns to step 509.

[0100] At step 507, the device 100 may substitute the weighting pair op(x, α) of the convolutional layer Conv(x) having the minimal weighting parameter α with its decomposed convolution layer DConv(x).

[0101] An exemplarily illustration substituting a weighting pair of a convolutional layer with its decomposed convolution is shown in the diagram 700 of FIG. 7. The diagram 700 of FIG. 7 illustrates, for example, the device 100 changing Op.sub.l.sub.k (x.sub.l, α.sub.l.sub.k) to corresponding DConv.sub.l.sub.k(x.sub.l).

[0102] At step 508, the device 100 may increase k by 1, and may further return to step 503 K times. (for example, K=10)

[0103] At step 509, the device 100 may change the remaining L−k Op.sub.l(x.sub.l, α.sub.l) to a corresponding convolutional layer Conv.sub.l(x.sub.l).

[0104] An exemplarily illustration of changing a set of weighting pairs with their corresponding convolutional layers is illustrated in FIG. 8. For example, the device 100 may obtain the training loss based on determining the cross-entropy according to:

[00007]$ℒ\left(D\right)=-\underset{x,y\sim D}{.Math.}\log \frac{e^{ne{t\left(x\right)}_y}}{{.Math.}_{j=1}^c{e}^{ne{t\left(x\right)}_j}}$

where net(x) is a neural network's output, D—data of training examples (x, y).

[0105] At step 510, the device 100 may train the model for m epochs. For example, the device 100 may perform an additional training iteration of the CNN A, based on substituting a weighting pair op(x, α) of the convolutional layer Conv(x) having the minimal weighting parameter α with its decomposed convolution layer DConv(x), a remaining of the at least one weighting pair op(x, α)) and a remaining of the at least one weighted convolutional layer α*Conv(x) from a previous iteration.

[0106] At step 511, the device 100 may evaluate a model M on test data.

[0107] At step 512, the device 100 may return trained model M with k decomposed layers. For example, the device 100 may obtain the decomposed CNN M comprising the second number of convolutional layers and a third number k of decomposed convolutional layers.

[0108] In the following, an example of the performance of the device 100 is discussed, without limiting the present discourse to this specific example.

[0109] At first, the device 100 selects the ranks R.sub.1, R.sub.2 for 3×3 convolutional layer, and R for the 1×1 convolution.

[0110] The device 100 may perform matrix multiplication operations. For example, the device 100 may split large matrices to parts of predefined size (e.g., 16, but any device-specific number can be used), and may further perform multiplication operation part-by-part. Furthermore, if channel number is not divisible by 16, channels may be padded with zeros until their number is divisible by 16.

[0111] The device 100 may further use R 2=16, because the last convolution in the tensor train convolution operates with this channel number, and for R.sub.1=S/(4*R.sub.2). So, the device 100 may use the following condition:

[00008]$R_1*R_2=\frac{S}{4}$

[0112] For example, if C=512, S=512, l=3: [0113] The first convolution is a mapping from 512 channels to 128 channels. [0114] The second convolution is a 3×3 group convolution from 128 channels to 16 channels, where number of groups is 16. So in this convolution, the device 100 shares 3×3×8×1 shape weight between 16 groups. [0115] The last convolution is a mapping from 16 channels to 512 channels.

[0116] Furthermore, a comparison of a total number of floating point operations of obtained by the device 100 and some conventional devices, respectively, is presented, without limiting the present disclosure. The following notation are thereby used: N is a batch size, C is a number of input channels, S is the number of output channels, l is the kernel size, R.sub.1, . . . , R.sub.d are original tensor-train decomposition operation (TTConv) ranks, R.sub.1, R.sub.2 are related to the TTConv ranks obtained by the device 100, R is the TRConv (tensor-ring convolution) rank obtained by conventional devices.

TABLE-US-00001 FLOP (computation) FLOP (data transfer) Usual Conv NHWl.sup.2CS NHW(C + S) Original TTConv [00009]$NHW\left(C{l}^2{R}_1+\underset{k=1}{\overset{d}{.Math.}}{R}_k{R}_{k+1}\underset{m\ge k}{.Math.}{C}_m\underset{m\le k}{.Math.}{S}_m\right)$ [00010]$\mathrm{NHW}\left(3C+7C{R}_1+\underset{k=1}{\overset{d}{.Math.}}\underset{m>k}{.Math.}{C}_m\underset{m<k}{.Math.}{S}_m\left(3R_k{C}_k+5R_{k+1}{S}_k\right)+4S\right)$ TRConv NHW(R.sup.2C + R.sup.3l.sup.2 + R.sup.2S) NHW(C + 4R.sup.2 + S) Device 100 NHW(R.sub.1R.sub.2C + R.sub.1R.sub.2l.sup.2 + R.sub.2S) NHW(C + 2R.sub.1R.sub.2 + 2R.sub.2 + S)
Some examples (for convenienceN=1, l=3):

TABLE-US-00002 FLOP FLOP (data FLOP H, W ranks C,S (computation) transfer) (total) Usual Conv 7, 7 — 512, 512 115.6M  0.05M   116M Original 7, 7 R.sub.1 = R.sub.2 = C.sub.1 = C.sub.2 = 10.5M  3.2M 13.7M TTConv R.sub.3 = 4; C.sub.3 = 8; S.sub.1 = S.sub.2 = S.sub.3 = 8; TRConv 7, 7 R = 16 512, 512 14.6M  0.1M 14.7M Device 100 7, 7 R.sub.1 = 8, R.sub.2 = 16 512, 512 3.7M 0.064M   3.8M Usual Conv 14, 14 — 256, 256 115.6M  0.1M 115.7M  Original 14, 14 R.sub.1 = R.sub.2 = C.sub.1 = C.sub.2 = 8, 4.9M 3.5M  8.4M TTConv R.sub.3 = 2; C.sub.3 = 4; S.sub.1 = S.sub.2 = 8, S.sub.3 = 4; TRConv 14, 14 R = 8 256, 256 7.3M 0.15M   7.4M Device 100 14, 14 R.sub.1 = 4, R.sub.2 = 16 256, 256 4.1M 0.13M  4.23M

[0117] Next, a comparison of the results obtained by the device 100 (based on performing the tensor train decomposition operation TTConv) with the previous implementation on object detection task is presented. A YOLO-based model is used, and the last three layers are decomposed in the following procedure: [0118] Converting last three convolutional layers from a pertained model to TTConv using TT-SVD algorithm with fixed ranks. One of the convolutions has C=256 and S=512 channels, and other two convolutions have has C=512 and S=512 channels, respectively. [0119] Training this model with three TTConv layers. [0120] Inference time has been measured by the device 100.

[0121] Results show that using the device 100 (implementing the tensor train decomposition operation or the TTConv) is more justified than original operation.

TABLE-US-00003 Pedestrian Inference Model Face AP AP (bs = 16), ms YOLO-baseline 85.1 88.9 3.175 YOLO-TTConv base 85.4 88.1 70 YOLO-TTConv our 85.7 88.6 2.963(−6.7%)

[0122] At next, the inference improvement is computed for individual layers using the device 100. This layers are part of ResNet50 backbone model. Further, the original convolutional layer is compared with the result obtained by the device 100.

TABLE-US-00004 Usual Conv Our TTConv C, S 1, stride (s) inference inference 256, 512 l = 1, s = 2 0.046 ms 0.033 ms (−28%) 512, 512 l = 3, s = 1 0.059 ms 0.03 ms (−47%) 512, 512 l = 3, s = 2 0.058 ms 0.32 ms (−45%) 512, 2048 l = 1, s = 1 0.042 ms 0.023 ms (−45%) 1024, 2048 l = 1, s = 2 0.056 ms 0.023 ms (−59%)

[0123] The results show that using TTConv accelerate individual convolutional layer in real device. So it may be concluded that the TTConv performed by the device 100 is hardware-friendly.

[0124] Moreover, the training operation performed by the device 100 may also improve the model quality. For example, ResNet34 is chosen as a model which has a good quality on ImageNet dataset. ResNet models comprise four 4 stages, where number of channels grow with stage, in case of ResNet34, the fourth stage comprises only 512 channel convolutions.

TABLE-US-00005 stages model TOP1 (accuracy) Inference, ms — ResNet34 (baseline) 73.36 1.15 4 ResNet34_stage 71.06 0.95 (−17.5%) ResNet34_auto 72.16 0.98 (−15%) 3, 4 ResNet34_stage 64.5 0.702 (−39%) ResNet34_auto 73.07 0.957 (−17%) 2, 3, 4 ResNet34_stage 60.32 0.601 (−48%) ResNet34_auto 73.44 1.01 (−12%) all stages ResNet34_stage 58.77 0.699 (−39%) ResNet34_auto 72.89 0.977 (−15%)

[0125] As ResNet34 stage, the device 100 uses a model, where all convolutions in these stages are replaced by TTConv, and the ResNet34_auto—model, where all convolutions in these stages are replaced by op (x, α) and are trained by our training procedure.

[0126] Furthermore, it may be concluded that using proposed TTConv improves model inference, for example, as it can be derived from the data presented on the last column. Furthermore, it may be concluded that using the training performed by the device 100, the optimal layers may be determined.

[0127] FIG. 9 shows a method 900 according to an embodiment of the disclosure for implementing a tensor-train decomposition operation for a convolutional layer of a convolutional neural network. The method 900 may be carried out by the device 100, as it is described above.

[0128] The method 900 comprises a step 901 of receiving input data 110 comprising a first number of channels.

[0129] The method 900 further comprises a step 902 of performing a 1×1 convolution on the input data 110, to obtain a plurality of data groups 120, the plurality of data groups 120 comprising a second number of channels.

[0130] The method 900 further comprises a step 903 of performing a group convolution on the plurality of data groups 120, to obtain intermediate data 130 comprising a third number of channels.

[0131] The method 900 further comprises a step 904 of performing a 1×1 convolution on the intermediate data 130, to obtain output data 140 comprising a fourth number of channels.

[0132] The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed disclosure, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word “comprising” does not exclude other elements or steps and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.