Matrix Multiply Accelerator For Variable Bitwidth Operands

Abstract

A system and method for multiplying first and second matrices are provided. For the first matrix, a number of bit slice vectors for each row are generated based on the bit resolution, and a first bit slice tensor is generated based on the bit slice vectors for each row. For the second matrix, a number of bit slice vectors for each column are generated based on the bit resolution, and a second bit slice tensor is generated based on the bit slice vectors for each row. The first and second bit slice tensors are multiplied by a matrix multiply accelerator (MMA) to generate an output matrix.

Claims

1. A system, comprising: a memory configured to store at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution; a processor, coupled to the memory, configured to: for the weight matrix: generate, based on the bit resolution, a number of bit slice vectors for each row, generate a bit slice weight tensor based on the bit slice vectors for each row, for the input data matrix: generate, based on the bit resolution, a number of bit slice vectors for each column, and generate a bit slice input data tensor based on the bit slice vectors for each column; and a matrix multiply accelerator (MMA), coupled to the processor and the memory, configured to: receive the bit slice weight tensor and the bit slice input data tensor, and multiply the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

2. The system according to claim 1, where: the number of columns of the weight matrix is the same as the number of rows of the input data matrix; for each row of the weight matrix: each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.

3. The system according to claim 2, where: for each column of the input data matrix: each bit slice vector includes one bit from each element within the column; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the input data matrix.

4. The system according to claim 3, where the MMA includes: a local memory; a controller coupled to the local memory; a first register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice input data tensor; a second register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice weight tensor; a third register, coupled to the controller and the local memory, configured to store at least a portion of the output data matrix; and an array of bit slice dot product (BSDP) elements, coupled to the controller and the first, second and third registers, configured to multiply the bit slice weight tensor and the bit slice input data tensor, each BSDP element configured to generate a dot product between one row of the weight matrix and one column of the input data matrix.

5. The system according to claim 4, where each BSDP element includes: a bit-wise AND circuit configured to input a first operand from the first register, input a second operand from the second register, and output a resultant value; a popcount circuit configured to receive the resultant value and output an intermediate value; an ADDER circuit configured to add the intermediate value to an accumulated value; and an accumulation register configured to store the accumulated value, and output a final accumulated value to the third register.

6. The system according to claim 5, where: the first operand is a bit slice vector from the bit slice input data tensor having an index k equal to the associated bit position of the bit slice vector; and the second operand is a bit slice vector from the bit slice weight tensor having an index j equal to the associated bit position of the bit slice vector.

7. The system according to claim 6, where the popcount circuit is configured to: receive an index value from the second register, the index value being equal to j+k; count a number of bits set to one in the resultant value to generate a population count value; and left-shift the population count value based on the index value to generate the intermediate value.

8. A system, comprising: a memory configured to store at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution; a processor coupled to the memory; and a matrix multiply accelerator (MMA), coupled to the processor and the memory, including a local memory, an array of bit slice dot product (BSDP) elements, and a controller coupled to the local memory and the array, where: the controller is configured to: receive the weight matrix and the input data matrix, for the weight matrix: generate, based on the bit resolution, a number of bit slice vectors for each row, generate a bit slice weight tensor based on the bit slice vectors for each row, for the input data matrix: generate, based on the bit resolution, a number of bit slice vectors for each column, and generate a bit slice input data tensor based on the bit slice vectors for each column, and the array is configured to: multiply the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

9. The system according to claim 8, where: the number of columns of the weight matrix is the same as the number of rows of the input data matrix; for each row of the weight matrix: each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.

10. The system according to claim 9, where: for each column of the input data matrix: each bit slice vector includes one bit from each element within the column; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the input data matrix.

11. The system according to claim 10, where the MMA further includes: a first register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice input data tensor; a second register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice weight tensor; and a third register, coupled to the controller and the local memory, configured to store at least a portion of the output data matrix, where the array is coupled to the first, second and third registers, and where each BSDP element is configured to generate a dot product between one row of the weight matrix and one column of the input data matrix.

12. The system according to claim 11, where each BSDP element includes: a bit-wise AND circuit configured to input a first operand from the first register, input a second operand from the second register, and output a resultant value; a popcount circuit configured to receive the resultant value and output an intermediate value; an ADDER circuit configured to add the intermediate value to an accumulated value; and an accumulation register configured to store the accumulated value, and output a final accumulated value to the third register.

13. The system according to claim 1.sub.2, where: the first operand is a bit slice vector from the bit slice input data tensor having an index k equal to the associated bit position of the bit slice vector; and the second operand is a bit slice vector from the bit slice weight tensor having an index j equal to the associated bit position of the bit slice vector.

14. The system according to claim 13, where the popcount circuit is configured to: receive an index value from the second register, the index value being equal to j+k; count a number of bits set to one in the resultant value to generate a population count value; and left-shift the population count value based on the index value to generate the intermediate value.

15. A method, comprising: at a memory: storing at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution; at a processor or a matrix multiply accelerator (MMA): for the weight matrix: generating, based on the bit resolution, a number of bit slice vectors for each row, generating a bit slice weight tensor based on the bit slice vectors for each row, for the input data matrix: generating, based on the bit resolution, a number of bit slice vectors for each column, and generating a bit slice input data tensor based on the bit slice vectors for each column; and at the MMA: multiplying the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

16. The method according to claim 15, where: the number of columns of the weight matrix is the same as the number of rows of the input data matrix; for each row of the weight matrix: each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.

17. The method according to claim 16, where: for each column of the input data matrix: each bit slice vector includes one bit from each element within the column; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the input data matrix.

18. The method according to claim 17, where: the MMA includes: a local memory; a controller coupled to the local memory; a first register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice input data tensor; a second register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice weight tensor; a third register, coupled to the controller and the local memory, configured to store at least a portion of the output data matrix; an array of bit slice dot product (BSDP) elements, coupled to the controller and the first, second and third registers, configured to multiply the bit slice weight tensor and the bit slice input data tensor; and the method further comprises: at each BSDP element: generating a dot product between one row of the weight matrix and one column of the input data matrix.

19. The method according to claim 18, where each BSDP element includes: a bit-wise AND circuit configured to input a first operand from the first register, input a second operand from the second register, and output a resultant value; a popcount circuit configured to receive the resultant value and output an intermediate value; an ADDER circuit configured to add the intermediate value to an accumulated value; and an accumulation register configured to store the accumulated value, and output a final accumulated value to the third register.

20. The method according to claim 19, where: the first operand is a bit slice vector from the bit slice input data tensor having an index k equal to the associated bit position of the bit slice vector; the second operand is a bit slice vector from the bit slice weight tensor having an index j equal to the associated bit position of the bit slice vector; and the method further comprises: at each popcount circuit: receiving an index value from the second register, the index value being equal to j+k; counting a number of bits set to one in the resultant value to generate a population count value; and left-shifting the population count value based on the index value to generate the intermediate value.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. 1 depicts an ANN, in accordance with embodiments of the present disclosure.

[0006] FIG. 2 depicts a CNN, in accordance with embodiments of the present disclosure.

[0007] FIG. 3A depicts convolutional layer calculation for a CNN, FIG. 3B depicts a converted convolutional layer calculation for a CNN, and FIG. 3C depicts a converted input data matrix, in accordance with an embodiment of the present disclosure.

[0008] FIG. 4 depicts a data flow diagram for a multiply-and-accumulate (MAC) array.

[0009] FIG. 5 depicts the computation of the dot product between vector A and vector B using a MAC unit, in accordance with an embodiment of the present disclosure.

[0010] FIG. 6A depicts the creation of bit slice vectors from the vector A depicted in FIG. 5, in accordance with an embodiment of the present disclosure.

[0011] FIG. 6B depicts the creation of bit slice vectors from the vector B depicted in FIG. 5, in accordance with an embodiment of the present disclosure.

[0012] FIG. 6C depicts the computation of the 1-bit dot product between two bit slice vectors using a 1-bit dot product unit, in accordance with an embodiment of the present disclosure.

[0013] FIGS. 6D, 6E and 6F depict examples of the computation of the dot product between vector A and vector B using a 1-bit dot product unit, in accordance with an embodiment of the present disclosure.

[0014] FIGS. 7A and 7B depict the creation of a bit slice tensor from a matrix X, in accordance with an embodiment of the present disclosure.

[0015] FIGS. 7C and 7D depict the creation of a bit slice tensor from a matrix Y, in accordance with an embodiment of the present disclosure.

[0016] FIG. 8A depicts a data flow diagram for a BSDP array, while FIG. 8B depicts a BSDP unit, in accordance with embodiments of the present disclosure.

[0017] FIGS. 8C, 8D, 8E, 8F, 8G and 8H depict examples of the multiplication of matrix X and matrix Y to generate matrix Z using a BSDP array, in accordance with an embodiment of the present disclosure.

[0018] FIG. 9 depicts a block diagram of an MMA, in accordance with embodiments of the present disclosure.

[0019] FIG. 10 depicts a block diagram of system, in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will now be described with reference to the drawing figures, in which like reference numerals refer to like parts throughout.

[0021] Embodiments of the present disclosure advantageously provide a system and method for multiplying first and second matrices with variable bit-width operands using an MMA with an array of bitslice dot product (BSDP) units. For the first matrix, a number of bit slice vectors for each row are generated based on the bit resolution, and a first bit slice tensor is generated based on the bit slice vectors for each row. For the second matrix, a number of bit slice vectors for each column are generated based on the bit resolution, and a second bit slice tensor is generated based on the bit slice vectors for each row. The first and second bit slice tensors are multiplied by the MMA to generate an output matrix.

[0022] An ANN models the relationships between input data or signals and output data or signals using a network of interconnected nodes that is trained through a learning process. The nodes are arranged into various layers, including, for example, an input layer, one or more hidden layers, and an output layer. The input layer receives input data, such as, for example, image data, and the output layer generates output data, such as, for example, a probability that the image data contains a known object. Each hidden layer provides at least a partial transformation of the input data to the output data. A DNN has multiple hidden layers in order to model complex, nonlinear relationships between input data and output data.

[0023] In a fully-connected, feedforward ANN, each node is connected to all of the nodes in the preceding layer, as well as to all of the nodes in the subsequent layer. For example, each input layer node is connected to each hidden layer node, each hidden layer node is connected to each input layer node and each output layer node, and each output layer node is connected to each hidden layer node. Additional hidden layers are similarly interconnected. Each connection has a weight value, and each node has an activation function, such as, for example, a linear function, a step function, a sigmoid function, a tanh function, a rectified linear unit (ReLU) function, etc., that determines the output of the node based on the weighted sum of the inputs to the node. The input data propagates from the input layer nodes, through respective connection weights to the hidden layer nodes, and then through respective connection weights to the output layer nodes.

[0024] More particularly, at each input node, input data is provided to the activation function for that node, and the output of the activation function is then provided as an input data value to each hidden layer node. At each hidden layer node, the input data value received from each input layer node is multiplied by a respective connection weight, and the resulting products are summed or accumulated into an activation value that is provided to the activation function for that node. The output of the activation function is then provided as an input data value to each output layer node. At each output layer node, the output data value received from each hidden layer node is multiplied by a respective connection weight, and the resulting products are summed or accumulated into an activation value that is provided to the activation function for that node. The output of the activation function is then provided as output data. Additional hidden layers may be similarly configured to process data.

[0025] FIG. 1 depicts ANN 10, in accordance with an embodiment of the present disclosure.

[0026] ANN 10 includes input layer 20, one or more hidden layers 30, 40, 50, etc., and output layer 60. Input layer 20 includes one or more input nodes 21, 22, 23, etc. Hidden layer 30 includes one or more hidden nodes 31, 32, 33, 34, 35, etc. Hidden layer 40 includes one or more hidden nodes 41, 42, 43, 44, 45, etc. Hidden layer 50 includes one or more hidden nodes 51, 52, 53, 54, 55, etc. Output layer 60 includes one or more output nodes 61, 62, etc. Generally, ANN 10 includes N hidden layers, input layer 20 includes “i” nodes, hidden layer 30 includes “j” nodes, hidden layer 40 includes “k” nodes, hidden layer 50 includes “m” nodes, and output layer 60 includes “o” nodes.

[0027] In one embodiment, N equals 3, i equals 3, j, k and m equal 5 and o equals 2 (depicted in FIG. 1). Input node 21 is coupled to hidden nodes 31 to 35, input node 22 is coupled to hidden nodes 31 to 35, and input node 23 is coupled to hidden nodes 31 to 35. Hidden node 31 is coupled to hidden nodes 41 to 45, hidden node 32 is coupled to hidden nodes 41 to 45, hidden node 33 is coupled to hidden nodes 41 to 45, hidden node 34 is coupled to hidden nodes 41 to 45, and hidden node 35 is coupled to hidden nodes 41 to 45. Hidden node 41 is coupled to hidden nodes 51 to 55, hidden node 42 is coupled to hidden nodes 51 to 55, hidden node 43 is coupled to hidden nodes 51 to 55, hidden node 44 is coupled to hidden nodes 51 to 55, and hidden node 45 is coupled to hidden nodes 51 to 55. Hidden node 51 is coupled to output nodes 61 and 62, hidden node 52 is coupled to output nodes 61 and 62, hidden node 53 is coupled to output nodes 61 and 62, hidden node 54 is coupled to output nodes 61 and 62, and hidden node 55 is coupled to output nodes 61 and 62.

[0028] Many other variations of input, hidden and output layers are clearly possible, including hidden layers that are locally-connected, rather than fully-connected, to one another.

[0029] Training an ANN includes optimizing the connection weights between nodes by minimizing the prediction error of the output data until the ANN achieves a particular level of accuracy. One method is backpropagation, or backward propagation of errors, which iteratively and recursively determines a gradient descent with respect to the connection weights, and then adjusts the connection weights to improve the performance of the network.

[0030] A multi-layer perceptron (MLP) is a fully-connected ANN that has an input layer, an output layer and one or more hidden layers. MLPs may be used for natural language processing applications, such as machine translation, speech recognition, etc. Other ANNs include recurrent neural networks (RNNs), long short-term memories (LSTMs), sequence-to-sequence models that include an encoder RNN and a decoder RNN, shallow neural networks, etc.

[0031] A CNN is a variation of an MLP that may be used for classification or recognition applications, such as image recognition, speech recognition, etc. A CNN has an input layer, an output layer and multiple hidden layers including convolutional layers, pooling layers, normalization layers, fully-connected layers, etc. Each convolutional layer applies a sliding dot product or cross-correlation to an input volume, applies an activation function to the results, and then provides the activation or output volume to the next layer. Convolutional layers typically use the ReLU function as the activation function. In certain embodiments, the activation function is provided in a separate activation layer, such as, for example, a ReLU layer. A pooling layer reduces the dimensions of the output volume received from the preceding convolutional layer, and may calculate an average or a maximum over small clusters of data, such as, for example, 2×2 matrices. In certain embodiments, a convolutional layer and a pooling layer may form a single layer of a CNN. The fully-connected layers follow the convolutional and pooling layers, and include a flatten layer and a classification layer, followed by a normalization layer that includes a normalization function, such as the SoftMax function. The output layer follows the last fully-connected layer; in certain embodiments, the output layer may include the normalization function.

[0032] FIG. 2 depicts CNN 100, in accordance with an embodiment of the present disclosure. CNN 100 includes input layer 120, one or more hidden layers, such as convolutional layer 130-1, pooling layer 130-2, hidden (flatten) layer 140, hidden (classification) layer 150, etc., and output layer 160. Many other variations of input, hidden and output layers are contemplated.

[0033] Input layer 120 includes one or more input nodes 121, etc., that present the input data, such as a color image, as an input volume to the first convolutional layer, e.g., convolutional layer 130-1. The input volume is a three-dimensional matrix that has a width, a height and a depth. For example, input data that represent a color image are presented as an input volume that is 512 pixels×512 pixels×3 channels (red, green, blue); other input volume dimensions may also be used, such as 32×32×3, 64×64×3, 128×128×3, etc., 32×32×1, 64×64×1, 128×128×1, 512×512×1, etc.

[0034] Convolutional layer 130-1 is locally-connected to input layer 120, and includes a plurality of nodes that are connected to local regions in the input volume (not depicted for clarity). For a CNN that uses a standard convolution, each node computes a dot product between the node's weights and the respective local region of the input volume. An activation function is then applied to the results of each convolution calculation to produce an output volume that is provided as an input volume to the subsequent layer. The activation function may be applied by each convolutional layer node or by the nodes of a subsequent locally-connected ReLU layer.

[0035] Pooling layer 130-2 is locally-connected to convolutional layer 130-1, and includes a plurality of nodes that are connected to local regions in the input volume (not depicted for clarity). Pooling layer 130-2 also produces an output volume that is provided as the input volume to the subsequent layer, such as, for example, another convolutional layer 130-1, a flatten layer 140, etc. In certain embodiments, convolutional layer 130-1 and pooling layer 130-2 form a single hidden layer 130. Similarly, in certain embodiments, convolutional layer 130-1, a ReLU layer and pooling layer 130-2 form a single hidden layer 130. Generally, the output volumes of the convolutional and pooling layers may be described as feature maps, and one or more single hidden layers 130 form a feature learning portion of CNN 100.

[0036] Hidden layer 140 is a “flatten” layer that is locally-connected to pooling layer 130-2, and includes one or more hidden (flatten) nodes 141, 142, 143, 144, 145, etc. Hidden (flatten) layer 140 “flattens” the output volume produced by the preceding pooling layer 130-2 into a column vector, which is provided to the subsequent, fully-connected hidden layer 150.

[0037] Hidden layer 150 is a classification layer that is fully-connected to hidden (flatten) layer 140, and includes one or more hidden (classification) nodes 151, 152, 153, 154, 155, etc.

[0038] Output layer 160 includes one or more output nodes 161, 162, etc., and is fully-connected to hidden (classification) layer 150. Fully-connected output layer 160 receives the classification results output by hidden (classification) layer 150, and each node outputs a predicted class score. A normalization function, such as a SoftMax function, may be applied to the predicted class scores by output layer 160, or, alternatively, by an additional layer interposed between hidden (classification) layer 150 and output layer 160.

[0039] Similar to ANNs, training a CNN includes optimizing the connection weights between nodes by minimizing the prediction error of the output data until the CNN achieves a particular level of accuracy. As noted above, backpropagation may be used to iteratively and recursively determines a gradient descent with respect to the connection weights, and then adjusts the connection weights to improve the performance of the network. Matrix multiplication operations, and, more particularly, multiply-and-accumulate (MAC) operations, are used extensively by CNNs, as well as other ANNs.

[0040] FIG. 3A depicts convolutional layer calculation 200 for a CNN, in accordance with an embodiment of the present disclosure.

[0041] Input feature maps 204 include four channels and one input data matrix for each channel, i.e., input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4. Filter 202 includes four filter or weight sets 202.sup.1, 202.sup.2, 202.sup.3 and 202.sup.4, and each filter or weight set includes four weight matrices, one weight matrix for each channel. Output feature maps 206 include four channels and one output data matrix for each filter or weight set, i.e., output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4. Convolutional layer calculation 200 convolves filter 202 with input feature maps 204 to produce output feature maps 206.

[0042] Generally, input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 form an input tensor, each weight set 202.sup.1, 202.sup.2, 202.sup.3 and 202.sup.4 forms a weight tensor, and output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4 form an output tensor. In this embodiment, each tensor has a height, a width and a depth. The depth of the input tensor is equal to the number of channels, the depth of each weight tensor is equal to the number of channels, and the depth of the output tensor is equal to the number of weight tensors (i.e., weight sets). While particular dimensions for the tensors and matrices have been selected for clarity of illustration and explanation, embodiments of the present disclosure are not so limited.

[0043] In one embodiment, input data matrix 204.sup.1 is a 5×5 matrix associated with the first channel and includes activations a.sup.1.sub.1, a.sup.1.sub.2, a.sup.1.sub.3, a.sup.1.sub.4, a.sup.1.sub.5, a.sup.1.sub.6, a.sup.1.sub.7, a.sup.1.sub.8, a.sup.1.sub.9, a.sup.1.sub.10, a.sup.1.sub.11, a.sup.1.sub.12, a.sup.1.sub.13, a.sup.1.sub.14, a.sup.1.sub.15, a.sup.1.sub.16, a.sup.1.sub.17, a.sup.1.sub.18, a.sup.1.sub.19, a.sup.1.sub.20, a.sup.1.sub.21, a.sup.1.sub.22, a.sup.1.sub.23, a.sup.1.sub.24 and a.sup.1.sub.25. Input data matrix 204.sup.2 is a 5×5 matrix associated with the second channel and includes activations a.sup.2.sub.1, a.sup.2.sub.2, a.sup.2.sub.3, a.sup.2.sub.4, a.sup.2.sub.5, a.sup.2.sub.6, a.sup.2.sub.7, a.sup.2.sub.8, a.sup.2.sub.9, a.sup.2.sub.10, a.sup.2.sub.11, a.sup.2.sub.12, a.sup.2.sub.13, a.sup.2.sub.14, a.sup.2.sub.15, a.sup.2.sub.16, a.sup.2.sub.17, a.sup.2.sub.18, a.sup.2.sub.19, a.sup.2.sub.20, a.sup.2.sub.21, a.sup.2.sub.22, a.sup.2.sub.23, a.sup.2.sub.24 and a.sup.2.sub.25. Input data matrix 204.sup.3 is a 5×5 matrix associated with the third channel and includes activations a.sup.3.sub.1, a.sup.3.sub.2, a.sup.3.sub.3, a.sup.3.sub.4, a.sup.3.sub.5, a.sup.3.sub.6, a.sup.3.sub.7, a.sup.3.sub.8, a.sup.3.sub.9, a.sup.3.sub.10, a.sup.3.sub.11, a.sup.3.sub.12, a.sup.3.sub.13, a.sup.3.sub.14, a.sup.3.sub.15, a.sup.3.sub.16, a.sup.3.sub.17, a.sup.3.sub.18, a.sup.3.sub.19, a.sup.3.sub.20, a.sup.3.sub.21, a.sup.3.sub.22, a.sup.3.sub.23, a.sup.3.sub.24 and a.sup.3.sub.25. Input data matrix 204.sup.4 is a 5×5 matrix associated with the fourth channel and includes activations a.sup.4.sub.1, a.sup.4.sub.2, a.sup.4.sub.3, a.sup.4.sub.4, a.sup.4.sub.5, a.sup.4.sub.6, a.sup.4.sub.7, a.sup.4.sub.8, a.sup.4.sub.9, a.sup.4.sub.10, a.sup.4.sub.10, a.sup.4.sub.12, a.sup.4.sub.13, a.sup.4.sub.14, a.sup.4.sub.15, a.sup.4.sub.16, a.sup.4.sub.17, a.sup.4.sub.18, a.sup.4.sub.19, a.sup.4.sub.20, a.sup.4.sub.21, a.sup.4.sub.22, a.sup.4.sub.23, a.sup.4.sub.24 and a.sup.4.sub.25.

[0044] In this embodiment, weight set 202.sup.1 includes four weight matrices 202.sup.1.sub.1, 202.sup.1.sub.2, 202.sup.1.sub.3 and 202.sup.1.sub.4. Weight matrix 202.sup.1.sub.1 is a 2×2 matrix associated with the first channel, and includes weights w.sup.1.sub.1, w.sup.1.sub.2, w.sup.1.sub.3 and w.sup.1.sub.4. Weight matrix 202.sup.1.sub.2 is a 2×2 matrix associated with the second channel, and includes weights w.sup.1.sub.5, w.sup.1.sub.6, w.sup.1.sub.7 and w.sup.1.sub.8. Weight matrix 202.sup.1.sub.3 is a 2×2 matrix associated with the third channel, and includes weights w.sup.1.sub.9, w.sup.1.sub.10, w.sup.1.sub.11 and w.sup.1.sub.12. Weight matrix 202.sup.1.sub.4 is a 2×2 matrix associated with the fourth channel, and includes weights w.sup.1.sub.13, w.sup.1.sub.14, w.sup.1.sub.15 and w.sup.1.sub.16.

[0045] Weight set 202.sup.2 includes four weight matrices 202.sup.2.sub.1, 202.sup.2.sub.2, 202.sup.2.sub.3 and 202.sup.2.sub.4. Weight matrix 202.sup.2.sub.1 is a 2×2 matrix associated with the first channel, and includes weights w.sup.2.sub.1, w.sup.2.sub.2, w.sup.2.sub.3 and w.sup.2.sub.4. Weight matrix 202.sup.2.sub.2 is a 2×2 matrix associated with the second channel, and includes weights w.sup.2.sub.5, w.sup.2.sub.6, w.sup.2.sub.7 and w.sup.2.sub.8. Weight matrix 202.sup.2.sub.3 is a 2×2 matrix associated with the third channel, and includes weights w.sup.2.sub.9, w.sup.2.sub.10, w.sup.2.sub.11 and w.sup.2.sub.12. Weight matrix 202.sup.2.sub.4 is a 2×2 matrix associated with the fourth channel, and includes weights w.sup.2.sub.13, w.sup.2.sub.14, w.sup.2.sub.15 and w.sup.2.sub.16.

[0046] Weight set 202.sup.3 includes four weight matrices 202.sup.3.sub.1, 202.sup.3.sub.2, 202.sup.3.sub.3 and 202.sup.3.sub.4. Weight matrix 202.sup.3.sub.1 is a 2×2 matrix associated with the first channel, and includes weights w.sup.3.sub.1, w.sup.3.sub.2, w.sup.3.sub.3 and w.sup.3.sub.4. Weight matrix 202.sup.3.sub.2 is a 2×2 matrix associated with the second channel, and includes weights w.sup.3.sub.5, w.sup.3.sub.6, w.sup.3.sub.7 and w.sup.3.sub.8. Weight matrix 202.sup.3.sub.3 is a 2×2 matrix associated with the third channel, and includes weights w.sup.3.sub.9, w.sup.3.sub.10, w.sup.3.sub.11 and w.sup.3.sub.12. Weight matrix 202.sup.3.sub.4 is a 2×2 matrix associated with the fourth channel, and includes weights w.sup.3.sub.13, w.sup.3.sub.14, w.sup.3.sub.15 and w.sup.3.sub.16.

[0047] Weight set 202.sup.4 includes four weight matrices 202.sup.4.sub.1, 202.sup.4.sub.2, 202.sup.4.sub.3 and 202.sup.4.sub.4. Weight matrix 202.sup.4.sub.1 is a 2×2 matrix associated with the first channel, and includes weights w.sup.4.sub.1, w.sup.4.sub.2, w.sup.4.sub.3 and w.sup.4.sub.4. Weight matrix 202.sup.4.sub.2 is a 2×2 matrix associated with the second channel, and includes weights w.sup.4.sub.5, w.sup.4.sub.6, w.sup.4.sub.7 and w.sup.4.sub.8. Weight matrix 202.sup.4.sub.3 is a 2×2 matrix associated with the third channel, and includes weights w.sup.4.sub.9, w.sup.4.sub.10, w.sup.4.sub.11 and w.sup.4.sub.12. Weight matrix 202.sup.4.sub.4 is a 2×2 matrix associated with the fourth channel, and includes weights w.sup.4.sub.13, w.sup.4.sub.14, w.sup.4.sub.15 and w.sup.4.sub.16.

[0048] In this embodiment, output data matrix 206.sup.1 is a 4×4 matrix associated with weight set 202.sup.1 and includes activations o.sup.1.sub.1, o.sup.1.sub.2, o.sup.1.sub.3, o.sup.1.sub.4, o.sup.1.sub.5, o.sup.1.sub.6, o.sup.1.sub.7, o.sup.1.sub.8, o.sup.1.sub.9, o.sup.1.sub.10, o.sup.1.sub.11, o.sup.1.sub.12, o.sup.1.sub.13, o.sup.1.sub.14, o.sup.1.sub.15 and o.sup.1.sub.16. Output data matrix 206.sup.2 is a 4×4 matrix associated with weight set 202.sup.2 and includes activations o.sup.2.sub.1, o.sup.2.sub.2, o.sup.2.sub.3, o.sup.2.sub.4, o.sup.2.sub.5, o.sup.2.sub.6, o.sup.2.sub.7, o.sup.2.sub.8, o.sup.2.sub.9, o.sup.2.sub.10, o.sup.2.sub.11, o.sup.2.sub.12, o.sup.2.sub.13, o.sup.2.sub.14, o.sup.2.sub.15 and o.sup.2.sub.16. Output data matrix 206.sup.3 is a 4×4 matrix associated with weight set 202.sup.3 and includes activations o.sup.3.sub.1, o.sup.3.sub.2, o.sup.3.sub.3, o.sup.3.sub.4, o.sup.3.sub.5, o.sup.3.sub.6, o.sup.3.sub.7, o.sup.3.sub.8, o.sup.3.sub.9, o.sup.3.sub.10, o.sup.3.sub.11, o.sup.3.sub.12, o.sup.3.sub.13, o.sup.3.sub.14, o.sup.3.sub.15 and o.sup.3.sub.16. Output data matrix 206.sup.4 is a 4×4 matrix associated with weight set 202.sup.4 and includes activations o.sup.4.sub.1, o.sup.4.sub.2, o.sup.4.sub.3, o.sup.4.sub.4, o.sup.4.sub.5, o.sup.4.sub.8, o.sup.4.sub.7, o.sup.4.sub.8, o.sup.4.sub.9, o.sup.4.sub.10, o.sup.4.sub.11, o.sup.4.sub.12, o.sup.4.sub.13, o.sup.4.sub.14, o.sup.4.sub.15 and o.sup.4.sub.16.

[0049] For ease of explanation, each input data matrix 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4 may be divided into four quadrants. The first quadrant spans the top (first) row and the second row, the second quadrant spans the second row and the third row, the third quadrant spans the third row and the fourth row, and the fourth quadrant spans the fourth row and the fifth (bottom) row. The first quadrant for input data matrix 204.sup.1 (a.sup.1.sub.q1), the first quadrant for input data matrix 204.sup.2 (a.sup.2.sub.q1), the first quadrant for input data matrix 204.sup.3 (a.sup.3.sub.q1), and the first quadrant for input data matrix 204.sup.4 (a.sup.4.sub.q1) are depicted; the remaining three quadrants for each input data matrix are not depicted for clarity.

[0050] First quadrant a.sup.1.sub.q1 includes elements a.sup.1.sub.1, a.sup.1.sub.2, a.sup.1.sub.3, a.sup.1.sub.4, a.sup.1.sub.5, a.sup.1.sub.6, a.sup.1.sub.7, a.sup.1.sub.8, a.sup.1.sub.9 and a.sup.1.sub.10, from which four blocks of elements are formed, i.e., a first block (a.sup.1.sub.1, a.sup.1.sub.2, a.sup.1.sub.6 and a.sup.1.sub.7), a second block (a.sup.1.sub.2, a.sup.1.sub.3, a.sup.1.sub.7 and a.sup.1.sub.8), a third block (a.sup.1.sub.3, a.sup.1.sub.4, a.sup.1.sub.8 and a.sup.1.sub.9), and a fourth block (a.sup.1.sub.4, a.sup.1.sub.5, a.sup.1.sub.9 and a.sup.1.sub.10). First quadrant a.sup.2.sub.q1 includes elements a.sup.2.sub.1, a.sup.2.sub.2, a.sup.2.sub.3, a.sup.2.sub.4, a.sup.2.sub.5, a.sup.2.sub.6, a.sup.2.sub.7, a.sup.2.sub.8, a.sup.2.sub.9 and a.sup.2.sub.10, from which four blocks of elements are formed, i.e., a first block (a.sup.2.sub.1, a.sup.2.sub.2, a.sup.2.sub.6 and a.sup.2.sub.7), a second block (a.sup.2.sub.2, a.sup.2.sub.3, a.sup.2.sub.7 and a.sup.2.sub.8), a third block (a.sup.2.sub.3, a.sup.2.sub.4, a.sup.2.sub.8 and a.sup.2.sub.9), and a fourth block (a.sup.2.sub.4, a.sup.2.sub.5, a.sup.2.sub.9 and a.sup.2.sub.10). First quadrant a.sup.3.sub.q1 includes elements a.sup.3.sub.1, a.sup.3.sub.2, a.sup.3.sub.3, a.sup.3.sub.4, a.sup.3.sub.5, a.sup.3.sub.6, a.sup.3.sub.7, a.sup.3.sub.8, a.sup.3.sub.9 and a.sup.3.sub.10, from which four blocks of elements are formed, i.e., a first block (a.sup.3.sub.1, a.sup.3.sub.2, a.sup.3.sub.6 and a.sup.3.sub.7), a second block (a.sup.3.sub.2, a.sup.3.sub.3, a.sup.3.sub.7 and a.sup.3.sub.8), a third block (a.sup.3.sub.3, a.sup.3.sub.4, a.sup.3.sub.8 and a.sup.3.sub.9), and a fourth block (a.sup.3.sub.4, a.sup.3.sub.5, a.sup.3.sub.9 and a.sup.3.sub.10). First quadrant a.sup.4.sub.q1 includes elements a.sup.4.sub.1, a.sup.4.sub.2, a.sup.4.sub.3, a.sup.4.sub.4, a.sup.4.sub.5, a.sup.4.sub.6, a.sup.4.sub.7, a.sup.4.sub.8, a.sup.4.sub.9 and a.sup.4.sub.10, from which four blocks of elements are formed, i.e., a first block (a.sup.4.sub.1, a.sup.4.sub.2, a.sup.4.sub.6 and a.sup.4.sub.7), a second block (a.sup.4.sub.2, a.sup.4.sub.3, a.sup.4.sub.7 and a.sup.4.sub.8), a third block (a.sup.4.sub.3, a.sup.4.sub.4, a.sup.4.sub.8 and a.sup.4.sub.9), and a fourth block (a.sup.4.sub.4, a.sup.4.sub.5, a.sup.4.sub.9 and a.sup.4.sub.10).

[0051] Second quadrant a.sup.1.sub.q2 includes elements a.sup.1.sub.67 a.sup.1.sub.7, a.sup.1.sub.8, a.sup.1.sub.9, a.sup.1.sub.10, a.sup.1.sub.11, a.sup.1.sub.12, a.sup.1.sub.13, a.sup.1.sub.14 and a.sup.1.sub.15, from which four blocks of elements are formed, i.e., a first block (a.sup.1.sub.6, a.sup.1.sub.7, a.sup.1.sub.11 and a.sup.1.sub.12), a second block (a.sup.1.sub.7, a.sup.1.sub.8, a.sup.1.sub.12 and a.sup.1.sub.13), a third block (a.sup.1.sub.8, a.sup.1.sub.9, a.sup.1.sub.13 and a.sup.1.sub.14), and a fourth block (a.sup.1.sub.9, a.sup.1.sub.10, a.sup.1.sub.14 and a.sup.1.sub.15). Second quadrant a.sup.2.sub.q2 includes elements a.sup.2.sub.6, a.sup.2.sub.7, a.sup.2.sub.8, a.sup.2.sub.9, a.sup.2.sub.10, a.sup.2.sub.11, a.sup.2.sub.12, a.sup.2.sub.13, a.sup.2.sub.14 and a.sup.2.sub.15, from which four blocks of elements are formed, i.e., a first block (a.sup.2.sub.6, a.sup.2.sub.7, a.sup.2.sub.11 and a.sup.2.sub.12), a second block (a.sup.2.sub.7, a.sup.2.sub.8, a.sup.2.sub.12 and a.sup.2.sub.13), a third block (a.sup.2.sub.8, a.sup.2.sub.9, a.sup.2.sub.13 and a.sup.2.sub.14), and a fourth block (a.sup.2.sub.9, a.sup.2.sub.10, a.sup.2.sub.14 and a.sup.2.sub.15). Second quadrant a.sup.3.sub.q2 includes elements a.sup.3.sub.6, a.sup.3.sub.7, a.sup.3.sub.8, a.sup.3.sub.9, a.sup.3.sub.10, a.sup.3.sub.11, a.sup.3.sub.12, a.sup.3.sub.13, a.sup.3.sub.14 and a.sup.3.sub.15, from which four blocks of elements are formed, i.e., a first block (a.sup.3.sub.6, a.sup.3.sub.7, and a.sup.3.sub.12), a second block (a.sup.3.sub.7, a.sup.3.sub.8, a.sup.3.sub.12 and a.sup.3.sub.13), a third block (a.sup.3.sub.8, a.sup.3.sub.9, a.sup.3.sub.13 and a.sup.3.sub.14), and a fourth block (a.sup.3.sub.9, a.sup.3.sub.10, a.sup.3.sub.14 and a.sup.3.sub.15). Second quadrant a.sup.4.sub.q2 includes elements a.sup.4.sub.6, a.sup.4.sub.7, a.sup.4.sub.8, a.sup.4.sub.9, a.sup.4.sub.10, a.sup.4.sub.11, a.sup.4.sub.12, a.sup.4.sub.13, a.sup.4.sub.14 and a.sup.4.sub.15, from which four blocks of elements are formed, i.e., a first block (a.sup.4.sub.6, a.sup.4.sub.7, a.sup.4.sub.11 and a.sup.4.sub.12), a second block (a.sup.4.sub.7, a.sup.4.sub.8, a.sup.4.sub.12 and a.sup.4.sub.13), a third block (a.sup.4.sub.8, a.sup.4.sub.9, a.sup.4.sub.13 and a.sup.4.sub.14), and a fourth block (a.sup.4.sub.9, a.sup.4.sub.10, a.sup.4.sub.14 and a.sup.4.sub.15).

[0052] Third quadrant a.sup.1.sub.q3 includes elements a.sup.1.sub.11, a.sup.1.sub.12, a.sup.1.sub.13, a.sup.1.sub.14, a.sup.1.sub.15, a.sup.1.sub.16, a.sup.1.sub.17, a.sup.1.sub.18, a.sup.1.sub.19 and a.sup.1.sub.20, from which four blocks of elements are formed, i.e., a first block (a.sup.1.sub.11, a.sup.1.sub.12, a.sup.1.sub.16 and a.sup.1.sub.17), a second block (a.sup.1.sub.12, a.sup.1.sub.13, a.sup.1.sub.17 and a.sup.1.sub.18), a third block (a.sup.1.sub.13, a.sup.1.sub.14, a.sup.1.sub.18 and a.sup.1.sub.19), and a fourth block (a.sup.1.sub.14, a.sup.1.sub.15, a.sup.1.sub.19 and a.sup.1.sub.20). Third quadrant a.sup.2.sub.q3 includes elements a.sup.2.sub.11, a.sup.2.sub.12, a.sup.2.sub.13, a.sup.2.sub.14, a.sup.2.sub.15, a.sup.2.sub.16, a.sup.2.sub.17, a.sup.2.sub.18, a.sup.2.sub.19 and a.sup.2.sub.20, from which four blocks of elements are formed, i.e., a first block (a.sup.2.sub.11, a.sup.2.sub.12, a.sup.2.sub.16 and a.sup.2.sub.17), a second block (a.sup.2.sub.12, a.sup.2.sub.13, a.sup.2.sub.17 and a.sup.2.sub.18), a third block (a.sup.2.sub.13, a.sup.2.sub.14, a.sup.2.sub.18 and a.sup.2.sub.19), and a fourth block (a.sup.2.sub.14, a.sup.2.sub.15, a.sup.2.sub.19 and a.sup.2.sub.20). Third quadrant a.sup.3.sub.q3 includes elements a.sup.3.sub.11, a.sup.3.sub.12, a.sup.3.sub.13, a.sup.3.sub.14, a.sup.3.sub.15, a.sup.3.sub.16, a.sup.3.sub.17, a.sup.3.sub.18, a.sup.3.sub.19 and a.sup.3.sub.20, from which four blocks of elements are formed, i.e., a first block (a.sup.3.sub.11, a.sup.3.sub.12, a.sup.3.sub.16 and a.sup.3.sub.17), a second block (a.sup.3.sub.12, a.sup.3.sub.13, a.sup.3.sub.17 and a.sup.3.sub.18), a third block (a.sup.3.sub.13, a.sup.3.sub.14, a.sup.3.sub.18 and a.sup.3.sub.19), and a fourth block (a.sup.3.sub.14, a.sup.3.sub.15, a.sup.3.sub.19 and a.sup.3.sub.20). Third quadrant a.sup.4.sub.q3 includes elements a.sup.4.sub.11, a.sup.4.sub.12, a.sup.4.sub.13, a.sup.4.sub.14, a.sup.4.sub.15, a.sup.4.sub.16, a.sup.4.sub.17, a.sup.4.sub.18, a.sup.4.sub.19 and a.sup.4.sub.20, from which four blocks of elements are formed, i.e., a first block (a.sup.4.sub.11, a.sup.4.sub.12, a.sup.4.sub.16 and a.sup.4.sub.17), a second block (a.sup.4.sub.12, a.sup.4.sub.13, a.sup.4.sub.17 and a.sup.4.sub.18), a third block (a.sup.4.sub.13, a.sup.4.sub.14, a.sup.4.sub.18 and a.sup.4.sub.19), and a fourth block (a.sup.4.sub.14, a.sup.4.sub.15, a.sup.4.sub.19 and a.sup.4.sub.20).

[0053] Fourth quadrant a.sup.1.sub.q4 includes elements a.sup.1.sub.10, a.sup.1.sub.17, a.sup.1.sub.18, a.sup.1.sub.19, a.sup.1.sub.20, a.sup.1.sub.21, a.sup.1.sub.22, a.sup.1.sub.23, a.sup.1.sub.24 and a.sup.1.sub.25, from which four blocks of elements are formed, i.e., a first block (a.sup.1.sub.16, a.sup.1.sub.17, a.sup.1.sub.21 and a.sup.1.sub.22), a second block (a.sup.1.sub.17, a.sup.1.sub.18, a.sup.1.sub.22 and a.sup.1.sub.23), a third block (a.sup.1.sub.18, a.sup.1.sub.19, a.sup.1.sub.23 and a.sup.1.sub.24), and a fourth block (a.sup.1.sub.19, a.sup.1.sub.20, a.sup.1.sub.24 and a.sup.1.sub.25). Fourth quadrant a.sup.2.sub.q4 includes elements a.sup.2.sub.16, a.sup.2.sub.17, a.sup.2.sub.18, a.sup.2.sub.19, a.sup.2.sub.20, a.sup.2.sub.21, a.sup.2.sub.22, a.sup.2.sub.23, a.sup.2.sub.24 and a.sup.2.sub.25, from which four blocks of elements are formed, i.e., a first block (a.sup.2.sub.16, a.sup.2.sub.17, a.sup.2.sub.21 and a.sup.2.sub.22), a second block (a.sup.2.sub.17, a.sup.2.sub.18, a.sup.2.sub.22 and a.sup.2.sub.23), a third block (a.sup.2.sub.18, a.sup.2.sub.19, a.sup.2.sub.23 and a.sup.2.sub.24), and a fourth block (a.sup.2.sub.19, a.sup.2.sub.20, a.sup.2.sub.24 and a.sup.2.sub.25). Fourth quadrant a.sup.3.sub.q4 includes elements a.sup.3.sub.16, a.sup.3.sub.17, a.sup.3.sub.18, a.sup.3.sub.19, a.sup.3.sub.20, a.sup.3.sub.21, a.sup.3.sub.22, a.sup.3.sub.23, a.sup.3.sub.24 and a.sup.3.sub.25, from which four blocks of elements are formed, i.e., a first block (a.sup.3.sub.16, a.sup.3.sub.17, a.sup.3.sub.21 and a.sup.3.sub.22), a second block (a.sup.3.sub.17, a.sup.3.sub.18, a.sup.3.sub.22 and a.sup.3.sub.23), a third block (a.sup.3.sub.18, a.sup.3.sub.19, a.sup.3.sub.23 and a.sup.3.sub.24), and a fourth block (a.sup.3.sub.19, a.sup.3.sub.20, a.sup.3.sub.24 and a.sup.3.sub.25). Fourth quadrant a.sup.4.sub.q4 includes elements a.sup.4.sub.16, a.sup.4.sub.17, a.sup.4.sub.18, a.sup.4.sub.19, a.sup.4.sub.20, a.sup.4.sub.21, a.sup.4.sub.22, a.sup.4.sub.23, a.sup.4.sub.24 and a.sup.4.sub.25, from which four blocks of elements are formed, i.e., a first block (a.sup.4.sub.16, a.sup.4.sub.17, a.sup.4.sub.21 and a.sup.4.sub.22), a second block (a.sup.4.sub.17, a.sup.4.sub.18, a.sup.4.sub.22 and a.sup.4.sub.23), a third block (a.sup.4.sub.18, a.sup.4.sub.19, a.sup.4.sub.23 and a.sup.4.sub.24), and a fourth block (a.sup.4.sub.19, a.sup.4.sub.20, a.sup.4.sub.24 and a.sup.4.sub.25).

[0054] Output feature maps 206 may also be divided into four quadrants; in this case, each quadrant spans all four output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4. The first quadrant spans the top (first) row of each output data matrix, the second quadrant spans the second row of each output data matrix, the third quadrant spans the third row of each output data matrix, and the fourth quadrant spans the fourth (bottom) row of each output data matrix. The first quadrant for output feature maps 206 (o.sub.q1), is depicted; the remaining three quadrants are not depicted for clarity.

[0055] First quadrant o.sub.q1 includes o.sup.1.sub.1, o.sup.1.sub.2, o.sup.1.sub.3, o.sup.1.sub.4, o.sup.2.sub.1, o.sup.2.sub.2, o.sup.2.sub.3, o.sup.2.sub.4, o.sup.3.sub.1, o.sup.3.sub.2, o.sup.3.sub.3, o.sup.3.sub.4, o.sup.4.sub.1, o.sup.4.sub.2, o.sup.4.sub.3 and o.sup.4.sub.4. Second quadrant o.sub.q2 includes o.sup.1.sub.5, o.sup.1.sub.6, o.sup.1.sub.7, o.sup.1.sub.8, o.sup.2.sub.5, o.sup.2.sub.6, o.sup.2.sub.7, o.sup.2.sub.8, o.sup.3.sub.5, o.sup.3.sub.6, o.sup.3.sub.7, o.sup.3.sub.8, o.sup.4.sub.5, o.sup.4.sub.6, o.sup.4.sub.7 and o.sup.4.sub.8. Third quadrant o.sub.q3 includes o.sup.1.sub.9, o.sup.1.sub.10, o.sup.1.sub.11, o.sup.1.sub.12, o.sup.2.sub.9, o.sup.2.sub.10, o.sup.2.sub.11, o.sup.2.sub.12, o.sup.3.sub.9, o.sup.3.sub.10, o.sup.3.sub.11, o.sup.3.sub.12, o.sup.4.sub.9, o.sup.4.sub.10, o.sup.4.sub.11 and o.sup.4.sub.12. Fourth quadrant o.sub.q4 includes o.sup.1.sub.13, o.sup.1.sub.14, o.sup.1.sub.15, o.sup.1.sub.16, o.sup.2.sub.13, o.sup.2.sub.14, o.sup.2.sub.15, o.sup.2.sub.16, o.sup.3.sub.13, o.sup.3.sub.14, o.sup.3.sub.15, o.sup.3.sub.16, o.sup.4.sub.13, o.sup.4.sub.14, o.sup.4.sub.15 and o.sup.4.sub.16.

[0056] Generally, each output element within output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4 is the sum of the dot products of one of the weight sets 202.sup.1, 202.sup.2, 202.sup.3 and 202.sup.4 and a block of activation elements within a particular quadrant of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4.

[0057] The calculation of the output elements in quadrant o.sub.q1 follows.

[0058] Output element o.sup.1.sub.1 of output data matrix 206.sup.1 is the sum of the dot products of weight set 202.sup.1 and the first block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. The first block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 includes a.sup.1.sub.1, a.sup.1.sub.2, a.sup.1.sub.6 and a.sup.1.sub.7; a.sup.1.sub.7; a.sup.2.sub.2, a.sup.2.sub.6 and a.sup.2.sub.7; a.sup.3.sub.1, a.sup.3.sub.2, a.sup.3.sub.6 and a.sup.3.sub.7; and a.sup.4.sub.1, a.sup.4.sub.2, a.sup.4.sub.6 and a.sup.4.sub.7, respectively.

[0059] More particularly, the following dot products are summed to generate output element o.sup.1.sub.1: the dot product of the first weight matrix of weight set 202.sup.1 and the first block of quadrant a.sup.1.sub.q1 (i.e., w.sup.1.sub.1.Math.a.sup.1.sub.1+w.sup.1.sub.2.Math.a.sup.1.sub.2+w.sup.1.sub.3.Math.a.sup.1.sub.6+w.sup.1.sub.4.Math.a.sup.1.sub.7), the dot product of the second weight matrix of weight set 202.sup.1 and the first block of quadrant a.sup.2.sub.q1 (i.e., w.sup.1.sub.5.Math.a.sup.2.sub.1+w.sup.1.sub.6.Math.a.sup.2.sub.2+w.sup.1.sub.7.Math.a.sup.2.sub.6+w.sup.1.sub.8.Math.a.sup.2.sub.7), the dot product of the third weight matrix of weight set 202.sup.1 and the first block of quadrant a.sup.3.sub.q1 (i.e., w.sup.1.sub.9.Math.a.sup.3.sub.1+w.sup.1.sub.10.Math.a.sup.3.sub.2+w.sup.1.sub.11.Math.a.sup.3.sub.6+w.sup.1.sub.12.Math.a.sup.3.sub.7), and the dot product of the fourth weight matrix of weight set 202.sup.1 and the first block of quadrant a.sup.4.sub.q1 (i.e., w.sup.1.sub.13.Math.a.sup.4.sub.1+w.sup.1.sub.14.Math.a.sup.4.sub.2+w.sup.1.sub.15.Math.a.sup.4.sub.6+w.sup.1.sub.16.Math.a.sup.4.sub.7).

[0060] Similarly, output element o.sup.2.sub.1 of output data matrix 206.sup.2 is the sum of the dot products of weight set 202.sup.2 and the first block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.3.sub.1 of output data matrix 206.sup.3 is the sum of the dot products of weight set 202.sup.3 and the first block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And, output element o.sup.4.sub.1 of output data matrix 206.sup.4 is the sum of the dot products of weight set 202.sup.4 and the first block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively.

[0061] Output element o.sup.1.sub.2 of output data matrix 206.sup.1 is the sum of the dot products of weight set 202.sup.1 and the second block of activation elements within the first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. The second block of activation elements within the first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 includes a.sup.1.sub.2, a.sup.1.sub.3, a.sup.1.sub.7 and a.sup.1.sub.8; a.sup.2.sub.2, a.sup.2.sub.3, a.sup.2.sub.7 and a.sup.2.sub.8; a.sup.3.sub.2, a.sup.3.sub.3, a.sup.3.sub.7 and a.sup.3.sub.8; and a.sup.4.sub.2, a.sup.4.sub.3, a.sup.4.sub.7 and a.sup.4.sub.8, respectively.

[0062] More particularly, the following dot products are summed to generate output element o.sup.1.sub.2: the dot product of the first weight matrix of weight set 202.sup.1 and the second block of quadrant a.sup.1.sub.q1 (i.e., w.sup.1.sub.1.Math.a.sup.1.sub.2+w.sup.1.sub.2.Math.a.sup.1.sub.3+w.sup.1.sub.3.Math.a.sup.1.sub.7+w.sup.1.sub.4.Math.a.sup.1.sub.8), the dot product of the second weight matrix of weight set 202.sup.1 and the second block of quadrant a.sup.2.sub.q1 (i.e., w.sup.1.sub.5.Math.a.sup.2.sub.2+w.sup.1.sub.6.Math.a.sup.2.sub.3+w.sup.1.sub.7.Math.a.sup.2.sub.7+w.sup.1.sub.8.Math.a.sup.2.sub.8), the dot product of the third weight matrix of weight set 202.sup.1 and the second block of quadrant a.sup.3.sub.q1 (i.e., w.sup.1.sub.9.Math.a.sup.3.sub.2+w.sup.1.sub.10.Math.a.sup.3.sub.3+w.sup.1.sub.11.Math.a.sup.3.sub.7+w.sup.1.sub.12.Math.a.sup.3.sub.8), and the dot product of the fourth weight matrix of weight set 202.sup.1 and the second block of quadrant a.sup.4.sub.q1 (i.e., w.sup.1.sub.13.Math.a.sup.4.sub.2+w.sup.1.sub.14.Math.a.sup.4.sub.3+w.sup.1.sub.15.Math.a.sup.4.sub.7+w.sup.1.sub.16.Math.a.sup.4.sub.8).

[0063] Similarly, output element o.sup.2.sub.2 of output data matrix 206.sup.2 is the sum of the dot products of weight set 202.sup.2 and the second block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.3.sub.2 of output data matrix 206.sup.3 is the sum of the dot products of weight set 202.sup.3 and the second block of activation elements within first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And, output element o.sup.4.sub.2 of output data matrix 206.sup.4 is the sum of the dot products of weight set 202.sup.4 and the second block of activation elements within the quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively.

[0064] And so on for output elements o.sup.1.sub.3 and o.sup.1.sub.4, o.sup.2.sub.3 and o.sup.2.sub.4, o.sup.3.sub.3 and o.sup.3.sub.4, and o.sup.4.sub.3 and o.sup.4.sub.4 of the first rows of output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.

[0065] With respect to quadrant o.sub.q2, output element o.sup.1.sub.5 of output data matrix 206.sup.1 is the sum of the dot products of weight set 202.sup.1 and the first block of activation elements within second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.2.sub.5 of output data matrix 206.sup.2 is the sum of the dot products of weight set 202.sup.2 and the first block of activation elements within second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.3.sub.5 of output data matrix 206.sup.3 is the sum of the dot products of weight set 202.sup.3 and the first block of activation elements within second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2 a.sup.3.sub.q2 and a.sup.4.sub.q2 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And, output element o.sup.4.sub.5 of output data matrix 206.sup.4 is the sum of the dot products of weight set 202.sup.4 and the first block of activation elements within second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And so on for output elements o.sup.1.sub.6, o.sup.1.sub.7 and o.sup.1.sub.8, o.sup.2.sub.6, o.sup.2.sub.7 and o.sup.2.sub.8, o.sup.3.sub.6, o.sup.3.sub.7 and o.sup.3.sub.8, and o.sup.4.sub.6, o.sup.4.sub.7 and o.sup.4.sub.8 of the second rows of output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.

[0066] With respect to quadrant o.sub.q3, output element o.sup.1.sub.9 of output data matrix 206.sup.1 is the sum of the dot products of weight set 202.sup.1 and the first block of activation elements within third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.2.sub.9 of output data matrix 206.sup.2 is the sum of the dot products of weight set 202.sup.2 and the first block of activation elements within third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.3.sub.9 of output data matrix 206.sup.3 is the sum of the dot products of weight set 202.sup.3 and the first block of activation elements within third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And, output element o.sup.4.sub.9 of output data matrix 206.sup.4 is the sum of the dot products of weight set 202.sup.4 and the first block of activation elements within third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And so on for output elements o.sup.1.sub.10, o.sup.1.sub.11 and o.sup.1.sub.12, o.sup.2.sub.10, o.sup.2.sub.11 and o.sup.2.sub.12, o.sup.3.sub.10, o.sup.3.sub.11 and o.sup.3.sub.12, and o.sup.4.sub.10, o.sup.4.sub.11 and o.sup.4.sub.12 of the third rows of output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.

[0067] With respect to quadrant o.sub.q4, output element o.sup.1.sub.13 of output data matrix 206.sup.1 is the sum of the dot products of weight set 202.sup.1 and the first block of activation elements within fourth quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.2.sub.13 of output data matrix 206.sup.2 is the sum of the dot products of weight set 202.sup.2 and the first block of activation elements within fourth quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. Output element o.sup.3.sub.13 of output data matrix 206.sup.3 is the sum of the dot products of weight set 202.sup.3 and the first block of activation elements within fourth quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And, output element o.sup.4.sub.13 of output data matrix 206.sup.4 is the sum of the dot products of weight set 202.sup.4 and the first block of activation elements within third quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4 of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, respectively. And so on for output elements o.sup.1.sub.14, o.sup.1.sub.15 and o.sup.1.sub.16, o.sup.2.sub.14, o.sup.2.sub.15 and o.sup.2.sub.16, o.sup.3.sub.14, o.sup.3.sub.15 and o.sup.3.sub.16, and o.sup.4.sub.14, o.sup.4.sub.15 and o.sup.4.sub.16 of the fourth rows of output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4.

[0068] FIG. 3B depicts converted convolutional layer calculation 210 for a CNN, while FIG. 3C depicts converted input data matrix 214, in accordance with an embodiment of the present disclosure.

[0069] In one embodiment, the convolutional layer calculations for CNNs may be converted into generic matrix multiplication (GEMM) operations for processing by one or more MMAs. Convolution layer calculation 200 is converted into a GEMM operation by converting filters 202 into converted weight matrix 212, converting input feature maps 204 into converted input data matrix 214, and then multiplying converted weight matrix 212 and converted input data matrix 214 to generate converted output data matrix 216. Because simple matrix multiplication is performed rather than a convolution operation, each output element within converted output data matrix 216 is the dot product of one row of converted weight matrix 212 and one column of converted input data matrix 214. Converted output data matrix 216 is then reformed into output feature maps 206.

[0070] Converted weight matrix 212 is a 4×16 matrix, and includes converted weight sets 212.sup.1, 212.sup.2, 212.sup.3 and 212.sup.4. Weight set 202.sup.1 is flattened to form converted weight set 212.sup.1, i.e., the first row, and includes weights w.sup.1.sub.1, w.sup.12, w.sup.1.sub.3, w.sup.1.sub.4, w.sup.1.sub.5, w.sup.1.sub.6, w.sup.1.sub.7, w.sup.1.sub.8, w.sup.1.sub.9, w.sup.1.sub.10, w.sup.1.sub.11, w.sup.1.sub.12, w.sup.1.sub.13, w.sup.1.sub.14, w.sup.1.sub.15 and w.sup.1.sub.16. Weight set 202.sup.2 is flattened to form converted weight set 212.sup.2, i.e., the second row, and includes weights w.sup.2.sub.1, w.sup.2.sub.2, w.sup.2.sub.3, w.sup.2.sub.4, w.sup.2.sub.5, w.sup.2.sub.6, w.sup.2.sub.7, w.sup.2.sub.8, w.sup.2.sub.9, w.sup.2.sub.10, w.sup.2.sub.11, w.sup.2.sub.12, w.sup.2.sub.13, w.sup.2.sub.14, w.sup.2.sub.15 and w.sup.2.sub.16. Weight set 202.sup.3 is flattened to form converted weight set 212.sup.3, i.e., the third row, and includes weights w.sup.3.sub.1, w.sup.3.sub.2, w.sup.3.sub.3, w.sup.3.sub.4, w.sup.3.sub.5, w.sup.3.sub.6, w.sup.3.sub.7, w.sup.3.sub.8, w.sup.3.sub.9, w.sup.3.sub.10, w.sup.3.sub.11, w.sup.3.sub.12, w.sup.3.sub.13, w.sup.3.sub.14, w.sup.3.sub.15 and w.sup.3.sub.16. And, weight set 202.sup.4 is flattened to form converted weight set 212.sup.4, i.e., the fourth row, and includes weights w.sup.4.sub.1, w.sup.4.sub.2, w.sup.4.sub.3, w.sup.4.sub.4, w.sup.4.sub.5, w.sup.4.sub.6, w.sup.4.sub.7, w.sup.4.sub.5, w.sup.4.sub.6, w.sup.4.sub.10, w.sup.4.sub.11, w.sup.4.sub.12, w.sup.4.sub.13, w.sup.4.sub.14, w.sup.4.sub.15 and w.sup.4.sub.16.

[0071] Converted input data matrix 214 is a 16×16 matrix, and includes the blocks of each quadrant of input data matrices 204.sup.1, 204.sup.2, 204.sup.3 and 204.sup.4, i.e., quadrants a.sup.1.sub.q1, a.sup.1.sub.q2 a.sup.1.sub.q3, a.sup.1.sub.q4, a.sup.2.sub.q1, a.sup.2.sub.q2, a.sup.2.sub.q3, a.sup.2.sub.q4, a.sup.3.sub.q1, a.sup.3.sub.q2, a.sup.3.sub.q3, a.sup.3.sub.q4, a.sup.4.sub.q1, a.sup.4.sub.q2, a.sup.4.sub.q3 and a.sup.4.sub.q4, respectively. Generally, each block is flattened to form a portion of a single column of converted input data matrix 214.

[0072] More particularly, the first column of converted input matrix 214 includes the first blocks from quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1, i.e., activations a.sup.1.sub.1, a.sup.1.sub.2, a.sup.1.sub.6, a.sup.1.sub.7, a.sup.2.sub.1, a.sup.2.sub.2, a.sup.2.sub.6, a.sup.2.sub.7, a.sup.3.sub.1, a.sup.3.sub.2, a.sup.3.sub.6, a.sup.3.sub.7, a.sup.4.sub.1, a.sup.4.sub.2, a.sup.4.sub.6, and a.sup.4.sub.7. The second column of converted input matrix 214 includes the second blocks from quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1, i.e., activations a.sup.1.sub.1, a.sup.1.sub.3, a.sup.1.sub.7, a.sup.1.sub.8, a.sup.2.sub.2, a.sup.2.sub.3, a.sup.2.sub.7, a.sup.2.sub.8, a.sup.3.sub.2, a.sup.3.sub.3, a.sup.3.sub.7, a.sup.3.sub.8, a.sup.4.sub.2, a.sup.4.sub.3, a.sup.4.sub.7, and a.sup.7.sub.8. The third column of converted input matrix 214 includes the third blocks from quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1, i.e., activations a.sup.1.sub.3, a.sup.1.sub.4, a.sup.1.sub.8, a.sup.1.sub.9, a.sup.2.sub.3, a.sup.2.sub.4, a.sup.2.sub.8, a.sup.2.sub.9, a.sup.3.sub.3, a.sup.3.sub.4, a.sup.3.sub.8, a.sup.3.sub.9, a.sup.4.sub.3, a.sup.4.sub.4, a.sup.4.sub.8, and a.sup.4.sub.9. And, the fourth column of converted input matrix 214 includes the fourth blocks from quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1, i.e., activations a.sup.1.sub.4, a.sup.1.sub.5, a.sup.1.sub.9, a.sup.1.sub.10, a.sup.2.sub.4, a.sup.2.sub.5, a.sup.2.sub.9, a.sup.2.sub.10, a.sup.3.sub.4, a.sup.3.sub.5, a.sup.3.sub.9, a.sup.3.sub.10, a.sup.4.sub.4, a.sup.4.sub.5, a.sup.4.sub.9, and a.sup.4.sub.10.

[0073] The remaining columns of converted input data matrix 214 are formed in a similar manner. The fourth to the eighth columns are formed from the blocks of quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2, the ninth to the twelfth columns are formed from the blocks of quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3, and the thirteenth to the sixteenth columns are formed from the blocks of quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4.

[0074] Converted output data matrix 216 is a 4×16 matrix, and includes flattened versions of output data matrices 206.sup.1, 206.sup.2, 206.sup.3 and 206.sup.4, i.e., converted output data matrices 216.sup.1, 216.sup.2, 216.sup.3 and 216.sup.4. Converted output data matrix 216 may also be arranged into four quadrants o.sub.q1, o.sub.q2, o.sub.q3 and o.sub.q4, which include the same output elements as the four quadrants o.sub.q1, o.sub.q2, o.sub.q3 and o.sub.q4 of output feature maps 206.

[0075] The calculation of the output elements in the first row of quadrant o.sub.q1 of converted output data matrix 216 follows.

[0076] Output element o.sup.1.sub.1 is the dot product of the first row of converted weight matrix 212, i.e., converted weight set 212.sup.1, and the first column of converted input data matrix 214. More particularly, output element o.sup.1.sub.1 is equal to w.sup.1.sub.1.Math.a.sup.1.sub.1+w.sup.1.sub.2.Math.a.sup.1.sub.2+w.sup.1.sub.3.Math.a.sup.1.sub.6+w.sup.1.sub.4.Math.a.sup.1.sub.7+w.sup.1.sub.5.Math.a.sup.2.sub.1+w.sup.1.sub.6.Math.a.sup.2.sub.2+w.sup.1.sub.7.Math.a.sup.2.sub.6+w.sup.1.sub.8.Math.a.sup.2.sub.7+w.sup.1.sub.9.Math.a.sup.3.sub.1+w.sup.1.sub.10.Math.a.sup.3.sub.2+w.sup.1.sub.11.Math.a.sup.3.sub.6+w.sup.1.sub.12.Math.a.sup.3.sub.7+w.sup.1.sub.13.Math.a.sup.4.sub.1+w.sup.1.sub.14.Math.a.sup.4.sub.2+w.sup.1.sub.15.Math.a.sup.4.sub.6+w.sup.1.sub.16.Math.a.sup.4.sub.7. As shown above, output element o.sup.1.sub.1 of converted output data matrix 216 is equal to output element o.sup.1.sub.1 of output feature maps 206.

[0077] Output element o.sup.1.sub.2 is the dot product of the first row of converted weight matrix 212, i.e., converted weight set 212.sup.1, and the second column of converted input data matrix 214. More particularly, output element o.sup.1.sub.2 is equal to w.sup.1.sub.1.Math.a.sup.1.sub.2+w.sup.1.sub.2.Math.a.sup.1.sub.3+w.sup.1.sub.3.Math.a.sup.1.sub.7+w.sup.1.sub.4.Math.a.sup.1.sub.8+w.sup.1.sub.5.Math.a.sup.2.sub.2+w.sup.1.sub.6.Math.a.sup.2.sub.3+w.sup.1.sub.7.Math.a.sup.2.sub.7+w.sup.1.sub.8.Math.a.sup.2.sub.8+w.sup.1.sub.9.Math.a.sup.3.sub.2+w.sup.1.sub.10.Math.a.sup.3.sub.3+w.sup.1.sub.11.Math.a.sup.3.sub.7+w.sup.1.sub.12.Math.a.sup.3.sub.8+w.sup.1.sub.13.Math.a.sup.4.sub.2+w.sup.1.sub.14.Math.a.sup.4.sub.3+w.sup.1.sub.15.Math.a.sup.4.sub.7+w.sup.1.sub.16.Math.a.sup.4.sub.8. As shown above, output element o.sup.1.sub.2 of converted output data matrix 216 is equal to output element o.sup.1.sub.2 of output feature maps 206.

[0078] Output element o.sup.1.sub.3 is the dot product of the first row of converted weight matrix 212, i.e., converted weight set 212.sup.1, and the third column of converted input data matrix 214. More particularly, output element o.sup.1.sub.3 is equal to w.sup.1.sub.1.Math.a.sup.1.sub.3+w.sup.1.sub.2.Math.a.sup.1.sub.4+w.sup.1.sub.3.Math.a.sup.1.sub.8+w.sup.1.sub.4.Math.a.sup.1.sub.9+w.sup.1.sub.5.Math.a.sup.2.sub.3+w.sup.1.sub.6.Math.a.sup.2.sub.4+w.sup.1.sub.7.Math.a.sup.2.sub.8+w.sup.1.sub.8.Math.a.sup.2.sub.9+w.sup.1.sub.9.Math.a.sup.3.sub.3+w.sup.1.sub.10.Math.a.sup.3.sub.4+w.sup.1.sub.11.Math.a.sup.3.sub.8+w.sup.1.sub.12.Math.a.sup.3.sub.9+w.sup.1.sub.13.Math.a.sup.4.sub.3+w.sup.1.sub.15.Math.a.sup.4.sub.8+w.sup.1.sub.16.Math.a.sup.4.sub.9. As shown above, output element o.sup.1.sub.3 of converted output data matrix 216 is equal to output element o.sup.1.sub.3 of output feature maps 206.

[0079] Output element o.sup.1.sub.4 is the dot product of the first row of converted weight matrix 212, i.e., converted weight set 212.sup.1, and the fourth column of converted input data matrix 214. More particularly, output element o.sup.1.sub.4 is equal to w.sup.1.sub.1.Math.a.sup.1.sub.4+w.sup.1.sub.2.Math.a.sup.1.sub.5+w.sup.1.sub.3.Math.a.sup.1.sub.9+w.sup.1.sub.4.Math.a.sup.1.sub.10+w.sup.1.sub.5.Math.a.sup.2.sub.4+w.sup.1.sub.6.Math.a.sup.2.sub.5+w.sup.1.sub.7.Math.a.sup.2.sub.9+w.sup.1.sub.8.Math.a.sup.2.sub.10+w.sup.1.sub.9.Math.a.sup.3.sub.4+w.sup.1.sub.10.Math.a.sup.3.sub.5+w.sup.1.sub.11.Math.a.sup.3.sub.9+w.sup.1.sub.12.Math.a.sup.3.sub.10+w.sup.1.sub.13.Math.a.sup.4.sub.4+w.sup.1.sub.14.Math.a.sup.4.sub.5+w.sup.1.sub.15.Math.a.sup.4.sub.9+w.sup.1.sub.16.Math.a.sup.4.sub.10. As shown above, output element o.sup.1.sub.4 of converted output data matrix 216 is equal to output element o.sup.1.sub.4 of output feature maps 206.

[0080] For the second row of quadrant o.sub.q1, output element o.sup.2.sub.1 is the dot product of the second row of converted weight matrix 212, i.e., converted weight set 212.sup.2, and the first column of converted input data matrix 214, output element o.sup.2.sub.2 is the dot product of the second row of converted weight matrix 212, i.e., converted weight set 212.sup.2, and the second column of converted input data matrix 214, output element o.sup.2.sub.3 is the dot product of the second row of converted weight matrix 212, i.e., converted weight set 212.sup.2, and the third column of converted input data matrix 214, and output element o.sup.2.sub.4 is the dot product of the second row of converted weight matrix 212, i.e., converted weight set 212.sup.2, and the fourth column of converted input data matrix 214.

[0081] For the third row of quadrant o.sub.q1, output element o.sup.3.sub.1 is the dot product of the third row of converted weight matrix 212, i.e., converted weight set 212.sup.3, and the first column of converted input data matrix 214, output element o.sup.3.sub.2 is the dot product of the third row of converted weight matrix 212, i.e., converted weight set 212.sup.3, and the second column of converted input data matrix 214, output element o.sup.3.sub.3 is the dot product of the third row of converted weight matrix 212, i.e., converted weight set 212.sup.3, and the third column of converted input data matrix 214, and output element o.sup.3.sub.4 is the dot product of the third row of converted weight matrix 212, i.e., converted weight set 212.sup.3, and the fourth column of converted input data matrix 214.

[0082] For the fourth row of quadrant o.sub.q1, output element o.sup.4.sub.1 is the dot product of the fourth row of converted weight matrix 212, i.e., converted weight set 212.sup.4, and the first column of converted input data matrix 214, output element o.sup.4.sub.2 is the dot product of the fourth row of converted weight matrix 212, i.e., converted weight set 212.sup.4, and the second column of converted input data matrix 214, output element o.sup.4.sub.3 is the dot product of the fourth row of converted weight matrix 212, i.e., converted weight set 212.sup.4, and the third column of converted input data matrix 214, and output element o.sup.4.sub.4 is the dot product of the fourth row of converted weight matrix 212, i.e., converted weight set 212.sup.4, and the fourth column of converted input data matrix 214.

[0083] The elements of the quadrants o.sub.q2, o.sub.q3 and o.sub.q4 are calculated in a similar manner.

[0084] FIG. 4 depicts data flow diagram 220 for MAC array 218.

[0085] As noted above, GEMM operations may be implemented in one or more MMAs, which are dedicated ANN hardware accelerators that include one or more arrays of MAC units. In this embodiment, MAC array 218 is a systolic, output stationary array that implements converted convolution operation 210 using a 4×4 array of MAC units m.sub.1, m.sub.2, m.sub.3, m.sub.4, m.sub.5, m.sub.6, m.sub.7, m.sub.8, mg, m.sub.10, m.sub.11, m.sub.12, m.sub.13, m.sub.14, m.sub.15 and m.sub.16. The orientation of transposed converted weight matrix 222, transposed converted input data matrix 224, and transposed converted output data matrix 226 relative to MAC array 218 simplifies illustration; other orientations are also contemplated.

[0086] Each MAC unit calculates a dot product, between a row of converted weight matrix 212 and a column of converted input data matrix 214, to generate an element of converted output data matrix 216. Generally, a MAC unit includes, inter alia, a multiplier, an adder and a storage register. Each MAC unit is reset by clearing or zeroing its storage register prior to, or at the start of, a new dot product calculation.

[0087] Generally, the rows from converted weight matrix 212 are read from local memory, enter MAC array 218 at the first row of MAC units m.sub.1, m.sub.2, m.sub.3 and m.sub.4, and propagate one MAC unit down at the beginning of each processing cycle. Similarly, the columns from converted input data matrix 214 are read from local memory, enter MAC array 218 at the first column of MAC units m.sub.1, m.sub.5, m.sub.9 and m.sub.13, and propagate one MAC unit to the right at the beginning of each processing cycle.

[0088] The dot product calculations performed by MAC unit m.sub.1 for the blocks of the first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of converted input data matrix 214 are discussed in detail below, while the dot product calculations performed by the remaining MAC units of MAC array 218 are summarized below.

[0089] MAC unit m.sub.1 calculates the dot product of the first row of converted weight matrix 212 (i.e., converted weight set 212.sup.1) and the first column of converted input data matrix 214 to generate element o.sup.1.sub.1 of converted output data matrix 216. During the processing cycle 1, MAC unit m.sub.1 receives a.sub.1 and w.sup.1.sub.1 from local memory, multiplies a.sub.1 and w.sup.1.sub.1 to generate an intermediate product, adds the intermediate product to the value stored in the storage register (i.e., 0), and stores the accumulated result back in the storage register. During processing cycle 2, MAC unit m.sub.1 transmits a.sub.1 to MAC unit m.sub.2 and w.sup.1.sub.1 to MAC unit m.sub.5, receives a.sub.2 and w.sup.1.sub.2 from local memory, multiplies a.sub.2 and w.sup.1.sub.2 to generate an intermediate product, adds the intermediate product to the value stored in the storage register, and stores the accumulated result back in the storage register.

[0090] During processing cycle 3, MAC unit m.sub.1 transmits a.sub.2 to MAC unit m.sub.2 and w.sup.1.sub.2 to MAC unit m.sub.5, receives as and w.sup.1.sub.3 from local memory, multiplies as and w.sup.1.sub.3 to generate an intermediate product, adds the intermediate product to the value stored in the storage register, and stores the accumulated result back in the storage register. During processing cycle 4, MAC unit m.sub.1 transmits as to MAC unit m.sub.2 and w.sup.1.sub.3 to MAC unit m.sub.5, receives a.sub.7 and w.sup.1.sub.4 from the local memory, multiplies a.sub.7 and w.sup.1.sub.4 to generate an intermediate product, adds the intermediate product to the value stored in the storage register, and stores the accumulated result back in the storage register.

[0091] Processing cycles 5 through 16 multiply and accumulate the remaining 12 elements of the first row of converted weight matrix 212 and the first column of converted input data matrix 214. At the end of the processing cycle 16, MAC unit m.sub.1 outputs element o.sup.1.sub.1.

[0092] The remainder of the first row of MAC array 218 includes MAC units m.sub.2, m.sub.3 and m.sub.4.

[0093] After an initial delay of one processing cycle, MAC unit m.sub.2 receives weights from the first delay register ff.sub.1 and input data from MAC unit m.sub.1, transmits weights to MAC unit m.sub.6 and input data to MAC unit m.sub.3, and calculates the dot product of the second row of converted weight matrix 212 (i.e., converted weight set 212.sup.2) and the first column of converted input data matrix 214 to generate element o.sup.2.sub.1 of converted output data matrix 216. The initial delay of one processing cycle allows the delay pipeline (i.e., delay register ff.sub.1) to be filled with weights transferred from memory, and the input data to become available from MAC unit m.sub.1. At the end of the processing cycle 17, MAC unit m.sub.2 outputs element o.sup.2.sub.1.

[0094] After an initial delay of two processing cycles, MAC unit m.sub.3 receives weights from the second delay register ff.sub.2 and input data from MAC unit m.sub.2, transmits weights to MAC unit m.sub.7 and input data to MAC unit m.sub.4, and calculates the dot product of the third row of converted weight matrix 212 (i.e., converted weight set 212.sup.3) and the first column of converted input data matrix 214 to generate element o.sup.3.sub.1 of converted output data matrix 216. The initial delay of two processing cycles allows the delay pipeline (i.e., delay registers ff.sub.1 and ff.sub.2) to be filled with weights transferred from memory, and the input data to become available from MAC unit m.sub.2. At the end of processing cycle 18, MAC unit m.sub.3 outputs element o.sup.3.sub.1.

[0095] After an initial delay of three processing cycles, MAC unit m.sub.4 receives weights from the third delay register ff.sub.3 and input data from MAC unit m.sub.3, transmits weights to MAC unit ma, and calculates the dot product of the fourth row of converted weight matrix 212 (i.e., converted weight set 212.sup.4) and the first column of converted input data matrix 214 to generate element o.sup.4.sub.1 of converted output data matrix 216. The initial delay of three processing cycles allows the delay pipeline (i.e., delay registers ff.sub.2 and ff.sub.3) to be filled with weights transferred from memory, and the input data to become available from MAC unit m.sub.3. At the end of processing cycle 19, MAC unit m.sub.4 outputs element o.sup.4.sub.1.

[0096] The second row of MAC array 218 includes MAC units m.sub.5, m.sub.6, m.sub.7 and m.sub.8.

[0097] After an initial delay of one processing cycle, MAC unit m.sub.5 receives weights from MAC unit m.sub.1 and input data from a first delay register ff.sub.1, transmits weights to MAC unit m.sub.9 and input data to MAC unit ma, and calculates the dot product of the first row of converted weight matrix 212 (i.e., converted weight set 212.sup.1) and the second column of converted input data matrix 214 to generate element o.sup.1.sub.2 of converted output data matrix 216. The initial delay of one processing cycle allows the delay pipeline (i.e., delay register ff.sub.1) to be filled with input data transferred from memory, and the weights to become available from MAC unit m.sub.1. At the end of processing cycle 17, MAC unit m.sub.5 outputs element o.sup.1.sub.2.

[0098] After an initial delay of two processing cycles, MAC unit m.sub.6 receives weights from MAC unit m.sub.2 and input data from MAC unit m.sub.5, transmits weights to MAC unit m.sub.10 and input data to MAC unit m.sub.7, and calculates the dot product of the second row of converted weight matrix 212 (i.e., converted weight set 212.sup.2) and the second column of converted input data matrix 214 to generate element o.sup.2.sub.2 of converted output data matrix 216. The initial delay of two processing cycles allows the weights to become available from MAC unit m.sub.2, and the input data to become available from MAC unit m.sub.5. At the end of processing cycle 18, MAC unit m.sub.6 outputs element o.sup.2.sub.2.

[0099] After an initial delay of three processing cycles, MAC unit m.sub.7 receives weights from MAC unit m.sub.3 and input data from MAC unit ma, transmits weights to MAC unit m.sub.11 and input data to MAC unit ma, and calculates the dot product of the third row of converted weight matrix 212 (i.e., converted weight set 212.sup.3) and the second column of converted input data matrix 214 to generate element o.sup.3.sub.2 of converted output data matrix 216. The initial delay of three processing cycles allows the weights to become available from MAC unit m.sub.3, and the input data to become available from MAC unit ma. At the end of processing cycle 19, MAC unit m.sub.7 outputs element o.sup.3.sub.2.

[0100] After an initial delay of four processing cycles, MAC unit ma receives weights from MAC unit m.sub.4 and input data from MAC unit m.sub.7, transmits weights to MAC unit m.sub.12, and calculates the dot product of the fourth row of converted weight matrix 212 (i.e., converted weight set 212.sup.4) and the second column of converted input data matrix 214 to generate element o.sup.4.sub.2 of converted output data matrix 216. The initial delay of four processing cycles allows the weights to become available from MAC unit m.sub.4, and the input data to become available from MAC unit m.sub.7. At the end of processing cycle 20, MAC unit ma outputs element o.sup.4.sub.2.

[0101] The third row of MAC array 218 includes MAC units m.sub.9, m.sub.10, mu and m.sub.12.

[0102] After an initial delay of two processing cycles, MAC unit m.sub.9 receives weights from MAC unit m.sub.5 and input data from a second delay register ff.sub.2, transmits weights to MAC unit m.sub.13 and input data to MAC unit m.sub.10, and calculates the dot product of the first row of converted weight matrix 212 (i.e., converted weight set 212.sup.1) and the third column of converted input data matrix 214 to generate element o.sup.1.sub.3 of converted output data matrix 216. The initial delay of two processing cycles allows the delay pipeline (i.e., delay registers ff.sub.1 and ff.sub.2) to be filled with input data transferred from memory, and the weights to become available from MAC unit m.sub.5. At the end of processing cycle 18, MAC unit m.sub.9 outputs element o.sup.1.sub.3.

[0103] After an initial delay of three processing cycles, MAC unit m.sub.10 receives weights from MAC unit m.sub.6 and input data from MAC unit m.sub.9, transmits weights to MAC unit m.sub.14 and input data to MAC unit mu u, and calculates the dot product of the second row of converted weight matrix 212 (i.e., converted weight set 212.sup.2) and the third column of converted input data matrix 214 to generate element o.sup.2.sub.3 of converted output data matrix 216. The initial delay of three processing cycles allows the weights to become available from MAC unit ma, and the input data to become available from MAC unit m.sub.9. At the end of processing cycle 19, MAC unit m.sub.10 outputs element o.sup.2.sub.3.

[0104] After an initial delay of four processing cycles, MAC unit m.sub.11 receives weights from MAC unit m.sub.7 and input data from MAC unit m.sub.10, transmits weights to MAC unit m.sub.15 and input data to MAC unit m.sub.12, and calculates the dot product of the third row of converted weight matrix 212 (i.e., converted weight set 212.sup.3) and the third column of converted input data matrix 214 to generate element o.sup.3.sub.3 of converted output data matrix 216. The initial delay of four processing cycles allows the weights to become available from MAC unit m.sub.7, and the input data to become available from MAC unit m.sub.10. At the end of processing cycle 20, MAC unit mu outputs element o.sup.3.sub.3.

[0105] After an initial delay of five processing cycles, MAC unit m.sub.12 receives weights from MAC unit ma and input data from MAC unit mu u, transmits weights to MAC unit m.sub.16, and calculates the dot product of the fourth row of converted weight matrix 212 (i.e., converted weight set 212.sup.4) and the third column of converted input data matrix 214 to generate element o.sup.4.sub.3 of converted output data matrix 216. The initial delay of five processing cycles allows the weights to become available from MAC unit ma, and the input data to become available from MAC unit mu u. At the end of processing cycle 21, MAC unit m.sub.12 outputs element o.sup.4.sub.3.

[0106] The fourth row of MAC array 218 includes MAC units m.sub.13, m.sub.14, m.sub.15 and m.sub.16.

[0107] After an initial delay of three processing cycles, MAC unit m.sub.13 receives weights from MAC unit m.sub.9 and input data from a third delay register ff.sub.3, transmits input data to MAC unit m.sub.14, and calculates the dot product of the first row of converted weight matrix 212 (i.e., converted weight set 212.sup.1) and the fourth column of converted input data matrix 214 to generate element o.sup.1.sub.4 of converted output data matrix 216. The initial delay of three processing cycles allows the delay pipeline (i.e., delay registers ff.sub.1, ff.sub.2 and ff.sub.3) to be filled with input data transferred from memory, and the weights to become available from MAC unit m.sub.9. At the end of processing cycle 19, MAC unit m.sub.13 outputs element o.sup.1.sub.4.

[0108] After an initial delay of four processing cycles, MAC unit m.sub.14 receives weights from MAC unit m.sub.10 and input data from MAC unit m.sub.13, transmits input data to MAC unit m.sub.15, and calculates the dot product of the second row of converted weight matrix 212 (i.e., converted weight set 212.sup.2) and the fourth column of converted input data matrix 214 to generate element o.sup.2.sub.4 of converted output data matrix 216. The initial delay of four processing cycles allows the weights to become available from MAC unit m.sub.10, and the input data to become available from MAC unit m.sub.13. At the end of processing cycle 20, MAC unit m.sub.14 outputs element o.sup.2.sub.4.

[0109] After an initial delay of five processing cycles, MAC unit m.sub.15 receives weights from MAC unit mu and input data from MAC unit m.sub.14, transmits input data to MAC unit m.sub.16, and calculates the dot product of the third row of converted weight matrix 212 (i.e., converted weight set 212.sup.3) and the fourth column of converted input data matrix 214 to generate element o.sup.3.sub.4 of converted output data matrix 216. The initial delay of five processing cycles allows the weights to become available from MAC unit mu u, and the input data to become available from MAC unit mud. At the end of processing cycle 21, MAC unit m.sub.15 outputs element o.sup.3.sub.4.

[0110] After an initial delay of six processing cycles, MAC unit m.sub.1 receives weights from MAC unit m.sub.11 and input data from MAC unit m.sub.15, and calculates the dot product of the fourth row of converted weight matrix 212 (i.e., converted weight set 212.sup.4) and the fourth column of converted input data matrix 214 to generate element o.sup.4.sub.4 of converted output data matrix 216. The initial delay of six processing cycles allows the weights to become available from MAC unit m.sub.11, and the input data to become available from MAC unit m.sub.15. At the end of processing cycle 22, MAC unit m.sub.1 outputs element o.sup.4.sub.4.

[0111] After the blocks of the first quadrants a.sup.1.sub.q1, a.sup.2.sub.q1, a.sup.3.sub.q1 and a.sup.4.sub.q1 of converted input data matrix 214 have been processed, the next sequence of operations processes the blocks of the second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2. After the blocks of the second quadrants a.sup.1.sub.q2, a.sup.2.sub.q2, a.sup.3.sub.q2 and a.sup.4.sub.q2 have been processed, the next sequence of operations processes the blocks of the third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3. And, after the blocks of the third quadrants a.sup.1.sub.q3, a.sup.2.sub.q3, a.sup.3.sub.q3 and a.sup.4.sub.q3 have been processed, the final sequence of operations processes the blocks of the fourth quadrants a.sup.1.sub.q4, a.sup.2.sub.q4, a.sup.3.sub.q4 and a.sup.4.sub.q4. Converted weight matrix 212 is accessed for each sequence of operations.

[0112] Many Machine Learning (ML) inference applications employ quantized ANNs, such as quantized CNNs, that require high-throughput, low-precision matrix multiplication operations. A conventional ANN has fixed bit-width dot product datapaths, such as, for example, 8 bits, 16 bits, 32 bits, etc. MMAs that support conventional ANNs include one or more MAC unit arrays that multiply operands having corresponding fixed bit-widths, such as, for example, 8 bits, 16 bits, 32 bits, etc.

[0113] A quantized ANN may have smaller bit-width dot product datapaths, such as 3 bits, 4 bits, 5 bits, etc. For example, one matrix for a particular CNN layer may contain weight data having a resolution of 3 bits, while another matrix for this particular CNN layer may contain input data having a resolution of 5 bits. Generally, a quantized ANN may have dot product datapaths with bit-widths that vary from 1 bit to 8 bits (or more).

[0114] MMAs that support conventional ANNs may be used to support quantized ANNs.

[0115] FIG. 5 depicts the computation of the dot product between vector A 310 and vector B 320 using MAC unit 300, in accordance with an embodiment of the present disclosure.

[0116] Vector A 310 includes sixteen 3-bit elements, i.e., A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15 and A16. Vector A 310 may represent, for example, one row from converted weight matrix 212. Vector B 310 includes sixteen 5-bit elements, i.e., B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15 and B16. Vector B 320 may represent, for example, one column from converted input data matrix 214. MAC unit 300 calculates the dot product between vector A 310 and vector B 320 by multiplying corresponding pairs of elements as 8-bit unsigned operands (i.e., UINT8), accumulating the intermediate products into a 32-bit accumulator register (ACC), and then outputting 32-bit scalar C 330 (e.g., UINT32, etc.), which may represent, for example, one element from converted output data matrix 216.

[0117] More particularly, during the first processing cycle, MAC unit 300 multiplies A1 and B1 as 8-bit operands to generate an intermediate product (i.e., A1 B1), adds the intermediate product to the value stored in the accumulator register (i.e., 0), and then stores the accumulated value back to the accumulator register (i.e., A1 B1). During the second processing cycle, MAC unit 300 multiplies A2 and B2 as 8-bit operands to generate an intermediate product (i.e., A2 B2), adds the intermediate product to the value stored in the accumulator register (i.e., A1 B1) and then stores the accumulated value back to the accumulator register (i.e., A1 B1+A2 B2). MAC unit 300 processes the remaining 14 pairs of elements from vector A 310 and vector B 320 in the same manner, and, after MAC unit 300 has processed A16 and B16, MAC unit 300 outputs the accumulated value stored in the accumulator register as 32-bit scalar C 330 (i.e., A1.Math.B1+A2.Math.B2+A3.Math.B3+A4.Math.B4+A5.Math.B5+A6.Math.B6+A7.Math.B7+A8.Math.B8+A9.Math.B9+A10.Math.B10+A11.Math.B11+A12.Math.B12+A13.Math.B13+A14.Math.B14+A15.Math.B15+A16.Math.B16).

[0118] However, using a wide datapath MAC unit array to multiply narrower operands is inefficient because the upper bits of the wide datapath are wasted. For example, a MAC unit that multiplies 3-bit operands and 5-bit operands as 8-bit operands operates much less efficiently that a MAC unit that multiplies 3-bit operands and 5-bit operands at their native resolution. Unfortunately, it is impractical to deploy narrow 1 bit-width to 8 bit-width MAC units in hardware to achieve maximal power and area efficiency.

[0119] Embodiments of the present disclosure advantageously provide a system and method for efficiently multiplying matrices with variable bit-width operands using an MMA with an array of BSDP units.

[0120] FIG. 6A depicts the creation of bit slice vectors 410 from vector A 310 depicted in FIG. 5, in accordance with an embodiment of the present disclosure.

[0121] The elements of vector A 310 are first arranged in bit vector form as bit vector A 312. The bit vector for each element of vector A 310 is a sequence of bits from the LSB (i.e., bit position “0”) to the MSB (i.e., bit position “2”). For example, the bit vector for element A1 is {A1[0], A1[1], A1[2]}, where A1[0] is the value of the bit at the first bit position (i.e., the LSB), A1[1] is the value of the bit at the second bit position, and A1[2] is the value of the bit at the third bit position (i.e., the MSB). Similarly, the bit vector for element A2 is {A2[0], A2[1], A2[2]}, where A2[0] is the value of the bit at the first bit position (i.e., the LSB), A2[1] is the value of the bit at the second bit position, and A2[2] is the value of the bit at the third bit position (i.e., the MSB). The remaining elements of bit vector A 312 are formed in a similar manner from the remaining elements of vector A 310.

[0122] Bit slice vectors 410 are then formed from bit vector A 312. Bit slice vector 410.sup.0 is a sequence of bits formed from the bit at the first bit position of each element of bit vector A 312, i.e., {A1[0], A2[0], A3[0], A4[0], A5[0], A6[0], A7[0], A8[0], A9[0], A10[0], A11[0], A12[0], A13[0], A14[0], A15[0], A16[0]}. Bit slice vector 410.sup.1 is a sequence of bits formed from the bit at the second bit position of each element of bit vector A 312, i.e., {A1[1], A2[1], A3[1], A4[1], A5[1], A6[1], A7[1], A8[1], A9[1], A10[1], A11[1], A12[1], A13[1], A14[1], A15[1], A16[1]}. Bit slice vector 410.sup.2 is a sequence of bits formed from the bit at the third bit position of each element of bit vector A 312, i.e., {A1[2], A2[2], A3[2], A4[2], A5[2], A6[2], A7[2], A8[2], A9[2], A10[2], A11[2], A12[2], A13[2], A14[2], A15[2], A16[2]}.

[0123] FIG. 6B depicts the creation of bit slice vectors 420 from vector B 320 depicted in FIG. 5, in accordance with an embodiment of the present disclosure.

[0124] The elements of vector B 320 are first arranged in bit vector form as bit vector B 322. The bit vector for each element of vector B 320 is a sequence of bits from the LSB (i.e., bit position “0”) to the MSB (i.e., bit position “4”). For example, the bit vector for element B1 is {B1[0], B1[1], B1[2], B1[3], B1[4]}, where B1[0] is the value of the bit at the first bit position (i.e., the LSB), B1[1] is the value of the bit at the second bit position, B1[2] is the value of the bit at the third bit position, B1[3] is the value of the bit at the fourth bit position, and B1[4] is the value of the bit at the fifth bit position (i.e., the MSB). Similarly, the bit vector for element B2 is {B2[0], B2[1], B2[2], B2[3], B2[4]}, where B2[0] is the value of the bit at the first bit position (i.e., the LSB), B2[1] is the value of the bit at the second bit position, B2[2] is the value of the bit at the third bit position, B2[3] is the value of the bit at the fourth bit position, and B2[4] is the value of the bit at the fifth bit position (i.e., the MSB). The remaining elements of bit vector B 312 are formed in a similar manner from the remaining elements of vector B 320.

[0125] Bit slice vectors 420 are then formed from bit vector B 322. Bit slice vector 420.sup.0 is a sequence of bits formed from the bit at the first bit position of each element of bit vector B 312, i.e., {B1[0], B2[0], B3[0], B4[0], B5[0], B6[0], B7[0], B8[0], B9[0], B10[0], B11[0], B12[0], B13[0], B14[0], B15[0], B16[0]}. Bit slice vector 420.sup.1 is a sequence of bits formed from the bit at the second bit position of each element of bit vector B 312, i.e., {B1[1], B2[1], B3[1], B4[1], B5[1], B6[1], B7[1], B8[1], B9[1], B10[1], B11[1], B1.sub.2[1], B13[1], B14[1], B15[1], B16[1]}. Bit slice vector 410.sup.2 is a sequence of bits formed from the bit at the third bit position of each element of bit vector B 312, i.e., {B1[2], B2[2], B3[2], B4[2], B5[2], B6[2], B7[2], B8[2], B9[2], B10[2], B11[2], B1.sub.2[2], B13[2], B14[2], B15[2], B16[2]}. Bit slice vector 410.sup.3 is a sequence of bits formed from the bit at the fourth bit position of each element of bit vector B 312, i.e., {B1[3], B2[3], B3[3], B4[3], B5[3], B6[3], B7[3], B8[3], B9[3], B10[3], B11[3], B1.sub.2[3], B13[3], B14[3], B15[3], B16[3]}. Bit slice vector 410.sup.4 is a sequence of bits formed from the bit at the fifth bit position of each element of bit vector B 312, i.e., {B1[4], B2[4], B3[4], B4[4], B5[4], B6[4], B7[4], B8[4], B9[4], B10[4], B11[4], B1.sub.2[4], B13[4], B14[4], B15[4], B16[4]}.

[0126] FIG. 6C depicts the computation of the 1-bit dot product between bit slice vectors 410 and bit slice vectors 420 using 1-bit dot product unit 400, in accordance with an embodiment of the present disclosure.

[0127] One-bit dot product unit 400 calculates the dot product between vector A 310 and vector B 320 by multiplying bit slice vectors 410 and 420 in a particular sequence, and then outputting 32-bit scalar C 330. Generally, 1-bit dot product unit 400 multiplies each bit slice vector 410.sup.0, 410.sup.1 and 410.sup.2 with each bit slice vector 412.sup.0, 412.sup.1, 412.sup.2, 412.sup.3 and 412.sup.4, accumulates the intermediate products and then generates the 32-bit scalar C 330.

[0128] Advantageously, 1-bit dot product unit 400 calculates the dot product between any two vectors A and B with the same or different bit-width elements.

[0129] In one embodiment, the bit slice vector multiplication process is a nested loop, in which an outer loop index j selects a particular bit slice vector 410 (i.e., BA[j]), while an inner loop index k selects a particular bit slice vector 420k (i.e., BB[k]). Each iteration of the inner loop multiplies a particular bit slice vector BA[j] and a particular bit slice vector BB[k] by performing a bit-wise AND operation and then counting the number of ones that are generated using, for example, a population count function, a sequence of adders including 32 1-bit adders, 50% full adders and 50% half adders, etc. In certain embodiments, the partial reduction may be used for the count.

[0130] The nested loop may be given by Equation 1:

TABLE-US-00001 for ( j = 0; j < 3; j++ ) {  for ( k = 0; k < 5; k++ ) {   n = j + k;   int t = DP1( BA[ j ], BB[ k ] );   S += t << n;  } } Eq. 1

[0131] The function DP1( ) represents the bit-wise AND operation followed by the counting operation, the variable t stores the count value, and the variable S accumulates the values of the intermediate products. Due to the nature of the bit multiplication process, the variable t is left-shifted by the sum of the indices j and k prior to accumulation. As described above, indices j and k represent the respective bit positions of the bits in each bit slice.

[0132] For the first iteration of the nested loop, index j is 0, index k is 0, and n is 0. The function DP1( ) first performs the bit-wise AND operation between BA[0] and BB[0] to generate an intermediate bit vector b, as follows:

[00001]$b=\left\{A1\left[0\right]\&B1\left[0\right],A2\left[0\right]\&B2\left[0\right],A3\left[0\right]\&B3\left[0\right],A4\left[0\right]\&B4\left[0\right],A5\left[0\right]\&B5\left[0\right],A6\left[0\right]\&B6\left[0\right],A7\left[0\right]\&B7\left[0\right],A8\left[0\right]\&B8\left[0\right],A9\left[0\right]\&B9\left[0\right],A10\left[0\right]\&B10\left[0\right],A11\left[0\right]\&B11\left[0\right],A12\left[0\right]\&B12\left[0\right],A13\left[0\right]\&B13\left[0\right],A14\left[0\right]\&B14\left[0\right],A15\left[0\right]\&B15\left[0\right],A16\left[0\right]\&B16\left[0\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0133] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 0 bits and then added to the variable S.

[0134] For the 2.sup.nd iteration of the nested loop, index j is 0, index k is 1, and n is 1. The function DP1( ) first performs the bit-wise AND operation between BA[0] and BB[1] to generate the intermediate bit vector b, as follows:

[00002]$b=\left\{A1\left[0\right]\&B1\left[1\right],A2\left[0\right]\&B2\left[1\right],A3\left[0\right]\&B3\left[1\right],A4\left[0\right]\&B4\left[1\right],A5\left[0\right]\&B5\left[1\right],A6\left[0\right]\&B6\left[1\right],A7\left[0\right]\&B7\left[1\right],A8\left[0\right]\&B8\left[1\right],A9\left[0\right]\&B9\left[1\right],A10\left[0\right]\&B10\left[1\right],A11\left[0\right]\&B11\left[1\right],A12\left[0\right]\&B12\left[1\right],A13\left[0\right]\&B13\left[1\right],A14\left[0\right]\&B14\left[1\right],A15\left[0\right]\&B15\left[1\right],A16\left[0\right]\&B16\left[1\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0135] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 1 bit and then added to the variable S.

[0136] For the 3.sup.rd iteration of the nested loop, index j is 0, index k is 2, and n is 2. The function DP1( ) first performs the bit-wise AND operation between BA[0] and BB[2] to generate the intermediate bit vector b, as follows:

[00003]$b=\left\{A1\left[0\right]\&B1\left[2\right],A2\left[0\right]\&B2\left[2\right],A3\left[0\right]\&B3\left[2\right],A4\left[0\right]\&B4\left[2\right],A5\left[0\right]\&B5\left[2\right],A6\left[0\right]\&B6\left[2\right],A7\left[0\right]\&B7\left[2\right],A8\left[0\right]\&B8\left[2\right],A9\left[0\right]\&B9\left[2\right],A10\left[0\right]\&B10\left[2\right],A11\left[0\right]\&B11\left[2\right],A12\left[0\right]\&B12\left[2\right],A13\left[0\right]\&B13\left[2\right],A14\left[0\right]\&B14\left[2\right],A15\left[0\right]\&B15\left[2\right],A16\left[0\right]\&B16\left[2\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0137] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 2 bits and then added to the variable S.

[0138] For the 4.sup.th iteration of the nested loop, index j is 0, index k is 3, and n is 3. The function DP1( ) first performs the bit-wise AND operation between BA[0] and BB[3] to generate the intermediate bit vector b, as follows:

[00004]$b=\left\{A1\left[0\right]\&B1\left[3\right],A2\left[0\right]\&B2\left[3\right],A3\left[0\right]\&B3\left[3\right],A4\left[0\right]\&B4\left[3\right],A5\left[0\right]\&B5\left[3\right],A6\left[0\right]\&B6\left[3\right],A7\left[0\right]\&B7\left[3\right],A8\left[0\right]\&B8\left[3\right],A9\left[0\right]\&B9\left[3\right],A10\left[0\right]\&B10\left[3\right],A11\left[0\right]\&B11\left[3\right],A12\left[0\right]\&B12\left[3\right],A13\left[0\right]\&B13\left[3\right],A14\left[0\right]\&B14\left[3\right],A15\left[0\right]\&B15\left[3\right],A16\left[0\right]\&B16\left[3\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0139] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 3 bits and then added to the variable S.

[0140] For the 5th iteration of the nested loop, index j is 0, index k is 4, and n is 4. The function DP1( ) first performs the bit-wise AND operation between BA[0] and BB[4] to generate the intermediate bit vector b, as follows:

[00005]$b=\left\{A1\left[0\right]\&B1\left[4\right],A2\left[0\right]\&B2\left[4\right],A3\left[0\right]\&B3\left[4\right],A4\left[0\right]\&B4\left[4\right],A5\left[0\right]\&B5\left[4\right],A6\left[0\right]\&B6\left[4\right],A7\left[0\right]\&B7\left[4\right],A8\left[0\right]\&B8\left[4\right],A9\left[0\right]\&B9\left[4\right],A10\left[0\right]\&B10\left[4\right],A11\left[0\right]\&B11\left[4\right],A12\left[0\right]\&B12\left[4\right],A13\left[0\right]\&B13\left[4\right],A14\left[0\right]\&B14\left[4\right],A15\left[0\right]\&B15\left[4\right],A16\left[0\right]\&B16\left[4\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0141] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 4 bits and then added to the variable S.

[0142] For the 6.sup.th iteration of the nested loop, index j is 1, index k is 0, and n is 1. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[0] to generate an intermediate bit vector b, as follows:

[00006]$b=\left\{A1\left[1\right]\&B1\left[0\right],A2\left[1\right]\&B2\left[0\right],A3\left[1\right]\&B3\left[0\right],A4\left[1\right]\&B4\left[1\right],A5\left[1\right]\&B5\left[0\right],A6\left[1\right]\&B6\left[0\right],A7\left[1\right]\&B7\left[0\right],A8\left[1\right]\&B8\left[0\right],A9\left[1\right]\&B9\left[1\right],A10\left[1\right]\&B10\left[0\right],A11\left[1\right]\&B11\left[0\right],A12\left[1\right]\&B12\left[0\right],A13\left[1\right]\&B13\left[0\right],A14\left[1\right]\&B14\left[0\right],A15\left[1\right]\&B15\left[0\right],A16\left[1\right]\&B16\left[0\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0143] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 1 bit and then added to the variable S.

[0144] For the 7.sup.th iteration of the nested loop, index j is 1, index k is 1, and n is 2. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[1] to generate the intermediate bit vector b, as follows:

[00007]$b=\left\{A1\left[1\right]\&B1\left[1\right],A2\left[1\right]\&B2\left[1\right],A3\left[1\right]\&B3\left[1\right],A4\left[1\right]\&B4\left[1\right],A5\left[1\right]\&B5\left[1\right],A6\left[1\right]\&B6\left[1\right],A7\left[1\right]\&B7\left[1\right],A8\left[1\right]\&B8\left[1\right],A9\left[1\right]\&B9\left[1\right],A10\left[1\right]\&B10\left[1\right],A11\left[1\right]\&B11\left[1\right],A12\left[1\right]\&B12\left[1\right],A13\left[1\right]\&B13\left[1\right],A14\left[1\right]\&B14\left[1\right],A15\left[1\right]\&B15\left[1\right],A16\left[1\right]\&B16\left[1\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0145] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 2 bits and then added to the variable S.

[0146] For the 8.sup.th iteration of the nested loop, index j is 1, index k is 2, and n is 3. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[2] to generate the intermediate bit vector b, as follows:

[00008]$b=\left\{A1\left[1\right]\&B1\left[2\right],A2\left[1\right]\&B2\left[2\right],A3\left[1\right]\&B3\left[2\right],A4\left[1\right]\&B4\left[2\right],A5\left[1\right]\&B5\left[2\right],A6\left[1\right]\&B6\left[2\right],A7\left[1\right]\&B7\left[2\right],A8\left[1\right]\&B8\left[2\right],A9\left[1\right]\&B9\left[2\right],A10\left[1\right]\&B10\left[2\right],A11\left[1\right]\&B11\left[2\right],A12\left[1\right]\&B12\left[2\right],A13\left[1\right]\&B13\left[2\right],A14\left[1\right]\&B14\left[2\right],A15\left[1\right]\&B15\left[2\right],A16\left[1\right]\&B16\left[2\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0147] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 3 bits and then added to the variable S.

[0148] For the 9.sup.th iteration of the nested loop, index j is 1, index k is 3, and n is 4. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[3] to generate the intermediate bit vector b, as follows:

[00009]$b=\left\{A1\left[1\right]\&B1\left[3\right],A2\left[1\right]\&B2\left[3\right],A3\left[1\right]\&B3\left[3\right],A4\left[1\right]\&B4\left[3\right],A5\left[1\right]\&B5\left[3\right],A6\left[1\right]\&B6\left[3\right],A7\left[1\right]\&B7\left[3\right],A8\left[1\right]\&B8\left[3\right],A9\left[1\right]\&B9\left[3\right],A10\left[1\right]\&B10\left[3\right],A11\left[1\right]\&B11\left[3\right],A12\left[1\right]\&B12\left[3\right],A13\left[1\right]\&B13\left[3\right],A14\left[1\right]\&B14\left[3\right],A15\left[1\right]\&B15\left[3\right],A16\left[1\right]\&B16\left[3\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0149] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 4 bits and then added to the variable S.

[0150] For the 10.sup.th iteration of the nested loop, index j is 1, index k is 4, and n is 5. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[4] to generate the intermediate bit vector b as follows:

[00010]$b=\left\{A1\left[1\right]\&B1\left[4\right],A2\left[1\right]\&B2\left[4\right],A3\left[1\right]\&B3\left[4\right],A4\left[1\right]\&B4\left[4\right],A5\left[1\right]\&B5\left[4\right],A6\left[1\right]\&B6\left[4\right],A7\left[1\right]\&B7\left[4\right],A8\left[1\right]\&B8\left[4\right],A9\left[1\right]\&B9\left[4\right],A10\left[1\right]\&B10\left[4\right],A11\left[1\right]\&B11\left[4\right],A12\left[1\right]\&B12\left[4\right],A13\left[1\right]\&B13\left[4\right],A14\left[1\right]\&B14\left[4\right],A15\left[1\right]\&B15\left[4\right],A16\left[1\right]\&B16\left[4\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0151] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 5 bits and then added to the variable S.

[0152] For the 11.sup.th iteration of the nested loop, index j is 2, index k is 0, and n is 2. The function DP1( ) first performs the bit-wise AND operation between BA[2] and BB[0] to generate an intermediate bit vector b, as follows:

[00011]$b=\left\{A1\left[2\right]\&B1\left[0\right],A2\left[2\right]\&B2\left[0\right],A3\left[2\right]\&B3\left[0\right],A4\left[2\right]\&B4\left[0\right],A5\left[2\right]\&B5\left[0\right],A6\left[2\right]\&B6\left[0\right],A7\left[2\right]\&B7\left[0\right],A8\left[2\right]\&B8\left[0\right],A9\left[2\right]\&B9\left[1\right],A10\left[2\right]\&B10\left[0\right],A11\left[2\right]\&B11\left[0\right],A12\left[2\right]\&B12\left[0\right],A13\left[2\right]\&B13\left[0\right],A14\left[2\right]\&B14\left[0\right],A15\left[2\right]\&B15\left[0\right],A16\left[2\right]\&B16\left[0\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0153] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 2 bits and then added to the variable S.

[0154] For the 12.sup.th iteration of the nested loop, index j is 2, index k is 1, and n is 3. The function DP1( ) first performs the bit-wise AND operation between BA[2] and BB[1] to generate the intermediate bit vector b, as follows:

[00012]$b=\left\{A1\left[2\right]\&B1\left[1\right],A2\left[2\right]\&B2\left[1\right],A3\left[2\right]\&B3\left[1\right],A4\left[2\right]\&B4\left[1\right],A5\left[2\right]\&B5\left[1\right],A6\left[2\right]\&B6\left[1\right],A7\left[2\right]\&B7\left[1\right],A8\left[2\right]\&B8\left[1\right],A9\left[2\right]\&B9\left[1\right],A10\left[2\right]\&B10\left[1\right],A11\left[2\right]\&B11\left[1\right],A12\left[2\right]\&B12\left[1\right],A13\left[2\right]\&B13\left[2\right],A14\left[2\right]\&B14\left[1\right],A15\left[2\right]\&B15\left[1\right],A16\left[2\right]\&B16\left[1\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0155] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 3 bits and then added to the variable S.

[0156] For the 13.sup.th iteration of the nested loop, index j is 2, index k is 2, and n is 4. The function DP1( ) first performs the bit-wise AND operation between BA[1] and BB[2] to generate the intermediate bit vector b, as follows:

[00013]$b=\left\{A1\left[2\right]\&B1\left[2\right],A2\left[2\right]\&B2\left[2\right],A3\left[2\right]\&B3\left[2\right],A4\left[2\right]\&B4\left[2\right],A5\left[2\right]\&B5\left[2\right],A6\left[2\right]\&B6\left[2\right],A7\left[2\right]\&B7\left[2\right],A8\left[2\right]\&B8\left[2\right],A9\left[2\right]\&B9\left[2\right],A10\left[2\right]\&B10\left[2\right],A11\left[2\right]\&B11\left[2\right],A12\left[2\right]\&B12\left[2\right],A13\left[2\right]\&B13\left[2\right],A14\left[2\right]\&B14\left[2\right],A15\left[2\right]\&B15\left[2\right],A16\left[2\right]\&B16\left[2\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0157] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 4 bits and then added to the variable S.

[0158] For the 14.sup.th iteration of the nested loop, index j is 2, index k is 3, and n is 5. The function DP1( ) first performs the bit-wise AND operation between BA[2] and BB[3] to generate the intermediate bit vector b, as follows:

[00014]$b=\left\{A1\left[2\right]\&B1\left[3\right],A2\left[2\right]\&B2\left[3\right],A3\left[2\right]\&B3\left[3\right],A4\left[2\right]\&B4\left[3\right],A5\left[2\right]\&B5\left[3\right],A6\left[2\right]\&B6\left[3\right],A7\left[2\right]\&B7\left[3\right],A8\left[2\right]\&B8\left[3\right],A9\left[2\right]\&B9\left[3\right],A10\left[2\right]\&B10\left[3\right],A11\left[2\right]\&B11\left[3\right],A12\left[2\right]\&B12\left[3\right],A13\left[2\right]\&B13\left[3\right],A14\left[2\right]\&B14\left[3\right],A15\left[2\right]\&B15\left[3\right],A16\left[2\right]\&B16\left[3\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0159] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 5 bits and then added to the variable S.

[0160] For the 15.sup.th and final iteration of the nested loop, index j is 2, index k is 4, and n is 6. The function DP1( ) first performs the bit-wise AND operation between BA[2] and BB[4] to generate the intermediate bit vector b, as follows:

[00015]$b=\left\{A1\left[2\right]\&B1\left[4\right],A2\left[2\right]\&B2\left[4\right],A3\left[2\right]\&B3\left[4\right],A4\left[2\right]\&B4\left[4\right],A5\left[2\right]\&B5\left[4\right],A6\left[2\right]\&B6\left[4\right],A7\left[2\right]\&B7\left[4\right],A8\left[2\right]\&B8\left[4\right],A9\left[2\right]\&B9\left[4\right],A10\left[2\right]\&B10\left[4\right],A11\left[2\right]\&B11\left[4\right],A12\left[2\right]\&B12\left[4\right],A13\left[1\right]\&B13\left[4\right],A14\left[2\right]\&B14\left[4\right],A15\left[2\right]\&B15\left[4\right],A16\left[2\right]\&B16\left[4\right]\right\}$$=\left\{b1,b2,b3,b4,b5,b6,b6,b8,b9,b10,b11,b12,b13,b14,b15,b16\right\}$

[0161] The function DP1( ) then performs the population count operation on the intermediate bit vector b to count the number of bits bn that have a value of one. The result is returned and assigned to the variable t, which is left-shifted by 6 bits and then added to the variable S.

[0162] After the last loop has completed, 1-bit dot product unit 400 outputs the final value of S as 32-bit scalar C 330. For this embodiment, there are a total of 15 loops, and, optionally, a loop iteration may be skipped if either BA[j] or BB[k] has a value of zero in each bit position. While vector A 310 and vector B 320 are 16 element vectors, any vectors with the same number of elements may be accommodated.

[0163] FIG. 6D depicts a first example of the computation of the dot product between vector A 310 and vector B 320 using 1-bit dot product unit 400, in accordance with an embodiment of the present disclosure.

[0164] Vector A 310 includes sixteen 3-bit elements, i.e., A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15 and A16, all of which are equal to 1 (i.e., binary “001”). Vector B 310 includes sixteen 5-bit elements, i.e., B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15 and B16, all of which are equal to 1 (i.e., binary “00001”). Bit slice vectors 410.sup.0, 410.sup.1 and 410.sup.2 are depicted, as well as bit slice vectors 420.sup.0, 420.sup.1, 420.sup.2, 420.sup.3, and 420.sup.4. Scalar C 330 is equal to 16. Result 332 is the result of the calculation of the decimal dot product, and is also equal to 16.

[0165] FIG. 6E depicts a second example of the computation of the dot product between vector A 310 and vector B 320 using 1-bit dot product unit 400, in accordance with an embodiment of the present disclosure.

[0166] Vector A 310 includes sixteen 3-bit elements, i.e., A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15 and A16, all of which are equal to 7 (i.e., binary “111”). Vector B 310 includes sixteen 5-bit elements, i.e., B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15 and B16, all of which are equal to 31 (i.e., binary “11111”). Bit slice vectors 410.sup.0, 410.sup.1 and 410.sup.2 are depicted, as well as bit slice vectors 420.sup.0, 420.sup.1, 420.sup.2, 420.sup.3, and 420.sup.4. Scalar C 330 is equal to 3,472, and result 332 is also equal to 3,472.

[0167] FIG. 6F depicts a third example of the computation of the dot product between vector A 310 and vector B 320 using 1-bit dot product unit 400, in accordance with an embodiment of the present disclosure.

[0168] Vector A 310 includes sixteen 3-bit elements, i.e., A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15 and A16. A1 is equal to 0 (i.e., binary “000”), A2 is equal to 1 (i.e., binary “001”), A3 is equal to 1 (i.e., binary “001”), A4 is equal to 0 (i.e., binary “000”), A5 is equal to 3 (i.e., binary “011”), A6 is equal to 7 (i.e., binary “111”), A7 is equal to 7 (i.e., binary “111”), A8 is equal to 3 (i.e., binary “011”), A9 is equal to 3 (i.e., binary “011”), A10 is equal to 7 (i.e., binary “111”), A11 is equal to 7 (i.e., binary “111”), A12 is equal to 3 (i.e., binary “011”), A13 is equal to 0 (i.e., binary “000”), A14 is equal to 1 (i.e., binary “001”), A15 is equal to 1 (i.e., binary “001”), and A16 is equal to 0 (i.e., binary “000”).

[0169] Vector B 310 includes sixteen 5-bit elements, i.e., B1, B2, B3, B4, B5, B6, B7, B8, B9, B10, B11, B12, B13, B14, B15 and B16. B1 is equal to 1 (i.e., binary “00001”), B2 is equal to 2 (i.e., binary “00010”), B3 is equal to 2 (i.e., binary “00010”), B4 is equal to 1 (i.e., binary “00001”), B5 is equal to 3 (i.e., binary “00011”), B6 is equal to 6 (i.e., binary “00110”), B7 is equal to 6 (i.e., binary “00110”), B8 is equal to 3 (i.e., binary “00011”), B9 is equal to 3 (i.e., binary “00011”), B10 is equal to 9 (i.e., binary “01001”), B11 is equal to 9 (i.e., binary “01001”), B12 is equal to 3 (i.e., binary “00011”), B13 is equal to 1 (i.e., binary “001”), B14 is equal to 2 (i.e., binary “0010”), B15 is equal to 2 (i.e., binary “00010”), and B16 is equal to 1 (i.e., binary “00001”).

[0170] Bit slice vectors 410.sup.0, 410.sup.1 and 410.sup.2 are depicted, as well as bit slice vectors 420.sup.0, 420.sup.1, 420.sup.2, 420.sup.3, and 420.sup.4. Scalar C 330 is equal to 254, and result 332 is also equal to 254.

[0171] In one embodiment, the conversion of vectors A and B to bit slice representation may be performed by a system processor, such as, for example, a central processing unit (CPU), etc. In another embodiment, the conversion of vectors A and B to bit slice representation may be performed by an MMA processor, such as, for example a processor or processor core, microprocessor, controller, microcontroller, etc.

[0172] Embodiments of the present disclosure advantageously break down variable bit-width vectors to 1-bit operations to increase power efficiency for variable bit-width matrix multiplications. The power reduction for the embodiment described above would be approximately (8.Math.8)/(3.Math.5)=64/15=4.3×.

[0173] In another embodiment, a first matrix and a second matrix are multiplied to generate a third matrix. The multiplication of each row of the first matrix with each column of the second matrix is a dot product operation that generates one element of the third matrix.

[0174] FIGS. 7A and 7B depict the creation of bit slice tensor 455 from matrix X 340, in accordance with an embodiment of the present disclosure.

[0175] Matrix X 340 and matrix Y 360 are multiplied to generate matrix Z 380. Matrix X 340 is a 4×4 matrix having 16 3-bit elements. The first row includes elements x.sup.1.sub.1, x.sup.1.sub.2, x.sup.1.sub.3 and x.sup.1.sub.4, the second row includes elements x.sup.2.sub.1, x.sup.2.sub.2, x.sup.2.sub.3 and x.sup.2.sub.4, the third row includes elements x.sup.3.sub.1, x.sup.3.sub.2, x.sup.3.sub.3 and x.sup.3.sub.4, and the fourth row includes elements x.sup.4.sub.1, x.sup.4.sub.2, x.sup.4.sub.3 and x.sup.4.sub.4.

[0176] Matrix Y 360 is a 4×4 matrix having 16 5-bit elements. The first column includes elements y.sup.1.sub.1, y.sup.2.sub.1, y.sup.3.sub.1 and y.sup.4.sub.1, the second column includes elements y.sup.1.sub.2, y.sup.2.sub.2, y.sup.3.sub.2 and y.sup.4.sub.2, the third column includes elements y.sup.1.sub.3, y.sup.2.sub.3, y.sup.3.sub.3 and y.sup.4.sub.3, and the fourth column includes elements y.sup.1.sub.4, y.sup.2.sub.4, y.sup.3.sub.4 and y.sup.4.sub.4.

[0177] Matrix Z 380 is a 4×4 matrix having 16 32-bit elements. The first row includes elements z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3 and z.sup.1.sub.4, the second row includes elements z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3 and z.sup.2.sub.4, the third row includes elements z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3 and z.sup.3.sub.4, and the fourth row includes elements z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4.

[0178] Generally, the elements of the rows of matrix X 340 are first arranged in bit vector form. The elements of the first row of matrix X 340 are arranged in bit vector form as bit vector X 341, the elements of the second row of matrix X 340 are arranged in bit vector form as bit vector X 342, the elements of the third row of matrix X 340 are arranged in bit vector form as bit vector X 343, and the elements of the fourth row of matrix X 340 are arranged in bit vector form as bit vector X 344.

[0179] The bit vector for each element of bit vectors X 341, 342, 343 and 344 is a sequence of bits from the LSB (i.e., bit position “0”) to the MSB (i.e., bit position “2”). With respect to bit vector X 341, the bit vector for element x.sup.1.sub.1 is {x.sup.1.sub.1[0], x.sup.1.sub.1[1], x.sup.1.sub.1[2]}, where x.sup.1.sub.1[0] is the value of the bit at the first bit position (i.e., the LSB), x.sup.1.sub.1[1] is the value of the bit at the second bit position, and x.sup.1.sub.1[2] is the value of the bit at the third bit position (i.e., the MSB). Similarly, the bit vector for element x.sup.1.sub.2 is {x.sup.1.sub.2[0], x.sup.1.sub.2[1], x.sup.1.sub.2[2]}, the bit vector for element x.sup.1.sub.3 is {x.sup.1.sub.3[0], x.sup.1.sub.3[1], x.sup.1.sub.3[2]}, and the bit vector for element x.sup.1.sub.4 is {x.sup.1.sub.4[0], x.sup.1.sub.4[1], x.sup.1.sub.4[2]}. Bit vectors X 342, 343 and 343 are formed in a similar manner from the second, third and fourth rows of matrix X 340, respectively.

[0180] Bit slice vector set 440 includes bit slice vectors 441, 442, 443 and 444, which are formed from bit vectors X 341, 342, 343 and 344, respectively. Bit slice vector 441.sup.0 is a sequence of bits formed from the bit at the first bit position of each element of bit vector X 341, i.e., {x.sup.1.sub.1[0], x.sup.1.sub.2[0], x.sup.1.sub.3[0], x.sup.1.sub.4[0]}. Bit slice vector 441.sup.1 is a sequence of bits formed from the bit at the second bit position of each element of bit vector X 341, i.e., {x.sup.1.sub.1[1], x.sup.1.sub.2[1], x.sup.1.sub.3[1], x.sup.1.sub.4[1]}. Bit slice vector 441.sup.2 is a sequence of bits formed from the bit at the third bit position of each element of bit vector X 341, i.e., {x.sup.1.sub.1[2], x.sup.1.sub.2[2], x.sup.1.sub.3[2], x.sup.1.sub.4[2]}.

[0181] Bit slice vectors 442, 443 and 444 are formed in a similar manner from bit vectors X 342, 343 and 344, respectively. Bit slice vectors 442 include bit slice vectors 442.sup.0, 442.sup.1 and 442.sup.2, bit slice vectors 443 include bit slice vectors 443.sup.0, 443.sup.1 and 443.sup.2, and bit slice vectors 444 include bit slice vectors 444.sup.0, 444.sup.1 and 444.sup.2.

[0182] Bit slice tensor set 450 includes bit slice tensors 451, 452, 453 and 454, which are formed from bit slice vectors 441, 442, 443 and 444, respectively. Bit slice tensor 451 is formed from the sequence of bit slice vectors 441.sup.0, 441.sup.1, and 441.sup.2. Bit slice tensor 452 is formed from the sequence of bit slice vectors 442.sup.0, 442.sup.1, and 442.sup.2. Bit slice tensor 453 is formed from the sequence of bit slice vectors 443.sup.0, 443.sup.1, and 443.sup.2. Bit slice tensor 454 is formed from the sequence of bit slice vectors 444.sup.0, 444.sup.1, and 444.sup.2.

[0183] X bit slice tensor 455 is formed from bit slice tensors 451, 452, 453 and 454.

[0184] FIGS. 7C and 7D depict the creation of bit slice tensor 475 from matrix Y 360, in accordance with an embodiment of the present disclosure.

[0185] Generally, the elements of the columns of matrix Y 360 are first arranged in bit vector form. The elements of the first column of matrix Y 360 are arranged in bit vector form as bit vector Y 361, the elements of the second column of matrix Y 360 are arranged in bit vector form as bit vector Y 362, the elements of the third column of matrix Y 360 are arranged in bit vector form as bit vector Y 363, and the elements of the fourth column of matrix Y 360 are arranged in bit vector form as bit vector Y 364.

[0186] The bit vector for each element of bit vectors Y 361, 362, 363 and 364 is a sequence of bits from the LSB (i.e., bit position “0”) to the MSB (i.e., bit position “4”). With respect to bit vector Y 361, the bit vector for element y.sup.1.sub.1 is {y.sup.1.sub.1[0], y.sup.1.sub.1[1], y.sup.1.sub.1[2], y.sup.1.sub.1[3], y.sup.1.sub.1[4]}, where y.sup.1.sub.1[0] is the value of the bit at the first bit position (i.e., the LSB), y.sup.1.sub.1[1] is the value of the bit at the second bit position, y.sup.1.sub.1[2] is the value of the bit at the third bit position, y.sup.1.sub.1[3] is the value of the bit at the fourth bit position, and y.sup.1.sub.1[4] is the value of the bit at the fifth bit position (i.e., the MSB). Similarly, the bit vector for element y.sup.2.sub.1 is {y.sup.2.sub.1[0], y.sup.2.sub.1[1], y.sup.2.sub.1[2], y.sup.2.sub.1[3], y.sup.2.sub.1[4]}, the bit vector for element y.sup.3.sub.1 is {y.sup.3.sub.1[0], y.sup.3.sub.1[1], y.sup.3.sub.1[2], y.sup.3.sub.1[3], y.sup.3.sub.1[4]}, and the bit vector for element y.sup.4.sub.1 is {y.sup.4.sub.1[0], y.sup.4.sub.1[1], y.sup.4.sub.1[2], y.sup.4.sub.1[3], y.sup.4.sub.1[4]}. Bit vectors Y 362, 363 and 363 are formed in a similar manner from the second, third and fourth columns of matrix Y 360, respectively.

[0187] Bit slice vector set 460 includes bit slice vectors 461, 462, 463 and 464, which are formed from bit vectors Y 361, 362, 363 and 364, respectively. Bit slice vector 461.sup.0 is a sequence of bits formed from the bit at the first bit position of each element of bit vector Y 361, i.e., {y.sup.1.sub.1[0], y.sup.2.sub.1[0], y.sup.3.sub.1[0], y.sup.4.sub.1[0]}. Bit slice vector 461.sup.1 is a sequence of bits formed from the bit at the second bit position of each element of bit vector Y 361, i.e., {y.sup.1.sub.1[.sup.1], y.sup.2.sub.1[.sup.1], y.sup.3.sub.1[.sup.1], y.sup.4.sub.1[.sup.1]}. Bit slice vector 461.sup.2 is a sequence of bits formed from the bit at the third bit position of each element of bit vector Y 361, i.e., {y.sup.1.sub.1[.sup.2], y.sup.2.sub.1[.sup.2], y.sup.3.sub.1[.sup.2], y.sup.4.sub.1[.sup.2]}. Bit slice vector 461.sup.3 is a sequence of bits formed from the bit at the fourth bit position of each element of bit vector Y 361, i.e., {y.sup.1.sub.1[3], y.sup.2.sub.1[3], y.sup.3.sub.1[3], y.sup.4.sub.1[3]}. Bit slice vector 461.sup.4 is a sequence of bits formed from the bit at the fifth bit position of each element of bit vector Y 361, i.e., {y.sup.1.sub.1[4], y.sup.2.sub.1[4], y.sup.3.sub.1[4], y.sup.4.sub.1[4]}.

[0188] Bit slice vectors 462, 463 and 464 are formed in a similar manner from bit vectors Y 362, 363 and 364, respectively. Bit slice vectors 462 include bit slice vectors 462.sup.0, 462.sup.1, 462.sup.2, 462.sup.3 and 462.sup.4, bit slice vectors 463 include bit slice vectors 463.sup.0, 463.sup.1, 463.sup.2, 463.sup.3 and 463.sup.4, and bit slice vectors 464 include bit slice vectors 464.sup.0, 464.sup.1, 464.sup.2, 464.sup.3 and 464.sup.4.

[0189] Bit slice tensor set 470 includes bit slice tensors 471, 472, 473 and 474, which are formed from bit slice vectors 461, 462, 463 and 464, respectively. Bit slice tensor 471 is formed from the sequence of bit slice vectors 461.sup.0, 461.sup.1, 461.sup.2, 461.sup.3 and 461.sup.4. Bit slice tensor 472 is formed from the sequence of bit slice vectors 462.sup.0, 462.sup.1, 462.sup.2, 462.sup.3 and 462.sup.4. Bit slice tensor 473 is formed from the sequence of bit slice vectors 463.sup.0, 463.sup.1, 463.sup.2, 463.sup.3 and 463.sup.4. Bit slice tensor 474 is formed from the sequence of bit slice vectors 464.sup.0, 464.sup.1, 464.sup.2, 464.sup.3 and 464.sup.4.

[0190] Y bit slice tensor 475 is formed from bit slice tensors 471, 472, 473 and 474.

[0191] FIG. 8A depicts a data flow diagram for BSDP array 650, while FIG. 8B depicts BSDP unit 500, in accordance with embodiments of the present disclosure.

[0192] In this embodiment, BSDP array 650 is an output stationary array that implements a bit slice dot product operation using a 4×4 array of BSDP units 500, i.e., BSDP.sub.1, BSDP.sub.2, BSDP.sub.3, BSDP.sub.4, BSDP.sub.5, BSDP.sub.6, BSDP.sub.7, BSDP.sub.8, BSDP.sub.9, BSDP.sub.10, BSDP.sub.11, BSDP.sub.12, BSDP.sub.13, BSDP.sub.14, BSDP.sub.15 and BSDP.sub.16. Each BSDP unit 500 calculates a dot product between one row of matrix X and one column of matrix Y by multiplying certain elements of X bit slice tensor 455 and certain elements of Y bit slice tensor 475, in a particular sequence, and then outputting the result.

[0193] For example, BSDP.sub.1 multiplies bit slice tensors 451 and 471, accumulates the intermediate products and then generates the result. As described above, bit slice tensor 451 represents the elements of the first row of matrix X 340 (i.e., x.sup.1.sub.1, x.sup.1.sub.2, x.sup.1.sub.3 and x.sup.1.sub.4), and bit slice tensor 471 represents the elements of the first column of matrix Y 360 340 (i.e., y.sup.1.sub.1, y.sup.2.sub.1, y.sup.3.sub.1 and y.sup.4.sub.1), and the result is z.sup.1.sub.1. In addition to the bit slice vectors of bit slice tensor 451 and the bit slice vectors of bit slice tensor 471, the sum of indices j and k, i.e., “n”, is provided to BSDP.sub.1.

[0194] BSDP array 650 may be a systolic or non-systolic array. FIG. 8A depicts the data flow for a non-systolic array. During each processing cycle, the appropriate element of X bit slice tensor 455 is provided to each BSDP unit 500 in each row, and the appropriate element of Y bit slice tensor 475 is provided to each BSDP unit 500 in each column. For example, during the first processing cycle (i.e., Cycle 1), bit slice vector 441.sup.0 (i.e., BX.sup.1[0]) is provided to BSDP.sub.1, BSDP.sub.2, BSDP.sub.3 and BSDP.sub.4, while bit slice vector 461.sup.0 (i.e., BY.sub.1[0]) is provided to BSDP.sub.1, BSDP.sub.5, BSDP.sub.9 and BSDP.sub.13.

[0195] Advantageously, BSDP unit 500 calculates the dot product between a row of a first matrix and a column of a second matrix with the same or different bit-width elements.

[0196] BSDP unit 500 includes bitwise AND circuit 510, intermediate product circuit 520, adder circuit 530 and accumulator register 540. BSDP unit 500 receives a bit slice vector BX[j], a bit slice vector BY[k], and “n”. Bitwise AND circuit 510 performs a bitwise AND on BX[j] and BX[k] to generate an intermediate bit vector z. Intermediate product circuit 520 determines the number of ones in the intermediate bit vector z, left-shifts this count by index sum “n” to generate an intermediate product. Adder circuit 530 adds the intermediate value to the value stored in accumulator register 540, and then stores the accumulated value in accumulator register 540.

[0197] In many embodiment, the elements of matrix X 340 and matrix Y 360 are unsigned integer values (e.g., UINT8, UINT32, etc.). In certain embodiments, the elements of matrix X 340 and matrix Y 360 may be signed or unsigned integer values, and a sign signal may be generated for each processing cycle and provided to each BSDP unit 500 to correct the accumulated value for the sign of the matrix elements, which advantageously supports processing signed operations as well as mixed unsigned and signed operations.

[0198] FIGS. 8C and 8D depict a first example of the multiplication of matrix X 340 and matrix Y 360 to generate matrix Z 380 using BSDP array 650, in accordance with an embodiment of the present disclosure.

[0199] Matrix X 340 includes sixteen 3-bit elements, i.e., x.sup.1.sub.1, x.sup.1.sub.2, x.sup.1.sub.3, x.sup.1.sub.4, x.sup.2.sub.1, x.sup.2.sub.2, x.sup.2.sub.3, x.sup.2.sub.4, x.sup.3.sub.1, x.sup.3.sub.2, x.sup.3.sub.3, x.sup.3.sub.4, x.sup.4.sub.1, x.sup.4.sub.2, x.sup.4.sub.3 and x.sup.4.sub.4, all of which are equal to 1 (i.e., binary “001”). Matrix Y 360 includes sixteen 5-bit elements, i.e., y.sup.1.sub.1, y.sup.2.sub.1, y.sup.3.sub.1 y.sup.4.sub.1, y.sup.1.sub.2, y.sup.2.sub.2, y.sup.3.sub.2, y.sup.4.sub.2, y.sup.1.sub.3, y.sup.2.sub.3, y.sup.3.sub.3, y.sup.4.sub.3, y.sup.1.sub.4, y.sup.2.sub.4, y.sup.3.sub.4 and y.sup.4.sub.4, all of which are equal to 1 (i.e., binary “00001”). Matrix Z 380 includes sixteen 32-bit elements, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4. Result matrix 382 presents the result of multiplying the decimal values of matrix X 340 and matrix Y 360; the values of all of the elements of result matrix 382 are equal to 4.

[0200] Bit slice vectors 441.sup.0, 441.sup.1 and 441.sup.2 of bit slice tensor 451, bit slice vectors 442.sup.0, 442.sup.1 (not labeled for clarity) and 442.sup.2 of bit slice tensor 452, bit slice vectors 443.sup.0, 443.sup.1 (not labeled for clarity) and 443.sup.2 of bit slice tensor 453, and bit slice vectors 444.sup.0, 444.sup.1 (not labeled for clarity) and 444.sup.2 of bit slice tensor 453 are depicted.

[0201] Similarly, bit slice vectors 461.sup.0, 461.sup.1 (not labeled for clarity), 461.sup.2 (not labeled for clarity), 461.sup.3 (not labeled for clarity) and 461.sup.4 of bit slice tensor 471, bit slice vectors 462.sup.0, 462.sup.1 (not labeled for clarity), 462.sup.2 (not labeled for clarity), 462.sup.3 (not labeled for clarity) and 462.sup.4 of bit slice tensor 472, bit slice vectors 463.sup.0, 463.sup.1 (not labeled for clarity), 463.sup.2 (not labeled for clarity), 463.sup.3 (not labeled for clarity) and 463.sup.4 of bit slice tensor 473, and bit slice vectors 464.sup.0, 464.sup.1, 464.sup.2, 464.sup.3 and 464.sup.4 of bit slice tensor 474 are depicted.

[0202] Computation array 384 depicts the computation of the bit slice dot product between a respective row of matrix X 340 and a respective column of matrix Y 360 by each BSDB unit 500 in BSDP array 650. The dot product computation is described above with respect to 1-bit dot product unit 400.

[0203] The value for each element of matrix z 380 depicted in FIG. 8D, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4, are depicted in a box directly beneath the element name. The values of all of the elements of matrix z 380 are equal to 4, and match the values of the elements of results matrix 382 depicted in FIG. 8C.

[0204] FIGS. 8E and 8F depict a second example of the multiplication of matrix X 340 and matrix Y 360 to generate matrix Z 380 using BSDP array 650, in accordance with an embodiment of the present disclosure.

[0205] Matrix X 340 includes sixteen 3-bit elements, i.e., x.sup.1.sub.1, x.sup.1.sub.2, x.sup.1.sub.3, x.sup.1.sub.4, x.sup.2.sub.1, x.sup.2.sub.2, x.sup.2.sub.3, x.sup.2.sub.4, x.sup.3.sub.1, x.sup.3.sub.2, x.sup.3.sub.3, x.sup.3.sub.4, x.sup.4.sub.1, x.sup.4.sub.2, x.sup.4.sub.3 and x.sup.4.sub.4, all of which are equal to 7 (i.e., binary “111”). Matrix Y 360 includes sixteen 5-bit elements, i.e., y.sup.1.sub.1, y.sup.2.sub.1, y.sup.3.sub.1, y.sup.4.sub.1, y.sup.1.sub.2, y.sup.2.sub.2, y.sup.3.sub.2, y.sup.4.sub.2, y.sup.1.sub.3, y.sup.2.sub.3, y.sup.3.sub.3, y.sup.4.sub.3, y.sup.1.sub.4, y.sup.2.sub.4, y.sup.3.sub.4 and y.sup.4.sub.4, all of which are equal to 31 (i.e., binary “11111”). Matrix Z 380 includes sixteen 32-bit elements, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4. Result matrix 382 presents the result of multiplying the decimal values of matrix X 340 and matrix Y 360; the values of all of the elements of result matrix 382 are equal to 868.

[0206] Bit slice vectors 441.sup.0, 441.sup.1 and 441.sup.2 of bit slice tensor 451, bit slice vectors 442.sup.0, 442.sup.1 (not labeled for clarity) and 442.sup.2 of bit slice tensor 452, bit slice vectors 443.sup.0, 443.sup.1 (not labeled for clarity) and 443.sup.2 of bit slice tensor 453, and bit slice vectors 444.sup.0, 444.sup.1 (not labeled for clarity) and 444.sup.2 of bit slice tensor 453 are depicted.

[0207] Similarly, bit slice vectors 461.sup.0, 461.sup.1 (not labeled for clarity), 461.sup.2 (not labeled for clarity), 461.sup.3 (not labeled for clarity) and 461.sup.4 of bit slice tensor 471, bit slice vectors 462.sup.0, 462.sup.1 (not labeled for clarity), 462.sup.2 (not labeled for clarity), 462.sup.3 (not labeled for clarity) and 462.sup.4 of bit slice tensor 472, bit slice vectors 463.sup.0, 463.sup.1 (not labeled for clarity), 463.sup.2 (not labeled for clarity), 463.sup.3 (not labeled for clarity) and 463.sup.4 of bit slice tensor 473, and bit slice vectors 464.sup.0, 464.sup.1, 464.sup.2, 464.sup.3 and 464.sup.4 of bit slice tensor 474 are depicted.

[0208] Computation array 384 depicts the computation of the bit slice dot product between a respective row of matrix X 340 and a respective column of matrix Y 360 by each BSDB unit 500 in BSDP array 650. The dot product computation is described above with respect to 1-bit dot product unit 400.

[0209] The value for each element of matrix z 380 depicted in FIG. 8F, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4, are depicted in a box directly beneath the element name. The values of all of the elements of matrix z 380 are equal to 868, and match the values of the elements of results matrix 382 depicted in FIG. 8E.

[0210] FIGS. 8G and 8H depict a third example of the multiplication of matrix X 340 and matrix Y 360 to generate matrix Z 380 using BSDP array 650, in accordance with an embodiment of the present disclosure.

[0211] Matrix X 340 includes sixteen 3-bit elements, i.e., x.sup.1.sub.1, x.sup.1.sub.2, x.sup.1.sub.3, x.sup.1.sub.4, x.sup.2.sub.1, x.sup.2.sub.2, x.sup.2.sub.3, x.sup.2.sub.4, x.sup.3.sub.1, x.sup.3.sub.2, x.sup.3.sub.3, x.sup.3.sub.4, x.sup.4.sub.1, x.sup.4.sub.2, x.sup.4.sub.3 and x.sup.4.sub.4. Element x.sup.1.sub.1 is equal to 0 (i.e., binary “000”), x.sup.1.sub.2 is equal to 1 (i.e., binary “001”), x.sup.1.sub.3 is equal to 1 (i.e., binary “001”), x.sup.1.sub.4 is equal to 0 (i.e., binary “000”), x.sup.2.sub.1 is equal to 3 (i.e., binary “011”), x.sup.2.sub.2 is equal to 7 (i.e., binary “111”), x.sup.2.sub.3 is equal to 7 (i.e., binary “111”), x.sup.2.sub.4 is equal to 3 (i.e., binary “011”), x.sup.3.sub.1 is equal to 3 (i.e., binary “011”), x.sup.3.sub.2 is equal to 7 (i.e., binary “111”), x.sup.3.sub.3 is equal to 7 (i.e., binary “111”), x.sup.3.sub.4 is equal to 3 (i.e., binary “011”), x.sup.4.sub.1 is equal to 0 (i.e., binary “000”), x.sup.4.sub.2 is equal to 1 (i.e., binary “001”), x.sup.4.sub.3 is equal to 1 (i.e., binary “001”), and x.sup.4.sub.4 is equal to 0 (i.e., binary “000”).

[0212] Matrix Y 360 includes sixteen 5-bit elements, i.e., y.sup.1.sub.1, y.sup.2.sub.1, y.sup.3.sub.1, y.sup.4.sub.1, y.sup.1.sub.2, y.sup.2.sub.2, y.sup.3.sub.2, y.sup.4.sub.2, y.sup.1.sub.3, y.sup.2.sub.3, y.sup.3.sub.3, y.sup.4.sub.3, y.sup.1.sub.4, y.sup.2.sub.4, y.sup.3.sub.4 and y.sup.4.sub.4. Element y.sup.1.sub.1 is equal to 1 (i.e., binary “00001”), y.sup.2.sub.1 is equal to 2 (i.e., binary “00010”), y.sup.3.sub.1 is equal to 2 (i.e., binary “00010”), y.sup.4.sub.1 is equal to 1 (i.e., binary “00001”), y.sup.1.sub.2 is equal to 3 (i.e., binary “00011”), y.sup.2.sub.2 is equal to 6 (i.e., binary “00110”), y.sup.3.sub.2 is equal to 6 (i.e., binary “00110”), y.sup.4.sub.2 is equal to 3 (i.e., binary “00011”), y.sup.1.sub.3 is equal to 3 (i.e., binary “00011”), y.sup.2.sub.3 is equal to 9 (i.e., binary “01001”), y.sup.3.sub.3 is equal to 9 (i.e., binary “01001”), y.sup.4.sub.3 is equal to 3 (i.e., binary “00011”), y.sup.1.sub.4 is equal to 1 (i.e., binary “001”), y.sup.2.sub.4 is equal to 2 (i.e., binary “0010”), y.sup.3.sub.4 is equal to 2 (i.e., binary “00010”), and y.sup.4.sub.4 is equal to 1 (i.e., binary “00001”).

[0213] Matrix Z 380 includes sixteen 32-bit elements, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4. Result matrix 382 presents the result of multiplying the decimal values of matrix X 340 and matrix Y 360.

[0214] Bit slice vectors 441.sup.0, 441.sup.1 and 441.sup.2 of bit slice tensor 451, bit slice vectors 442.sup.0, 442.sup.1 (not labeled for clarity) and 442.sup.2 of bit slice tensor 452, bit slice vectors 443.sup.0, 443.sup.1 (not labeled for clarity) and 443.sup.2 of bit slice tensor 453, and bit slice vectors 444.sup.0, 444.sup.1 (not labeled for clarity) and 444.sup.2 of bit slice tensor 453 are depicted.

[0215] Similarly, bit slice vectors 461.sup.0, 461.sup.1 (not labeled for clarity), 461.sup.2 (not labeled for clarity), 461.sup.3 (not labeled for clarity) and 461.sup.4 of bit slice tensor 471, bit slice vectors 462.sup.0, 462.sup.1 (not labeled for clarity), 462.sup.2 (not labeled for clarity), 462.sup.3 (not labeled for clarity) and 462.sup.4 of bit slice tensor 472, bit slice vectors 463.sup.0, 463.sup.1 (not labeled for clarity), 463.sup.2 (not labeled for clarity), 463.sup.3 (not labeled for clarity) and 463.sup.4 of bit slice tensor 473, and bit slice vectors 464.sup.0, 464.sup.1, 464.sup.2, 464.sup.3 and 464.sup.4 of bit slice tensor 474 are depicted.

[0216] Computation array 384 depicts the computation of the bit slice dot product between a respective row of matrix X 340 and a respective column of matrix Y 360 by each BSDB unit 500 in BSDP array 650. The dot product computation is described above with respect to 1-bit dot product unit 400.

[0217] The value for each element of matrix z 380 depicted in FIG. 8H, i.e., z.sup.1.sub.1, z.sup.1.sub.2, z.sup.1.sub.3, z.sup.1.sub.4, z.sup.2.sub.1, z.sup.2.sub.2, z.sup.2.sub.3, z.sup.2.sub.4, z.sup.3.sub.1, z.sup.3.sub.2, z.sup.3.sub.3, z.sup.3.sub.4, z.sup.4.sub.1, z.sup.4.sub.2, z.sup.4.sub.3 and z.sup.4.sub.4, are depicted in a box directly beneath the element name, i.e., 4, 12, 18, 4, 34, 102, 144, 34, 34, 102, 144, 34, 4, 12, 18 and 4, respectively. The values of all of the elements of matrix z 380 match the values of the elements of results matrix 382 depicted in FIG. 8G.

[0218] FIG. 9 depicts a block diagram of MMA 600, in accordance with embodiments of the present disclosure.

[0219] MMA 600 includes I/O interface 605, controller 610, memory 615, register 620, register 630, register 640 and BSDP array 650.

[0220] In this embodiment, BSDP array 650 includes 16 BSDP units 500 arranged in a 4×4 array; other numbers of BSDP units 500 and arrangements are also contemplated, such as, for example, four BSDP units 500 arranged in a 2×2 array, nine BSDP units 500 arranged in a 3×3 array, 25 BSDP units 500 arranged in a 5×5 array, 36 BSDP units 500 arranged in a 6×6 array, 49 BSDP units 500 arranged in a 7×7 array, 64 BSDP units 500 arranged in a 8×8 array, etc. Non-symmetric arrangements, such as a 2×3 array, a 3×4 array, a 4×5 array, a 4×6 array, etc., may be advantageous for certain applications. Each BSDP unit 500 is coupled to register 620, register 630 and register 640, and calculates a dot product for one element of converted output data matrix 216.

[0221] For example, the BSDP unit 500 located in the first row and the first column (i.e., BSDP.sub.1) of BSDP array 650 may calculate the dot products of the 1.sup.st row of converted weight matrix 212 and the 1.sup.st, 5.sup.th, 9.sup.th and 13.sup.th columns of converted input data matrix 214, using bit slice tensor matrices, to generate the o.sup.1.sub.1, o.sup.1.sub.5, o.sup.1.sub.9 and o.sup.1.sub.13 elements of converted output data matrix 216.

[0222] I/O interface 605 is coupled to bus 710, controller 610 and memory 615. I/O interface 605 includes a microcontroller that sends data to, and receives data and commands from, processor 720, memory 730, etc. The microcontroller implements a set of instructions that controls the data flow and the operation of BSDP units 500.

[0223] In some embodiments, a dedicated controller, microcontroller, field programmable gate array (FPGA), etc., may control the data flow and the operation of MMA 600. For example, the controller may implement load/store (L/S) instructions, memory mapped I/O (MMIO), direct memory access (DMA), etc., to load elements of X bit slice tensor 455 and associated data into register 620, to load elements of Y bit slice tensor 475 and associated data into register 630, start the matrix multiply operation, read back the output matrix from register 640, etc. In one embodiment, a software module executing on a CPU calculates the bit slice tensors and related data for each matrix, and then sends these data and the appropriate commands to MMA 600 to upload memory 615, registers 620 and 630, start the matrix multiply operation, read back the results from register 640, etc. In another embodiment, the software module sends the matrices to MMA 600, and then controller 610 calculates the bit slice tensor data and related data (i.e., n) for each matrix, upload registers 620 and 630, start the matrix multiply operation, read back the results from register 640, etc.

[0224] Generally, register 620 simultaneously provides certain data from X bit slice tensor 455 to each row of BSDP units 500 in BSDP array 650, register 630 simultaneously provides certain data from Y bit slice tensor 475 and other related data (i.e., n) to each column of BSDP units 500 in BSDP array 650, and register 640 stores the elements of the output matrix in the multiplication operation.

[0225] FIG. 10 depicts a block diagram of system 700, in accordance with an embodiment of the present disclosure.

[0226] Computer 702 includes bus 710 coupled to one or more processors 720, memory 730, I/O interfaces 740, display interface 750, one or more communication interfaces 760 and one or more MMAs 600. Generally, I/O interfaces 740 are coupled to I/O devices 742 using a wired or wireless connection, display interface 750 is coupled to display 752, and communication interface 760 is connected to network 762 using a wired or wireless connection.

[0227] Bus 710 is a communication system that transfers data between processor 720, memory 730, I/O interfaces 740, display interface 750, communication interface 760, MMA 600, as well as other components not depicted in FIG. 10. Power connector 712 is coupled to bus 710 and a power supply (not shown).

[0228] Processor 720 includes one or more general-purpose or application-specific microprocessors that executes instructions to perform control, computation, input/output, etc. functions for computer 702. Processor 720 may include a single integrated circuit, such as a micro-processing device, or multiple integrated circuit devices and/or circuit boards working in cooperation to accomplish the functions of processor 720. In addition, processor 720 may execute computer programs or modules, such as operating system 732, software modules 734, etc., stored within memory 730. For example, software modules 734 may include an ML application, an ANN application, a CNN application, etc.

[0229] Generally, storage element or memory 730 stores instructions for execution by processor 720 and data. Memory 730 may include a variety of non-transitory computer-readable medium that may be accessed by processor 720. In various embodiments, memory 730 may include volatile and nonvolatile medium, non-removable medium and/or removable medium. For example, memory 730 may include any combination of random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), read only memory (ROM), flash memory, cache memory, and/or any other type of non-transitory computer-readable medium.

[0230] Memory 730 contains various components for retrieving, presenting, modifying, and storing data. For example, memory 730 stores software modules that provide functionality when executed by processor 720. The software modules include operating system 732 that provides operating system functionality for computer 702. Software modules 734 provide various functionality, such as image classification using convolutional neural networks, etc. Data 736 may include data associated with operating system 732, software modules 734, etc.

[0231] I/O interfaces 740 are configured to transmit and/or receive data from I/O devices 742. I/O interfaces 740 enable connectivity between processor 720 and I/O devices 742 by encoding data to be sent from processor 720 to I/O devices 742, and decoding data received from I/O devices 742 for processor 720. Generally, data may be sent over wired and/or wireless connections. For example, I/O interfaces 740 may include one or more wired communications interfaces, such as USB, Ethernet, etc., and/or one or more wireless communications interfaces, coupled to one or more antennas, such as WiFi, Bluetooth, cellular, etc.

[0232] Generally, I/O devices 742 provide input to computer 702 and/or output from computer 702. As discussed above, I/O devices 742 are operably connected to computer 702 using a wired and/or wireless connection. I/O devices 742 may include a local processor coupled to a communication interface that is configured to communicate with computer 702 using the wired and/or wireless connection. For example, I/O devices 742 may include a keyboard, mouse, touch pad, joystick, etc.

[0233] Display interface 750 is configured to transmit image data from computer 702 to monitor or display 752.

[0234] Communication interface 760 is configured to transmit data to and from network 762 using one or more wired and/or wireless connections. Network 762 may include one or more local area networks, wide area networks, the Internet, etc., which may execute various network protocols, such as, for example, wired and/or wireless Ethernet, Bluetooth, etc. Network 762 may also include various combinations of wired and/or wireless physical layers, such as, for example, copper wire or coaxial cable networks, fiber optic networks, Bluetooth wireless networks, WiFi wireless networks, CDMA, FDMA and TDMA cellular wireless networks, etc.

[0235] MMA 600 is configured to multiply matrices and generate output matrices to support various applications implemented by software modules 734.

[0236] The embodiments described herein are combinable.

[0237] In one embodiment, a system includes a memory, a processor coupled to the memory, and a matrix multiply accelerator (MMA) coupled to the processor and the memory. The memory is configured to store at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution. The processor is configured to, for the weight matrix, generate, based on the bit resolution, a number of bit slice vectors for each row, and generate a bit slice weight tensor based on the bit slice vectors for each row; and, for the input data matrix, generate, based on the bit resolution, a number of bit slice vectors for each column, and generate a bit slice input data tensor based on the bit slice vectors for each column. The MMA is configured to receive the bit slice weight tensor and the bit slice input data tensor, and multiply the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

[0238] In another embodiment of the system, the number of columns of the weight matrix is the same as the number of rows of the input data matrix; and, for each row of the weight matrix, each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.

[0239] In another embodiment of the system, for each column of the input data matrix, each bit slice vector includes one bit from each element within the column; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the input data matrix.

[0240] In another embodiment of the system, the MMA includes a memory; a controller coupled to the memory; a first register, coupled to the controller and the memory, configured to store at least a portion of the bit slice input data tensor; a second register, coupled to the controller and the memory, configured to store at least a portion of the bit slice weight tensor; a third register, coupled to the controller and the memory, configured to store at least a portion of the output data matrix; and an array of bit slice dot product (BSDP) elements, coupled to the controller and the first, second and third registers, configured to multiply the bit slice weight tensor and the bit slice input data tensor, each BSDP element configured to generate a dot product between one row of the weight matrix and one column of the input data matrix.

[0241] In another embodiment of the system, each BSDP element includes a bit-wise AND circuit configured to input a first operand from the first register, input a second operand from the second register, and output a resultant value; a popcount circuit configured to receive the resultant value and output an intermediate value; an ADDER circuit configured to add the intermediate value to an accumulated value; and an accumulation register configured to store the accumulated value, and output a final accumulated value to the third register.

[0242] In another embodiment of the system, the first operand is a bit slice vector from the bit slice input data tensor having an index k equal to the associated bit position of the bit slice vector; and the second operand is a bit slice vector from the bit slice weight tensor having an index j equal to the associated bit position of the bit slice vector.

[0243] In another embodiment of the system, the popcount circuit is configured to receive an index value from the second register, the index value being equal to j+k; count a number of bits set to one in the resultant value to generate a population count value; and left-shift the population count value based on the index value to generate the intermediate value.

[0244] In one embodiment, a further system includes a memory, a processor coupled to the memory, and a matrix multiply accelerator (MMA) coupled to the processor and the memory. The memory is configured to store at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution. The MMA includes a local memory, an array of bit slice dot product (BSDP) elements, and a controller coupled to the local memory and the array. The controller is configured to receive the weight matrix and the input data matrix; for the weight matrix, generate, based on the bit resolution, a number of bit slice vectors for each row, and generate a bit slice weight tensor based on the bit slice vectors for each row; for the input data matrix, generate, based on the bit resolution, a number of bit slice vectors for each column, and generate a bit slice input data tensor based on the bit slice vectors for each column. The array is configured to multiply the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

[0245] In another embodiment of the further system, the number of columns of the weight matrix is the same as the number of rows of the input data matrix; and, for each row of the weight matrix, each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.

[0246] In another embodiment of the further system, for each column of the input data matrix, each bit slice vector includes one bit from each element within the column; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the input data matrix.

[0247] In another embodiment of the further system, the MMA further includes a first register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice input data tensor; a second register, coupled to the controller and the local memory, configured to store at least a portion of the bit slice weight tensor; and a third register, coupled to the controller and the local memory, configured to store at least a portion of the output data matrix. The array is coupled to the first, second and third registers, and each BSDP element is configured to generate a dot product between one row of the weight matrix and one column of the input data matrix.

[0248] In another embodiment of the further system, each BSDP element includes a bit-wise AND circuit configured to input a first operand from the first register, input a second operand from the second register, and output a resultant value; a popcount circuit configured to receive the resultant value and output an intermediate value; an ADDER circuit configured to add the intermediate value to an accumulated value; and an accumulation register configured to store the accumulated value, and output a final accumulated value to the third register.

[0249] In another embodiment of the further system, the first operand is a bit slice vector from the bit slice input data tensor having an index k equal to the associated bit position of the bit slice vector; and the second operand is a bit slice vector from the bit slice weight tensor having an index j equal to the associated bit position of the bit slice vector.

[0250] In another embodiment of the further system, the popcount circuit is configured to receive an index value from the second register, the index value being equal to j+k; count a number of bits set to one in the resultant value to generate a population count value; and left-shift the population count value based on the index value to generate the intermediate value.

[0251] In one embodiment, a method includes, at a memory, storing at least one weight matrix and at least one input data matrix, the weight matrix having a number of rows, a number of columns, a number of elements and a bit resolution, the input data matrix including a number of rows, a number of columns, a number of elements and a bit resolution. At a processor or a matrix multiply accelerator (MMA), for the weight matrix, generating, based on the bit resolution, a number of bit slice vectors for each row, generating a bit slice weight tensor based on the bit slice vectors for each row; for the input data matrix, generating, based on the bit resolution, a number of bit slice vectors for each column, generating a bit slice input data tensor based on the bit slice vectors for each column. At the MMA, multiplying the bit slice weight tensor and the bit slice input data tensor to generate an output data matrix.

[0252] In another embodiment of the method, the number of columns of the weight matrix is the same as the number of rows of the input data matrix; and, for each row of the weight matrix, each bit slice vector includes one bit from each element within the row; each bit slice vector is associated with a different bit position; and the number of bit slice vectors is the same as the bit resolution of the weight matrix.