Power Optimization In An Artificial Intelligence Processor

Abstract

In one embodiment, the present disclosure includes a method of reducing power in an artificial intelligence processor. For each cycle, over a plurality of cycles, an AI model is translated into operations executable on an artificial intelligence processor. The translating is based on power parameters that correspond to power consumption and performance of the artificial intelligence processor. The AI processor is configured with the executable operations, and input activation data sets are processed. Accordingly, result sets, power consumption data, and performance data are generated and stored over the plurality of cycles. The method further includes training an AI algorithm using the stored parameters, the power consumption data, and the performance data. A trained AI algorithm outputs a plurality of optimized parameters to reduce power consumption of the AI processor. The AI model is then translated into optimized executable operations based on the plurality of optimized parameters.

Claims

1. A non-transitory computer-readable storage medium having stored thereon computer executable instructions, which when executed by a compiler operating on at least one computer processor, cause the at least one computer processor to be operable for: receiving a set of parameters produced using at least power consumption data and performance data generated by executing a first plurality of operations on an artificial intelligence (AI) processor over a first time period; and translating a model for the AI processor into a second plurality of operations by using at least a subset of the parameters from the set controlling utilization of resources in the AI processor over a second time period during execution of the second operations on the AI processor, wherein a power consumed for executing the second operations on the AI processor over the second time period is less than another power consumed for executing the first operations on the AI processor over the first time period, the power consumed over the second time period maintained below a threshold level.

2. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for training an AI algorithm using at least the power consumption data and the performance data to produce the set of parameters.

3. The computer-readable storage medium of claim 2, wherein the trained AI algorithm comprises a plurality of elements representing values and mathematical operations associated with the second operations.

4. The computer-readable storage medium of claim 1, wherein the compiler comprises a plurality of stages, and the at least one computer processor is further operable for: reconfiguring the model into another model for the AI processor; and translating the other model into the second operations.

5. The computer-readable storage medium of claim 4, wherein the at least one computer processor is further operable for: reconfiguring the model into the other model by selectively mapping a plurality of elements of the model modeling other resources in the AI processor utilized over the first time period into a plurality of elements of the other model modeling the resources utilized over the second time period using at least the subset of the parameters to reduce an amount of power consumed by the AI processor over the second time period.

6. The computer-readable storage medium of claim 4, wherein the at least one computer processor is further operable for: translating the other model into the second operations by selectively scheduling the second operations for execution on the AI processor over the second time period using at least the subset of the parameters.

7. The computer-readable storage medium of claim 4, wherein the at least one computer processor is further operable for: translating the other model into the second operations by scheduling one or more no operation (NOOP) operations using at least the subset of the parameters to reduce an amount of power consumed by the AI processor over the second time period.

8. The computer-readable storage medium of claim 4, wherein the at least one computer processor is further operable for: reconfiguring the model into the other model by converting convolutions into matrix multiplications performed by a first portion of the resources and into vector operations performed by a second portion of the resources resulting into fewer matrix multiplications, more vector operations, and the consumed power over the second time period below the threshold level.

9. The computer-readable storage medium of claim 4, wherein the at least one computer processor is further operable for: reconfiguring the model into the other model by selectively controlling modifications of the model using least the subset of the parameters, the modifications reducing an amount of power consumed at the AI processor over the second time period.

10. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for: resolving dependencies in the model to determine a plurality of operations for parallel execution on the AI processor; and including the plurality of operations into the other model.

11. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for: dynamically configuring the resources for performing the second operations over the second time period, based on at least the subset of the parameters.

12. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for: determining data paths to be utilized in the AI processor during the execution of the second operations based on at least the subset of the parameters.

13. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for: determining sub-circuits in a plurality of circuits to be utilized in the AI processor during the execution of the second operations based on at least the subset of the parameters.

14. The computer-readable storage medium of claim 1, wherein the at least one computer processor is further operable for: dynamically turning on and off the resources at predetermined time during the execution of the second operations on the AI processor, based on at least the subset of the parameters.

15. A system comprising: an artificial intelligence (AI) processor comprising a plurality of circuits for inferring a solution to an AI problem defined by a model for the AI processor by executing a plurality of operations to perform convolutions; and a dynamic compiler having means for: receiving a set of parameters produced using at least power consumption data and performance data generated by executing a first plurality of operations on the AI processor over a first time period, and translating a model for the AI processor into a second plurality of operations by using at least a subset of the parameters from the set controlling utilization of resources in the AI processor over a second time period during execution of the second operations on the AI processor, wherein a power consumed for executing the second operations on the AI processor over the second time period is less than another power consumed for executing the first operations on the AI processor over the first time period, the power consumed over the second time period maintained below a threshold level.

16. The system of claim 15, wherein the model comprises a tensor flow model for the AI processor.

17. The system of claim 15, wherein the AI processor comprises a matrix multiplier circuit, a memory circuit, a vector processor circuit, and a switch matrix circuit.

18. A method comprising: receiving, at a compiler operating on a computer, a set of parameters produced using at least power consumption data and performance data generated by executing a first plurality of operations on an artificial intelligence (AI) processor over a first time period; and translating, in the compiler, a model for the AI processor into a second plurality of operations by using at least a subset of the parameters from the set controlling utilization of resources in the AI processor over a second time period during execution of the second operations on the AI processor, wherein a power consumed for executing the second operations on the AI processor over the second time period is less than another power consumed for executing the first operations on the AI processor over the first time period, the power consumed over the second time period maintained below a threshold level.

19. The method of claim 18, wherein an AI algorithm is trained using at least the power consumption data and the performance data to produce the set of parameters, the trained AI algorithm comprises a plurality of elements representing values and mathematical operations associated with the second operations.

20. The method of claim 18, further comprising: reconfiguring the model into the other model by selectively mapping a plurality of elements of the model modeling other resources in the AI processor utilized over the first time period into a plurality of elements of the other model modeling the resources utilized over the second time period using at least the subset of the parameters to reduce an amount of power consumed by the AI processor over the second time period; and translating the other model into the second operations.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG. 1 illustrates an AI processor system according to one embodiment.

[0018] FIG. 2 illustrates a method of optimizing power and performance in an AI processor according to an embodiment.

[0019] FIG. 3 illustrates power parameters for a first stage of an AI compiler according to an embodiment.

[0020] FIG. 4A illustrates an artificial intelligence (AI) model represented as a graph and being collapsed to have fewer elements according to an embodiment.

[0021] FIG. 4B Illustrates a convolution according to an embodiment.

[0022] FIG. 5 illustrates power parameters for a third stage of an AI compiler according to an embodiment.

[0023] FIG. 6 illustrates executable operations from a third stage of a compiler according to an embodiment.

[0024] FIG. 7 illustrates dynamic programming in an AI processor according to an embodiment.

[0025] FIG. 8 illustrates dynamically turning different circuit block subsystems on and off at predetermined time periods according to an embodiment.

DETAILED DESCRIPTION

[0026] In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present disclosure. Such examples and details are not to be construed as unduly limiting the elements of the claims or the claimed subject matter as a whole. It will be evident to one skilled in the art, based on the language of the different claims, that the claimed subject matter may include some or all of the features in these examples, alone or in combination, and may further include modifications and equivalents of the features and techniques described herein.

[0027] FIG. 1 illustrates an AI processor system according to one embodiment. Features and advantages of the present disclosure include translating an artificial intelligence model based on parameters (sometimes referred to herein as power parameters) to produce executable operations that optimize power and performance when executed in an AI processor. For example, an AI model 102 may be translated into executable instructions using a compiler 103 executing on server computer system 101, for example. One example AI model 102 is a neural network model comprising a network of weights. A more specific example may include a Tensorflow model. Compiler 103 translates AI model 102 into executable operations performed on an artificial intelligence processor circuit 110. The term compiler is used here broadly to refer to one or more executable software components that receive the AI model, perform a number of processing steps, and output instructions for configuring the AI processor 110. In one embodiment, compiler 103 may comprise a plurality of stages. For example, a first stage 104 may receive the AI model and perform model optimization to produce an optimized AI model (e.g., with fewer elements or elements that have been reconfigured to achieve the same result more efficiently). In this example, the optimized model is received by a second stage 105, which may convert the AI model to executable operations. As described in more detail below, the executable operations may be linked to particular circuit blocks in the AI processor 110. For example, operations may be linked to memory reads or writes in a memory circuit 111, matrix multiplications in a matrix multiplier 112, vector multiplications in a vector processor 114, data transfers over bus 115, or other data transformations and manipulations in other circuit blocks 115 of AI processor 110, for example. In this example, second stage 105 may perform a scheduling function, which may deterministically schedule each executable operation to implement the AI model on input values 150 (activations) received in an input/output circuit 113, for example, and produce output results 151.

[0028] Advantageously, compiler 103 may be configured with parameters to modify the translation process to optimize power and performance. For example, translating the AI model into the executable operations may be based on a plurality of parameters. The parameters may modify various steps of the translation process that impact power and performance. Translating an AI model into executable operations may entail a wide range of choices that impact power and performance (e.g., what operations can be performed in parallel, the structure of the optimized AI model itself, the number of matrix multiplications versus the number of vector operations, how the model is mapped to the particular operations, and the scheduling of such operations in the AI processor). Power parameters may be used to modify a wide range of decisions made by compiler 103, thereby changing the performance and power achieved by AI processor 110. Power and performance are observable results stemming from different values of parameters and different combinations of such values, for example. Accordingly, embodiments of the disclosure may advantageously gather and store power and performance data, and the parameter values that generated them, and train an AI algorithm 141 to optimize the parameters such that power and performance are optimized.

[0029] For example, an AI model 102 may be received and translated by compiler 103 to produce executable operations using one set of power parameters. The AI processor may then be configured with the executable operations. Once configured, a plurality of input activation data sets may be processed in the AI processor 110 to generate result sets, power consumption data, and performance data.

[0030] The power parameters may be used as features input to an AI algorithm. Accordingly, a corpus of power parameters, power consumption data, and performance data over many execution cycles (e.g., where activations are processed and results produced) may be stored at 142 (e.g., in a data repository on computer system 101) to form a training set, for example. The training set may be used to train AI algorithm 141. The trained AI algorithm, in turn, may output a plurality of optimized parameters to reduce power consumption of the artificial intelligence processor, for example. The parameters may be trained to optimize power consumption and performance based on the corpus 142 of past power and performance for different parameters, for example. Accordingly, AI model 102 may be translated again into a new set of executable operations based on the optimized parameters from AI algorithm 141. When the new executable operations are used to configure AI processor 110, input activations may be processed more efficiently.

[0031] FIG. 2 illustrates a method of optimizing power and performance in an AI processor according to an embodiment. At 201, an AI model is translated into a first plurality of executable operations based on a plurality of parameters (e.g., power parameters that modify the translation process to change the power and performance of the AI processor when the operations are executed). At 202, the AI processor is configured with the operations. At 203, a plurality of input activation data sets are processed in the AI processor to generate results sets, power consumption data, and performance data based on the operations. At 204, at least a portion of the parameters, the power consumption data, and the performance data are stored (e.g., in a data repository). At 205, an artificial intelligence algorithm is trained using the stored parameters, the power consumption data, and the performance data for a plurality of said translating and processing steps. For example, the stored data set may be based on translating the same or different AI models over numerous different translation cycles using different parameters to obtain different power and performance data from processing numerous activations. The trained artificial intelligence algorithm outputs a plurality of optimized parameters to reduce power consumption of the AI processor. At 206, the AI model is again translated into a second plurality of operations based on the plurality of optimized parameters. The second plurality of executable operations may be deployed to the AI processor to process input activations with improved power and performance, for example.

[0032] FIG. 3 illustrates power parameters for a first stage of an AI compiler according to an embodiment. As mentioned above, the first stage 301 of an AI compiler may receive an AI model, reconfigure the model, and output an optimized AI model. Translating the AI model into an optimized AI model may include pruning the model to remove training nodes, collapsing nodes that merely involve constant values (e.g., with no dependencies on activations), normalizing the nodes (e.g., include zero padding or do not include zero paddingsometimes referred to as same and valid conversion), converting elements of the model to matrix multiplications or vector operations, and other data transformations and manipulations required to execute the AI model, for example. In some embodiments, AI model 302 may be represented as a graph, such as a Tensorflow graph, for example. The graph may model an AI algorithm as a series of elements comprising values and mathematical operations, for example. An example optimization is shown in FIG. 4A, which shows a model being collapsed to have fewer elements. In FIG. 4A, constant values 5 and 2 do not depend on any input activation values, x. Accordingly, the model may be optimized as shown. This is sometimes referred to as constant folding.

[0033] As illustrated in FIG. 3, stage 301 receives first parameters for selectively mapping a plurality of input elements of the AI model (e.g., nodes of a graph) into a plurality of output elements of the output AI model. For instance, mapping the input elements to one set of output elements may correspond to a first power consumption and mapping the input elements to a different set of output elements may correspond to a second power consumptioneven though the resulting AI processing of the activations produces the same result, for example. In this example, the input AI model 302 and optimized output AI model 303 may be neural network models. The parameters may selectively control how AI model 302 is modified to obtain output AI model 303. The parameters may selectively control modifications of the input model 302 that impact power and performance. For example, stage 301 may resolves dependencies in the AI model 302 to determine model operations that can be performed in parallel. One or more of the parameters may adjust the number of parallel model operations included in the output AI model 303, for example. More parallelism may increase performance but also increase power consumption at a given moment. Thus, adjusting parallelism using the parameters may optimize the performance of the system, for example.

[0034] As another example, translating model 302 into model 303 may include converting convolutions into operations supported by circuit blocks of the AI processor. An example of matrix convolution of a kernel of weights in an AI model and an array of input values is shown in FIG. 4B. In this illustrative example, an MN kernel of weight values 402 is applied across an array of input values 401. The kernel 402 is applied sequentially across the width and height of the array of input values 401. At each step, the overlapping weight values in the kernel are multiplied by the input values, and the results are summed. Accordingly, matrix convolutions may be decomposed into multiply and add operations, which may be assigned to different circuit blocks in the AI processor, such as a matrix multiplier or vector processor. Multiplier arrays are faster but typically consume more power than a vector processor. Accordingly, assigning operations in a convolution may impact performance and power. In one embodiment, one translation of an AI model into an optimized AI model may use parameter values for stage 301 which result in more matrix multiplications, fewer vector operations, and a higher power consumption with higher performance (e.g., faster results). However, to reduce power consumption, another translation of an AI model into an optimized AI model may be based on first values of the parameters which result in fewer matrix multiplications, more vector operations, and a lower power. The AI algorithm 141 of FIG. 1 may determine, through training, that an optimum performance/power tradeoff may include reducing power (e.g., for certain operations or at certain times in the processing) by setting the parameter values so that a lower power vector processor circuit block is used rather than a more power hungry matrix multiplier, for example. As another example, convolutions may be performed in a variety of ways (or modes). Some modes may be faster and consume more power (e.g., by performing convolutions in parallel) while other modes may be slower but consume less power. In an AI process that supports multiple convolution algorithms, embodiments of the disclosure may include parameters to selecting the convolution mode. Accordingly, such parameters, and the resulting performance and power, may be stored and used to train and optimize (together with the other parameters) an AI algorithm to produce optimum power and performance, for example.

[0035] FIG. 5 illustrates power parameters for a second stage of an AI compiler according to an embodiment. In one embodiment, the compiler comprises a stage 701 for translating the optimized AI model into scheduled executable operands (aka instructions). The executable operands may be deployed to an AI processor to configure the AI processor to execute the AI model on an input data set and produce a result set. Stage 701 may further receive parameters for selectively mapping the AI model to executable operands and selectively scheduling the executable operands, for example.

[0036] In various embodiments, an AI processor may comprise a variety of hardware resources, the utilization of which may involve a power/performance tradeoff. For example, in one embodiment, an AI processor may comprise numerous data streams (or data paths. Utilizing more data paths may improve performance, but increase power consumption. Conversely, utilizing fewer data paths may reduce performance, but decrease power consumption. Accordingly, one or more parameters may control the number of data paths used in the AI processor. Additionally, in one embodiment, various processing circuit blocks in the AI processor may comprise multiple subblocks (e.g., computational elements). For example, a memory, vector processor, switch matrix, or matrix multiplier may comprise numerous subblocks (e.g., tiles or cells). If more subblocks are used, performance may improve, but more power is consumed. Conversely, if fewer subblocks are used, performance may be reduced, but less power is consumed. Accordingly, one or more parameters may control how many subblocks in each of the different circuit blocks of an AI processor are used during various times during processing.

[0037] Furthermore, selective scheduling may be used to throttle power and performance up or down based on the parameters. For example, a first schedule of the executable operands may correspond to a first power consumption. However, changing the parameters may change various aspects of the scheduling, which may impact power and performance. Accordingly, a second schedule of the executable operands may correspond to different power consumption. As but one example, stage 701 may selectively schedule one or more no operation (NOOP) operations based on the parameters to control the amount of power consumed during execution. This technique is illustrated in FIG. 6. FIG. 6 illustrates concurrent operations 801-803 being performed in an AI processor. At time 810, operations are being performed at 801 and 802. Additional operations may be performed during time 810. However, to reduce power consumption, a NOOP may be scheduled at 803 during time 810 rather than an additional operation so that the power is reduced or maintained below a threshold level, for example, during a particular time period. Similarly, at time 811 operations may be occurring at 802 and 803. Thus, to reduce power consumption, stage 701 may schedule a NOOP instruction at 801 during time 811, for example.

[0038] FIG. 7 illustrates dynamic programming an AI processor according to an embodiment. In one example embodiment, parameters received by stage 701 in FIG. 5 may produce executable operations that (sometimes referred to as instruction based power control or programmable power control (Prog Pwr Ctrl)) to turn different subsystems of various circuit blocks on and off to reduce power consumption, for example. In one embodiment, an AI processor 901 may include a matrix multiplier circuit 902, memory circuit 903, vector processor circuit 904, and switch matrix circuit 905, for example. Each circuit block 902-905 may include a number of subsystems as mentioned above. For instance, matrix multiplier 902 may be divided into a plurality of multipliers 921-922, memory 903 may be divided into a plurality of memory subcells 923-924, vector processor 904 may be divided into a plurality of units for performing vector operations 925-926, and switch matrix 905 may be comprise subsystems for performing different switching operations 927-928, for example. Additionally, each circuit block 902-905 may receive executable instructions 910-917, which may be dynamically programmed to change power consumption based on parameters received in stage 701, for example. The instructions may include data bits for selectively turning subsystems 921-928 in each circuit block on and off to reduce power consumption, for example.

[0039] FIG. 8 illustrates dynamically turning different circuit block subsystems on and off at predetermined time periods according to an embodiment. As mentioned above, executable operations are generated and scheduled by stage 701. Parameters received by stage 701 may control the bits in instructions sent to the AI processor so that different subsystems are turned on and off or are used (or not) in an optimal way determined by the trained AI algorithm 141. FIG. 8 illustrates an example of turning subblocks of a matrix multiplier (MM) on and off (or used/not used). Different subsystems of the MM may correspond to groups of multiplier cells, for example. Here, operations configured in the AI processor dynamically program the MM to turn different circuit block subsystems on and off at predetermined time periods during processing of input activation data sets in the AI processor to reduce power. For example, an instruction may turn on multiplier subsections 1001 and turn off the other subsection as shown in FIG. 8 during a first time period. Similarly, during a later time period, a smaller subsection 1002 is turned on and other subsections may be turned off. From FIG. 8, one advantageous embodiment of the disclosure is illustrated. In particular, subsystems of a matrix multiplier may be turned on and off in a contiguous manner for more efficient processing, for example. In addition to turning subsections on and off to save power, stage 701 may allocate multiplication operations based on which subsystems are already on so that fewer subsystems need to be turned on and off because using a subsystem that is already on may be more power efficient than turning subsystems on and off, for example. While performance may, at times, be reduced by not utilizing the full resources of the AI processor, overall power and performance may be optimized when the parameters used to configure the dynamic programming are trained by AI algorithm, for example.

[0040] The above description illustrates various embodiments of the present disclosure along with examples of how aspects of the particular embodiments may be implemented. The above examples should not be deemed to be the only embodiments, and are presented to illustrate the flexibility and advantages of the particular embodiments as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents may be employed without departing from the scope of the present disclosure as defined by the claims.