MEMORY LOOKUP COMPUTING MECHANISMS

Abstract

According to some example embodiments of the present disclosure, in a method for a memory lookup mechanism in a high-bandwidth memory system, the method includes: using a memory die to conduct a multiplication operation using a lookup table (LUT) methodology by accessing a LUT, which includes floating point operation results, stored on the memory die; sending, by the memory die, a result of the multiplication operation to a logic die including a processor and a buffer; and conducting, by the logic die, a matrix multiplication operation using computation units.

Claims

1. A storage system comprising: a first device configured to store matrix data; and a second device configured to: receive intermediate matrix multiplication result data based on the matrix data from the first device; and output a matrix multiplication result based on the intermediate matrix multiplication result data.

2. The storage system of claim 1, wherein the first device supports in-memory computing operations.

3. The storage system of claim 1, wherein the first device further stores a lookup table, and wherein the first device is further configured to output the intermediate matrix multiplication result data based on the lookup table.

4. The storage system of claim 1, wherein the first device corresponds to a first storage device, the storage system further comprising a second storage device distinct from the first storage device.

5. The storage system of claim 4, wherein the second storage device comprises dynamic random access memory.

6. The storage system of claim 1, wherein the second device is configured to execute an artificial intelligence application based on access to the first device.

7. A storage device comprising: storage media configured to store matrix data; and an interface configured to send intermediate matrix multiplication result data, based on the matrix data, to a second device configured to output a matrix multiplication result based on the intermediate matrix multiplication result data.

8. The storage device of claim 7, wherein the intermediate matrix data is distinct from the matrix data.

9. The storage device of claim 7, wherein the storage media supports in-memory computing operations.

10. The storage device of claim 7, wherein the storage media further stores a lookup table, and wherein the storage media is further configured to output the intermediate matrix multiplication result data based on the lookup table.

11. The storage device of claim 7, wherein the storage media corresponds to a first storage device, the storage device further comprising a second storage device distinct from the first storage device.

12. The storage device of claim 11, wherein the second storage device comprises dynamic random access memory.

13. The storage device of claim 7, wherein the second device is configured to execute an artificial intelligence application based on access to the storage media.

14. A method comprising: storing matrix data at a storage device; and sending, from the storage device, intermediate matrix multiplication result data, based on the matrix data, to a second device configured to output a matrix multiplication result based on the intermediate matrix multiplication result data.

15. The method of claim 14, wherein the storage device supports in-memory computing operations.

16. The method of claim 14, further comprising storing at the storage device a lookup table, and wherein the storage device is further configured to output the intermediate matrix multiplication result data based on the lookup table.

17. The method of claim 14, wherein the storage device corresponds to a first storage device, and wherein a second storage device is distinct from the first storage device.

18. The method of claim 17, wherein the second storage device comprises dynamic random access memory.

19. The method of claim 14, wherein the second device is configured to execute an artificial intelligence application based on access to the storage device.

20. The method of claim 14, wherein the intermediate matrix data is distinct from the matrix data.

21. A method comprising: performing, by a memory device, a first operation using a lookup table (LUT) stored on the memory die; sending, by the memory device, a result of the first operation to a processor device; and performing, by the processor device, a matrix multiplication operation using the result of the first operation performed on the memory device.

22. The method of claim 21, wherein the first operation includes a multiplication operation.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Some embodiments can be understood in more detail from the following description taken in conjunction with the accompanying drawings, in which:

[0027] FIGS. 1A and 1B are block diagrams of an architecture of a second generation high-bandwidth memory HBM2 system according to one or more embodiments of the present disclosure;

[0028] FIGS. 2A-2D are block diagrams illustrating design options in accordance with one or more embodiments of the disclosure.

[0029] FIG. 3 provides an illustration of how memory LUTs perform scalar-vector multiplication.

[0030] FIG. 4 displays a flow diagram 400 outlining different components of the floating points such as the floating point exponent, fraction, or sign.

[0031] FIG. 5 illustrates a block diagram 500 illustrating redundant sections of a LUT, and a table 510 displaying the number of bits used by the row index, column index, and result size.

[0032] FIG. 6 displays a visual representation of an algorithm that can save memory space by “folding” the LUT.

[0033] FIG. 7 shows a block diagram 700 representing a LUT, and a table 710 for displaying the number of bits used by a row index, column index, and result size of a LUT storing only partial product multiplication results.

[0034] FIG. 8 shows a block diagram 800 representing a LUT that contains fraction partial product results; and

[0035] FIG. 9 provides a visual representation of a layer-by-layer convolution operation.

DETAILED DESCRIPTION

[0036] Features of the inventive concept and methods of accomplishing the same may be understood more readily by reference to the following detailed description of embodiments and the accompanying drawings. Hereinafter, embodiments will be described in more detail with reference to the accompanying drawings, in which like reference numbers refer to like elements throughout. The present disclosure, however, may be embodied in various different forms, and should not be construed as being limited to only the illustrated embodiments herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of the present disclosure to those skilled in the art. Accordingly, processes, elements, and techniques that are not necessary to those having ordinary skill in the art for a complete understanding of the aspects and features of embodiments of the present disclosure may not be described. Unless otherwise noted, like reference numerals denote like elements throughout the attached drawings and the written description, and thus, descriptions thereof will not be repeated. In the drawings, the relative sizes of elements, layers, and regions may be exaggerated for clarity.

[0037] In the following description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of various embodiments. It is apparent, however, that various embodiments may be practiced without these specific details or with one or more equivalent arrangements. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring various embodiments.

[0038] It will be understood that when an element, layer, region, or component is referred to as being “on,” “connected to,” or “coupled to” another element, layer, region, or component, it can be directly on, connected to, or coupled to the other element, layer, region, or component, or one or more intervening elements, layers, regions, or components may be present. However, “directly connected/directly coupled” refers to one component directly connecting or coupling another component without an intermediate component. Meanwhile, other expressions describing relationships between components such as “between,” “immediately between” or “adjacent to” and “directly adjacent to” may be construed similarly. In addition, it will also be understood that when an element or layer is referred to as being “between” two elements or layers, it can be the only element or layer between the two elements or layers, or one or more intervening elements or layers may also be present.

[0039] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the singular forms “a” and “an” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “have,” “having,” “includes,” and “including,” when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

[0040] As used herein, the term “substantially,” “about,” “approximately,” and similar terms are used as terms of approximation and not as terms of degree, and are intended to account for the inherent deviations in measured or calculated values that would be recognized by those of ordinary skill in the art. “About” or “approximately,” as used herein, is inclusive of the stated value and means within an acceptable range of deviation for the particular value as determined by one of ordinary skill in the art, considering the measurement in question and the error associated with measurement of the particular quantity (i.e., the limitations of the measurement system). For example, “about” may mean within one or more standard deviations, or within ±30%, 20%, 10%, 5% of the stated value. Further, the use of “may” when describing embodiments of the disclosure refers to “one or more embodiments of the disclosure.” As used herein, the terms “use,” “using,” and “used” may be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Also, the term “exemplary” is intended to refer to an example or illustration.

[0041] When a certain embodiment may be implemented differently, a specific process order may be performed differently from the described order. For example, two consecutively described processes may be performed substantially at the same time or performed in an order opposite to the described order.

[0042] Various embodiments are described herein with reference to sectional illustrations that are schematic illustrations of embodiments and/or intermediate structures. As such, variations from the shapes of the illustrations as a result, for example, of manufacturing techniques and/or tolerances, are to be expected. Further, specific structural or functional descriptions disclosed herein are merely illustrative for the purpose of describing embodiments according to the concept of the present disclosure. Thus, embodiments disclosed herein should not be construed as limited to the particular illustrated shapes of regions, but are to include deviations in shapes that result from, for instance, manufacturing. For example, an implanted region illustrated as a rectangle will, typically, have rounded or curved features and/or a gradient of implant concentration at its edges rather than a binary change from implanted to non-implanted region. Likewise, a buried region formed by implantation may result in some implantation in the region between the buried region and the surface through which the implantation takes place. Thus, the regions illustrated in the drawings are schematic in nature and their shapes are not intended to illustrate the actual shape of a region of a device and are not intended to be limiting.

[0043] The electronic or electric devices and/or any other relevant devices or components according to embodiments of the disclosure described herein may be implemented utilizing any suitable hardware, firmware (e.g., an application-specific integrated circuit), software, or a combination of software, firmware, and hardware. For example, the various components of these devices may be formed on one integrated circuit (IC) chip or on separate IC chips. Further, the various components of these devices may be implemented on a flexible printed circuit film, a tape carrier package (TCP), a printed circuit board (PCB), or formed on one substrate. Further, the various components of these devices may be a process or thread, running on one or more processors, in one or more computing devices, executing computer program instructions and interacting with other system components for performing the various functionalities described herein. The computer program instructions are stored in a memory which may be implemented in a computing device using a standard memory device, such as, for example, a random access memory (RAM). The computer program instructions may also be stored in other non-transitory computer readable media such as, for example, a CD-ROM, flash drive, or the like. Also, a person of skill in the art should recognize that the functionality of various computing devices may be combined or integrated into a single computing device, or the functionality of a particular computing device may be distributed across one or more other computing devices without departing from the spirit and scope of the exemplary embodiments of the disclosure.

[0044] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and/or the present specification, and should not be interpreted in an idealized or overly formal sense, unless expressly so defined herein.

[0045] Large data batch sizes are used to shorten total training time without losing accuracy. This approach can scale-up by increasing utilization of computation/memory utilization on a single high-performance process. Alternatively, this approach can scale out by distributing learning on multiple high-performance processors using model parallelism and data parallelism.

[0046] Deep learning applications exhibit various compute-to-memory ratios. Convolutional layers have high compute-to-memory ratios. Fully connected layers/recurrent layers have medium-low compute-to-memory ratios. Elementwise/scalar operations have low compute-to-memory ratios. Application wise compute-to-memory ratio depends on the above combination.

[0047] For low compute-to-memory ratio workloads, compute-centric throughput processors may suitably use large memory capacity and bandwidth. On-chip memory cannot contain large batch size data nor large layer size data. State of the art processors, such as graphics processing units (GPUs), central processing units (CPUs), and field-programmable gate arrays (FGPAs), employ High Bandwidth Memory (HBM). For high computer-to-memory ratio workloads, if the workload is computation bound, memory bandwidth and memory capacity is under-utilized.

[0048] FIGS. 1A and 1B are block diagrams of an architecture of a second generation high-bandwidth memory (“HBM2”) system according to one or more embodiments of the present disclosure. FIG. 1A provides an aerial view of the architecture. FIG. 1B provides a side view of the architecture.

[0049] Referring to FIG. 1A, embodiments of the present disclosure provide a system for memory lookup computer mechanism, or a second generation of High Bandwidth Memory (HBM2) system 100. The HBM2 system 100 contains a multiple host die 110 connected to multiple three-dimensional HBM2 memory stacks 120. According to various embodiments, the multiple host die 110 can be connected to up to four HBM2 memory stacks 120.

[0050] Referring to FIG. 1B, the three-dimensional HBM2 memory stack 120 contains multiple HBM2 memory dies 130. As illustrated, the stacks 120 may have 4 to 8 memory dies 130 per stack 120. Each stack 120 may contain up to eight channels per stack 120. In some embodiments, the memory stack 120 may be positioned upon a logic die 150. In some embodiments, an operational accelerator die 160 may be placed on top of the stack 120.

[0051] Part of the memory stacks 120 are configured as a lookup table (“LUT”) to utilize bank-level parallelism. Lookup tables are stored on the memory dies 130. The LUTs contain results (e.g., pre-computed results) of floating point operations. The results are real number values. The results contained in the LUTs on the memory dies 130 can be streamed to the logic die 150.

[0052] FIGS. 2A-2D are block diagrams illustrating design options in accordance with one or more embodiments of the disclosure.

[0053] FIG. 2A shows one embodiment of the memory lookup mechanism. According to the present embodiment, the LUT 220A is located on the DRAM die 210A. The LUT 220A contains real number value results C′ of one or more pre-computed floating point operations. In the current embodiments, the real number value results C′ may include whole scalar floating point values. The DRAM die 210A retrieves the value/result C′ from the LUT 220A. The DRAM die 210A uses input values A and B to identify the results C′ in the LUT 220A. The input value A represents the row address of the LUT 220A, and the input value B represents the column address of the LUT 220A. According to this embodiment, the input value A and input value B may include whole scalar floating point values.

[0054] The DRAM die 210A transmits the result C′ to the buffer (“BUF”) 230A located on the logic die 280A. The buffer 230A stores the results C′. The buffer 230A allows for quicker access of the results C′ for use in matrix multiplication operations conducted by the logic die 280A in conjunction with the DRAM die 210A. The buffer 230A transmits the result C′ to an accumulator (“ACC”) 240A. In various embodiments, the accumulator 240A will gather results C′ for use in calculating intermediate results of matrix multiplication operations conducted by the logic die 280A and the DRAM die 210A.

[0055] FIG. 2B illustrates a second embodiment of the memory lookup mechanism. According to the present embodiment, the LUT 220B is located on the DRAM die 210B. The LUT 220B stores partial product multiplication results “C.fraction” of pre-computed floating point operations. The DRAM die 210B uses input values “A.fraction” and “B.fraction” to identify the result C.fraction in the LUTs. The A.fraction value represents the row address of the LUT and is a fraction part of input value A. The B.fraction represents the column address of the LUT and is a fraction part of input value B. According to the present embodiment, the A.fraction value and the B.fraction value may include fraction scalar floating point values.

[0056] In the present embodiment, the sign logic and exponent addition and subtraction logic are not implemented by the LUT to enable reduction in the size of the LUT. Because input values A.fraction and B.fraction are symmetrical, it is suitable to store only half of the results on the LUT 220B. The reduced size of the LUT 220B allows for more storage space on the DRAM die 210B.

[0057] The DRAM die 210B transmits the result C.fraction to the buffer 230B located on the logic die 280B. The buffer 230B stores the result C.fraction. The buffer 230B allows for quicker access of the result C.fraction for use in matrix multiplication operations conducted by the logic die 280B. The buffer 230B streams the result C.fraction to an adjustor (“ADJ”) 250. The adjustor 250 aligns/processes the result C.fraction to produce result C′. After alignment, the adjustor 250 transmits the result C′ to the accumulator 240B, which is able to effectively achieve the same tasks as the accumulator 240A of FIG. 2A. The accumulator 240B will gather result C′ for use in calculating intermediate results of matrix multiplication operations conducted by the logic die 280B.

[0058] FIG. 2C shows a third embodiment of the memory lookup mechanism. According to the present embodiment, the LUT 220C is located on the DRAM die 210C. The LUT 220C contains tensor value results Tensor C′ of pre-computed floating point operations. The DRAM die 210C retrieves the tensor value results Tensor C′ from the LUT 220C. The DRAM die 210C uses the row address and column address of the LUT 220C to identify the tensor value results Tensor C′. For the multiplication of Tensor A and Tensor B, the row address is the concatenation of every element in each row of input value A, and the column address is every element in each column of input value B. Input value A and input value B may include tensor floating point values. The row address is equal to the number of elements multiplied by the length of the element. The row index and the column index are assumed to be unsigned integers. Because input values A and B are symmetrical, the LUT 220C only needs to store half of the tensor value results Tensor C′.

[0059] In some embodiments, the size of the LUT 220C can be further reduced. Before concatenation, the control engine on DRAM die 210C may sort the tensor value results Tensor C′ in the rows and columns according to their absolute value and to their pre-computer sign bit logic. If the row index is larger than the column index, the control engine on DRAM die 210C may switch the row index and the column index. The control engine on DRAM die 210C may assume the row index and column index are unsigned integers. Further, the control engine on DRAM die 210 C may use pre-computer sign bit logic to select the LUT 220C for concatenation of the row and column address lookup.

[0060] The DRAM die 210C transmits the tensor value results Tensor C′ to the buffer 230C located on the logic die 280C. The buffer 230C stores the tensor value results Tensor C′. The buffer 230C allows for quicker access of the tensor value results Tensor C′ for use in matrix multiplication operations conducted by the logic die 280C. The buffer 230C transmits the Tensor result C′ to the Tensor Post Processing Unit (“TPU”) 260, which may contain one or more accumulators, such as the accumulators 240A and 240B of FIGS. 2A and 2B. The TPU 260 functions as an accumulator for tensor results such as Tensor result C′. In various embodiments, the TPU 260 will gather Tensor results C′ for us in calculating intermediate results of operations conducted by the logic die 280C.

[0061] FIG. 2D displays a fourth embodiment of the memory lookup mechanism. According to the present embodiment, the LUT 220D is located on the DRAM die 210D. Furthermore, the present embodiment also includes a buffer die 270. The LUT 220D contains real number value results C″ of pre-computed floating point operations. In the current embodiments, the real number value results C″ may include whole scalar floating point values. The DRAM die 210D retrieves the real number value results C″ from the LUT 220D. The DRAM die 210D uses the input values A′ and B′ to identify the real number value results C″ in the LUT 220D. The input value A′ represents the row address of the LUT 220D, and the input value B′ represents the column address of the LUT 220D. According to this embodiment, the input value A′ and input value B′ may include whole scalar floating point values.

[0062] The DRAM die 210D transmits the real number value result C″ to the buffer 230D located on the logic die 280D. The buffer 230D stores the real number value results C″. The buffer 230D allows for quicker access of the real number value results C″ for use in the matrix multiplication operations conducted by the logic die 280D. The buffer 230D transmits the real number value results C″ to a fused-layer post processor (FPU) 290. In situations where the real number value result C″ is an intermediate value in a series of operations conducted by the logic die 280D, the FPU 290 may transmit the real number value result C″ to a buffer die 270. In some embodiments the buffer die 270 may contain multiple buffers that are similar to the buffer 230D of the logic die 280D. The separate buffer die 270 will allow for wider stores of intermediate results that are suitable for larger and more complex operations.

[0063] FIG. 3 provides an illustration of how memory LUTs perform scalar-vector multiplication.

[0064] Referring to FIG. 3, in the process of the present embodiment, scalar A, which is unrelated to input value A, is multiplied by vectors B.sub.1-B.sub.n. The results of the vector-scalar multiplication are represented by vector products C.sub.1-C.sub.n. The scalar A is used as a row address to activate one row from the memory LUT 310. The vectors B.sub.1-B.sub.n are used as the column addresses to read the corresponding lookup results C.sub.1-C.sub.n from the row buffer 320. The results C.sub.1-C.sub.n are transferred to the logic die 280 for accumulation, as shown previously in FIGS. 2A-2D.

[0065] FIG. 4 displays a flow diagram 400 outlining different components of the floating points such as the floating point exponent, fraction, or sign.

[0066] Referring to FIG. 4, the table 410 displays the number of bits used by each component of the floating point. For example, a 64-bit floating point will have 52 bits for the fraction component, and 11 bits for the exponent component.

[0067] FIG. 5 illustrates a block diagram 500 illustrating redundant sections of a LUT, and a table 510 displaying the number of bits used by the row index, column index, and result size.

[0068] Referring to FIG. 5, in an example of the present embodiment, a 64-bit floating point has a 63-bit row index, 63-bit column index, and a 63-bit result size.

[0069] FIG. 6 displays a visual representation of an algorithm that can save memory space by “folding” the LUT.

[0070] Referring to FIG. 6, in the example of the present embodiment, the input values are represented as A and B. As in previous figures, the input value A represents the row address of a LUT and the input value B represents the column address of the LUT. In some situations, the input value B may be greater than the input value A. In those cases, the control engine on DRAM die may switch the input values A and B. Additionally, if the first bit of the input value B is equal to 1, then the input value A is equal to the reverse of all of the bits of input value A, and input value B is equal to the reverse of all of the bits of input value B. The control engine on DRAM die may use the final value of A as the row index and the final value of B as the column index. This algorithm guarantees that the actual LUT access is half of the original table.

[0071] FIG. 7 shows a block diagram 700 representing a LUT, and a table 710 for displaying the number of bits used by a row index, column index, and result size of a LUT storing only partial product multiplication results.

[0072] Referring to FIG. 7, in the example of the present embodiment, a 64-bit floating point has a 52-bit row index, a 52-bit column index, and a 54-bit result size.

[0073] FIG. 8 shows a block diagram 800 representing a LUT that contains fraction partial product results. The fraction part can be partitioned to multiple segments with equal size (e.g., fraction_length=fraction_segment*fraction_part). It may be suitable for the LUT to implement only the multiplication table of fraction segment multiplication. Intermediate results are not rounded and normalized. Instead, full precision of an intermediate result is kept, and is sent to an adder to shift or sum up the final result. An example of the fraction segment multiplication is provided below:

(A.sub.1*2.sup.12+A.sub.0)*(B.sub.1*2.sup.12+B.sub.0)==2.sup.12*A.sub.1*B.sub.1+2.sup.12*(A.sub.1*B.sub.0+B.sub.1*A.sub.0)+A.sub.0*B.sub.0

The normal expression can be displayed as follows:

24=12+12

2.sup.12=4K

[0074] FIG. 9 provides a visual representation of layer-by-layer convolution operation.

[0075] Referring to FIG. 9, the representation of the present embodiment is consistent with the embodiment illustrated in FIG. 2D. In conventional convolution operations, each layer must be calculated one by one. As shown in FIG. 9, the layer 1 kernel 900 will slide along (e.g., reading) every possible location in the layer 1 input feature map 910, convolve with the layer 1 input feature map 910 and generate one result in layer 2 input feature map 930. As a result, every element in layer 2 input feature map 930 is the result of the convolution of layer 1 kernel 900 with layer 1 input feature map 910. In the next step, the layer 2 kernel 920 will, similarly, slide along every possible location in layer 2 input feature map 930 to generate the results in layer 3 input feature map 950. This convolution operation will continue layer-by-layer for N layers. However, the storage of each of these intermediate results (e.g. layer 1, layer 2, layer 3 . . . layer N) becomes problematic. Storing all of the intermediate results on a DRAM die may require a relatively large amount of energy and storage space.

[0076] To address energy consumption and storage issues, the present embodiment focuses on a small computation pyramid. The small computation pyramid is represented by the shaded areas of the input feature maps (910, 930, 950). As each layer progresses, the shaded area decreases. When storing the intermediate results, only the shaded areas of the input feature maps are stored. Each of these intermediate results are stored on a buffer die rather than the DRAM die. The buffer die is illustrated as buffer die 270 in FIG. 2D. Retrieving the intermediate results from the buffer die uses less energy than retrieving the intermediate results from the DRAM die. In some embodiments the buffer could be a buffer on a memory die, a buffer on a logic die, or a separate buffer die.

[0077] The foregoing is illustrative of example embodiments, and is not to be construed as limiting thereof. Although a few example embodiments have been described, those skilled in the art will readily appreciate that many modifications are possible in the example embodiments without materially departing from the novel teachings and advantages of example embodiments. Accordingly, all such modifications are intended to be included within the scope of example embodiments as defined in the claims. In the claims, means-plus-function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents but also equivalent structures. Therefore, it is to be understood that the foregoing is illustrative of example embodiments and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed example embodiments, as well as other example embodiments, are intended to be included within the scope of the appended claims. The inventive concept is defined by the following claims, with equivalents of the claims to be included therein.