Computer-Implemented Method of Executing SoftMax

Abstract

The present disclosure concerns a method of executing a SoftMax function, the method comprising: (i) pre-storing in memory M fraction components (fc.sub.j) in binary form, derived from the expression 2.sup.(j/M), said fc.sub.j forming a lookup table (T) of size M; (ii) calculating, for each z.sub.i, an element y.sub.i of a number of the form 2.sup.y.sup.i; (iii) separating y.sub.i into an integral part (int.sub.i) and a fractional part (fract.sub.i); (iv) determining a lookup index (ind.sub.i) that corresponds to fract.sub.i scaled by the size M; (v) retrieving a fraction component fc.sub.i from T with ind.sub.i; (vi) generating, in a result register, a binary number representative of the exponential value of said z.sub.i, by combining said fc.sub.i retrieved from T and said int.sub.i; (v) adding the K result registers corresponding to z.sub.i into a sum register R7; and (vi) determining the K probability values p.sub.i from the K result registers and the sum register.

Claims

1. A computer-implemented method of executing a SoftMax function, the computer-implemented method comprising: pre-storing, in a memory, M fraction components (fc.sub.j) in binary form, derived from an expression 2.sup.(j/M), where j is an integer varying from 0 to M−1, and where the fc.sub.j form a lookup table (T) of size M; calculating, for each input number (z.sub.i), an element y.sub.i of a number of a form 2.sup.y.sub.i, where the element y.sub.i represents $\frac{z_i}{\ln \left(2\right)};$ separating the element y.sub.i into an integral part (int.sub.i) and a fractional part (fract.sub.i); determining a lookup index (ind.sub.i) that corresponds to fract.sub.i scaled by the size M; retrieving a fraction component (fc.sub.i) from T with the ind.sub.i; generating, in a result register, a binary number (q.sub.i) representative of an exponential value for each z.sub.i, by combining fc.sub.i retrieved from the T and int.sub.i; adding K result registers corresponding to K input numbers z.sub.i into a sum register; and determining K probability values p.sub.i from the K result registers and the sum register.

2. The computer-implemented method as described in claim 1, wherein each z.sub.i in binary form is scaled by an input scaling factor (S.sub.in) effective to produce a corresponding scaled input number z.sub.i, where S.sub.in=2.sup.σ is stored in a first N-bit register.

3. The computer-implemented method as described in claim 2, wherein S.sub.in is determined depending on an expected smallest and largest values of z.sub.i and a size N of the first N-bit registers.

4. The computer-implemented method as described in claim 3, wherein calculating each element y.sub.i further comprises: providing the corresponding scaled input number z.sub.i in the first N-bit register; right-shifting the corresponding scaled input number z.sub.i by N/2−1, effective to produce a right-shifted scaled input number z.sub.i; and processing the right-shifted scaled input number z.sub.i effective to provide a transformed input number z.sub.i″ into a second N/2-bit register.

5. The computer-implemented method as described in claim 4, wherein processing the right-shifted scaled input number z.sub.i is processed according to at least one of the following conditions: if the scaled input number z.sub.i, after right-shifting by N/2−1, does not overflow the second N/2-bit register, fitting the shifted scaled input number z.sub.i into said second N/2-bit register; or if the scaled input number z.sub.i, after right-shifting by N/2−1, overflows the second N/2-bit register, saturating said N/2 bit register, providing the value of $\left(\frac{1}{\ln \left(2\right)}-1\right)$ scaled by 2.sup.N/2-1 in binary form in a third N/2-bit register.

6. The computer-implemented method as described in claim 5, further comprising: calculating a product of the second and third N/2-bit registers; and storing the product into a fourth N-bit register.

7. The computer-implemented method as described in claim 6, further comprising adding the first N-bit register and the fourth N-bit register by implementing a saturating addition, to obtain an element y.sub.i that is scaled by S.sub.in in a fifth N-bit register.

8. The computer-implemented method as described in claim 7, wherein calculating each element y.sub.i, further includes rounding the right-shifted scaled input number z.sub.i″ by adding 2N/2-2 to the scaled input number z.sub.i before right-shifting by N/2−1.

9. The computer-implemented method as described in claim 1, further comprising: computing the fc.sub.j of T, with j varying from 0 to M−1, by using the following formula: ${\mathrm{fc}}_j=a*2^{\left(b*\left(\frac{j}{M}\right)+c\right)}-d,$ and wherein a, b, c, and d are constant parameters for which a includes an output scaling factor S.sub.out is equal to .sup.2B and B is a number of desired bits for the computed exponential values, b is either 1 or −1, and d is a multiple of a, and the constant parameters a, b, c, and d for computing the fraction components the fc.sub.j are adjusted in a way that the largest fraction component is equal or close to 2.sup.B−1.

10. The computer-implemented method as described in claim 9, wherein the binary number q.sub.i representative of the exponential value for each z.sub.i is generated by inputting a corresponding fc.sub.i retrieved from T into the result register and right-shifting fc.sub.i by int.sub.i in the result register.

11. The computer-implemented method as described in claim 10, wherein parameters a, b, c, and d for computing fraction components of T are adjusted in a way that the smallest fraction component is close to 2.sup.B/2, and parameter d is equal to zero.

12. The computer-implemented method as described in claim 11, wherein determining the K probability values p.sub.i derived from the K input numbers z.sub.i comprises: adding the result registers with i varying from 1 to K to obtain a sum number; and obtaining a normalization factor (f.sub.n) by scaling a value V.sub.100, obtained by setting to 1 all bits in a result register and corresponding to a result q.sub.i giving a probability value of 100% by a normalization scaling factor (S.sub.n).

13. The computer-implemented method as described in claim 12, wherein obtaining f.sub.n is obtained using the following formula: $f_n=\frac{\left(V_{100}\ll S_n\right)}{\Sigma }.$

14. The computer-implemented method as described in claim 1, wherein q.sub.i is generated in the result register in a form of an Institute of Electrical and Electronics Engineers (IEEE) 754 floating-point number including an exponent and an IEEE 754 mantissa.

15. The computer-implemented method as described in claim 14, wherein the exponent is a combination of int.sub.i in binary form and an IEEE 754 exponent bias.

16. The computer-implemented method as described in claim 14, wherein the IEEE 754 mantissa is derived from fc.sub.i retrieved from T.

17. The computer-implemented method as described in claim 14, wherein parameters a, b, c, and d for computing fc.sub.i of T are adjusted in a way that the fc.sub.i of T match the IEEE 754 mantissa.

18. The computer-implemented method as described in claim 1, further comprising: selecting a maximal input number x.sub.max; and performing, for each z.sub.i, at least one of the following: subtracting x.sub.max from an original input number to obtain a negative or zero input number; or subtracting the original input number from x.sub.max to obtain a zero or positive input number.

19. A non-transitory computer-readable storage medium storing one or more programs comprising instructions, which when executed by a processor, cause the processor to perform operations including: pre-storing, in a memory, M fraction components (fc.sub.j) in binary form, derived from an expression 2.sup.(j/M), where j is an integer varying from 0 to M−1, and where the fc.sub.j form a lookup table (T) of size M; calculating, for each input number (z.sub.i), an element y.sub.i of a number of a form 2.sup.y.sub.i, where the element y.sub.i represents $\frac{z_i}{\ln \left(2\right)};$ separating the element y.sub.i into an integral part (int.sub.i) and a fractional part (fract.sub.i); determining a lookup index (ind.sub.i) that corresponds to fract.sub.i scaled by the size M; retrieving a fraction component (fc.sub.i) from T with the ind.sub.i; generating, in a result register, a binary number (q.sub.i) representative of an exponential value for each z.sub.i, by combining fc.sub.i retrieved from the T and int.sub.i; adding K result registers corresponding to K input numbers z.sub.i into a sum register; and determining K probability values p.sub.i from the K result registers and the sum register.

20. A system comprising: one or more processors; and a memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions that, when executed by the one or more processors, cause the one or more processors to: pre-store, in a memory, M fraction components (fc.sub.j) in binary form, derived from an expression 2.sup.(j/M), where j is an integer varying from 0 to M−1, and where the fc.sub.j form a lookup table (T) of size M; calculate, for each input number (z.sub.i), an element y.sub.i of a number of a form 2.sup.y.sub.i, where the element y.sub.i represents $\frac{z_i}{\ln \left(2\right)};$ separate the element yi into an integral part (inti) and a fractional part (fracti); determine a lookup index (ind.sub.i) that corresponds to fract.sub.i scaled by the size M; retrieve a fraction component (fc.sub.i) from T with the ind.sub.i; generate, in a result register, a binary number (q.sub.i) representative of an exponential value for each z.sub.i, by combining fc.sub.i retrieved from the T and int.sub.i; add K result registers corresponding to K input numbers z.sub.i into a sum register; and determine K probability values p.sub.i from the K result registers and the sum register.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Other features, purposes and advantages of the disclosure may become more explicit by means of reading the detailed statement of the non-restrictive Implementations made with reference to the accompanying drawings.

[0030] FIG. 1 illustrates a method of executing a SoftMax function on a processor, according to the prior art;

[0031] FIG. 2 illustrates a method of executing a SoftMax function, according to a first implementation;

[0032] FIG. 3A is a table illustrating an example of original input numbers x.sub.i in the first column and derived positive input numbers z.sub.i in the second column;

[0033] FIG. 3B illustrates a table of numerical values;

[0034] FIG. 3C illustrates the registers R6.sub.i before and after normalization S20;

[0035] FIG. 4 is a flowchart of the step of the method of FIG. 2;

[0036] FIG. 5 illustrates a method of executing a SoftMax function, according to a second implementation;

[0037] FIG. 6 is a flowchart of the step of the method of FIG. 5;

[0038] FIG. 7 is a functional block diagram of a data processing device for executing the SoftMax function according to the first implementation; and

[0039] FIG. 8 illustrates a layout for a 32-bit floating point as defined by the IEEE standard for floating-point arithmetic (IEEE 754).

DETAILED DESCRIPTION

[0040] The SoftMax function is a function that turns a vector of K input numbers (real values) into a vector of K output numbers (real values) that add up to 1. The input numbers can be positive, negative or zero. The SoftMax function transforms them into values between 0 and 1, that can be interpreted as probabilities. If one of the input number is small or negative, the SoftMax can turn it into a small probability, and if an input is positive and large, then it can turn it into a large probability.

[0041] A multi-layer neural network may end in a penultimate layer which can output real-valued scores that are not conveniently scaled. The SoftMax function can be used in the final layer, or last activation layer, of the neural network to convert the scores to a normalized probability distribution, which can be displayed to a user or used as input to another system.

[0042] Further, the SoftMax function can use the following mathematical formula to compute output values between 0 and 1:

[00012]$\begin{array}{cc}{\sigma \left(z\right)}_i=\frac{e^{z_i}}{{.Math.}_{j=1}^K{e}^{z_j}}\mathrm{for}i=1,.Math.,K\mathrm{and}z=\left(z_1,.Math.,z_K\right)\in {ℝ}^K& \left(1\right)\end{array}$

where z.sub.i may represent an input number (real value); σ(z).sub.i may represent an output value (real value); and K can be the total number of input numbers.

[0043] The implementation of the SoftMax function can involve a floating-point calculation of the exponential function for each input number z.sub.i. The calculation of the exponentials e.sup.z.sup.i is often costly in terms of computing resources and energy consumption. It takes a lot of computing cycles and runtime.

[0044] The present disclosure concerns a computer-implemented method of executing the SoftMax function on a data processing device 100, that requires fewer computing efforts and less runtime.

[0045] In the present disclosure, the exponentiation is turned into an exponentiation of two according to the following mathematical relationship:

[00013]$\begin{array}{cc}{e}^{z_i}={\left(2^{{\log }_{2}e}\right)}^{z_i}=2^{{\log }_{2}e*z_i}=2^{y_i}\mathrm{where}{y}_i={\log }_{2}e*z_i=\frac{1}{\ln \left(2\right)}*z_i\mathrm{and}\frac{1}{\ln \left(2\right)}\cong 1.442695041.& \left(2\right)\end{array}$

[0046] A computer (or a processor) may be more efficient and perform fewer computing cycles at calculating two to the power of a given exponent than the exponential function.

[0047] In the present disclosure, the computer-implemented method of executing the SoftMax function on the data processing device 100 is based on an exponentiation of two and uses a lookup table T of a small size to reduce the calculation efforts. The present method replaces the floating-point calculation of the exponential function by some shifts in registers, integer additions and integer multiplications, and limits the floating-point operations.

[0048] The computer-implemented method of the present disclosure can be implemented on a data processing device 100, represented in FIG. 7, that has a processor 101, a memory 200 for storing the lookup table T, and registers. In an implementation, the registers include N-bit registers and N/2-bit registers. The number N is may be of the form 2.sup.n, n being an integer with n≥1. A N/2-bit register can be implemented using half of a N-bit register. For example, N may be equal to 32. However, N can have any other value, for example 16 or 64.

[0049] In a preliminary step S10 of the method, a lookup table T can be computed and stored in the memory 200 in the data processing device 100. The lookup table T can be computed by the data processing device 100 itself, or by any other data processing device and then transferred to the data processing device 100.

[0050] The lookup table T can contain M fraction components fc.sub.j in binary form, all derived from the expression

[00014]$2^{\left(\frac{j}{M}\right)}$

where j can be a lookup index consisting in an integer varying from 0 to M−1. For example, M can be equal to 256. However, any other size M of the lookup table T, preferably of the kind M=2.sup.m (m being an integer), can be used.

[0051] The computation of the M fraction components fc.sub.j of the lookup table T (with 0≤j≤M−1) can be calculated using the following expression:

[00015]$\begin{array}{cc}f{c}_j=a*2^{\left(b*\left(\frac{j}{M}\right)+c\right)}-d& \left(3\right)\end{array}$

where a, b, c and d are constant parameters; b can be either 1 or −1; a can include an output scaling factor S.sub.out, where S.sub.out is equal to 2.sup.B and B is a number of desired bits for the computed exponential values; and d can be a multiple of a.

[0052] The lookup table T can store, in order, M fraction components fc.sub.j with j going from 0 to M−1, the index j giving the position of the fraction component fc.sub.j in the lookup table T. The expression (3) covers different variants of the lookup table T. In different implementations that may be described later, different variants of the lookup table T defined by the expression (3), based on different sets of constant parameters a, b, c, and d that are used.

[0053] In an implementation, the number B of desired bits for the results, namely for the exponential values computed by the present method, can be derived from the size N (in number of bits) in registers. For example, B can be equal to N/2. In case that N is 32, B can be equal to 16. B can be equal to any other value, preferably of the type 2.sup.m, m being an integer.

[0054] A first implementation of the computer-implemented method of executing the SoftMax function on the data processing device 100, that can be termed as a positive and fixed-point approach, may now be described with reference to FIGS. 2 and 3A-3C.

[0055] The first implementation uses a first variant of the lookup table T1. The constant parameters for computing the fraction components fc.sub.j of the lookup table T1 are set as follows:

a=S.sub.out−1=2.sup.B−1; b=−1; c=0; and d=0.

[0056] For example, the fraction components fc.sub.j of the lookup table T1 are computed using the expression:

[00016]$\begin{array}{cc}f{c}_j=\left(2^B-1\right)*2^{\left(\frac{-j}{M}\right)}& \left(4\right)\end{array}$

[0057] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the largest fraction component is fc.sub.0=2.sup.B−1=S.sub.out−1 and the smallest fraction component is

[00017]$f{c}_{M-1}=\left(2^B-1\right)*2\left(\frac{-\left(M-1\right)}{M}\right)\approx \frac{2^B}{2}=\frac{s_{out}}{2}\left(e.g.,\mathrm{close}\mathrm{to}\frac{s_{out}}{2}\right).$

[0058] In a first step S11, the data processing device 100 receives a vector of K original input numbers x.sub.i. The original input numbers x.sub.i may be real values. For example, they may be the real-valued scores at the output of a multi-layer neural network, such as a classification neural network.

[0059] In an implementation, typically in a case that the input into the data processing device 100 is an output from a neural network, the original input numbers x.sub.i received in the step S11 can be already scaled by an input scaling factor S.sub.in=2.sup.σ and rounded to an integer (the nearest integer) so as to be stored in N-bit registers. For example, the scaling happens in the input to the neural network which delivers data into a SoftMax operator. In such a case, the scaling happens outside the SoftMax operator. The input scaling factor S.sub.in can be chosen depending on the expected smallest and largest values of the original input numbers and the size of the N-bit registers. After scaling by S.sub.in, all the scaled numbers x.sub.i can advantageously fit −2.sup.N-1 and 2.sup.N-1−1 (in the signed integer representation).

[0060] In another implementation, the original input numbers are scaled in the input to the data processing device 100. For example, the original input values x.sub.i in the table of FIG. 3A are scaled by 2.sup.27 and rounded to the nearest integer to be stored in 32-bit registers. The corresponding decimal values are indicated in the table of FIG. 3B, as an illustrative example.

[0061] In a step S12, the processor 101 derives an input number z.sub.i from each original (already scaled by S.sub.in) input number x.sub.i and obtains K input number z.sub.i corresponding respectively to the K input numbers x.sub.i (positive, negative or zero) with i=1, . . . , K. The first implementation is a positive approach, which means that the input numbers z.sub.i are positive or zero. Thus, in the first implementation, deriving each input numbers z.sub.i from the original input number x.sub.i includes selecting the maximal original input number x.sub.max and subtracting x.sub.i from x.sub.max so as to obtain a input number z.sub.i that is positive or equal to zero. For example, the input numbers z.sub.i are calculated based on the expression z.sub.i=x.sub.max−x.sub.i. The K input numbers z.sub.i can be stored in the first N-bit registers R1.sub.i by overwriting the original input numbers x.sub.i. In further implementations, the K input numbers z.sub.i can be stored in other N-bit registers.

[0062] FIG. 3A is a table illustrating an example of original input numbers x.sub.i in the first column and derived positive input numbers z.sub.i (=x.sub.max−x.sub.i) in the second column. The numbers are represented by decimal values without the scaling by S.sub.in, for the sake of clarity.

[0063] In a next step S14, the input numbers z.sub.i (scaled by S.sub.in=2.sup.σ) are multiplied by 1/ln(2) (≅1.442695041) to obtain the numbers y.sub.i (corresponding to

[00018]$y_i==\frac{z_i}{\ln \left(2\right)},$

scaled by S.sub.in=2.sup.σ). In the first implementation, the multiplication of each z.sub.i stored in a N-bit register R1.sub.i, by the constant value 1/ln(2), is optimized by using a cheaper N/2-bit*N/2-bit multiplication resulting in a N-bit result, as explained below. As an illustrative example, N is equal to 32 bits.

[0064] The step S14 includes the following steps (or sub-steps), carried out by the processor 101 for each (scaled) input number z.sub.i. In a step S140, the processor 101 provides the (scaled) input number z.sub.i in the first N-bit register R1.sub.i (in the given example: in the first 32-bit register R1.sub.i). In a step S142, the processor 101 performs a right-shifting of the (scaled) input number z.sub.i by N/2−1 into a second N/2-bit register R2.sub.i (in the given example: it is a right-shifting by 15 bits in a second 16-bit register R2.sub.i). For example, the N/2−1 (15 in the given example) less significant bits are discarded by right-shifting. The N/2+1 (17 in the given example) most significant bits are processed according to the following processing rules: (i) if the scaled input number z.sub.i, after right-shifting by N/2−1, does not overflow the second N/2-bit register R2.sub.i, fitting the shifted number z.sub.i into said second N/2-bit register R2.sub.i. It means that, when N is equal to 32 for example, if the most significant bit (the 32.sup.th bit) of the number z.sub.i is 0, the 16 least significant of the 17 bits of the scaled number z.sub.i right-shifted by 15 are simply transferred into the second N/2 bit register R2.sub.i; and (ii) if the scaled input number z.sub.i, after right-shifting by N/2−1, overflows the second N/2-bit register R2.sub.i saturating said second N/2-bit register R2.sub.i. It means that if the most significant (the N.sup.th bit) bit of z.sub.i is 1, each of the other N/2 (e.g., 16) most significant bits (from the (N−1).sup.th to the N/2.sup.th bit) are set to 1 and transferred into the second N/2-bit register R2.sub.i that is consequently saturated.

[0065] The step S142 results in storing a number referenced as z.sub.i″ in the second N/2-bit register R2.sub.i. In the domain of decimal numbers, the step of right-shifting the scaled input number z.sub.i by N/2−1 bits corresponds to a division by 2.sup.N/2-1.

[0066] In an implementation, the right-shifting by N/2−1 of the (scaled) input number z.sub.i may be preceded by a rounding step S141 of adding 2.sup.N/2-2 to the (scaled) input number z.sub.i. In this way, the right-shifted number z.sub.i, referenced as z.sub.i″, can be rounded.

[0067] In another step S143, the processor 101 provides (or loads) the constant value of

[00019]$\left(\frac{1}{\ln \left(2\right)}-1\right)$

scaled by 2.sup.N/2-1 in binary form in a third N/2-bit register R3. The value of

[00020]$\left(\frac{1}{\ln \left(2\right)}-1\right)$

can be precalculated and stored (In an implementation after scaling by 2.sup.N/2-1) in memory in the data processing device 100 or calculated in real time, when needed.

[0068] In a step S144, the processor 101 calculates the product of the second N/2-bit register R2.sub.i and the third N/2-bit register R3 (e.g., the multiplication of z.sub.i″ by

[00021]$\left(\frac{1}{\ln \left(2\right)}-1\right)$

scaled by 2.sup.N/2-1) and stores the product result into a fourth N-bit register R4.sub.i. The product result corresponds to

[00022]$\left(\frac{1}{\ln \left(2\right)}-1\right)*z_i\cong 0.442695041*z_i,$

scaled by the input scaling factor S.sub.in=2.sup.σ.

[0069] In a step S145, the processor 101 adds the first N-bit register R1.sub.i and the fourth N-bit register R4.sub.i to obtain the element y.sub.i, scaled by the input scaling factor S.sub.in, that can be stored into a fifth N-bit register R5.sub.i. In an implementation, the addition S145 can be a saturating addition, which means that if the result of the addition overflows the N-bit register R5.sub.i (e.g., greater than the maximum value in the N-bit register R5.sub.i) all the bits are set to 1 in the N-bit register R5.sub.i (e.g., the N-bit register R5.sub.i can be set to the maximum). The saturation allows to avoid some unreasonable results, such as low probabilities getting very large by numerical overflow due to the two's complement representation. In further implementations, the result of the addition of the step S145 can be overwritten in one of the two N-bit registers R1.sub.i and R4.sub.i.

[0070] The step S14 ends by providing the element (number) y.sub.i in the binary domain scaled by S.sub.in (=2.sup.σ). In a first variant, the step S14 of calculating each element y.sub.i corresponding to y.sub.i==

[00023]$\frac{z_i}{\ln \left(2\right)}$

(here already scaled by S.sub.in=2.sup.σ) is executed as described below.

[0071] As previously described, the scaled input numbers z.sub.i are multiplied by 1/ln(2) (≅1.442695041) to obtain the numbers y.sub.i. In the first variant, the step S14 includes the following steps (or sub-steps), carried out by the processor 101 for each (scaled) input number z.sub.i.

[0072] In a step S140′ (identical to the step S140), the processor 101 provides the (scaled) input number z.sub.i in the first N-bit register R1.sub.i (in the given example, a 32-bit register).

[0073] In a step S142′, the processor 101 performs a right-shifting of the scaled input number z.sub.i by N/2-2 in a second N-bit register R2.sub.i (in the given example: it is a right-shifting by 14 bits in a 32-bit register R2.sub.i). For example, the N/2-2 (14 in the given example) less significant bits are discarded by right-shifting. The N/2+2 (18 in the given example) most significant bits are maintained in the first N-bit register R2.sub.i as the N/2+2 least significant bits (the N/2-2 most significant bits are set to 0). The first and the second N-bit register R1.sub.i and R2.sub.i can be the same N-bit register and two distinct N-bit registers. In an implementation, the step S142′ is preceded by a step S141′ of rounding the right-shifted scaled input number z.sub.i by adding 2.sup.N/2-3 to the scaled input number z.sub.i before right-shifting by N/2-2.

[0074] In a step S143′, the processor 101 provides the value of

[00024]$\frac{1}{\ln \left(2\right)}-1$

in binary form scaled by 2.sup.N/2-2 in a third register R3.

[0075] In a step S144′, the processor 101 multiplies each second N-bit register R2.sub.i and the third register R3 and stores the result into a fourth N-bit register R4.sub.i. This multiplication is more costly in computing resources than a N/2-bit*N/2-bit multiplication as described in step S144. In further implementations, the second N-bit register R2.sub.i (that can be the first register R1.sub.i in case that z.sub.i s right-shifted in the same register in the step S142′) can be overwritten by the result of the multiplication (instead of using another N-bit register). For example, the fourth N-bit register R4.sub.i can be the N-bit register R2.sub.i.

[0076] In a step S145′, the processor 101 adds the first and the fourth N-bit register R1.sub.i and R4.sub.i to obtain the element y.sub.i (scaled by the input scaling factor S.sub.in). The addition result y.sub.i is stored into a fifth N-bit register R5.sub.i. The processor 101 can implement a saturating addition of the two registers R1.sub.i and R4.sub.i. If the addition result is greater than the maximum value that can be stored in the register R5.sub.i, the N-bit register R5.sub.i is set to the maximum (all the N bits are set to 1 in the register R5.sub.i). Any value of the scaled input number z.sub.i or derived from z.sub.i can be maintained in the register R1.sub.i until its last use in the step S145′.

[0077] In the first variant, there is no need for saturation. The calculation is more precise (because there 1 more bit of precision for the input number) but also a little more costly in computing resources.

[0078] In a second variant, the step S14 of calculating each element y.sub.i (y.sub.i corresponding to

[00025]$y_i==\frac{z_i}{\ln \left(2\right)},$

already scaled by S.sub.in=2.sup.σ) is analogous to the first implementation illustrated in FIG. 4 but differs from it by the following features: (i) in the step S142 of right-shifting, the processor 101 performs a right-shifting of the scaled input number z.sub.i by N/2 in a second N/2-bit register R2.sub.i (in the given example: it is a right-shifting by 16 bits in a 16-bit register R2.sub.i); and (ii) in the step S143, the processor 101 provides (loads) the constant value of

[00026]$\left(\frac{1}{\ln \left(2\right)}-1\right)$

scaled by 2.sup.N/2 in binary form in a third N/2-bit register R3. In at least some implementations, this second variant may be less precise than the first variant.

[0079] In a following step S15, the processor 101 separates the integral part int.sub.i and the fractional part frac.sub.i of the element y.sub.i. The integral part int.sub.i can be an integer. It can be derived from the register R5.sub.i by right-shifting by a bits (which results in discarding all fractional bits due to the right-shift). The fractional part frac.sub.i can be a value that is less than 1, and more than zero or equal to zero (e.g., 0≤frac.sub.i<1). It is also derived from the register R5.sub.i for example by masking off the bits corresponding to the integral part.

[0080] In a step S16, the processor 101 scales (e.g., multiplies) the fractional part frac.sub.i by the size M of the lookup table T1 (the multiplication or scaling result being rounded to the nearest integer) to obtain a lookup index ind.sub.i, that is an integer between 0 and M−1. The bits for the lookup index ind.sub.i can be extracted directly from the internal representation of y.sub.i (e.g., y.sub.i scaled by 2.sup.σ which is the content of the register R5.sub.i), via a formula indicating which bits to use and which bits to discard to form the index ind.sub.i directly.

[0081] In the table of FIG. 3A, the third column contains the numerical values of the element y.sub.i in the decimal domain, the fourth column contains the corresponding integral parts int.sub.i in the decimal domain and the fifth column contains the corresponding fractional part frac.sub.i scaled by the size M of the lookup table T1 in the decimal domain, For example the lookup indexes ind.sub.i (j in the expression (4)).

[0082] In a step S17, the processor 101 retrieves from the lookup table T1 the fraction component fc.sub.i using the obtained lookup index ind.sub.i (e.g., the fraction component fc.sub.i located in the lookup table T1 at a position j between 0 and M−1 with j=ind.sub.i).

[0083] In the table of FIG. 3A, the sixth column contains the decimal values of the fraction components fc.sub.i corresponding to the lookup indexes indicated in the fifth column. However, as previously explained, the fraction components fc.sub.i are stored in the lookup table T1 in binary form, scaled by the output scaling factor S.sub.out=2.sup.B (in the present implementation S.sub.out=2.sup.N/2, with N=32 in the given example).

[0084] In a step S18, for each input number z.sub.i, the processor 101 generates in a sixth N/2-bit register R6.sub.i, termed as a result register, a binary number q.sub.i representative of the exponential value of said input number z.sub.i, by combining said fraction component fc.sub.i retrieved from the lookup table T1 at the step S17 and said integral part int.sub.i determined in the step S15.

[0085] The first implementation may be a fixed-point or integer approach. It means that all the operations carried out by the processor 101 are fixed-point operations. In the first implementation, generating the binary number q.sub.i representative of the exponential value of each input number z.sub.i in the N/2-bit result register R6.sub.i includes: inputting the corresponding fraction component fc.sub.i retrieved from the lookup table T1 at the step S17 into said result register R6.sub.i, then, right-shifting the fraction component fc.sub.i by the integral part int.sub.i, in said result register R6.sub.i.

[0086] The right-shifting by the integral part int.sub.i results in discarding the r less significant bits of the fraction component fc.sub.i, where p is a number equal to int.sub.i. For large values of the integer component int.sub.i, all bits of the result register R6.sub.i become equal to zero. The number resulting from the right-shifting of the step S18 in each result register R6.sub.i, referenced as q.sub.i, can be expressed by the following formula:

[00027]$q_i=2^{-in{t}_i}*f{c}_i=2^{-in{t}_i}*\left(2^B-1\right)*2^{\left(\frac{-j}{M}\right)}=\left(2^B-1\right)*2^{-\left(in{t}_i+\frac{j}{M}\right)}=\left(2^B-1\right)*2^{-\left(in{t}_i+{\mathrm{frac}}_i\right)}=\left(2^B-1\right)*2^{-y_i}==\left(2^B-1\right)*e^{-z_i}=\left(2^B-1\right)*e^{x_i-x_{\max }}$

[0087] As shown by the above relationship, the result number q.sub.i is representative of the exponential value of the original input number x.sub.i. In the present implementation, it is proportional to the exponential of x.sub.i.

[0088] In FIG. 3A, the seventh column of the table contains the decimal values of the number q.sub.i in each result register R6.sub.i (after the right-shifting of fc.sub.i by int.sub.i). As shown in the seventh column, all the result registers R6.sub.i corresponding to the non-meaningful original input numbers x.sub.i, namely x.sub.3, x.sub.5, x.sub.7, x.sub.10 are set to zero.

[0089] In a step S19, the processor 101 adds all the result registers R6.sub.i with i varying from 1 to K to obtain a sum number E in a sum register R7 that is a N-bit register. In the illustrative example of the FIG. 3A, the decimal value of the sum Σ is 65993 as shown in FIG. 3C.

[0090] Then, in a step S20, the processor 101 normalizes the result registers R6.sub.i (e.g., the values q.sub.i in the result registers R6.sub.i), in order to determine the K probability values derived from the K input numbers z.sub.i (or from the K original input numbers x.sub.i).

[0091] The step S20 includes different steps (or sub-steps) S200 to S202. In the first step S200, the processor 101 obtains a normalization factor f.sub.n calculated according to the following expression:

[00028]$\begin{array}{cc}{f}_n=\frac{\left(v_{100}\ll s_n\right)}{.Math.}& \left(5\right)\end{array}$

where V.sub.100 represents the value obtained by setting to 1 all bits in a N/2-bit result register R6.sub.i and corresponds to the result q.sub.i giving a probability value of 100%; <<S.sub.n represents a left-shift by a normalization scaling factor S.sub.n used for the normalization, for example N/2 (which corresponds to a scale by 2.sup.N/2 in the decimal domain); and Σ represents the sum of all result registers R6.sub.i with i varying from 1 to K.

[0092] In an implementation, for precision improvement, the factor f.sub.n can be rounded using the modified expression:

[00029]$\begin{array}{cc}{f}_n=\frac{\left(v_{100}\ll s_n+\frac{.Math.}{2}\right)}{.Math.}& (5’)\end{array}$

[0093] The normalization factor f.sub.n can be computed by the processor 101 and stored in a memory 201. With reference to the example in FIGS. 3A-3C, the value V.sub.100 is 65535, the sum is 65993, the scaling factor is 16 and the normalization factor f.sub.n is given by the following expression:

[00030]$f_n=\frac{\left(65535\ll 16+\frac{65993}{2}\right)}{65993}=65082.$

[0094] Then, in a step S201, the normalization factor f.sub.n is applied to each result register R6.sub.i with a rescaling by the factor 1/S.sub.n, to obtain a normalized value q.sub.i_n in the result register R6.sub.i. The operation of the step S201 can be expressed as follows:

q.sub.i_n=(q.sub.i*f.sub.n)>>S.sub.n

where q.sub.i is the binary value in the result register R6.sub.i; f.sub.n is the normalization factor; and >>S.sub.n represents a right-shift by S.sub.n in the result register R6.sub.i.

[0095] The operation of step S201 can be combined with a rounding to the nearest integer by adding 2.sup.S.sup.n.sup.−1 to q.sub.i*f.sub.n before right-shifting by S.sub.n in the result register R6.sub.i.

[0096] FIG. 3C illustrates the registers R6.sub.i before and after normalization S20. It can be seen that the sum of the normalized elements q.sub.i_n is 65636 which corresponds to a total probability that is very close to 100% (only very slightly more than 100% by 1). The eighth column of the table contains the decimal values of the numbers q.sub.i_n.

[0097] In an optional step S202, the processor 101 can derive from the normalized values q.sub.i_n in the result registers R6.sub.i the respective probability values p.sub.i (that are decimal numbers) corresponding to the input values x.sub.i, by dividing the normalized values q.sub.i_n in the result registers R6.sub.i by the output scaling factor S.sub.out (2.sup.16 in the example of FIGS. 3A-3C). The ninth column of the table in FIG. 3A contains the probability values p.sub.i expressed in %. The sum of the percentage is very slightly more than 100% (100,0783%).

[0098] In the first implementation, the processor 101 executes mainly integer (fixed-point) operations and some shifts in registers, which allows to determine the probability values p.sub.i corresponding to the input numbers x.sub.i in a fast manner. The run time and energy consumption of the processor 101 are significantly reduced. Furthermore, a microcontroller without capacity for floating-point calculation can easily use the SoftMax function according to the present disclosure.

[0099] A second implementation, that can be termed as a positive and floating-point approach, may now be described with reference to FIG. 6. The second implementation is based on the first implementation and only varies from the first implementation by the features described below.

[0100] In the second implementation, the steps S10 to S17 described in connection with the first implementation are performed, but the lookup table T2 used in the second implementation is a different variant of the lookup table T defined by the expression (3).

[0101] In the second implementation, the constant parameters of the expression (3) for computing the fraction components fc.sub.j of the lookup table T2 are set as follows:

a=2*S.sub.out−2=2.sup.B+1−2; b=−1; c=0; d=S.sub.out−1=2.sup.B−1

[0102] For example, the fraction components fc.sub.j of the lookup table T2 are computed using the expression:

[00031]$\begin{array}{cc}f{c}_j=\left(2^{B+1}-2\right)*2^{\left(\frac{-j}{M}\right)}-\left(2^B-1\right)& \left(6\right)\end{array}$

[0103] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the largest fraction component is fc.sub.0=2.sup.B−1=S.sub.out−1 and the smallest fraction component is:

[00032]$f{c}_{M-1}=\left(2^{B+1}-2\right)*2^{\left(\frac{-\left(M-1\right)}{M}\right)}-\left(2^B-1\right)\approx 0.$

[0104] In the second implementation, the step S18′ of generating a binary number q.sub.i representative of the exponential value of each input number z.sub.i, in a N/2-bit result register R6.sub.i, by combining the fraction component fc.sub.i retrieved from the lookup table T2 at the step S17 and the integral part int.sub.i determined in the step S15 is different from the step S18 of the first implementation, due to the fact that the second implementation is a floating-point approach. It means that all the binary numbers representative of the exponential values of the input numbers z.sub.i are floating-point numbers, here IEEE 754 floating-point numbers. They are referred as q.sub.i.sup.ieee. According to the IEEE standard for Floating-Point Arithmetic (IEEE 754), a N-bit floating point has a layout defined by one sign bit (the most significant bit), a N1-bit exponent part and a N2-bit mantissa. FIG. 8 illustrates an example of a layout for a 32-bit floating point number (−248,75), as defined by the standard IEEE 754. The exponent is a 8-bit part and the mantissa is a 23-bit part. N-bit registers can be used to store the numbers q.sub.i.sup.ieee and, for example, N=32 bits.

[0105] In the second implementation, generating the binary floating-point number q.sub.i.sup.ieee representative of the exponential value of each input number z.sub.i in the N-bit result register R6.sub.i in the step S18 includes: (i) combining the integral part int.sub.i determined in the step S15 and the IEEE 754 exponent bias (here by subtracting the integral part int.sub.i from the bias) and transferring the integral part int.sub.i combined with the IEEE 754 exponent bias in the exponent part of the floating-point number q.sub.i.sup.ieee, in the N-bit result register R6.sub.i; and (ii) transferring the fraction component fc.sub.i retrieved from the lookup table T2 at the step S17 into the mantissa of the floating-point number q.sub.i.sup.ieee in the N-bit result register R6.sub.i, if needed by adding bits set at 0 on the least significant bits. For example, if the fraction component fc.sub.i is represented by 16 bits, it is transferred to the 16 most significant bits of the mantissa and the other least significant bits of the mantissa are set to 0.

[0106] Thus, the exponent of q.sub.i.sup.ieee is a combination of the integral part int.sub.i stored in binary form in said result register R6.sub.i and the IEEE 754 exponent bias, and the mantissa of q.sub.i.sup.ieee is directly derived from the fraction component fc.sub.i retrieved from the lookup table T2.

[0107] In the second implementation, the parameters a, b, c and d for computing the fraction components fc.sub.j of the lookup table T2 are adjusted in a way they match IEEE 754 fraction components, which means that the fraction components fc.sub.j are identical or directly transformable from one to the other, for example by simply adding 0 on the right.

[0108] Then, in a step S21, the processor 101 adds the result registers R6.sub.i with i varying from 1 to K by performing a floating-point addition, to obtain a sum Σ.sup.ieee (floating-point number) in a register R7.

[0109] In a step S22, the processor 101 calculates the inverse of the sum Σ.sup.ieee and stores the result

[00033]$\frac{1}{{.Math.}^{ieee}}$

in a register R8 (in floating-point).

[0110] In a step S23, the processor 101 multiplies each result register R6.sub.i by the register R8. For example, the product of each result element q.sub.i.sup.ieee by

[00034]$\frac{1}{{.Math.}^{\mathrm{ieee}}}$

is calculated. The result can be stored in a register R8.sub.i that is a N-bit register.

[0111] In a step S24, the decimal probability values p.sub.i corresponding to the original input numbers x.sub.i can be obtained by converting the floating-point numbers stored in the registers R8.sub.i into decimal values.

[0112] In the second implementation, the efforts to use floating points are also reduced. The processor 101 calculates more in fixed points and less in floating points than in the prior art.

[0113] A third implementation, that can be termed as a negative and integer approach, may now be described. The third implementation is based on the first implementation and only varies from the first implementation by the features described below.

[0114] The third implementation mainly differs from the first implementation by the features that each input number z.sub.i is negative or zero and derived from the original input number x.sub.i by subtracting x.sub.max from x.sub.i so as to obtain input number z.sub.i;

the lookup table T3 is another variant of the lookup table defined by the general expression (3).

[0115] In the third implementation, the constant parameters of the expression (3) for computing the fraction components fc.sub.i of the lookup table T3 are set as follows:

[00035]$a=\frac{s_{out}}{2}=2^{B-1};b=+1;c=0;d=0.$

[0116] For example, the fraction components fc.sub.j of the lookup table T2 are computed using the expression:

[00036]$\begin{array}{cc}f{c}_j=2^{B-1}*2^{\left(\frac{j}{M}\right)}& \left(7\right)\end{array}$

[0117] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the smallest fraction component is

[00037]$f{c}_0=2^{B-1}=\frac{s_{out}}{2}$

and the largest fraction component is:

[00038]$f{c}_{M-1}=2^{B-1}*2^{\left(\frac{M-1}{M}\right)}\approx 2^B=S_{\mathrm{out}}.$

[0118] In the third implementation, the steps S10 to S20 described in connection with the first implementation are performed, with the only differences indicated above.

[0119] A fourth implementation, that can be termed as a negative and fixed-point approach, may now be described. The fourth implementation is based on the second (fixed-point) implementation and only varies from the second implementation by the features described below.

[0120] The fourth implementation mainly differs from the second implementation by the features that each input number z.sub.i is negative or zero and derived from the original input number x.sub.i by subtracting x.sub.max from x.sub.i so as to obtain input number z.sub.i and the lookup table T4 is another variant of the lookup table defined by the general expression (3).

[0121] In the fourth implementation, the constant parameters of the expression (3) for computing the fraction components fc.sub.j of the lookup table T4 are set as follows:

a=S.sub.out=2.sup.B; b=+1; c=0; d=S.sub.out=2.sup.B.

[0122] For example, the fraction components fc.sub.j of the lookup table T4 are computed using the expression:

[00039]$f{c}_j=2^B*2^{\left(\frac{j}{M}\right)}-2^B$

[0123] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the smallest fraction component is fc.sub.0=0 and the largest fraction component is

[00040]${\mathrm{fc}}_{M-1}=2^B*2^{\left(\frac{M-1}{M}\right)}-2^B\approx 2^B=S_{\mathrm{out}}.$

[0124] In the fourth implementation, the steps S10 to S18 and S21 to S24 described in connection with the second implementation are performed, with the only differences indicated above.

[0125] In the step S18, combining the integral part int.sub.i and the IEEE 754 exponent bias may consist in either subtracting the integral part int.sub.i from the IEEE 754 exponent bias or adding the integral part int.sub.i and the IEEE 754 exponent bias, depending on the sign of the integral part int.sub.i. In case that the input value z.sub.i is positive (as in the second implementation), the int integral part int.sub.i is subtracted from the IEEE 754 exponent bias. But, in case that the input value z.sub.i is negative, the integral part int.sub.i is added to the IEEE 754 exponent bias, as in the fourth implementation.

[0126] A fifth implementation, that can be termed as a negative, 0.5, integer approach, may now be described. The fifth implementation is based on the first implementation and only varies from the first implementation by the use of another variant of the lookup table T5 defined by the general expression (3).

[0127] In the fifth implementation, the constant parameters of the expression (3) for computing the fraction components fc.sub.j of the lookup table T5 are set as follows:

[00041]$a=S_{\mathrm{out}}=2^B;b=-1;c=\frac{-0.5}{M};\mathrm{and}d=0.$

[0128] For example, the fraction components fc.sub.j of the lookup table T5 are computed using the expression:

[00042]$\begin{array}{cc}{\mathrm{fc}}_j=2^B*2^{\left(\frac{-j-0.5}{M}\right)}& \left(7\right)\end{array}$

[0129] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the largest fraction component is fc.sub.0≈2.sup.B=S.sub.out and the smallest fraction component is:

[00043]${\mathrm{fc}}_{M-1}=2^B*2^{\left(\frac{-M+0.5}{M}\right)}\approx \frac{2^B}{2}=\frac{S_{\mathrm{out}}}{2}.$

[0130] In the fifth implementation, the steps S10 to S20 described in connection with the first implementation are performed, with the only differences indicated above.

[0131] A sixth implementation, that can be termed as a negative, 0.5, fixed-point approach, may now be described. The sixth implementation is based on the second implementation and only varies from the second implementation by the use of another variant of the lookup table T6 defined by the general expression (3).

[0132] In the sixth implementation, the constant parameters of the expression (3) for computing the fraction components fc.sub.j of the lookup table T6 are set as follows:

[00044]$a=2*S_{\mathrm{out}}=2^{B+1};b=-1;c=\frac{-0.5}{M};\mathrm{and}d=S_{\mathrm{out}}=2^B.$

[0133] For example, the fraction components fc.sub.j of the lookup table T6 are computed using the expression:

[00045]$\begin{array}{cc}{\mathrm{fc}}_j=2^{B+1}*2^{\left(\frac{-j-0.5}{M}\right)}-2^B& \left(7\right)\end{array}$

[0134] Thus, the parameters a, b, c and d for computing the fraction components fc.sub.j are adjusted in a way that the largest fraction component is fc.sub.0≈2.sup.B=S.sub.out and the smallest fraction component is:

[00046]${\mathrm{fc}}_{M-1}=2^{B+1}*2^{\left(\frac{-M+0.5}{M}\right)}-2^B\approx 0.$

[0135] In the sixth implementation, the steps S10 to S18 and S21 to S24 described in connection with the second implementation are performed, with the only differences indicated above.

[0136] The lookup tables storing fraction components fc.sub.j derived from the expression

[00047]$2^{\left(\frac{j}{N}\right)},$

with b=+1, and the lookup tables storing fraction components fc.sub.j derived from the expression

[00048]$2^{\left(\frac{-j}{N}\right)},$

with b=−1, basically store identical numbers but in opposite order (potentially with some shift by the constant c).

[0137] In the fixed-point (or integer) approaches (first, third and fifth Implementations), the constant parameter d is set to zero.

[0138] For stability improvement, all operations with a potential to overflow a register (like additions, subtractions, clipping from N/2+1 to N/2 bits) should be implemented as saturating operations. For example, if the result of an operation in a register is greater than the maximum value that can be stored in said register, the register is set to the maximum (all the register bits are set to one.

[0139] For precision improvement, all divisions (including any division implemented by performing a right-shift in a register) are preferably rounded, by adding half the divisor to the dividend. For example, any division of the type “a divided by b” is advantageously executed by adding ‘b/2’ to the dividend ‘a’ before dividing by the divisor ‘b’ (or performing a corresponding right-shift in a register).

[0140] The present disclosure also concerns: a computer program including instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any of the implementations previously defined; a data processing device including a processor in charge of performing the steps of the method according to any of the implementations previously defined.