Method for realizing a neural network

Abstract

A method for realizing an artificial neural network via an electronic integrated circuit (FPGA), wherein artificial neurons grouped into different interlinked layers for the artificial neural network, where a functional description is created for each neuron of the artificial neural network, taking into account a specifiable starting weighting, a synthesis is performed for each neuron based on the associated functional description with the associated specified starting weighting, a network list is determined as the synthesis result, in which at least a base element and a starting configuration belonging to the base element are stored for each neuron, a base element is formed as a lookup table (LUT) unit and an associated dynamic configuration cell, in which a current configuration for the LUT unit or the base element is stored, and where the network list is implemented as a starting configuration of the artificial neural network in the electronic integrated circuit.

Claims

1. A method for realizing an artificial neural network via an electronic integrated circuit, the artificial neural network being formed from artificial neurons which are grouped into different layers and linked to each other, the method comprising: a. creating a functional description, taking into account a specifiable starting weighting for each neuron; b. performing a synthesis for each respective neuron based on a respective functional description with an associated specifiable starting weighting; c. creating a netlist as a synthesis result, at least one base element and a starting configuration belonging to the base element being stored in the netlist for each neuron, and the at least one base element being formed by a lookup table (LUT) unit and an associated dynamic configuration cell, in which a respective current configuration for the associated LUT unit is stored; d. implementing the netlist as a starting configuration of the artificial neural network in the electronic integrated circuit; wherein, starting from the starting configuration of the artificial neural network implemented in the electronic integrated circuit, a training phase of the artificial neural network is performed in which at least one of (i) the starting configuration and (ii) a respective current configuration of at least one of (i) at least one base element and (ii) at least one neuron is changed; and wherein at least one of (i) fixed, specified test data and (ii) test samples are utilized in the training phase of the artificial neural network, output data obtained with at least one of (i) the test data and (ii) test sample is compared with specified reference data, and a change to the respective current configuration of at least one of (i) at least one base unit and (ii) at least one neuron is performed iteratively until the output data obtained with at least one of (i) the test data and (ii) the specified test sample corresponds to the specified reference data within a specifiable tolerance.

2. The method as claimed in claim 1, wherein, during the creation of the functional description, taking into account the specifiable starting weighting of the respective neuron, the functionality of the respective neuron is reduced such that the respective neuron is mapped onto one base element.

3. The method as claimed in claim 1, wherein a specially designated interface of the electronic integrated circuit is utilized during the training phase of the artificial neural network to feed in a change to at least one of (i) the starting configuration and (ii) the respective current configuration of at least one of (i) the at least one base unit and (ii) the neuron.

4. The method as claimed in claim 1, wherein a memory unit of a configuration memory of the electronic integrated circuit is utilized as the configuration cell for storing the respective current configuration of the respective base element.

5. The method as claimed in claim 1, wherein that the configuration cell for storing the respective current configuration of the respective base element comprises static RAM.

6. The method as claimed in claim 1, wherein the electronic integrated circuit comprises a Field Programmable Gate Array (FPGA).

7. The method as claimed in claim 1, wherein the functional description of the respective neuron is created in a hardware description language (HDL).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The invention will now be described by way of example making reference to the accompanying figures, in which:

(2) FIG. 1a shows an exemplary and generalized structure of an artificial neural network;

(3) FIG. 1b shows an exemplary and generalized basic structure of an artificial neuron;

(4) FIG. 2 shows an exemplary, schematic implementation of a neuron with dedicated memory block for associated weighting factors during a training phase;

(5) FIG. 3 shows a schematic sequence of the method in accordance with the invention for realizing an artificial neural network via an integrated electronic circuit; and

(6) FIG. 4 shows an exemplary schematic representation of a base element for a neuron for use in the method in accordance with the invention.

DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS

(7) Artificial neural networks mimic natural neural networks. An artificial neural network therefore typically represents a highly simplified abstraction model of a natural neural network, where the artificial neural network is constructed of artificial neurons that are connected to each other in the manner of a network. A structure of an artificial neural network of this type, as is known, e.g., from Rey, Güdnter Daniel; Wender, Karl F.: Neuronale Netze—Eine Einführung in die Grundlagen, Anwendungen and Datenauswertung [Neural Networks—an Introduction to the Basics, Applications and Data Analysis], 2nd edition, Huber Verlag, 2010, and an exemplary basic structure of an artificial neuron, as presented, e.g., in the publication by Muthuramalingam, A, et al., Neural Network Implementation Using FPGA: Issues and Application, International Journal of Electrical and Computer Engineering Vol. 2, No. 12, 2008, will be described in further detail with the aid of FIGS. 1a and 1b.

(8) FIG. 1a shows, by way of example, the structure of a multi-layer “feedforward neural network”, where the structure of the network is linked directly, for example, to the learning or training methods used and a necessary scope of the neural network depends on the complexity of the particular task to be solved (e.g., text recognition, image recognition or pattern recognition, decision-making or a classification task). The neural network consists of multiple neurons N1 to N8 that are used to receive information from the environment or from other neurons N1 to N8 and to pass it on in modified form to other neurons N1 to N8 or to the environment. For this purpose, the neurons N1 to N8, for example, are grouped into different layers ES, VS, AS and linked to each other. A neural network may comprise, for example, an input layer ES consisting of input neurons N1, N2, N3, one or more hidden layers VS, and an output layer AS with output neurons N7, N8. Signals, stimuli or samples are received by the input layer ES or the input neurons N1, N2, N3 via inputs E1, E2, E3. The outputs of the input neurons N1, N2, N3 are linked to the neurons N4, N5, N6 of the hidden layer VS and the outputs thereof linked to the output neurons N7, N8, with signals or output data then being output via their outputs.

(9) FIG. 1b shows, by way of example, a generalized basic structure of an artificial neuron N1 to N8 with n input connections. Via the input connections, the neuron obtains its input values p1, p2, p3, . . . , pn, e.g., in the form of signals, stimuli or samples from the environment, or in the form of output data from other neurons. Each of these input values p1, p2, p3, . . . , pn are linked to a weighting or weighting factors w1, w2, w3, . . . , wn. Before a training phase for the neural network, for example, the weighting or the weighting factors w1, w2, w3, . . . , wn may be specified as a starting weighting w1, w2, w3, . . . , wn for the respective neuron, where a maximum possible value for the weighting or weighting factors w1, w2, w3, . . . , wn of the respective neuron may be used as the starting weighting, for example. The weighting w1, w2, w3, . . . , wn may be varied accordingly during the training phase. Furthermore, with an artificial neuron, a “bias input b” may be provided with an input value 1.

(10) The functionality of the neuron, which may be used to create the functional description for a realization by means of an integrated electronic circuit, is described in principle by a transfer function Σ and an activation function f(x). With the transfer function Σ, a network input x of the neuron is calculated with the aid of the weighting w1, w2, w3, . . . , wn of the input values p1, p2, p3, . . . , pn, possibly taking into consideration the input value of the bias input b. The output y of the neuron is determined by the activation function f(x), where different function types may be used for the activation function depending on the respective network topology. A linear function, a sigmoid function, a piecewise linear function or a step function, for example, may be used as the activation function f(x).

(11) The transfer function Σ and the activation function f(x) must be calculated for the implementation of the functionality of an individual neuron, as shown, e.g., in FIG. 1b. Different calculation units (e.g., summators, multipliers or complex evaluating units) may be used for this purpose, for example, which are implemented by different circuit blocks on the basis of hardware in the case of a realization by means of an electronic integrated circuit (e.g. FPGA). Currently known implementations of artificial neurons or neural networks by means of electronic integrated circuit, in particular by means of an FPGA, are known from the following publications: Muthuramalingam, A, et al., Neural Network Implementation Using FPGA: Issues and Application, International Journal of Electrical and Computer Engineering Vol. 2, No. 12, 2008; Nurviadhi, E. et al.: Accelerating Binarized Neural Networks: Comparison of FPGA, CPU, GPU and ASIC, 2016, IEEE; or Haruyoshi Yonekawa and Hiroki Nakahara: On-chip Memory Based Binarized Convolutional Deep Neural Network Applying Batch Normalization Free Technique on an FPGA, 2017, IEEE p. 98-105.

(12) As shown in simplified form and by way of example in FIG. 2, an artificial neuron realized nowadays on an electronic integrated circuit SE or on an FPGA may consist of at least, e.g., a lookup table or LUT unit LUT, and a memory block BS. Optionally a calculation unit RE may also be provided. A configuration cell KON belonging to the LUT unit LUT is used in the configuration memory KS of the electronic integrated circuit SE, for example, to configure the LUT unit LUT or to realize the function of the LUT unit LUT. In the case of neurons with complex functionality, multiple such units LUT, RE and BS as well as configuration cells KON assigned to the LUT units LUT may possibly be provided.

(13) The transfer function Σ and the activation function f(x) are realized, e.g., in the training phase and in live operation via the LUT unit LUT and any calculation unit RE that may be present. For this purpose, for example, the function of the LUT unit is fixed and specified by the configuration of the LUT unit stored in the associated configuration cell KON. The calculation unit RE may be realized as a digital signal processor (DSP) or constructed from, e.g., gates, shift registers, summator units and/or multiplier units. The weighting w1, w2, w3, . . . , wn for the calculation of the network input of the neuron via the transfer function Σ is provided by at least one block memory BS which may be formed, e.g., as block RAM or BRAM. The content of the at least one block memory BS, i.e., the respective weighting w1, w2, w3, . . . , wn of the respective neuron, may be changed for example during the training phase. That is, in order to train the neurons or neural network, the weighting w1, w2, w3, . . . , wn, which is stored in a dedicated memory block BS of at least one neuron, is modified. If an output result with the desired quality is obtained from the neural network during the training phase, then the weightings w1, w2, w3, . . . , wn stored in the respective memory blocks BS are fixed.

(14) FIG. 3 shows an exemplary and schematic sequence of the method in accordance with the invention for realizing an artificial neural network via an electronic integrated circuit SE formed, by way of example, as an FPGA. Here, the artificial neural network consists of neurons (as shown by way of example in FIG. 1b) which, as described by way of example with the aid of FIG. 1a, are grouped in layers and linked to each other. In a first step 101 of the method in accordance with the invention, a functional description is created for each neuron, where a specifiable starting weighting for the respective neuron is already taken into account in the functional description. The starting weighting that may be, for example, randomly selected or maximum possible values for the weighting w1, w2, w3, . . . , wn, may be used as the starting weighting. The functional description of the respective neuron is composed in, for example, a hardware description language (e.g., HDL, Very High Speed Integrated Circuit Hardware Description Language (VHDL), or Verilog).

(15) In a second step 102, a synthesis is performed based on the respective functional description of the respective neuron, in which the respective associated starting weighting is specified or has been taken into account, and in a third step 103 a netlist is created or obtained as the synthesis result. That is, the functional description of the neurons or neural network, which is usually available in a hardware description language, is converted into a netlist. The netlist typically comprises a description of the modules necessary for realization of the circuit (e.g., logic gates or memory blocks) and the corresponding links between the modules. In the netlist created in the third step 103, at least one base element BE1 and a starting configuration belonging to the base element BE1 are then stored for each neuron.

(16) A base element BE1 of this type is shown by way of example and schematically in FIG. 4. A neuron of a neural network realized in accordance with the method of the invention consists at least of one base element BE1 of this type. However, functionally more complex neurons may also have multiple base elements BE1, BE2, . . . . Ideally, however, in the first step 101, during the creation of the functional description, taking into account the respective starting weighting, the functionality of the respective neuron is reduced such that the respective neuron may be mapped onto precisely one base element BE1.

(17) A base element BE1 of this type, via which a neuron may be formed, is formed from a LUT unit LUT1 on the electronic integrated circuit SE and an associated dynamic configuration cell KON1. A respective current configuration for the LUT unit LUT1 is stored in the dynamic configuration cell KON1. With this configuration, the function of the LUT unit LUT1 is determined from which, in principle, the transfer function are implemented, taking into account a weighting w1, w2, w3, . . . , wn, and the activation function f(x). A memory unit of the configuration memory KS of the electronic integrated circuit SE is used as the configuration cell KON1, for example, which may be formed as static RAM or Flash memory, for example.

(18) In the case of a functionally more complex neuron, a further base unit BE2 may be provided, for example, which likewise consists of a further LUT unit LUT2 and a further associated dynamic configuration cell KON2 in which the respective current configuration for the further LUT unit LUT2 is stored. If, in the step 101, the neurons of the neural network are established via functional description such that each neuron is mapped only onto one base unit BE1, BE2, . . . , the further base unit BE2 may already represent a further neuron of the neural network.

(19) In a fourth step 104, the netlist is loaded and implemented, as the starting configuration of the neural network, into the electronic integrated circuit SE or into the FPGA SE. Subsequently a training phase, ideally during online operation or directly on the circuit SE, may then be started.

(20) For this purpose, when the circuit SE or FPGA is ramped up or started, for example, the starting configurations stored in the configuration cells KON1, KON2, . . . , are loaded into the LUT units LUT1, LUT2, . . . . In the training phase, fixed, specified test data and/or test samples are used to perform the task to be solved by the neural network (e.g., text recognition, image recognition or pattern recognition, decision-making or a classification task). The output data of the neural network obtained with the test data and/or test samples is then compared with specified reference data and a check is performed to determine whether the output data obtained corresponds to the reference data within a specifiable tolerance.

(21) If the specifiable tolerance is exceeded or the desired conformity to the reference data is not achieved, then starting from the starting configuration, the respective current configuration is modified for at least one base element BE1, BE2, . . . and/or for at least one neuron. That is, the configuration of the respective associated LUT unit LUT1, LUT2, . . . stored in the respective configuration cell KON1, KON2, . . . is changed. In this process, by way of example, only the functional description for the respective neuron may be modified, or only the weighting w1, w2, w3, . . . , wn taken into account in the functional description for the neuron, or both. The modified configuration for the respective LUT unit LUT1, LUT2, . . . may be fed in, for example, via a specially designated interface of the electronic integrated circuit SE or FPGA SE (e.g., special interface of a manufacturer of the circuit or FPGA).

(22) If the specifiable tolerance is achieved/not exceeded or the desired conformity is obtained between output data of the neural network and the reference data, then the respective current configurations of the LUT units LUT1, LUT2, . . . in the associated configuration cells KON1, KON2, . . . are fixed and the training phase is terminated.

(23) In a neural network realized via the method in accordance with the invention, the respective neurons are realized via at least one respective base element BE1, BE2, where the function of the respective neuron is determined by the respective LUT unit LUT1, LUT2, . . . . Ideally, the functional content of the respective LUT unit LUT1, LUT2, . . . may be varied via the associated configuration cell KON1, KON2, . . . . The respective weighting w1, w2, w3, . . . , wn of the respective neuron is mapped indirectly in the function of the respective LUT unit LUT1, LUT2, . . . . As a result, a consumption of resources by the electronic integrated circuit or FPGA SE may be significantly reduced and the processing speed, in particular during the training phase, may be increased.

(24) Thus, while there have been shown, described and pointed out fundamental novel features of the invention as applied to a preferred embodiment thereof, it will be understood that various omissions and substitutions and changes in the form and details of the methods described and the devices illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit of the invention. For example, it is expressly intended that all combinations of those elements and/or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Moreover, it should be recognized that structures and/or elements and/or method steps shown and/or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.