Filtering data in orthogonal directions through a convolutional neural network

Abstract

A method can be used to process an initial set of data through a convolutional neural network that includes a convolution layer followed by a pooling layer. The initial set is stored in an initial memory along first and second orthogonal directions. The method includes performing a first filtering of the initial set of data by the convolution layer using a first sliding window along the first direction. Each slide of the first window produces a first set of data. The method also includes performing a second filtering of the first sets of data by the pooling layer using a second sliding window along the second direction.

Claims

1. A method of processing an initial set of data through a convolutional neural network that includes a convolution layer followed by a pooling layer, the initial set being stored in an initial memory along first and second orthogonal directions, the method comprising: performing a first filtering of the initial set of data by the convolution layer using a first sliding window along the first direction, each slide of the first window producing a first set of data; and performing a second filtering of the first sets of data by the pooling layer using a second sliding window along the second direction.

2. The method according to claim 1, wherein the convolution layer comprises a plurality of convolution layers and the pooling layer comprises a plurality of pooling layers.

3. The method according to claim 1, wherein the initial set of data is stored in matrix fashion according to rows and columns, the rows extending along the first direction and the columns extending along the second direction, each individual row extending along the second direction and each individual column extending along the first direction.

4. The method according to claim 3, wherein the first sliding window slides by strides along the first direction and, at each stride, traverses the corresponding row.

5. The method according to claim 1, wherein the first filtering and the second filtering follow one another until all the data of the initial set is processed.

6. The method according to claim 1, wherein the neural network comprises a non-linear layer between the convolution layer and the pooling layer.

7. The method according to claim 1, wherein the neural network is defined by parameters implemented in floating point or in fixed precision.

8. A method of processing an initial set of data through a convolutional neural network that includes a convolution layer followed by a pooling layer, the initial set being stored in an initial memory along first and second orthogonal directions, the method comprising: performing a first filtering of the initial set of data by the convolution layer using a first sliding window along the first direction, each slide of the first window producing a first set of data; performing a second filtering of the first sets of data by the pooling layer using a second sliding window along the second direction, wherein the second sliding window has a dimension N counted along the first direction, the first filtering comprising N−1 slides of the first sliding window along the first direction; and storing N first sets of data in a first temporary memory.

9. The method according to claim 8, wherein the second filtering comprises slides of the second sliding window, along the second direction, on the first temporary memory storing the N first sets of data, each slide producing a second set of data, the second filtering comprising a second storage of M second sets of data in a second temporary memory.

10. The method according to claim 8, wherein the storing comprises a circular filling of the first temporary memory by the first set of data produced at each slide of the first sliding window.

11. The method according to claim 10, wherein the second filtering is performed after a number of circular fillings of the first memory.

12. A device comprising: an initial memory configured to store an initial set of data along first and second orthogonal directions; a first memory configured to store parameters defining a convolutional neural network configured to process the initial set of data, the network comprising a convolution layer followed by a pooling layer; and a computation unit configured to perform the operations of the layers of the neural network, the convolution layer being configured to perform a first filtering of the initial set of the data using a first sliding window along the first direction so as to produce a first set of data at each slide, and the pooling layer being configured to perform a second filtering of the first sets of data using a second sliding window along the second direction.

13. The device according to claim 12, wherein the second sliding window has a dimension N counted along the first direction, the device comprising a processor configured to limit the slides of the first sliding window to N−1 slides along the first direction, the device further comprising a first temporary memory configured to store N first sets of data.

14. The device according to claim 13, wherein the initial set of data is stored in matrix fashion according to rows and columns, the rows extending along the first direction and the columns extending along the second direction, each individual row extending along the second direction and each individual column extending along the first direction.

15. The device according to claim 14, wherein the processor is configured to slide the first sliding window by strides along the first direction and to, at each stride, make the first sliding window traverse the corresponding row.

16. The device according to claim 13, wherein processor configured to slide the second sliding window along the second direction on the first temporary memory storing the N first sets of data, so as to produce a second set of data at each slide, the device further comprising a second temporary memory configured to store M second sets of data.

17. The device according to claim 13, wherein the processor is configured to perform a circular filling of the first temporary memory with the first sets of data produced during the slides of the first sliding window.

18. The device according to claim 17, wherein the processor is configured to perform the second filtering once the first memory has performed a number of circular fillings.

19. The device according to claim 13, wherein processor is configured to perform the first filtering and the second filtering successively until the processing of all the data of the initial set.

20. The device according to claim 12, wherein the neural network comprises a non-linear layer between the convolution layer and the pooling layer.

21. The device according to claim 12, wherein the neural network is defined by parameters implemented in floating point or in fixed precision.

22. The device according to claim 12, wherein the device is a microcontroller.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Other advantages and characteristics of the invention will become apparent on reading the detailed description of the wholly non-limiting modes of implementation and embodiments, and the appended drawings in which:

(2) FIG. 1 schematically illustrates an embodiment of the invention,

(3) FIG. 2 schematically illustrates a mode of implementation and embodiment of the invention,

(4) FIG. 3 schematically illustrates a mode of implementation and embodiment of the invention,

(5) FIG. 4A schematically illustrates a mode of implementation and embodiment of the invention,

(6) FIG. 4B schematically illustrates a mode of implementation and embodiment of the invention, and

(7) FIG. 4C schematically illustrates a mode of implementation and embodiment of the invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

(8) In FIG. 1, the reference MC designates an electronic object, for example, a microcontroller, a code generator or any other object that can contain a hardware or embedded software architecture.

(9) The object MC comprises a device MO, for example, a module comprising an initial memory MV, for example, a RAM (for “Random Access Memory”) volatile memory configured to store an initial set of data DI.

(10) The device MO also comprises a non-volatile memory MNV, for example, a ROM (for “Read-Only Memory”) memory intended to store the parameters defining a convolutional neural network RN.

(11) By “parameters” is meant the weights of the neural network and the data defining each layer of the neural network.

(12) The parameters can be implemented in fixed point or in floating point.

(13) By “floating point” is meant a real number represented by a sign, a mantissa and an exponent.

(14) By “fixed point” is meant a number which possesses a fixed number of bits after the point.

(15) This configuration makes it possible to process in a specific manner the data DI stored in the initial memory MV and to deliver output data.

(16) Accordingly, the device MO comprises a computation unit UC, for example, a microprocessor configured to perform the operations defined by each layer of the neural network RN.

(17) The computation unit UC is consequently coupled to the non-volatile memory MNV so as to extract the parameters of each layer of the neural network RN. It is also coupled to the volatile memory MV so as to extract the initial set of data DI on which the processing will be performed.

(18) The device MO comprises a first temporary memory MEM1 and a second temporary memory MEM2.

(19) By “temporary” is meant a buffer memory (commonly referred to by the term “buffer”).

(20) The first and the second temporary memories MEM1 and MEM2 are coupled to the computation unit UC.

(21) The first and the second temporary memories MEM1 and MEM2 are configured to store data delivered by the computation unit UC.

(22) The device MO comprises a processor MT coupled to the computation unit UC and configured to control the number of operations to be performed for each layer and consequently, control the storage of the data delivered by the computation unit UC.

(23) The processor can be embodied, for example, by a software module within the microprocessor and/or by specific logic circuits.

(24) FIG. 2 schematically illustrates the various steps of processing of the initial set of data DI by the layers of the neural network RN.

(25) The neural network RN comprises at least one convolution layer CV followed by at least one pooling layer CMC.

(26) The convolution layer CV makes it possible to tag the presence of a set of specific data in the initial set of data DI received in step V1. Accordingly, the convolution layer carries out a convolution filtering in step V2 using a first sliding window F1 illustrated in FIGS. 4A, 4B and 4C.

(27) The first sliding window F1 makes it possible to compute the convolution product between the window and each portion of the initial set of data DI.

(28) The initial set of data DI arises as a matrix having rows and columns, the succession of rows extending along a first direction and therefore vertically and the succession of columns extending along a second direction, that is to say horizontally.

(29) The first sliding window F1 slides along the first direction and therefore vertically, while traversing, at each slide, a row of the initial set of data DI. Each slide produces a first set of data D1 in step V3.

(30) By “slide” is meant here that the first sliding window F1 traverses each row of the initial set of data DI.

(31) Each first set of data D1 is stored in the first temporary memory MEM1.

(32) The neural network can comprise a non-linear layer CNL, which makes it possible to improve the relevance of the result. It takes as input in step V4 a first set of data D1 and produces an intermediate set of data D11 in step V5.

(33) The intermediate sets of data D11 can be placed in an intermediate temporary memory.

(34) The non-linear layer CNL being optional, the first sets of data D1 can be dispatched directly to the pooling layer CMC.

(35) The pooling layer CMC therefore receives the first sets of data D1 to which a “pooling” operation is applied in step V6, which consists in reducing the size of the sets of data D1 while preserving their important characteristics.

(36) Accordingly, the pooling layer CMC uses a second sliding window F2 illustrated in FIGS. 4A, 4B and 4C.

(37) The second sliding window F2 slides along the second direction and produces a first set of data D2 in step V7.

(38) Each second set of data D2 is stored in the second temporary memory MEM2.

(39) FIG. 3 schematically illustrates an algorithm implemented in the processor MT.

(40) The algorithm allows the processor MT to control the number of operations to be performed for each layer and consequently, control the storage of the data delivered by the computation unit UC.

(41) When the processor MT detects the presence of a convolution layer CV in step S1, they initialize in step S2 a variable i to 0 and a variable N1 also to 0.

(42) The variable i represents an index traversing the list of the rows generated as output of the convolution layer.

(43) The variable N1 represents the number of first sets D1 stored in the first temporary memory MEM1.

(44) The first and the second temporary memory MEM1 and MEM2 are empty in step S2.

(45) In step S3, the first sliding window F1 slides along the first direction and therefore vertically, while traversing at each slide a row of the initial set of data DI. Each slide produces a first set of data D1. The first set of data D1 can be dispatched to the non-linearity layer CNL in step S4 which produces an intermediate set of data D11. This step S4 is optional.

(46) In step S5, the value of the variable N1 is incremented by 1 since a first set of data D1 has just been stored in the temporary memory MEM1.

(47) In step S6, a first comparison is carried out between the variable N1 and a third variable N representative of the dimension of the second sliding window F2 and of the maximum number of first sets of data D1 that the first temporary memory MEM1 can contain.

(48) If the value of the variable N1 is equal to the value N, this means that the first temporary memory MEM1 comprises N first sets of data D1. It can therefore no longer contain any more of them.

(49) If the value of the variable N1 is less than the value N, this means that the first temporary memory MEM1 can store more first sets of data D1.

(50) Given that in this example, only a first set of data D1 has been stored, the value of the variable N1 is less than the value N and therefore step S7 is passed to in which the value of the variable i is incremented by 1.

(51) In step S8, a second comparison is carried out between the value of the variable i and a variable Max representative of the maximum number of rows generated as output of the convolution layer.

(52) If the value of the variable i is greater than the value of the variable Max, this means that all the rows generated have been traversed. The processor MT passes to the following convolution layer CV if there remains any in step S9.

(53) In the converse case, steps S3, S4 and S5 are repeated until the value of the variable N1 is equal to the value of the variable N.

(54) We thereafter pass to step S10 in which the pooling layer CMC uses the second filtering window F2 along the second direction on the N first sets of data D1 contained in the first temporary memory MEM1 and produces a second set of data D2.

(55) In step S11, a defined number N2, which represents the number of slides of the first window F1 to be carried out before performing a second filtering, is subtracted from the value of the variable N1.

(56) The value of the variable N1 is equal to N1-N2.

(57) In step S12, the value of the variable i is incremented by 1 and in step S13 we verify whether the value of the variable i is less than the value of the variable Max.

(58) If no, we pass in step S14 to the following convolution layer CV which will use another first filtering window F1 and will traverse the initial set of data DI.

(59) If yes, we continue to filter the initial set of data D1 with the first filtering window N2 times, that is to say until N1 is equal to N.

(60) The N2 first sets of data D1 overwrite the oldest first sets of data D1 in the first temporary memory MEM1. The computation unit UC therefore performs a circular filling.

(61) The second sliding window F2 carries out a slide on the first temporary memory MEM1 comprising the two new first sets D1 and produces a second set D2.

(62) This algorithm is repeated until the processing of all the rows of the initial set of data DI.

(63) FIGS. 4A, 4B and 4C schematically illustrate the implementation of the algorithm.

(64) In FIG. 4A, the first sliding window F1 slides a first time over the first row of the initial set of data DI and produces a first set of data D1 which will be stored in the first temporary memory MEM1. N1 is now equal to 1.

(65) The first window F1 slides a second time over the second row of the initial set of data DI and produces a first set of data D1 which will be stored in the first temporary memory MEM1. N2 is now equal to 2.

(66) The sliding is repeated until N1 is equal to N here 5. N represents the height of the second sliding window.

(67) The first temporary memory therefore contains N first sets of data D1.

(68) The second sliding window F2 slides along the second direction on the first temporary memory MEM1 and produces a second set of data D2 which will be stored in the second temporary memory MEM2.

(69) N2 is subtracted from N1 which is here equal to 2. N1 is therefore equal to 3.

(70) The first sliding window F1 slides a sixth time over the sixth row of the initial set of data DI and produces a first set of data D1. N1 equals 4.

(71) The first set of data D1 overwrites the oldest first set of data D1.

(72) The first window F1 slides a seventh time over the seventh row of the initial set of data DI and produces a first set of data D1. N1 equals 5.

(73) The first set of data D1 overwrites the oldest first set of data D1.

(74) It should be noted that the first window F1 may, for example, not slide over each row of the initial set of data DI. The first window F1 may have a slide stride.

(75) In FIG. 4B, the second sliding window F2 slides over the first temporary memory MEM1 containing the two new first sets of data D1 and produces a second set of data D2 which will be stored in the second temporary memory MEM2.

(76) N2 is subtracted from N1 which is here equal to 2. N1 is therefore equal to 3.

(77) The same steps in FIGS. 4B and 4C are repeated until there are no longer any rows to be processed in the initial set of data DI.