HARDWARE ACCELERATOR EXTENSION TO TRANSFER LEARNING - EXTENDING/FINISHING TRAINING TO THE EDGE

Abstract

A computer-implemented method for training a neural network on a hardware accelerator of an edge device includes dividing a trained neural network into a domain independent portion and a domain dependent portion. The domain independent portion of the neural network is deployed onto a dedicated neural network processing unit of the hardware accelerator of the edge device, and the domain dependent portion of the neural network is deployed onto one or more additional processors of the hardware accelerator of the edge device. The domain dependent portion on the additional processors of the hardware accelerator is retrained using data collected at the edge device.

Claims

1. A computer-implemented method for training a neural network on a hardware accelerator of an edge device, the method comprising: dividing a trained neural network into a domain independent portion and a domain dependent portion; deploying the domain independent portion of the neural network onto a dedicated neural network processing unit of the hardware accelerator of the edge device; deploying the domain dependent portion of the neural network onto one or more additional processors of the hardware accelerator of the edge device; retraining the domain dependent portion on the additional processors of the hardware accelerator using data collected at the edge device.

2. The method of claim 1, wherein the trained neural network is divided at a remote computer system connected to the edge device over a network.

3. The method of claim 1, wherein the trained neural network is divided at the edge device using the additional processors.

4. The method of claim 1, wherein the number of layers of the trained neural network included in the domain dependent portion is selected based on hardware characteristics of the edge device.

5. The method of claim 1, wherein the domain independent portion performs feature extraction on a set of input data and the domain dependent portion performs one or more image processing tasks on outputs for the domain independent portion.

6. The method of claim 5, wherein the image processing tasking comprise one or more of object detection, object segmentation, image classification, or localization.

7. The method of claim 1, wherein the neural network is trained using a first set of data that is not specific to a factory operating environment and the domain dependent portion of the neural network is retrained using a second set of data that is specific to the factory operating environment.

8. The method of claim 7, wherein the first set of data and second set of data comprise image data.

9. The method of claim 7, wherein the first set of data and second set of data comprise audio data.

10. The method of claim 1, wherein the additional processors of the hardware accelerator are SHAVE vector processors.

11. The method of claim 1, wherein the additional processors of the hardware accelerator are graphical processing units (GPUs).

12. The method of claim 1, wherein the additional processors of the hardware accelerator are central processing units (CPUs).

13. An edge device connected to a remote computer system over a network, the edge device comprising: a hardware accelerator comprising: one or more communication buses, dedicated neural network processing unit executing a domain independent portion of a trained neural network, and one or more processors executing a domain dependent portion of the trained neural network connected to the domain independent portion over the communication buses, wherein the domain dependent portion is re-trained by the processors using data collected at the edge device.

14. The edge device of claim 13, wherein the trained neural network is divided at the remote computer system, and the domain independent portion and the domain dependent portion are delivered to the edge device separately over the network.

15. The edge device of claim 13, wherein the trained neural network is divided at the edge device using the additional processors.

16. The edge device of claim 13, wherein the number of layers of the trained neural network included in the domain dependent portion is selected based on hardware characteristics of the edge device.

17. The edge device of claim 13, wherein the additional processors of the hardware accelerator are SHAVE vector processors.

18. The edge device of claim 13, wherein the additional processors of the hardware accelerator are graphical processing units (GPUs).

19. The edge device of claim 13, wherein the additional processors of the hardware accelerator are central processing units (CPUs).

20. A system for training a neural network, the system comprising: a computer configured to divide a neural network into a domain independent portion and a domain dependent portion, wherein at least the domain independent portion is pre-trained; an edge device configured to: receive the domain independent portion and domain dependent portion from the computer via a network, deploy the domain independent portion of the neural network onto a dedicated neural network processing unit of the hardware accelerator of the edge device, deploy the domain dependent portion of the neural network onto one or more additional processors of the hardware accelerator of the edge device, train the domain dependent portion on the additional processors of the hardware accelerator using data collected at the edge device, and following the deployments and the retraining, use the domain independent portion and the domain dependent portion to perform one or more tasks.

21. A system for training a neural network, the system comprising: a computer configured to divide a neural network into a domain independent portion and a domain dependent portion, wherein at least the domain independent portion is pre-trained; an edge device configured to: receive the domain independent portion from the computer via a network, deploy the domain independent portion of the neural network onto a dedicated neural network processing unit of the hardware accelerator of the edge device, generate a new version of the domain dependent portion of the neural network onto one or more additional processors of the hardware accelerator of the edge device, train the new version of the domain dependent portion on the additional processors of the hardware accelerator using data collected at the edge device, and following the deployments and the training, use the new version of the domain independent portion and the domain dependent portion to perform one or more tasks.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[0013] FIG. 1A shows an example edge device architecture configured in accordance with some embodiments of the present invention;

[0014] FIG. 1B shows an alternative edge device architecture configured in accordance with some embodiments of the present invention; and

[0015] FIG. 2 illustrates a computer-implemented method for training a neural network on a hardware accelerator of an edge device, according to some embodiments.

DETAILED DESCRIPTION

[0016] The following disclosure describes the present invention according to several embodiments directed at methods, systems, and apparatuses related to hardware accelerator extension to transfer learning by extending/finishing training to the edge.

[0017] In typical neural network deployments, maximally large data sets are recorded at the edge, and uploaded to the cloud to retrain the neural network for optimal performance. The network is then downloaded and applied at the edge. This process is repeated whenever a new class needs to be added to the network. The reason for this approach is that neural network hardware accelerators are designed to maximize their inference performance. Fast, iterative changing of neural network weights, required for training, are not considered in the architecture, as the hardware is computationally not powerful enough for full-blown training and will not be exposed to millions of examples from large databases at the edge. An alternative is to collect the data and do the training at a local server for data privacy concerns. Thereafter, the updated network is uploaded back on the edge hardware for future inferences. Neither of these approaches allows training with minimal data, inside the edge device (without interfacing other systems), and while allowing continuous operation of the device inference (i.e., without stopping the factory processing line).

[0018] With the techniques described herein a neural network is divided into a fixed and a flexible part. The fixed part is pre-trained, for example, with millions of examples in the cloud and used as feature extraction layers. The fixed part can then be deployed on the inference-optimized parts of the hardware accelerators. The flexible part of the neural network is used to train new classes and adapt current classifiers on the fly. This flexible part can then be deployed in the flexible parts of the hardware accelerator such as Streaming Hybrid Architecture Vector Engine (SHAVE) processors, GPU, or CPUs. By performing neural network training/adaptation at the edge, on low power hardware we can enable more flexible applications including continuous learning based approaches. This will make the update and refinement process of neural networks significantly more efficient than for conventional solutions.

[0019] FIG. 1A shows an example edge device architecture configured in accordance with some embodiments of the present invention. This Edge Device 105 is connected to a Remote Computer 110 over a Network 115. The main component of the Edge Device 105 in this example is Artificial Intelligence (AI) Hardware Accelerator 120. The Hardware Accelerator 120 includes a Dedicated Neural Network Processing Unit 125 comprising one or more processing units optimized for executing neural networks at high speed and low power. The Hardware Accelerator 120 further includes one or more Additional Processors 130. These Additional Processors 130 may include, for example, SHAVE processors, Graphical Processing Units (GPUs), or even central processing units (CPUs). A single type of processor may be employed, or combinations of different types of processors (e.g., SHAVE processors and CPUs) may be used with the present invention. A Communication Bus 135 connects the Dedicated Neural Network Processing Unit 125 and the Additional Processors 130. Non-limiting examples of hardware accelerators that may be used with the present invention include Intel's Myriad™ X VPU. In the Myriad™ X VPU, the Dedicated Neural Network Processing Unit 125 is embodied in the neural compute engine, while the Additional Processors 130 are embodied by Myriad's SHAVE cores.

[0020] Continuing with reference to FIG. 1, at the Remote Computer 110 a Trained Neural Network 150 is divided into a Domain Independent Portion 140 and a Domain Dependent Portion 145. This division may be made, for example, by the original designer of the Trained Neural Network 150 or the Trained Neural Network 150 can be analyzed (e.g., using visualization tools known in the art) to ascertain the functional characteristics different portions of the Trained Neural Network 150.

[0021] The Domain Independent Portion 140 is the portion of the Trained Neural Network 150 which performs functions that can be used across multiple domains (e.g., feature extraction), while the Domain Dependent Portion 145 performs functions that are more specific to a particular domain or application (e.g., classification). The Hardware Accelerator 120 executes the Domain Independent Portion 140 and a Domain Dependent Portion 145 as a complete neural network using the Communication Bus 135 as necessary to transfer data between the two Portions 140, 145. Techniques for transferring data between different hardware accelerator components are generally known in the art and, in general, any such technique may be employed to facilitate the data transfer described above.

[0022] Each Portion 140, 145 comprises one or more layers. As is generally understood in the art, the term “layers” refers to a collection of nodes operating together at a specific depth within a neural network and different network architectures such as convolutional layers or polling layers. A neural network can be divided into different layers based on its underlying architecture. Research on deep neural networks has shown that the front layers in a network resemble Gabor filters or color blobs. Similar functions have been used in classical image processing approached for preprocessing and feature extraction. These features are then combined in the later layers of the network to distinguish between classes or perform other functions. The domain specificity of different layers of a neural network is analyzed in: J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3320-3328. Curran Associates, Inc., Dec. 2014. It has been shown that by using this approach, it is possible to generate significantly higher recognition accuracies with small training sets than when using the limited data for training. For example, Matlab published a comparison where they trained a network with 5000 images per class from scratch and received 75% accuracy and then used a transfer learning approach where they only retrained the last 3 layers of AlexNet with 500 images per class and achieved 85% accuracy. See “Deep Learning with MATLAB: Transfer Learning with Neural Networks in MATLAB,” available at https://www.mathwors.com/videos/transfer-learning-with-neural-networs-in-matlab-1492007175092.html (last accessed Jan. 9, 2019).

[0023] For example, a convolutional neural network may be designed with the following layers: an input layer; a series of convolutional and pooling layers, and one or more fully connected layers. The input layer receives the raw pixel values of the image. The convolutional layer comprises a set of filters (or kernels) that activate when a specific feature is detected in the input data. For example, the convolutional layers may perform edge detection and/or curve detection. The pooling layer reduces the spatial size of the outputs of the convolutional layer. In some CNNs, a pooling layer is inserted periodically between successive convolutional layers. After the series of convolutional and max pooling layers, the high-level reasoning of the neural network is performed via one or more fully connected layers. For example, the fully connected layer may perform image processing tasks such as object detection, object segmentation, image classification, or localization. Techniques for implementing these image processing techniques are generally known in the art, and thus the techniques are not described in detail herein.

[0024] With the transfer learning technique described herein, the CNN can be trained with a large dataset (e.g., the ImageNet dataset). The input layer and the series of convolutional and pooling layers are used as a domain independent feature extractor. The fully connected layers of the trained CNN are replaced with domain dependent fully connected layers on the Edge Device 105. Effectively, this can be understood as retaining the complete CNN, where only the weights of the fully connected layers are allowed to change when performing the gradient descent/optimization procedure.

[0025] In some embodiments, the division between the Domain Independent Portion 140 and a Domain Dependent Portion 145 of the Trained Neural Network 150 may be made on the basis of the hardware capabilities of the Hardware Accelerator 120. For example, as noted above, the Domain Dependent Portion 145 can just comprise the fully connected layers of the Trained Neural Network 150 that perform classification (or other tasks) on extracted feature data. Depending on the task involved, it may be desirable to also include some of the other layers (e.g., one or more convolutional and pooling layers) in the Domain Dependent Portion 145 to allow these layers to be retrained based on data that is local to the edge device. In order to assess how to divide the Trained Neural Network 150, the Remote Computer 110 may query the Edge Device 105 for its hardware characteristics and then divide the Trained Neural Network 150 accordingly. These hardware characteristics may include generally any information that would be indicative of the processing capabilities of the Edge Device 105. This information may include, without limitation, processing speed, available RAM, and bus speed of the Hardware Accelerator 120.

[0026] FIG. 1B shows a variation on the example of FIG. 1A. Here, rather than dividing the Trained Neural Network 150 on the Remote Computer 110 and send the two Portions to 140, 145 to the Edge Device, the Remote Computer 110 simply sends the complete Trained Neural Network 150. Then, the Edge Device 105 divides the Trained Neural Network 150 into the Domain Independent Portion 140 and the Domain Dependent Portion 145. In some embodiments, the Trained Neural Network 150 may be delivered to the Edge Device 105 with instructions specifying the hardware characteristics required to include certain layers in the Domain Dependent Portion 145. In this way, the Edge Device 105 can perform the division locally based on an evaluation of its hardware characteristics.

[0027] FIG. 2 illustrates a computer-implemented method 200 for training a neural network on a hardware accelerator of an edge device, according to some embodiments. This method 200 may be used to allow customization of a pre-trained neural network at an edge device. For example, in one embodiment, the neural network is trained using a first set of data that is not specific to a factory operating environment. In general this data can take any form and will vary based on the application. Non-limiting examples of the training data include generic image and/or audio data. A domain independent version is created and can be retrained at the edge device using a second set of data that is specific to the factory operating environment. It should be noted that “domain independent” does not necessarily imply that the original neural network is entirely generic. For example, a neural network can be trained based on library of sensor measurements collected from a group of factories. A portion of this neural network can then be re-trained at an edge device of a particular factory to refine the neural network with locally collected sensor measurements.

[0028] Starting at step 205, a trained neural network is divided into a domain independent portion and a domain dependent portion. In some embodiments, the trained neural network is divided at a remote computer system connected to the edge device over a network. For example, in some embodiments, the neural network portions are transferred as one or more Extensible Markup Language (XML) files describing the architecture of the network and weights for each node. In other embodiments, the neural network is transferred as one or more executable applications compiled for the architecture of the edge computer. In other embodiments, the trained neural network is divided at the edge device using the additional processors. The number of layers of the trained neural network included in the domain dependent portion may be selected, for example, based on hardware characteristics of the edge device.

[0029] Continuing with reference to FIG. 2, at step 210, the domain independent portion of the neural network is deployed onto a dedicated neural network processing unit of the hardware accelerator of the edge device. The hardware accelerator may include in silicon optimized neural network specific operations such as convolutions or matrix multiplication or extremely fast memory-compute connectivity optimized for common neural network design execution. The domain dependent portion of the neural network is deployed at step 215 onto one or more additional processors of the hardware accelerator of the edge device. The method of deployment can vary depending on how the neural network was developed and the architecture of the edge device. However, in general any deployment technique known in the art may be used.

[0030] The domain dependent portion is re-trained at step 220 on the additional processors of the hardware accelerator using data collected at the edge device. For example, in one embodiment, the edge device is a component with in a factory setting that acts as a controller. The edge device receives measurements, makes predictions based on those measurements, and then generates one or more outputs that control factory devices. In this case, the measurement data gathered during operations can be used to re-train the domain dependent portion of the neural network. Note that this method of re-training not only allows device-level customization of the neural network, but also training with higher data fidelity than may be available at remote computing system. For example, sensor measurements may be gathered at 0.01 second intervals and 10 seconds worth of data may be required to make a prediction. If the data needs to be transferred to a remote computer, storage and bandwidth requirements may require the data to be sampled on a per-second basis, thus requiring the training to be performed based on 10 measurements. However, at the edge device, every measurement can be used for training, thus potentially increasing the accuracy of overall model. Finally, once the neural network is re-trained, the edge device uses it to perform one or more tasks at step 225. For example, to continue with the examples set forth above, the edge device may classify image data and generate an output that is used to control a factory device.

[0031] In some embodiments, the remote computer system is not transferred at the domain dependent portion to the edge device; instead, the domain dependent portion is fully generated at the edge based on edge data. However, providing a pre-trained domain dependent portion with the same class outputs should help the conversion speed of the algorithms. This assumes that the difference between the domains is not very large and one can find a convex path to the optimum classifier in the new domain. The training can also try this approach and if no sufficient classifier can be adapted; then one can reinitiate the fully connected layers in the domain dependent network randomly and start training this part from scratch.

[0032] To further illustrate the applicability of the technique described in FIG. 2, imagine the use case where the output of an industrial process needs to be classified in good products and faulty products based on a camera image. It would be useful to define/specify the detected fault so that the next steps like a rework can be automated. As certain faults are very rare, there is no data available on these fault classes during the training phase. Using the approach applied in conventional systems one would need to stop production to ensure that no faults are missed, add the new data to the training and upload the new model before activating the production line again. This is a very costly and sometimes infeasible approach as there can be long ramp-up times of the machines until they produce at desired quality levels. This can be solved by using a Progressive Neural Network learning approach illustrated in: A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606.04671, 2016.

[0033] Similar to transfer learning, this approach borrows from previously found classifiers to minimize necessary parameter optimizers and take advantage of the larger dataset that was available to train the feature extraction parts of the network. The advantage of this approach is that previously learned classifiers are frozen and not affected by the training of a new classifier. This enables that the classification can continue during training without loss of accuracy. Again, the network can be split into a fixed, high performance optimized part and a flexible adaptive part. The training can be outsourced in the flexible layers of the hardware accelerator to enable operation in current edge hardware.

[0034] The training technique described above enables a large number of new applications through training at the edge. As noted above, one advantage of the disclosed techniques is that data does not need to be recorded and moved to a powerful machine for training resulting in a more efficient process. Given this minimal effort, the user will be able to adapt the training more frequently for finer adjustments. The user can give corrective feedback during operation for continuous learning at the edge. The performance of the classifier that uses large pre-trained networks from millions of examples as a basis and adapts to the use case by a small set of training data is significantly higher than a network that is trained from the few examples from scratch. Re-training and adding of classes can be done during online operation. This may result in reducing costly shutdowns of monitored production lines. In addition, training components on the edge generates a logical separation layer, enabling users to isolate parts of their processes from pre-training which may happen externally, such as on a cloud platform.

[0035] The embodiments of the present disclosure may be implemented with any combination of hardware and software. For example, aside from hardware accelerator architectures presented above with respect to FIGS. 1A and 1B, one can use the hardware accelerator as a co-processor, computing all flexible (i.e., domain dependent) parts in the main CPU and outsourcing the fixed (i.e., domain independent) part to the accelerator. In addition, the embodiments of the present disclosure may be included in an article of manufacture (e.g., one or more computer program products) having, for example, computer-readable, non-transitory media. The media may have embodied therein computer readable program code for providing and facilitating the mechanisms of the embodiments of the present disclosure. The article of manufacture can be included as part of a computer system or sold separately.

[0036] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

[0037] An executable application, as used herein, comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.

[0038] The functions and process steps herein may be performed automatically or wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.

[0039] The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and/or combinations thereof. No claim element herein is to be construed under the provisions of 35 U.S.C. 112(f) unless the element is expressly recited using the phrase “means for.”