MULTI-PROCESSOR NEURAL NETWORK PROCESSING APPARATUS

Abstract

A multi-processor neural network processing apparatus comprises: a plurality of network processing engines, each for processing one or more layers of a neural network according to a network configuration. A memory at least temporarily stores network configuration information, input image information, intermediate image information and output information for the network processing engines. At least one of the network processing engines is configured, when otherwise idle, to identify configuration information and input image information to be processed by another target network processing engine and to use the configuration information and input image information to replicate the processing of the target network processing engine. The apparatus is configured to compare at least one portion of information output by the target network processing engine with corresponding information generated by the network processing engine to determine if either the target network processing engine or the network processing engine is operating correctly.

Claims

1. A multi-processor neural network processing apparatus comprising: a plurality of network processing engines, each for processing one or more layers of a neural network according to a network configuration; a memory for at least temporarily storing network configuration information for said network processing engines, input image information for processing by one or more of said network processing engines, intermediate image information produced by said network processing engines and output information produced by said network processing engines; and a system bus across which said plurality of network processing engines access said memory, wherein at least one of said network processing engines is configured, when otherwise idle, to identify configuration information and input image information to be processed by another target network processing engine and to use said configuration information and input image information to replicate the processing of the target network processing engine, said apparatus being configured to compare at least one portion of information output by said target network processing engine with corresponding information generated by said one of said network processing engines to determine if at least one of said target network processing engine or said one of said network processing engines is operating correctly.

2. An apparatus as claimed in claim 1 wherein each network processing engine comprises a cluster of more than one individual network processing engine, each cluster comprising a common controller, said common controller being configured to identify said configuration information and input image information to be processed by another target network processing engine.

3. An apparatus according to claim 2 wherein said common controller for said one of said network processing engines is configured to compare said at least one portion of information output.

4. An apparatus as claimed in claim 1 further comprising a host controller configured to designate a given network processing engine as said one of said network processing engines.

5. An apparatus as claimed in claim 1 further comprising a host controller configured to compare said at least one portion of information output.

6. An apparatus as claimed in claim 1 wherein said one of said network processing engines is configured to identify configuration information and input image information to be processed by another target network processing engine either: in said memory or as said information is passed across the system bus.

7. An apparatus as claimed in claim 1 wherein said information output comprises any one or more of: intermediate image information produced by said network processing engines; output information produced by said network processing engines; and information derived from intermediate image information or output information.

8. An apparatus according to claim 7 wherein said output information comprises any combination of output classifications, output images or output maps.

9. An apparatus according to claim 1 wherein said input image information comprises any combination of visible image information; infra-red image information; thermal image information; or image maps derived from image acquisition device images.

10. An apparatus according to claim 1 wherein said network processing engines are configured to access information through a separate common shared memory.

11. A vehicle comprising a communication network and a plurality of image capture devices arranged to acquire images from the vehicle environment and to write said images across said communication network into said memory.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] An embodiment of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0015] FIG. 1 shows a typical architecture for a driver monitoring system (DMS);

[0016] FIG. 2 shows a multi-processor neural network processing apparatus operable according to an embodiment of the present invention;

[0017] FIG. 3 illustrates Programmable Convolutional Neural Network (PCNN) engines operating in independent mode; and

[0018] FIG. 4 illustrates PCNN engines operating in redundancy mode.

DESCRIPTION OF THE EMBODIMENT

[0019] Referring now to FIG. 2, there is shown a neural network processing apparatus of the type disclosed in the above referenced PCT Application No. PCT/EP2018/071046 (Ref: FN-618-PCT).

[0020] The apparatus includes a host CPU 50 comprising a bank of processors which can each independently control a number of programmable convolutional neural network

[0021] (PCNN) clusters 92 through a common internal Advanced High-performance Bus (AHB), with an interrupt request (IRQ) interface used for signalling from the PCNN cluster 92 back to the host CPU 50, typically to indicate completion of processing, so that the host CPU 50 can coordinate the configuration and operation of the PCNN clusters 92.

[0022] Each PCNN cluster 92 includes its own CPU 200 which communicates with the host CPU 50 and in this case 4 independently programmable CNNs 30-A . . . 30-D of the type disclosed in PCT Application WO 2017/129325 (Ref: FN-481-PCT), the disclosure of which is incorporated herein by reference. Note that within the PCNN cluster 92, the individual CNNs 30 do not have to be the same and for example, one or more individual CNNs might have different characteristics than the others. So for example, one CNN may allow a higher number of channels to be combined in a convolution than others and this information would be employed when configuring the PCNN accordingly. In the embodiment, each individual CNN 30-A . . . 30-D, as well as accessing either system memory 99 or 102 across system bus 91 can use a shared memory 40 through which information can be shared with other clusters 92. Thus, the host CPU 50 in conjunction with the cluster CPU 200 and a memory controller 210 arrange for the transfer of initial image information as well as network configuration information from either the memory 99 or 102 into the shared memory 40. In order to facilitate such transfer, each host CPU 50 can incorporate some cache memory 52.

[0023] An external interface block 95A with one or more serial peripheral interfaces (SPIs) enables the host processors 50 to connect to other processors within a vehicle network (not shown) and indeed a wider network environment. Communications between such host processors 50 and external processors can be provided either through the SPIs or through a general purpose input/output (GPIO) interface, possibly a parallel interface, also provided within the block 95A.

[0024] In the embodiment, the external interface block 95A also provides a direct connection to various image sensors including: a conventional camera (VIS sensor), a NIR sensitive camera, and a thermal imaging camera for acquiring images from the vehicle environment.

[0025] In the embodiment, a dedicated image signal processor (ISP) core 95B includes a pair of pipelines ISP0, ISP1. A local tone mapping (LTM) component within the core 95B can perform basic pre-processing on received images including for example: re-sampling the images; generating HDR (high dynamic range) images from combinations of successive images acquired from the image acquisition devices; generating histogram information for acquired imagessee PCT Application No. PCT/EP2017/062188 (Ref: FN-398-PCT2) for information on producing histogram of gradients; and/or producing any other image feature maps which might be used by PCNN clusters 92 during image processing, for example, Integral Image mapssee PCT Application WO2017/032468 (Ref: FN-469-PCT) for details of such maps. The processed images/feature maps can then be written to shared memory 40 where they are either immediately or eventually available for subsequent processing by the PCNN clusters 92 as well as or alternatively, providing received pre-processed image information to a further distortion correction core 95C for further processing or writing the pre-processed image information to memory 99 or 102 possibly for access by external processors.

[0026] The distortion correction core 95C includes functionality such as described in US Pat. No. 9,280,810 (Ref: FN-384-CIP) for flattening distorted images for example those acquired by wide field of view (WFOV) cameras, such as in-cabin cameras. The core 95C can operate either by reading image information temporarily stored within the core 95B tile-by-tile as described in U.S. Pat. No. 9,280,810 (Ref: FN-384-CIP) or alternatively, distortion correction can be performed while scanning raster image information provided by the core 95B. Again, the core 95C includes an LTM component so that the processing described in relation to the core 95B can also be performed if required on distortion corrected images.

[0027] Also note that in common with the PCNN clusters 92, each of the cores 95B and 95C has access to non-volatile storage 102 and memory 99 via a respective arbiter 220 and controller 93-A, 97 and volatile memory 40 through respective SRAM controllers 210.

[0028] Embodiments of the present invention can be implemented on systems such as shown in FIG. 2. In this case, each PCNN cluster 92 or CNN 30 can swap between operating in independent mode, FIG. 3, and redundant mode, FIG. 4.

[0029] Embodiments are based on a cluster 92 or an individual CNN 30 being provided with both input images or map(s), as well as the network configuration required to be executed by the CNN i.e. the definitions for each layer of a network and the weights to be employed within the various layers of the network, each time it is to process one or more input images/maps.

[0030] Normally, as shown in FIG. 3, a first PCNN_0 92/30 might be required to process one or more input maps_0 through a network defined by program_0 and using weights_0 to produce one or more output maps_0. Note that the output maps_0 can comprise output maps from intermediate layers of the network defined by program_0 or they can include one or more final classifications generated by output nodes of the network defined by program_0. Similarly, a second PCNN_1 92/30 might be required to process one or more input maps_1 through a network defined by program_1 and using weights_1 to produce one or more output maps_1. (Note that input maps_0 and input maps_1 can be the same or different images/maps or overlapping sets of images/maps.)

[0031] As will be appreciated, it may be desirable or necessary to execute different networks at different times and different frequencies. So for example, in a vehicle with one or more front facing cameras, a PCNN cluster 92 or CNN 30 dedicated to identifying pedestrians within the field of view of the camera may be executing at upwards of 30 frames per second, whereas in a vehicle with a driver facing camera, a PCNN cluster 92 or CNN 30 dedicated to identifying driver facial expressions may be executing at well below 30 frames per second. Similarly, some networks may be deeper or more extensive than others and so may involve different processing times even if executed at the same frequency.

[0032] Thus, it should be apparent that there will be periods of time when one or more of the multiple PCNN clusters 92 or individual CNNs 30 in a multi-processor neural network processing apparatus such as shown in FIG. 2 will be idle.

[0033] Embodiments of the present invention are based on at least some of such PCNN clusters 92 or CNNs 30 either under the control of the host CPU 50, independently or via their respective cluster CPUs 200 being able to identify program commands and data for other target PCNNs either in memory 99 or being passed across the system bus 91 (as well as possibly the AHB bus).

[0034] In these cases, as illustrated in FIG. 4, a PCNN cluster 92 or individual CNNs 30 can switch to operate in a redundant mode, where using the input maps, configuration information/program definition and weights for another target PCNN cluster 92 or CNN 30, they replicate the processing of the target PCNN cluster/CNN.

[0035] Each such PCNN cluster 92 or CNN 30 when operating in redundant mode can continue to execute the program for the target PCNN until either processing is completed or the PCNN cluster/CNN receives a command from a host CPU 50 requesting that it execute its own network program in independent mode.

[0036] The results of processing, to the extent this has occurred before completion or interruption, can then be compared by either: the cluster CPU 200 in the redundant PCNN cluster 92; the redundant CNN 30; the cluster CPU 200 in the target PCNN cluster 92 or CNN 30, if they know that their operation is being shadowed by a redundant PCNN cluster/CNN; or by a host CPU 50, as indicated by the decision box 400.

[0037] Using a CPU 200 common to a number of individual CNNs to conduct a redundancy check of either a PCNN cluster 92 or individual CNN 30 removes the burden from the host CPU 50 of identifying opportunities for conducting testing, but also lessens the amount of logic to be implemented vis-a-vis providing such functionality within each individual CNN 30. Similarly, as each CPU 200 in any case provides access for each individual CNN 30 to the system bus 91, it can readily act on their behalf to identify opportunities for conducting testing of other PCNN clusters 92 or individual CNNs 30.

[0038] The redundancy check functionality 400 can be implemented in a number of ways. It will be appreciated that during the course of processing a neural network, each layer in a succession of layers will produce one or more output maps. Typically, convolutional and pooling layers produce 2-dimensional output feature maps, whereas fully connected or similar classification layers produce 1-dimensional feature vectors. Typically, the size of output map decreases as network processing progresses until for example, a relatively small number of final classification values might be produced by an final network output layer. Nonetheless, it will be appreciated that other networks for example generative networks or those performing semantic segmentation may in fact produce large output maps. Thus, in particular, if a target PCNN cluster 92 or CNN 30 writes any such output map back to memory 99 during processing, this can be compared with the corresponding map produced by a redundant PCNN cluster 92 or CNN 30 to determine if there is a difference. For very large output maps, rather than a pixel-by-pixel comparison, a hash, CRC (cyclic redundancy check) or signature can be generated for an output map and these can be compared.

[0039] In any case, if the output maps or values derived from such output maps match, then it can be assumed that both the target PCNN cluster 92 or CNN 30 and the redundant PCNN cluster 92 or CNN 30 are functioning. If not, then at least one of the target or redundant PCNN clusters or CNNs can be flagged as being potentially faulty. Such a potentially faulty PCNN cluster or CNN can subsequently be set to run only in redundant mode until it has an opportunity to be checked against another target PCNN cluster or CNN. If one of the potentially faulty PCNN cluster or CNN checks out against another target PCNN cluster or CNN and the other does not, then that other PCNN cluster or CNN can be designated as faulty and disabled permanently. (The remaining potentially faulty PCNN cluster or CNN may need to run successfully in redundant mode a given number of times before it is undesignated as potentially faulty.)

[0040] It will be appreciated from the above description that multiple CNNs 30, whether within a single PCNN cluster 92 or spread across a number of PCNN clusters 92 especially lend themselves to this opportunistic testing because it is not essential that such CNNs complete the processing of an entire network for a fault analysis to be made. Indeed, it can be the case, that typically larger output maps from processing of earlier layers of a network can provide a more extensive test result of functionality within a PCNN cluster or individual CNN than what might be a single final classification from a network. On the other hand, writing too much of such intermediate layer information back across a system bus 91 to memory 99 rather than maintaining such information in a local cache only may unduly consume system resources. As such, network program configuration can be balanced between consuming only a minimum of system resources and providing sufficient intermediate layer output information that redundancy checking can be performed without a redundant PCNN cluster 92 or CNN 30 necessarily completing processing of a network during its otherwise idle time.

[0041] Similarly, it will be seen that in a multi-processor neural network processing apparatus such as shown in FIG. 2, the availability of a number of duplicate clusters 92 and cores 30 enables some of these to be shut down or not relied upon when they are determined to be faulty or potentially faulty and for the apparatus to continue processing, albeit with less opportunity for opportunistic testing. As such, the system might be programmed to warn a user that a fault had been detected and for example, limit system functionality (speed, range or, for example, autonomous driving level) until the fault is repaired.

[0042] It will be appreciated that in certain systems, the tasks performed by each CNN 30 can be deterministically scheduled and so the host CPU 50 or the cluster CPUs may know a priori when they are to operate in redundant mode and accordingly when and where to expect configuration information for a target PCNN cluster or CNN to appear in system memory 99. Other systems may operate more asynchronously with the host CPU 50 allocating PCNN clusters 92 and/or CNNs to perform tasks on demand. In either case, it will be appreciated that PCNN clusters 92 or CNNs 30 can be configured to identify opportunities to operate in redundant mode so that the functionality of another PCNN cluster 92 or CNN 30 can be tested.

[0043] It will also be appreciated that in some embodiments, all of the PCNN clusters 92 or CNNs 30 could be configured to opportunistically test any other of the PCNN clusters 92 or CNNs 30, whereas in other embodiments, there may be a limited number or even a designated PCNN cluster 92 or CNN 30 which is configured with the ability to switch into redundant mode. This is course has the advantage of providing some spare computing capacity in the event that any given PCNN cluster 92 or CNN 30 is identified as being faulty and still allow the system to perform at fully capacity.

[0044] It should also be noted that there may be specific times when it can be beneficial to test the functionality of PCNN clusters 92 or CNNs 30, for example, when a vehicle is static and perhaps less demand is being made of the processing apparatus or perhaps not in very dark or low contrast conditions when image information being processed may be less useful for testing. In any case, it is not essential that testing would run continuously or at rigid intervals.