Partitionable Networked Computer

Abstract

A computer comprising a plurality of processing nodes is provided. Each processing node has at least one processor configured to process input data to generate an array of data items. The processing nodes are arranged in cliques in which each processing node of a clique is connected to each other processing node in the clique by first and second clique links. The cliques are inter-connected in rings such that each processing node is a member of a single clique and a single ring. The processing nodes of all cliques are configured to exchange in each exchange step of a machine learning collective via the respective first and second clique links at least two data items with the other processing node(s) in its clique, and all processing nodes are configured to reduce each received data item with the data item in the corresponding position in the array on that processing node.

Claims

1. A computer comprising a plurality of processing nodes, each processing nodes having at least one processor configured to process input data to generate output data in the form of an array of data items; the plurality of processing nodes arranged in cliques in which each processing node of a clique is connected to each other processing node in the clique by first and second clique links, the cliques being inter-connected in rings such that each processing node is a member of a single clique and a single ring, the processing nodes being configured to exchange data items in respective exchange steps of a machine learning collective, wherein the processing nodes of all cliques are configured to exchange in each exchange step via the respective first and second clique links at least two data items with the other processing node(s) in its clique, and all processing nodes are configured to reduce each received data item with the data item in the corresponding position in the array on that processing node.

2. The computer according to claim 1, wherein the machine learning collective is an Allreduce collective and each processing node is configured to exchange data items in exchange steps of an Allgather phase, following a reduce scatter phase of the Allreduce collective, wherein in each step of the Allgather phase reduced data items are exchanged between processing nodes in a clique.

3. The computer according to claim 1, wherein each processing node comprises memory configured to store an array of data items ready to be exchanged in the reduce scatter phase, wherein each data item is respectively positioned in the array with corresponding data items being respectively positioned at corresponding locations in the arrays of other processing nodes.

4. The computer according to claim 1, wherein the processing nodes are each configured to transmit data items in a forwards direction to its adjacent processing node in the ring in at least some of the exchange steps in the reduce-scatter phase.

5. The computer according to claim 4, wherein the processing nodes are configured to transmit data items to their forwards adjacent processing node in the ring for all exchange steps of the reduce scatter phase apart from a first step, in which no data items are transmitted between processing nodes connected in a ring.

6. The computer according to claim 1, wherein the array at each processing node comprises two sub arrays and processing nodes are inter-connected by bi-directional links, wherein in each exchange step of the reduce scatter phase, all processing nodes are configured to exchange with the other processing node(s) of their clique, two data items from one sub array and two further data items from the other sub array wherein the two data items and the further two data items are exchanged over the same bi-directional link in opposite directions.

7. The computer according to claim 6, wherein the processing nodes are each configured to transmit data items in a forwards direction to its adjacent processing node in the ring in at least some of the exchange steps in the reduce-scatter phase and wherein in at least some exchange steps of the reduce-scatter phase each processing node is configured to transmit data items to its adjacent backwards processing node in the ring, wherein the transmission in each of the forwards and backwards direction from each processing node is carried out on the same bi-directional link.

8. The computer according to claim 3, wherein each array represents at least part of a vector of partial deltas, each partial delta representing an adjustment to a value stored at each processing node.

9. The computer according to claim 8, wherein each processing node is configured to generate the vector of partial deltas in a compute step.

10. The computer according to claim 9, wherein each processing node is configured to divide the vector into two sub arrays for separate exchange and reduction in the reduce-scatter phase.

11. The computer according to claim 9, wherein each processing node is configured to generate the vector of partial deltas by carrying out a compute function on a set of values and a batch of incoming deltas, the partial deltas being the output of the compute function.

12. The computer according to claim 11, which is configured to implement a machine learning model wherein the incoming batch data is training data, and the values are weights of the machine learning model.

13. A method of operating a computer comprising a plurality of processing nodes, each processing node having at least one processor configured to process input data to generate output data in the form of an array of data items, the plurality of processing nodes arranged in cliques in which each processing node of a clique is connected to each other processing node in the clique by first and second clique links, the cliques being interconnected in rings such that each processing node is a member of a single clique and a single ring, the method comprising exchanging data item in respect of exchange steps of a first phase of a machine learning collective, wherein in each exchange step the processing nodes of all cliques exchange via the respective first and second clique links at least two data items with the other processing nodes in its clique, and all processing nodes reduce each received data item with the data item in the corresponding position in the array on that processing node.

14. The method according to claim 13, wherein the machine learning collective is an Allreduce collective and each processing node exchanges data items in exchange steps of an Allgather phase, following a reduce scatter phase of the Allreduce collective, wherein in each step of the Allgather phase reduced data items are exchanged between processing nodes in a clique.

15. The method according to claim 13, wherein each processing node comprises memory configured to store an array of data items ready to be exchanged in the reduce scatter phase, wherein each data item is respectively positioned in the array with corresponding data items being respectively positioned at corresponding locations in the arrays of other processing nodes.

16. The method according to claim 13, wherein each processing node transmits data items in a forwards direction to its adjacent processing node in the ring in at least some of the exchange steps in the reduce-scatter phase.

17. The method according to claim 16, wherein each processing node transmits data items to their forwards adjacent processing node in the ring for all exchange steps of the reduce scatter phase apart from a first step, in which no data items are transmitted between processing nodes connected in a ring.

18. The method according to claim 13, wherein the array at each processing node comprises two sub arrays and processing nodes are inter-connected by bi-directional links, wherein in each exchange step of the reduce scatter phase, all processing nodes exchange with the other processing node(s) of their clique, two data items from one sub array and two further data items from the other sub array wherein the two data items and the further two data items are exchanged over the same bi-directional link in opposite directions.

19. The method according to claim 18, wherein each processing node transmits data items in a forwards direction to its adjacent processing node in the ring in at least some of the exchange steps in the reduce-scatter phase and wherein in at least some exchange steps of the reduce-scatter phase each processing node transmits data items to its adjacent backwards processing node in the ring, wherein the transmission in each of the forwards and backwards direction from each processing node is carried out on the same bi-directional link.

20. The method according to claim 15, wherein each array represents at least part of a vector of partial deltas, each partial delta representing an adjustment to a value stored at each processing node.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0054] For a better understanding of the present invention to show how the same may be carried into effect, reference will now be made by way of example to the accompanying drawings.

[0055] FIG. 1 is a schematic diagram illustrating distributed training in a neural net.

[0056] FIG. 1A is a schematic diagram showing a line of processing nodes for implementing a simple streaming line Allreduce algorithm.

[0057] FIG. 1B is a timing diagram of a streaming line Allreduce algorithm.

[0058] FIG. 1C shows a one-dimensional ring.

[0059] FIG. 1D illustrates the corresponding timing diagram for the forward and backward links.

[0060] FIG. 2 is a schematic diagram illustrating implementation of an Allreduce function by a reduce-scatter step followed by Allgather step.

[0061] FIG. 3A shows a computer with processing nodes connected in a 44 torus configuration.

[0062] FIG. 3B shows an asymmetrical twisted torus with 48 processing nodes.

[0063] FIG. 4 shows a configurable network in which rings are maintained in allocated portions.

[0064] FIG. 4A is a schematic diagram of a computer having processing nodes connected in a cushion topology.

[0065] FIG. 4B shows how the rings are connected in the cushion topology of FIG. 4A.

[0066] FIGS. 5A and 5B illustrate a bucket based algorithm for Allreduce for use on the cushion topology.

DETAILED DESCRIPTION

[0067] Aspects of the present invention have been developed in the context of a multi-tile processor which is designed to act as an accelerator for machine learning workloads. The accelerator comprises a plurality of interconnected processing nodes. Each processing node may be a single multi-tile chip, a package of multiple chips, or a rack of multiple packages. The aim herein is to devise a machine which is highly efficient at deterministic (repeatable) computation. Processing nodes are interconnected in a manner which enable collectives, especially broadcast and Allreduce, to be efficiently implemented.

[0068] One particular application is to update models when training a neural network using distributed processing. In this context, distributed processing utilises multiple processing nodes which are in different physical entities, such as chips or packages or racks. That is transmitting data between the processing nodes requires messages to be exchanged over physical links.

[0069] The challenges in developing a topology dedicated to machine learning differ from those in the general field of high performance computing (HPC) networks. HPC networks usually emphasise on demand asynchronous all-to-all personalised communication, so dynamic routing and bandwidth over provisioning are normal. Excess bandwidth may be provisioned in a HPC network with the aim of reducing latency rather than to provide bandwidth. Over provisioning of active communication links waste power which could contribute to compute performance. A commonly used link today draws power when activated whether or not it is being used to transmit data.

[0070] Torus computer networks have been discussed above as a machine topology which is particularly adapted to super computing machine workloads, such as ML workloads.

[0071] In ML workloads, inter chip communication is currently dominated by broadcast and Allreduce collectives. The broadcast collective can be implemented by a scatter collective followed by an Allgather collective, and the Allreduce collective can be implemented by a reduce-scatter collective followed by an Allgather collective. In this context, the term inter-chip denotes any communication between processing nodes which are connected via external communication links. As mentioned, these processing nodes may be chips, packages or racks.

[0072] One particularly useful algorithm which has been developed to process the Allreduce collective in machine learning algorithms has been described above. This algorithm uses rings in a torus structure to efficiently exchange and process fragments of partial vectors. One challenge which arises however is that such torus computer networks are not easy to partition. One objective of the novel computer network discussed herein is to enable a large network of many processing nodes to be partitioned into one or more smaller networks without any or any significant hardware involvement in the partitioning. To achieve the above objectives in ML applications, rings in two dimensions need to be preserved in the allocated portions. With present torus computer networks this is not possible, as any partitioning of the torus computer network would involve breaking one or more of the rings in the torus structure.

[0073] One reason for partitioning very large networks is to enable one or more tenant to use the network. A tenant may be described as a computer program distributed across processing nodes of the partition, the program being self-contained and controlled by a particular owner. Another requirement of partition networks is that there is no leakage between partitions. That is, it is highly undesirable for an owner of one tenant to be exposed to code or messages from another tenant of the partition. FIG. 4 is a schematic block diagram of a computer network topology which satisfies these criteria and which is referred to herein as a cushion.

[0074] FIG. 4 shows a topology which comprises a front layer 430 and a rear layer 432. It will be appreciated that the denotations front and rear do not imply any particular physical orientation when the network is in use. There are used to define the relationship between processing nodes in the individual layers.

[0075] The front layer comprises an nn array or grid of processing nodes which in the embodiment of FIG. 4 is a 66 array. The processing nodes N.sub.F denote the corner nodes of the front layer. The rear layer similarly comprises a 6 x 6 array of processing nodes, with the processing nodes N.sub.R denoting the corner nodes of the rear layer. Each layer has edgesthe vertical edges are defined by the processing nodes extending vertically between the corner nodes N.sub.F or N.sub.R respectively, and the horizontal edges are defined by the processing nodes extending horizontally between the corner nodes N.sub.F/N.sub.R respectively. Each layer also has a set of central nodes, forming a 44 array within the edge nodes. Reference numeral 400 denotes a typical central node in the front layer. All central nodes have the same connective characteristics. Each central node has six activatable links. Each link represents a physical connection between its central the node 400 and its adjacent nodes in the topology both within the array and between the layers. Links L1 and L4 connect the processing node 400 to its respective vertically adjacent nodes 402, 404 within the front layer. Links L2 and L3 connect the node 400 to its respective adjacent horizontal nodes 406, 408 within the front layer. Links L5 and L6 connect the node 400 to its corresponding facing node 410 in the rear layer. Each link, when activated, is a bi-directional link, for example a SERDES link as discussed further herein. Such links draw power whether or not they are transmitting data. In this context, an activated link is a link which is powered up such as to enable the transmission of data items over it. In a SERDES link, data is transmitted by changing a differential voltage on the link with respect to a base, powered up voltage. The amount of power consumed by the link is independent of the traffic on the link. Conversely, an inactive link (that is a link which is not activated) is not capable of transmitting data items because it has not been powered up to the base voltage level. Thus, an inactive link not only saves power, but also prevents unwanted access to data items from another node which is physically connected to it. This characteristic is important when considering the partitioning of the network.

[0076] If the 66 array were to be used as a single computer, the links between the front and rear faces on the central nodes would be rendered inactive. Furthermore, only one of the link between the facing layers would be activated at the edge nodes (see for example nodes 404 and its facing node 414). However, both of the link between the facing corner nodes PNF/PNR would be activated. That is, for each node in an active computer topology four of the six links would be active, and the remaining two would be inactive. It should be evident from this description that in the case of the central node 400 the inactive links are LF5 and LF6, and the active links are L1 through L4.

[0077] As a result of this connectivity, the word cushion has been coined to describe the computer network in its activated state. This is because it is joined only at its edges and not between its facing layers.

[0078] The activated computer network in this form provides two dimensions of n rings of length 2 n (where n=6 in the maximum configuration of FIG. 4).

[0079] This provides for an optimally-efficient implementation of an AllReduce algorithm as discussed earlier herein. Reference will now be made to FIGS. 4A and 4B which show the use of the cushion architecture when implementing a two dimensional AllReduce algorithm. FIG. 4A shows a three dimensional view of a 3 x 3 cushion connected network. There are three rings in the X plane and three rings in the Y plane. For the sake of clarity, not all nodes are labelled, but a ring in the horizontal plane and the ring in the vertical plane are fully defined. That is, the top ring in the horizontal plane comprises nodes Na1, Nb1 and Nc1 which are nodes in the front face of the cushion, and Na1, Nb1 and Nc1 which are nodes in the rear face of the cushion. As the links are bi-directional, there are two rings in different directions, one direction (starting at Na1): Na1, Nb1, Nc1, Nc1, Nb1, Na1. In the other direction, starting again at Na1, Na1, Na1, Nb1, Nc1, Nc1, Nb1.

[0080] In the vertical (Y) direction, the extreme most left hand ring is formed by nodes Na1, Na2, Na3 (in the front face) and Na3, Na2 and Na1 (in the rear face). Once again, the ring can operate in two directions due to the bi-directional links.

[0081] FIG. 4B shows the constructions of FIG. 4A laid out so that the configuration of the rings can be more clearly seen. The nodes in FIG. 4B are numbered correspondingly to the nodes in FIG. 4A. According to one embodiment, the AllReduce algorithm can be implemented by a one-dimensional algorithms as described below with reference to FIGS. 5A and 5B, operated in two dimensions in a manner similar to that discussed in the Jain paper referred to earlier. Note that, however, in contrast the application of the two-dimensional algorithm to an asymmetric toroid as discussed in Jain, the cushion configuration herein is fully bandwidth efficient. Moreover, there is a significant additional advantage with the cushion topology described herein, in that it lends itself readily to partitioning into smaller portions (cushions).

[0082] The nodes with a dark outline (the 4x4 group of nodes in the bottom left-hand corner of the array) represent a partitioned cushion that is an allocated set of nodes from the main set. The corner nodes of the front layer are labelled 420 and the corner nodes of the rear layer are labelled 422. The four active links for each node are shown in black or dark grey illustrating whether they form part of a vertical or horizontal link respectively. The facing corner nodes are each connected by two links, a black and a dark grey because these form the terminations of both horizontal and vertical rings. The remaining edge nodes on the horizontal edge connect the facing layers only through a single activated link (dark grey) which forms the termination of vertical rings only. Conversely, the edge nodes on the vertical edges are connected only through one black link which represents the termination of horizontal rings.

[0083] The links between corresponding facing central nodes are de-activated (light grey). Furthermore, the links of nodes outside the partitioned cushion (for example, the vertical links from nodes 420 and 422 are de-activated). Note, however, that the de-activation of these links would not prevent the forming of a second cushion comprising the nodes in the upper two rows of the array.

[0084] Partitioning of the array into one or more partitioned cushions can be carried out in any suitable way. FIG. 4 illustrates schematically an allocation engine 412 which carries out the partitioning by activating or de-activating the links accordingly. It will readily be appreciated that this allocation engine may be implemented by distributing appropriate code across processing nodes themselves, or by an external manager capable of issuing instructions to the network for the activation of links on each processing node. For example, each processing node could be provided with a control plane and a data plane. The data plane is responsible for compute on the data (such as the reductions described herein, and the generation of partials). The control plane may be responsible for exchange code for managing the transmission of data items as described herein. A host may be responsible for allocating code to the processing nodes in the computer. This is the case when the computer is being used as an accelerator, for which it is particularly suitable when implementing machine learning and other high-processing requirement applications. A host can load computer code into each of the processing nodes in a boot stage, and this code can include connectivity code which controls the connectivity to allocate the operating processing nodes in a particular cushion. For example, this could be done by allocating each node to a particular index (for example, an XY coordinate) and defining the connectivity of that processing node based on the index.

[0085] A particular advantage of being able to partition the network into separate cushions is that each cushion has multiple rings, which can be used to efficiently implement a ring Allreduce collective.

[0086] The Allreduce collective has been described above and is illustrated in FIG. 2. FIG. 2 shows a set (vector) of partial values or partial P.sub.0, P.sub.1, P.sub.2, P.sub.3 on each of four nodes in a starting state S1. In this context a node is a processing node in a network of processing nodes. Note that each node N.sub.0, N.sub.1, N.sub.2, N.sub.3 has four corresponding partials, which are hatched accordinglylarge diamond grid, wide downward diagonal stripe, large square grid, wide upward diagonal stripe. That is, each partial has a position in a vector such that P0(n) has the same position in its vector on node n as P0 (n+1) in its vector on node n+1. The suffix (n) is used to denote the node in which the partial residesthus P0(0) is the partial P0 on node N0. In a reduce-scatter pass, corresponding partials are reduced and the reduction provided to one of the nodes. For example, partials P0(0), P0(1), P0(2), P0(3) are reduced (to r.sub.0) and placed onto node N0. Similarly, partials P1(0), P1(1), P1(2) and P1(3) are reduced (to r.sub.1) and placed onto node N1. And so forth so that in an intermediate state S2, each node has one of the reductions r.sub.0, r.sub.1, r.sub.2 and r.sub.3. As explained, the reduction may be by any combinational function f (Pi.sub.0.sup.3)which could include independent operators (e.g. max) or associative operators=P1 (N0)*P1(N1)*P1(N2) *P1(N3). Then, in an Allgather pass, each reduction is provided to all nodes to activate a state S3 wherein each node now holds all four reductions. Note that in S1, the corresponding partials, e.g. P0(0), P0(1), P0(2) and P0(3) may all differ whereas, in state S3, each reduction, e.g. r.sub.0 is the same at all nodes, where r.sub.1=f{(P.sub.i(0), P.sub.i(1), P.sub.i(2) and P.sub.i(3))}. In machine learning, the set of partials P0, P1, P2, P3 is a vector. A vector of partials (e.g. updated weights) is produced on each pass of the model during training. The reduction r.sub.0 (diamond grid), r.sub.1 (downward diagonal stripe), r.sub.2 (square grid), r.sub.3 (upward diagonal stripe) on each node in state S3 is the full reduction vector. In the context of machine learning, each partial could be a set of updating deltas for a parameter in the model. Alternatively (in an arrangement not described further herein) it could be an updated parameter itself.

[0087] FIG. 5A and 5B illustrate a bucket based algorithm for reduce-scatter/Allgather that assumes six virtual 1D rings. These are termed logical rings herein. FIG. 5A is a schematic diagram illustrating the reduction of partials in multiple virtual rings. Each partial is split into six fragments. In FIG. 5A, the capital letters R, Y, G, B, P, L each denote a different fragment of a partial stored at each node indicated by hatching diamond grid, upward diagonal stripe, square grid, horizontal stripe, downward diagonal stripe, vertical stripe. The letters denote corresponding fragments which are to be reduced with each other, and define the virtual or logical ring for those fragments. Looking at FIG. 5A, the R fragments in each of the partials P0, P1, P2, P3 and P4 are reduced into a single fragment in the result vector (R Similarly for the Y, G, B, P and L fragments.

[0088] FIG. 5B shows a timing diagram with time on the horizontal axis indicating the data exchanges and computations in each step of the Allreduce process. In FIGS. 5A and 5B, the Allreduce process is accomplished by a reduce-scatter phase followed by an Allgather phase. In FIG. 5B each of the fragments are denoted by different hatchings as described above. The notation in FIGS. 5A and 5B is as follows. The partials are each denoted P0, P1, P2, P3, P4, P5. At the start of the process, each partial is stored on a respective node N0, N1, N2, N3, N4, N5. Each fragment is labelled according to its fragment ordinant and its position in the virtual ring in which it is deemed to be reduced. For example, RA0 denotes the R fragment in partial P0, because this is the first fragment in a virtual ring formed by nodes N0-N1-N2-N3-N4-N0. RA1 denotes the R fragment at node N1, which is in the second position in its virtual ring. YA0 denotes the Y fragment at node N1. The 0 suffix indicates it is the first fragment in its virtual ring, the Y-ring being N1-N2-N3-N4-N0-N1. Note in particular that the suffixes on A reflect the virtual rings, and do not correspond to the physical nodes (or the partials). Note that FIG. 5A shows only the virtual rings on the forward links. FIG. 5B shows that an equivalent process is occurring on the backward links, with the fragments denoted as B.

[0089] In step one, the first fragment (the A0) in each virtual ring is transferred from its node to the next adjacent node where it is reduced with the corresponding fragment at that node. That is, RA0 moves from N0 to N1 where it is reduced into R(A0+A1). Once again, the + sign is used here as a shorthand for any combinatorial function. Note that in the same step the A 0 fragments of each virtual ring will simultaneously be being transmitted. That is, the link between N1 and N2 is used to transmit YA0, the link between N2 and N3 is used to transmit GA0, et cetera. In the next step, the corresponding reduced fragments are transmitted over the forward links to their next adjacent node. For example, R(A0+A1) is transmitted from N1 to N2, and Y(A0+A1) is transmitted from N2 to N3. Note that for reasons of clarity not all fragments are numbered, nor are all transmissions numbered in FIG. 5A. The full set of fragments and numbers are shown in FIG. 5B. This process carries on for five steps. After five steps, there is a reduction of all fragments on each node. At the end of the fifth step, this reduction is on the last node of each corresponding ring for that fragment. For example the R reduction is on node N5.

[0090] The beginning of the Allgather phase starts by a transmission from the last to the first node in each virtual ring. Thus, the final reduction for the R fragments ends on node N5 ready for the first step of the Allgather phase. The final reduction of the Y fragments correspondingly ends up on the node N0. In the next step of the Allgather phase, the reduced fragments are transmitted again to their next adjacent node. Thus the fully reduced R fragment is now also at N2, the fully reduced Y fragment is now also at N3 and so on. In this way, each node ends up at the end of the Allgather phase with all fully reduced fragments R, Y, G, B, P, L of the partial.

[0091] Implementation of the algorithm can be achieved if the computation required for the reduction can be concealed behind the pipeline latency. Note that in forming suitable rings in a computer for implementation of Allreduce, a tour of the ring must visit each node in the ring only once. The algorithm described above uses six pipelined logical rings in a physical one-dimensional ring. The principle can be extended to two dimensions in the cushion. For full bandwidth utilisation, not only are rings used in a horizontal direction, but also in a vertical direction. Consider for example the node N.sub.a1. The partial at node N.sub.a1 is split into two halves. Note for the purpose of this discussion that the effect of bi-directionality in the links is ignored for now. In fact, everything that is described herein is mirrored in the opposite direction. Reverting to node N.sub.a1 the partial to be transmitted from this node is split into two halves. A first half of the partial is reduce-scattered (according to the one-dimensional line algorithm described above) around the ring in the X direction, and the other half is reduce-scattered around the ring in the Y direction. The same thing is happening at each node in the cushion. Note that (according to the one-dimensional algorithm described above), in fact, what is transmitted in each step on the X ring or Y ring respectively is a corresponding fragment for that step of the logical ring. When the reduce-scatter operations in the X and Y rings have been completed, the algorithm reverts back to the first node in each ring (consider node N.sub.a1) and swaps over the partials. That is, it now reduce-scatters the first set of the reductions resulting from the first pass round the X ring in the Y direction, and conversely reduce-scatters the first pass of reductions from the Y ring around the X ring.

[0092] After this phase, the fully reduced fragments are subject to an Allgather step, according to the one-dimensional ring algorithm described above. The first sets of the fully reduced fragments are Allgathered round the Y rings, and the second set of the fully reduced fragments are Allgathered around the X rings. Then, once again, the partially gathered sets are swapped and an Allgather process takes a first set around the X rings, and the second set around the Y rings.

[0093] The final result is a fully reduced complete vector at each node. As the number of nodes in the rings in both the X and Y directions are the same, the process has full bandwidth utilisation in all phases (that is all links are operating).

[0094] Each node is capable of implementing a processing or compute function. Each node could be implemented as a single processor. It is more likely, however, that each node will be implemented as a single chip or package of chips, wherein each chip comprises multiple processors. There are many possible different manifestations of each individual node. In one example, a node may be constituted by an intelligence processing unit of the type described in British applications with publication numbers GB2569843; GB2569430; GB2569275; the contents of which are herein incorporated by reference. However, the techniques described herein may be used on any type of processor constituting the nodes. What is outlined herein is a computer topology which can be partitioned in an efficient manner to maintain rings which are useful in executing collectives in machine learning models. Furthermore, the links could be manifest in any suitable way, subject only to the criteria that they are bi-directional. As mentioned, one particular category of communication link is a SERDES link which has a power requirement which is independent of the amount of data that is carried over the link, or the time spent carrying that data. SERDES is an acronym for Serializer/DeSerializer and such links are known. In order to transmit a signal on a wire of such links, power is required to be applied to the wire to change the voltage in order to generate the signal. A SERDES link has the characteristic that power is continually applied to the wire to maintain it at a certain voltage level, such that signals may be conveyed by a variation in that voltage level (rather than by a variation between 0 and an applied voltage level). Thus, there is a fixed power for a bandwidth capacity on a SERDES link whether it is used or not. A SERDES link is implemented at each end by circuitry which connects a link layer device to a physical link such as copper wires. This circuitry is sometimes referred to as PHY (physical layer). PCIe (Peripheral Component Interconnect Express) is an interface standard for connecting high speed computers.

[0095] Deactivation of the links in non-allocated partitions can save power. While in theory the links could be deactivated to consume effectively no power in an allocated cushion, in practice the activation time and non-deterministic nature of machine learning applications can render dynamic activation during program execution as problematic. As a consequence, the present inventor has determined that it is better to make use of the fact that the chip to chip link power consumption is essentially constant for any particular configuration, and that therefore the best optimisation is to maximise the use of the physical links by maintaining chip to chip traffic within a cushion concurrent with IPU activity as far as is possible.

[0096] SERDES PHYs are full duplex (that is a 16 Gbit per second PHY supports 16 Gbits per second in each direction simultaneously), so full link bandwidth utilisation implies balanced bi-directional traffic. Moreover, note that there is significant advantage in using direct chip to chip communication as compared with indirect communication such as via switches. Direct chip to chip communication is much more power efficient than switched communication.

[0097] Another factor to be taken into consideration is the bandwidth requirement between nodes. An aim is to have sufficient bandwidth to conceal inter node communication behind the computations carried out at each node for distributed machine learning.

[0098] The links are physical links provided by suitable buses or wires as mentioned above. In one manifestation, each processing node has a set of wires extending out of it for connecting it to another processing node. This may be done for example by one or more interface of each processing node having one or more port to which one or more physical wire is connected.

[0099] In another manifestation, the links may be constituted by on-board wires. For example, a single board may support a group of chips, for example four chips. Each chip has an interface with ports connectable to the other chips. Connections may be formed between the chips by soldering wires onto the board according to a predetermined method. Note that the concepts and techniques described herein are particularly useful in that context, because they make maximise use of links which have been pre soldered between chips on a printed circuit board.

[0100] The concepts and techniques described herein are particularly useful because they enable optimum use to be made of non-switchable links. A configuration may be built by connecting up the processing nodes as described herein using the fixed non switchable links between the nodes.

[0101] In some embodiments, in order to use the configuration, a set of parallel programs are generated. The set of parallel programs contain node level programs, that is programs designated to work on particular processing nodes in a configuration. The set of parallel programs to operate on a particular configuration may be generated by a compiler. It is the responsibility of the compiler to generate node level programs which correctly define the links to be used for each data transmission step for certain data. These programs include one or more instruction for effecting data transmission in a data transmission stage which uses a link identifier to identify the link to be used for that transmission stage. For example, a processing node may have two or three active links at any one time (double that if the links are simultaneously bidirectional). The link identifier causes the correct link to be selected for the data items for that transmission stage. Note that each processing node may be agnostic of the actions of its neighbouring nodesthe exchange activity is pre compiled for each exchange stage.

[0102] Note also that links do not have to be switchedthere is no need for active routing of the data items at the time at which they are transmitted, or to change the connectivity of the links.

[0103] As mentioned above, the configurations of computer networks described herein are to enhance parallelism in computing. In this context, parallelism is achieved by loading node level programs into the processing nodes of the configuration which are intended to be executed in parallel, for example to train an artificial intelligence model in a distributed manner as discussed earlier. It will be readily be appreciated however that this is only one application of the parallelism enabled by the configurations described herein. One scheme for achieving parallelism is known as bulk synchronous parallel (BSP) computing. According to a BSP protocol, each processing node performs a compute phase and an exchange phase which follows the compute phase. During the compute phase, each processing nodes performs its computation tasks locally but does not exchange the results of its computations with the other processing nodes. In the exchange phase, each processing node is permitted to exchange the results of its computations from the preceding compute phase with the other processing nodes in the configuration. A new compute phase is not commenced until the exchange phase has been completed on the configuration. In this form of BSP protocol, a barrier synchronisation is placed at the juncture transitioning from the compute phase into the exchange phase, or transitioning from the exchange phase into the compute phase or both.

[0104] In the present embodiments, when the exchange phase is initiated, each processing node executes an instruction to exchange data with its adjacent nodes, using the link identifier established by the compiler for that exchange phase. The nature of the exchange phase can be established by using the MPI message passing standard discussed earlier. For example, a collective may be recalled from a library, such as the all reduced collective. In this way, the compiler has precompiled node level programs which control the links over which the partial vectors are transmitted (or respective fragments of the partial vectors are transmitted).

[0105] It will readily be apparent that other synchronisation protocols may be utilised.

[0106] While particular embodiments have been described, other applications and variants of the disclosed techniques may become apparent to a person skilled in the art once given the disclosure herein. The scope of the present disclosure is not limited by the described embodiments but only by the accompanying claims.