COHERENT NODE CONTROLLER

Abstract

A cache coherent node controller at least includes one or more network interface controllers, each network interface controller includes at least one network interface, and at least two coherent interfaces each configured for communication with a microprocessor. A computer system includes one or more of nodes wherein each node is connected to at least one network switch, each node at least includes a cache coherent node controller.

Claims

1.-12. (canceled)

13. A cache coherent node controller comprising: a. one or more network interface controllers, NIC, each network interface controller includes at least one network interface, c. at least two coherent interfaces, CIF, each configured for communication with a microprocessor, d. two or more interconnected protocol engines, PE, and e. at least two coherent link layers, each coherent link layer at least connected to one protocol engine, each coherent link layer provides coherent interfaces, wherein each network interface controller is connected with two coherent link layers, each coherent link layer is connected with one neighbouring coherent link layer, and where the protocol engines are directly or indirectly connected with coherent link layers.

14. A computer system comprising one or more nodes wherein each node is connected to at least one network switch, each node comprising: a. a cache coherent node controller comprising: i. one or more network interface controllers, each network interface controller includes at least one network interface, ii. at least two coherent interfaces each connected to a microprocessor, and iii. one or more interconnected protocol engines, PE.

15. The computer system according to claim 14, wherein the cache coherent node controller further comprises: iv. one or more coherent link layers, each coherent link layer connected to one protocol engine, each coherent link layer provides coherent interfaces.

16. The computer system according to claim 15, wherein each network interface controller is connected with a coherent link layer, each coherent link layers is connected with one neighbouring coherent link layer, and where the protocol engines are directly or indirectly connected with coherent link layers.

17. The computer system according to claim 15, wherein connection between each node and the network switches is provided by the network interfaces.

18. The system according to claim 15, wherein one network interface of each cache coherent node controller is connected to one network switch to form a 1-dimensional fat tree topology.

19. The system according to claim 15, wherein two network interfaces of each cache coherent node controller are connected to two network switches to form a 2-dimensional fat tree topology.

20. The system according to claim 15, wherein three network interfaces of each cache coherent node controller are connected to three network switches to form a 3-dimensional fat tree topology.

21. The system according to claim 15, wherein four network interfaces of each cache coherent node controller are connected to four network switches to form a 4-dimensional fat tree topology.

22. The system according to claim 15, wherein n network interfaces of each cache coherent node controller are connected to n network switches to form a n-dimensional fat tree topology.

23. The system according to claim 17, wherein the network switches are one of: an Omnipath switch, an InfiniBand switch, a Gen-Z switch or an Ethernet switch.

24. The system according to claim 17, wherein connection between each node and the network switches is provided by fibre optic links to the network switches.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0031] Following is a brief description of the drawings in order to make the invention more readily understandable, the discussion that follows will refer to the accompanying drawings, in which

[0032] FIG. 1 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces. Configurations require one or more cache coherent ports and one or more network interfaces,

[0033] FIG. 2 shows 8 cache coherent links to CPUs and 4 network interfaces (NICs) to connect to HPC switches with various number of ports. The 4 NICs allow configurations with up to 4 dimensional fat trees,

[0034] FIG. 3 shows CPU connections,

[0035] FIG. 4 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces and 4 connected CPUs. Configurations require one or more cache coherent ports and one or more network interfaces. FIGS. 5, 7, 8a and 8b, use a simplified ring depicting a node with two or more CPUs and two or more NIC channels,

[0036] FIG. 5 shows a 2-D fat tree system with 32 nodes,

[0037] FIG. 6 shows a 1-D fat tree, with 8-node configurations, where top: 8 nodes is associated with one 8-port node controller and bottom 8 nodes are associated with two 8-port node controllers for double bandwidth and redundancy,

[0038] FIG. 7 shows a 2-D fat tree system with 64 nodes,

[0039] FIG. 8a shows a 2D face of with 48 nodes of a 3-D fat tree system,

[0040] FIG. 8b shows a 3-D fat tree system with 686 nodes,

[0041] FIG. 9 shows a fat tree configuration in a rack, where a node includes 8 cache coherent, a rack contains 32 nodes, 864 port Switches, Message Passing Total: 256 CPU's, and

[0042] FIG. 10 shows a simplified drawing of an 8-port cache coherent node controller with 4 integrated network interfaces. Configurations require one or more cache coherent ports and one or more network interfaces. Further drawing use a simplified ring depicting a node with two or more CPUs and two or more NIC channels.

DETAILED DESCRIPTION OF THE INVENTION

[0043] In the following it is firstly disclosed general embodiments in accordance to the present invention, thereafter particular exemplary embodiments will be described. Where possible reference will be made to the accompanying drawings and where possible using reference numerals in the drawings. It shall be noted however that the drawings are exemplary embodiments only and other features and embodiments may well be within the scope of the invention as described.

[0044] In the context of the present invention a processor is defined as a module containing one or more CPU cores, one or more memory channels, one or more I/O channels, one or more coherent links and memory in the form caches or with or without main memory in any form included in the processor module.

[0045] A node contains one or more processor modules, one or more memory banks, zero or one or more I/O modules and one or more coherent node controllers.

[0046] In the present invention it is not necessarily different bandwidths within a fat tree configuration as indicated in the figures.

[0047] The present invention is based on integrating several functions that in combination will produce a flexible and versatile interconnect component for building large scale computer systems. The key of the invention is the close connection between a packet-based network switch and a cache coherent switch for inter-processor communication inside the node. This connection enables building of scalable systems with low latency and high bandwidth communication with very low power consumption. As an example, a device with 8 coherent links to CPUs and 4 interconnect links to connect to other devices of the same kind will consume less than 100 W of power. Such a device will reduce the main system interconnect port requirement with a factor of four compared to the traditional configuration for HPC systems with 2 CPUs per network interface.

[0048] A Network Interface Controller, NIC, can be implemented for use with different types of high performance networks like Ethernet, InfiniBand, Omnipath (Intel), BXI (ATOS/Bull), and Gen-Z (Gen-Z consortium standard). Gen-Z defines a load/store architecture where CPU instructions can address memory directly without going through a message passing protocol through software driver layers. Omnipath has a similar function in addition to RDMA functionality for message passing. The ability for processors to use load and store instructions to operate directly on memory locations across the interconnect reduces the overhead for data movement. This is most important for short data transfers where latency and overhead becomes a significant factor for performance.

First Embodiment of the Present Invention

[0049] The present invention will now be described with a specified example. The invention introduces a new element, namely a cache coherent node controller, it also provides examples of the use of said new element in system configurations from 1-dimensional fat tree configurations to n-dimensional fat tree configurations.

[0050] The cache coherent node controller may at least comprise: [0051] a. one or more network interface controllers, each network interface controller includes at least one network interface. An example of such a coherent node controller is shown schematically in FIG. 1 and more detailed in FIG. 2. It can be seen in FIG. 2 that the node controller includes 4 NICs and each NIC includes at least one network interface port. The NICs are configured for communication with network switches. [0052] b. at least two coherent interfaces each configured for communication with a microprocessor. In FIG. 2 it is shown eight coherent interfaces, CLL. Each of the CLLs at least comprises a coherent interface port. [0053] c. one or more interconnected protocol engines. The protocol engines are shown in FIG. 2 as squares marked PE. FIG. 2 indicates 16 PEs. Each PE is connected directly or indirectly to a CLL. PEs are interconnected directly or indirectly.

[0054] The cache coherent node controller can be scaled up or down maintaining the same principle of incorporating one or more NICs.

[0055] Obviously, the design layout shown in FIG. 2 with NICs arranged in the corner is a design matter, also it makes the figure readily understandable.

Second Embodiment of the Present Invention

[0056] In the second embodiment of the present invention the coherent node controller with NICs described above is used in a computer system. The computer system comprises one or more nodes where each node is connected to at least one network switch, FIGS. 5, 7. 8a, 8b and 9. Each node is described in the section above, the first embodiment of the present invention.

[0057] In the event that only one NIC is included or only one NIC is connected to network switches one will have a one dimensional system as depicted in FIG. 6. In FIG. 5 it is shown a two-dimensional system, i.e. two NICs in each cache coherent node controller is connected to a network switch.

[0058] FIG. 7 shows a three dimensional system, that is; each NIC is connected with three networks switches. FIG. 8a shows a front layer of a three dimensional system whilst FIG. 8b shows a three dimensional system with all layers. The configuration in FIG. 8b is 686 nodes.

[0059] The cache coherent node controller of FIG. 2, is configured for four dimensional systems.

TABLE-US-00001 100a, 100b, A computer, a server or a blade server - above referred 100c, 100d to as a node 200 On-chip distributed switching, such as but not limited to. Ethernet switch, Omnipath switch, Infiniband switch, and Gen-Z switch. CIF Coherent Interface CLL Coherent Link Layer NIC Network Interface Controllers NIF Network Interface PE Protocol Engine HPC High performance computing CNCNIC Coherent Node Controller w/Integrated NICs CPU Central Processing Unit RDMA Remote direct memory access