Mouse over elephant

Abstract

An optical switch plane with one or more switch layers, each layer with multiple switches is provided. In a data center, an optical circuit switch plane is added between the device plane and packet switch plane. Direct speed of light connections may be created between devices, the data center temporally shrunk, remote devices localized, elephant flows kept out of mouse switches, mouse switch spend reduced, stranded resources recovered, layer 1 reconfigured and optimized, bare metal bent, secure tunnels created, networks physically isolated, failure resiliency increased, and packet switch congestion avoided.

Claims

1. An optical data network with up/down connectivity, including: an optical circuit switch plane, between an end device plane, and an optical packet switch plane; said end device plane includes a multitude of end devices; said end devices include: servers, GPUs, FPGAs, ASICs, neural networks, memory, or storage; said circuit switch plane includes two or more circuit switch layers with north/south connectivity; said packet switch plane includes two or more packet switch layers with north/south connectivity; and said two or more layers of said circuit plane are interconnected with up/down connectivity to respective layers of said two or more layers of said packet plane.

2. The network of claim 1, where said circuit switch plane includes multiple hierarchical optical switch layers; and each layer including at least two optical circuit switches.

3. The network of claim 2, where said circuit switch plane includes two or more switch layers; 200K or more ports; and The maximum insertion loss of said plane is <=3 dB.

4. The network of claim 2, where said circuit switch plane includes three or more switch layers; 1M or more ports; and the maximum insertion loss of said plane is <=5 dB.

5. The network of claim 2, where said plane includes 20K or more optical ports; and the maximum insertion loss of said plane is <=3 dB.

6. The network of claim 1, where a majority of ports from said device plane couple to said circuit switch plane.

7. The network of claim 1, where a majority of ports from a said packet switch plane couple to said circuit switch plane.

8. The network of claim 1, where a majority of said devices include one or more PSM optics modules.

9. The network of claim 1, where a majority of said devices include multiple optical network ports; and a said circuit switch couples some said ports from said devices to a said packet switch, without circuit switching said connections.

10. The network of claim 1, where two said devices, each located in a different rack, are optically connected via a circuit that does not traverse said packet plane.

11. The network of claim 1, where two said devices, each located in a different pod, are optically connected via a circuit that does not traverse said packet plane.

12. The network of claim 1, where southbound ports of a single said packet switch are optically coupled via said circuit switch plane to said devices located in different racks.

13. The network of claim 1, where one or more WAN ports are coupled to a said device plane via said circuit switch plane, without coupling to said packet switch plane.

14. The network of claim 1, where said packet switch ports are oversubscribed; and the oversubscription ratio of device ports to packet switch ports is 2:1 or greater.

15. A method to configure the network of claim 1, including: accepting a request for resources, determining a low-cost route, verifying the route meets requirements, and configuring said circuit network and said devices.

16. The method of claim 15, further including: configuring said circuit switched plane to order connections to said packet switch ports, as to produce a more desired packet switch latency.

17. The method of claim 15, further including: dynamically clustering a multitude of said devices by reconfiguring said circuit switched plane.

18. The method of claim 15, further including: direct connecting GPU, FPGA, or ASIC devices to a server or another GPU, FPGA, or ASIC.

19. The network of claim 1, where 33% or more of ports from said device plane couple to said circuit switch plane; where 33% or more of ports from said packet switch plane couple to said circuit switch plane; and where the sum of the number of ports of all said circuit switches is greater than 20,000, with a circuit switch plane insertion loss of <=5 dB.

20. The network of claim 1, where said up/down connectivity between said circuit/packet planes also has a north/south connectivity between different said layers of said planes.

Description

DESCRIPTION OF DRAWINGS

(1) FIG. 1 is a diagram of a packet switched network over a circuit switched network.

(2) FIG. 2 is a diagram of a packet switched network over a circuit switched network with A/B redundancy and fixed packet routes.

(3) FIG. 3 is a diagram of a packet switched network over a circuit switched network with 3 layers.

(4) FIG. 4a illustrates a GPU cluster with FPGA and ASIC assist and backup.

(5) FIG. 4b illustrates a server cluster and Hadoop replication.

(6) FIG. 4c illustrates packet congestion and latency reduction.

(7) FIG. 4d illustrates NFV.

(8) FIG. 4e illustrates a composable server.

(9) FIG. 4f illustrates an Al HPC cloud.

(10) FIG. 5 is a method to orchestrate a circuit switched network.

DETAILED DESCRIPTION

(11) FIG. 1 is a diagram of a packet switched fiber optic network over a circuit switched fiber optic network 100. A circuit switch plane 120 is added to the datacenter network between the device plane 110 and packet switch plane 130. Circuit switch plane 120 has a circuit leaf layer 122 with one or more leaf circuit switches 123, and packet switch plane 130 has a packet leaf layer 132 with one or more leaf packet switches 133. Circuit switch plane 120 may have a circuit spine layer 124 with one or more spine circuit switches 125, and packet switch plane 130 may have a packet spine layer 134 with one or more spine packet switches 135. A WAN 140 connection may exist.

(12) Devices 112 may be housed in racks 114, and racks 114 organized into pods 116. Pods 116 may be containers.

(13) Devices 112 have a network port. Ports may be Omni Path Architecture (OPA) with PSM4 optics. PSM ports may be switched independently, in 25 GBPS increments from a 100 GBPS module. Other protocols may be used, such as Ethernet, Infiniband, FibreChannel, or other protocol over fiber, or any mix of the above. Other optics may by used, such as CWDM4, coarse WDM, dense WDM, BX, LX, SX, or other optics module, or any mix of the above. Each end of a link must have compatible protocol and optics.

(14) Switches may have a number of ports for east/west connectivity to other switches in the same layer and a number of ports for north/south connectivity to higher or lower layers. Higher layers may span multiple physical locations. In the context used within this paper, east/west does not mean within a datacenter, and north/south outside of the datacenter. A novel up/down routing dimension may be available between circuit plane 120 and packet plane 130. This is a mouse network over an elephant network. East/west is illustrated on the page as left/right, north/south as up/down, and up/down with squiggly lines. East/west and north/south flows may exist on both the circuit and packet planes. A single circuit connection may have multiple east/west and/or north/south components.

(15) A packet layer may have the same or different topology as a circuit layer. Elephant traffic need not traverse a mouse switch, saving many mouse switch ports. The up/down direction may be oversubscribed by 7:1, 3:1, 2:1, or other circuit:packet ratio. The oversubscription ratio may have local variations. The number of packet switches over a circuit switch may vary. Some circuit switches may not have a directly corresponding packet switch. Connections to packet switches elsewhere in the network may be made.

(16) High port-count low insertion-loss non-blocking transparent optical circuit switches are required for large switch planes. An example of such a switch is described in copending Ser. No. 16/041,815 “Optical Switch”, and is incorporated by reference. 2 or 3 switch layers may have an insertion loss of <=3 dB, or <=5 dB. Switches may have 500, 1000, 2000, 4000, or more ports. Planes may have 20K, 50K, 200K, 1M, or more ports, with an insertion loss of <=3, 4, or 5 dB. Insertion losses exclude interconnect fiber. Existing low port-count switches have insufficient connectivity to form a large plane. Too many ports are used for up/down, east/west, north/south traffic compared to the number of device ports. Existing high insertion-loss switches limit hops through multiple optical switches.

(17) FIG. 2 is a diagram of a packet switched network over a circuit switched network with A/B redundancy and fixed packet routes 200. A/B redundant network 200 may be similar to network 100, with the addition of one or more layers have A and B sub-layers. A/B/C or A/B/C/D sub-layers are also possible. Not all layers may require the same level of redundancy. For instance, the leaf circuit layer may only have an A sub-layer and the spine circuit layer may have A and B sub-layers. Or, a sub-layer may be present in only a portion of a layer. For devices with a single PSM port, a cable may split fibers between sub-layers.

(18) Connections between A/B sub-layer add another routing dimension. This allows traffic to load balance between A/B sub-layers, and not be distinct redundant networks until necessary. This may be advantageous in relieving congestion. If more than one circuit switch is required in a pod, they may be partitioned horizontally as A/B layers, instead of vertically with each switch serving different racks.

(19) Dashed lines in circuit switch 223A indicate fixed (not switched) connections between a device and a packet switch. This may be desired if all nodes have some amount of packet switched traffic. Conversely, if most nodes have point to point traffic, such as GPU clusters, HPC, direct connection, or storage replication, fixed connections to a packet switch may waste ports. Fixed connections may also be used with non-redundant network 100. This split of fixed ports from a PSM4 fiber bundle may be done inside of the switch, but without connecting to the switch fabric, to simplify datacenter cabling.

(20) FIG. 3 is a diagram of a packet switched network over a circuit switched network with 3 layers 300. Three layer network 300 may be similar to network 100, but with an additional super spine layer 326 and 336. This allows a high port-count low insertion-loss switch of 1024 ports and 0.4 dB insertion loss, with 5 dB link budget optics, to realize a 268M port network. Or, a 512 port 0.4 db insertion loss switch to form a 33M port network. Compared to the 2 layer network 100, a 524K port and 131K port network may be realized, respectively. Preferably, the insertion loss of the circuit plane is <=3 dB, but may be <=3.5, 4, or 5 dB.

(21) FIG. 4a-e illustrates some possible configurations of network 100. Highlighted routes illustrate connections used in the example application. Preferably devices 112 may forward packets, for low bandwidth requirements this eliminates the need for a connection to a packet switch.

(22) FIG. 4a illustrates a GPU cluster with FPGA and ASIC assist. Also, backup is illustrated. As an example, GPU 112k may be fully meshed, memory 112e may be in a star topology with GPU 112k and server 112a, FPGA 112l may assist GPU 112k. GPU 112k and FPGA 112l need not use a standard packet format. GPU 112k may be added in rack scale increments. Specially configured racks with a mix of GPU 112k and server 112a are no longer necessary. This configuration may be useful for rendering and compressing video and allowing asymmetric hardware refresh.

(23) Backup network load need not burden a packet switch, or other network traffic. Circuit switched plane 120 may connect SSD 112f and cold store 112j. SSD 112f and cold store 112j are preferably able to stream files, with server 112a (not shown) running the backup application only needing to generate transfer requests and confirm completion. Optionally, some devices, such as cold store 112j, may have a low percentage of use within its pod and may be hard connected directly to the spine layer. Files may also be moved between store 112c, SSD 112f, JBOD, NVM, hard disk, hot storage 112h, warm storage 112i, cold storage 112j, backup, or other device.

(24) FIG. 4b illustrates a server cluster and Hadoop replication. Servers 112a may be clustered in a topology selected to best meet data or process load. Servers 112a may be located in different pods. The route cost of the north/south and east/west connections to the server in the remote pod are nearly identical. Server 112a may connect directly to a spine packet switch 135, avoiding the latency of leaf packet switch 133. Server 112b may be used interchangeably with server 112a when not used as a WAN interface.

(25) Storage 112c may connect to multiple servers 112a in various pods 116. Storage 112c may be intelligent storage, capable of processing Hadoop requests. Hadoop replication may be a logical function, moving the network and not moving the data. Additionally, storage 112c may connect to other storage 112c, allowing replication between devices without burdening servers 112a or the packet network 130. This may be necessary due to limited ports on storage 112c. Alternatively, a packet switch 133 may be used for aggregation.

(26) FIG. 4c illustrates packet congestion and latency reduction. Multiple ports of server 112a may connect to one or more switches 133 for added bandwidth. Leaf switches 132 may connect to other leaf switches 132, instead of a spine switch 134, adding additional hybridized layers of switching. Or, additional switch ports may be connected for added bandwidth.

(27) Packet switch 133 may be a Top Of Cluster (TOC) switch. All nodes of the cluster connect to the same switch, regardless of which rack of pod they may be in. Any node within the cluster may reach any other node within the cluster with a one hop latency.

(28) FIG. 4d illustrates Network Function Virtualization (NFV). Ethernet may be nearly eliminated from the network. An ethernet NIC and appropriate WAN optics may be added to server 112b. No other modifications are required, as the circuit switch plane 120 is wavelength and protocol agnostic. Ethernet WAN 140 may connect to the circuit switch plane 120 for server redundancy. Server 112b may take on routing functions and connect to spine switch 136 or leaf switch 133. Additional packet ports may be used as necessary from switches in other pods. Some network functions are more efficiently implemented in FPGA 1121 or ASIC 112m than in server 112b software. FPGA and servers may be used for switching functions during peak hours, and released to other customers off peak.

(29) FIG. 4e illustrates a composable server. A server may be disaggregated into building blocks, CPU 112d, memory 112e, storage 112f, and GPU 112g. Memory 112e may have a mix of SRAM, DDR, flash, or other memory types and a memory manager with a network port. These components may be composed to create servers better sized for a particular task.

(30) FIG. 4f illustrates an Al HPC cloud. An Al application may require a neural network ASIC 112m. ASIC 112m may be connected to memory 112e to record neural evolution. FPGA 1121 may assist ASIC 112m. Server cluster 112a may reside in a different pod. Given network latency constraints and using only an existing packet network, ASIC 112m may be too costly to deploy throughout the datacenter and reserving server 112a time within a pod may also be too costly.

(31) Other functionality (not illustrated) may include diagnostic 112o with TDR, power meter, or other diagnostic hardware which may be scripted to validate switch ports, fiber, and fiber connectivity. Human errors in fiber connectivity might be corrected by updating port mapping, instead of moving fiber to correct the wiring error.

(32) FIG. 5 is a method to compose and orchestrate a circuit switched network 500.

(33) Resource requests may be accepted in step 510. The request may contain: customer contract requirements; direct connect requests; hardware allocation request; drive mount/unmount requests; open/close stream request; Hadoop requests; packet switch traffic statistics; packet switch latency; composition script request; application request; and/or other sources. Certain OS functions may be modified to simplify request generation.

(34) Direct connect, drive mount/unmount, and open/close stream requests may configure a circuit switched route between source and destination devices.

(35) Hadoop replication requests may configure circuit switched routes between multiple servers and a single storage device, moving the network instead of moving the data.

(36) Latency within a packet switch is dependent on source and destination ports. Use of QOS or packet traffic statistics may be used to configure the circuit switched network to reorder packet switch ports.

(37) Backup applications may configure various routes between source and destination.

(38) FEA, HPC, or Al HPC applications may cluster servers into a network topology that best matches the data topology.

(39) Clustered devices may have a packet switch port. All packet switch ports for the cluster may connect to a single TOC packet switch.

(40) A composer may provide a requested resources and connection topology.

(41) Lowest cost of available resource allocations may be determined in step 520. Cost of resources required to meet the resource request may be calculated using: number and type of ports and devices consumed; packet switch port to port latency; wire latency; minimum quantity of unallocated resources; billable cost; and/or other factors. If a sufficiently low cost route is not available, existing routes may be moved to a higher cost route that still within acceptable cost.

(42) Verify proposed resource allocation in step 530. If the allocation fails to meet: latency requirements; physical isolation requirements; and/or other factors, the request may be rejected.

(43) Network and devices are configured in step 540.

(44) The orchestration request returns in step 550.

(45) The previous examples are intended to be illustrative. Countless additional variations and applications are readily envisioned. Planes and layers may be partially implemented. Resource orchestration will vary.