A network interface controller (NIC) capable of facilitating efficient memory address translation is provided. The NIC can be equipped with a host interface, a cache, and an address translation unit (ATU). During operation, the ATU can determine an operating mode. The operating mode can indicate whether the ATU is to perform a memory address translation at the NIC. The ATU can then determine whether a memory address indicated in the memory access request is available in the cache. If the memory address is not available in the cache, the ATU can perform an operation on the memory address based on the operating mode.

A network interface controller (NIC) capable of facilitating efficient memory address translation is provided. The NIC can be equipped with a host interface, a cache, and an address translation unit (ATU). During operation, the ATU can determine an operating mode. The operating mode can indicate whether the ATU is to perform a memory address translation at the NIC. The ATU can then determine whether a memory address indicated in the memory access request is available in the cache. If the memory address is not available in the cache, the ATU can perform an operation on the memory address based on the operating mode.

Methods and apparatus for split memory allocations in non-kernel space. Many modern networking technologies use asymmetric transmit and/or receive resource. Various aspects described herein split memory resources for transmit and receive, configuring each for their respective hardware optimizations. For example, a receive data paths that support batch processing and packet aggregation may be allocated large memory objects (32 KB) that can route data packets on a per-flow basis. In contrast, transmit data paths that support multiple concurrent network connections may be allocated small memory objects (2 KB) that can route data packets one at a time.

An apparatus for managing buffers and a method thereof are provided. The method for managing buffers includes: receiving a plurality of pieces of data, where the plurality of pieces of data includes a first piece of data and a second piece of data; allocating at least one buffer to establish a cluster buffer according to a data amount of the first piece of data; and if at least one of a first condition and a second condition is satisfied, ending a storage operation of the cluster buffer, where the first condition is that a total remaining space of the at least one buffer that has stored the data in the cluster buffer is less than a remaining space threshold, and the second condition is that the quantity of the at least one buffer that has stored the data in the cluster buffer reaches a cluster threshold.

A method for data transmission may be implemented on an electronic device having one or more processors. The one or more processors may include a master queue including a master queue head and a plurality of primary ports that are connected to each other using a serial link. The method may include operating the master queue head to obtain a message. The method may also include operating the master queue head to segment the message into a plurality of segments. The method may also include operating the master queue head to transmit the plurality of segments to a first primary port of the plurality of primary ports in the master queue. The method may also include operating the first primary port to transmit the plurality of segments to a second primary port of the plurality of primary ports in the master queue.

A first device as a buffer server in an Ethernet transmits a first buffer client querying packet from a port of enabling a distributed buffer function of the first device, receives a first buffer client registering packet from a second device through the port, and adds the second device into a distributed buffer group of the port. When the first device detects that a sum of sizes of packets entering the port and not transmitted reaches a preset first flow-splitting threshold in a first preset time period, the first device forwards a packet entering the port and not transmitted to a buffer client selected from the distributed buffer group of the port.

Techniques for generating a stream processing pipeline are provided. In one embodiment, a method includes receiving a configuration file from a data service. The configuration file represents a pipeline configuration of the stream processing pipeline, and the pipeline configuration includes representations of a plurality of different types of pipeline stages configured based on a respective customization of an entity. The method further includes generating a plurality of pipeline stages in accordance with the pipeline configuration of the stream processing pipeline; collecting, at one or more pipeline stages of a first-type in the stream processing pipeline, data items from one or more data sources; processing the collected data items at one or more pipeline stages of a second-type in the stream processing pipeline; and transmitting, at one or more pipeline stages of a third-type in the stream processing pipeline, the processed data items to the data service.

A data capture module includes a first port configured to receive first data transmitted from a first component to a second component of a substrate processing system, a second port configured to received second data transmitted from the second component to the first component, a first data stream forwarding module configured to duplicate the first data, forward the duplicated first data to the second port, and output the first data, and a second data stream forwarding module configured to duplicate the second data, forward the duplicated second data to the first port, and output the second data. The first port is configured to transmit the duplicated second data to the first component and the second port is configured to transmit the duplicated first data to the second component. A data compression module is configured to compress the first and second data. Data storage is configured to store the compressed data.

A switch equipped with a self-managing reduction engine is provided. During operation, the reduction engine can use a timeout mechanism to manage itself in different latency-induced or error scenarios. As a result, the network can facilitate an efficient and scalable environment for high performance computing.

A switch equipped with a self-managing reduction engine is provided. During operation, the reduction engine can use a timeout mechanism to manage itself in different latency-induced or error scenarios. As a result, the network can facilitate an efficient and scalable environment for high performance computing.