A compute device to manage workflow to disaggregated computing resources is provided. The compute device comprises a compute engine receive a workload processing request, the workload processing request defined by at least one request parameter, determine at least one accelerator device capable of processing a workload in accordance with the at least one request parameter, transmit a workload to the at least one accelerator device, receive a work product produced by the at least one accelerator device from the workload, and provide the work product to an application.

A matrix compression/decompression accelerator (MCA) system/method that coordinates lossless data compression (LDC) and lossless data decompression (LDD) transfers between an external data memory (EDM) and a local data memory (LDM) is disclosed. The system implements LDC using a 2D-to-1D transformation of 2D uncompressed data blocks (2DU) within LDM to generate 1D uncompressed data blocks (1DU). The 1DU is then compressed to generate a 1D compressed superblock (CSB) in LDM. This LDM CSB may then be written to EDM with a reduced number of EDM bus cycles. The system implements LDD using decompression of CSB data retrieved from EDM to generate a 1D decompressed data block (1DD) in LDM. A 1D-to-2D transformation is then applied to the LDM 1DD to generate a 2D decompressed data block (2DD) in LDM. This 2DD may then be operated on by a matrix compute engine (MCE) using a variety of function operators.

The described technology is generally directed towards reducing the amount of data stored in a sequence of data blocks by combining deduplication and compression. According to an embodiment, a system can comprise a memory that can store computer executable components, and a processor that can execute the components stored in the memory. The components can comprise a data block identifier that can identify, for a sequence of data blocks, a first data block that corresponds to a first data, resulting in a first identified data block, and a deduplication component that can identify a second data block that corresponds to the first data, resulting in a second identified data block, wherein the deduplication component can replace the second identified data block with a key value corresponding to the first identified data block. Further, a compression component can compress the first identified data block, resulting in a compressed data block.

One embodiment provides a computer implemented method of improving backup and restore performance including sending a compression job to a hardware accelerator using a compression thread; providing a callback pointer for the compression job; monitoring the hardware accelerator using a polling thread; calling the callback pointer to notify the compression thread when the hardware accelerator is available; and retrieving data from a destination buffer using the compression thread via a destination buffer pointer.

Embodiments of the present disclosure relate to a method, electronic device and computer program product for processing data. The method comprises determining a first hotness associated with a first compressed data block stored on a first storage device. The method also comprises: determining, based on the hotness, whether the first compressed data is stored to the second storage device, a type of the second storage device being different from a type of the first storage device. The method further comprises: in response to determining that the first compressed data block is stored to the second storage device, generating, based on a second compression level of the compression algorithm, a second compressed data block from the first compressed data block for storing to the second storage device, wherein the second compression level corresponds to the second storage device.

An aspect of managing data block compression in a storage system includes performing, for each block written to the storage system: bit-wise traversing the block, searching the block for a pattern indicating a repeating sequence of bits and, upon determining the pattern exists in the block and the repeating sequence of bits in the pattern exceeds a threshold value, removing the repeating sequence of bits from the block thereby yielding a reduced-size block.

Technologies for offloading acceleration task scheduling operations to accelerator sleds include a compute device to receive a request from a compute sled to accelerate the execution of a job, which includes a set of tasks. The compute device is also to analyze the request to generate metadata indicative of the tasks within the job, a type of acceleration associated with each task, and a data dependency between the tasks. Additionally the compute device is to send an availability request, including the metadata, to one or more micro-orchestrators of one or more accelerator sleds communicatively coupled to the compute device. The compute device is further to receive availability data from the one or more micro-orchestrators, indicative of which of the tasks the micro-orchestrator has accepted for acceleration on the associated accelerator sled. Additionally, the compute device is to assign the tasks to the one or more micro-orchestrators as a function of the availability data.

Techniques for compressing binary input data streams and files by reducing entropy of the input data prior to compression. Entropy reduction is achieved by first getting a stream of single-digit decimal pseudo random numbers and calculating the frequency of occurrence of each decimal number in the even and odd positions of the pseudo random number stream. Subsets of the frequencies of occurrence of the decimal digits are selected to best match the frequency of occurrence of 0 and 1 in the odd and even positions of the binary input data stream. The decimal digits of the subsets of frequencies of occurrence are selectively set to 0 or 1 thereby creating a binary pseudo random number (i.e. mapping) stream, which is XORed with the binary input stream and compressed. Decompression uses the same pseudo random number stream using the mapping stream and the seed number used during compression.

An information handling system may include at least one processor and a memory coupled to the at least one processor. The information handling system may be configured to receive data comprising a plurality of data chunks; perform deduplication on the plurality of data chunks to produce a plurality of unique data chunks; determine a compression ratio for respective pairs of the unique data chunks; determine a desired compression order for the plurality of unique data chunks based on the compression ratios; combine the plurality of unique data chunks in the desired compression order; and perform data compression on the combined plurality of unique data chunks.

There is provided a system and method for compression and decompression of a data stream used by machine learning networks. The method including: encoding each value in the data stream, including: determining a mapping to one of a plurality of non-overlapping ranges, each value encoded as a symbol representative of the range and a corresponding offset; and arithmetically coding the symbol using a probability count; storing a compressed data stream including the arithmetically coded symbols and the corresponding offsets; and decoding the compressed data stream with arithmetic decoding using the probability count, the arithmetic decoded symbols use the offset bits to arrive at a decoded data stream; and communicating the decoded data stream for use by the machine learning networks.