Aspects for generating compressed data streams with lookback pre-fetch instructions are disclosed. A data compression system is provided and configured to receive and compress an uncompressed data stream as part of a lookback-based compression scheme. The data compression system determines if a current data block was previously compressed. If so, the data compression system is configured to insert a lookback instruction corresponding to the current data block into the compressed data stream. Each lookback instruction includes a lookback buffer index that points to an entry in a lookback buffer where decompressed data corresponding to the data block will be stored during a separate decompression scheme. Once the data blocks have been compressed, the data compression system is configured to move a lookback buffer index of each lookback instruction in the compressed data stream into a lookback pre-fetch instruction located earlier than the corresponding lookback instruction in the compressed data stream.

Apparatus and systems, including computer program products, implementing and using techniques for predicate application using partial compression dictionary match. A search strategy is developed for each predicate to be applied to compressed data. The compressed data is searched using the search strategy to locate the compression symbols identified in the search strategy. In response to locating a compression symbol from the search strategy in the compressed data, a respective row and applying the predicate is decompressed and a respective row that matches the predicate is returned to a database engine or an application.

Technologies for data decompression include a computing device that reads a symbol tag byte from an input stream. The computing device determines whether the symbol can be decoded using a fast-path routine, and if not, executes a slow-path routine to decompress the symbol. The slow-path routine may include data-dependent branch instructions that may be unpredictable using branch prediction hardware. For the fast-path routine, the computing device determines a next symbol increment value, a literal increment value, a data length, and an offset based on the tag byte, without executing an unpredictable branch instruction. The computing device sets a source pointer to either literal data or reference data as a function of the tag byte, without executing an unpredictable branch instruction. The computing device may set the source pointer using a conditional move instruction. The computing device copies the data and processes remaining symbols. Other embodiments are described and claimed.

A mechanism is provided for high-throughput compression of data. Responsive to receiving an indication of a match of a current 4-byte sequence from an incoming data stream to stored hash values in a set of hash tables, numerous variables are set to initial values. Responsive to receiving a subsequent 4-byte sequence from the incoming data stream and determining that an active match variable is set to one, the subsequent 4-byte sequence is compared to data in a copy of the incoming data stream in memory at an active position with a predefined length offset. A constraint variable is set to a number of bytes for which the match is to be extended. Responsive to the constraint variable being below a predetermined number, a length, distance pair is output indicating a match to a previous pattern in the incoming data stream.

Technologies for dividing work across one or more accelerator devices include a compute device. The compute device is to determine a configuration of each of multiple accelerator devices of the compute device, receive a job to be accelerated from a requester device remote from the compute device, and divide the job into multiple tasks for a parallelization of the multiple tasks among the one or more accelerator devices, as a function of a job analysis of the job and the configuration of each accelerator device. The compute engine is further to schedule the tasks to the one or more accelerator devices based on the job analysis and execute the tasks on the one or more accelerator devices for the parallelization of the multiple tasks to obtain an output of the job.

A method for sharing a hardware decompression engine, including performing a compression type check on a first data stream to determine a compression type of the first data stream, wherein the first data stream is compressed using one selected from a group consisting of a first compression type and a second compression type; wherein, when the first data stream is compressed with the second compression type: receiving the second compression type at a selector; converting the first data stream compressed with the second compression type into a second data stream of the first compression type; inputting the converted second data stream into the selector; and decompressing the converted second data stream using the hardware decompression engine capable of decompressing a data stream compressed using the first compression type. In other aspects, a system for sharing a hardware decompression engine and a computing system are provided.

Apparatus and method for dictionary accelerator compression. For example, one embodiment of an apparatus comprises: a plurality of cores; a compression/decompression accelerator coupled to or integral to one or more of the plurality of cores, the compression/decompression accelerator to perform decompression and compression operations in response to read and write operations, respectively, wherein responsive to notification of a compression job to compress a memory page or a portion thereof, a history buffer associated with the compression/decompression accelerator to is to be initialized with pre-configured dictionary data, the compression/decompression accelerator to match portions of the pre-configured dictionary data with portions of the memory page to generate compressed output data.

A data transfer device calculates a compression performance value which represents a quantity of data that can be compressed per unit time and a transfer performance value which represents a quantity of data that can be transferred per unit time, and calculates, based on these values, a compression ratio which represents a ratio of data to be compressed and then transferred to total data to be transferred. The data transfer device extracts, from a storage unit which stores data, the data to be transferred, and then compresses part of the extracted data based on the compression ratio, and transfers the compressed data and remaining data to another device. The compression and transfer processes are performed in parallel.

A hardware data compressor that compresses an input block of characters by replacing strings of characters in the input block with back pointers to matching strings earlier in the input block. A hash table is used in searching for the matching strings in the input block. A plurality of hash index generators each employs a different hashing algorithm on an initial portion of the strings of characters to be replaced to generate a respective index. The hardware data compressor also includes an indication of a type of the input block of characters. A selector selects the index generated by of one of the plurality hash index generators to index into the hash table based on the type of the input block.

A system comprises an encoder configured to entropy encode a bitstream comprising both compressible and non-compressible symbols. The encoder parses the bitstream into a compressible symbol sub-stream and a non-compressible sub-stream. The non-compressible symbol sub-stream bypass an entropy encoding component of the encoder while the compressible symbol sub-stream is entropy encoded. When a quantity of bytes of entropy encoded symbols and bypass symbols is accumulated a chunk of fixed or known size is formed using the accumulated entropy encoded symbol bytes and the bypass bytes without waiting on the full bitstream to be processed by the encoder. In a complementary manner, a decoder reconstructs the bitstream from the packets or chunks.