A method for sharing a hardware decompression engine, including performing a compression type check on a first data stream to determine a compression type of the first data stream, wherein the first data stream is compressed using one selected from a group consisting of a first compression type and a second compression type; wherein, when the first data stream is compressed with the second compression type: receiving the second compression type at a selector; converting the first data stream compressed with the second compression type into a second data stream of the first compression type; inputting the converted second data stream into the selector; and decompressing the converted second data stream using the hardware decompression engine capable of decompressing a data stream compressed using the first compression type. In other aspects, a system for sharing a hardware decompression engine and a computing system are provided.

Compressed domain processors configured to perform operations on data compressed in a format that preserves order. The Compressed domain processors may include operations such as addition, subtraction, multiplication, division, sorting, and searching. In some cases, compression engines for compressing the data into the desired formats are provided.

Methods and apparatuses relating to data decompression are described. In one embodiment, a hardware processor includes a core to execute a thread and offload a decompression thread for an encoded, compressed data stream comprising a literal code, a length code, and a distance code, and a hardware decompression accelerator to execute the decompression thread to selectively provide the encoded, compressed data stream to a first circuit to serially decode the literal code to a literal symbol, serially decode the length code to a length symbol, and serially decode the distance code to a distance symbol, and selectively provide the encoded, compressed data stream to a second circuit to look up the literal symbol for the literal code from a table, look up the length symbol for the length code from the table, and look up the distance symbol for the distance code from the table.

A graph compression system includes a memory unit to store graph data, and an electronic hardware controller in signal communication with the memory unit. The electronic hardware controller determines a distribution of a set of vertices in a graph, and encodes each vertex included in the set of vertices as a variable length integer (VLI) that includes a variable number of bytes. The variable number of bytes of each vertex is based on the determined distribution. Accordingly, the memory unit stores each vertex having been encoded according to the distribution of the set of vertices in the graph.

A computer program product for data compression is provided. The computer program product includes a computer readable storage medium having program instructions embodied therewith. The program instructions are readable and executable by a processing circuit to cause the processing circuit to execute software compression for first requests for data compression that have respective sizes below a predefined threshold, forward second requests for data compression having respective sizes above the predefined threshold to a hardware accelerator and maintain a persistence of a compression dictionary used for executing the second requests across executions of the first and second requests.

A computerized method and apparatus compresses symbolic information, such as text. Symbolic information is compressed by recursively identifying pairs of symbols (e.g., pairs of words or characters) and replacing each pair with a respective replacement symbol. The number of times each symbol pair appears in the uncompressed text is counted, and pairs are only replaced if they appear more than a threshold number of times. In recursive passes, each replaced pair can include a previously substituted replacement symbol. The method and apparatus can achieve high compression especially for large datasets. Metadata, such as the number of times each pair appears, generated during compression of the documents can be used to analyze the documents and find similarities between two documents.

In one embodiment, an apparatus comprises a memory; and a compression engine comprising circuitry, the compression engine to assign weights to a plurality of first symbols of a data set, a weight representing a frequency of a corresponding first symbol in the data set; perform a partial sort of the first symbols based on the assigned weights; generate at least a portion of a Huffman tree based on the partial sort; and create a plurality of Huffman codes for the plurality of first symbols based on the Huffman tree.

Detailed herein are embodiments of systems, methods, and apparatuses for decompression using hardware and software. In hardware, an input buffer stores incoming input records from a compressed stream. A plurality of decoders decode at least one input record from the input buffer out output an intermediate record from the decoded data and a subset of the plurality of decoders to output a stream of literals. Finally, a reformat circuit formats an intermediate record into one of two types of tokens.

Aspects for generating compressed data streams with lookback pre-fetch instructions are disclosed. A data compression system is provided and configured to receive and compress an uncompressed data stream as part of a lookback-based compression scheme. The data compression system determines if a current data block was previously compressed. If so, the data compression system is configured to insert a lookback instruction corresponding to the current data block into the compressed data stream. Each lookback instruction includes a lookback buffer index that points to an entry in a lookback buffer where decompressed data corresponding to the data block will be stored during a separate decompression scheme. Once the data blocks have been compressed, the data compression system is configured to move a lookback buffer index of each lookback instruction in the compressed data stream into a lookback pre-fetch instruction located earlier than the corresponding lookback instruction in the compressed data stream.

Methods and apparatus are described by for compressing data using LZ77 compression. Embodiments determine an initial run from input data. The initial run includes repeating data at a first location and has a first length. A hash chain is updated with a proper set of hashes from prefixes from the initial run. A first search engine determines a second run that includes the repeating data at a second location. The second run has a second length less than the first length. A first matching location is determined within the input data having the repeating data using the hash chain and the second run. The first matching location is the first location. The first matching location, the second location, and the second length are written to an output buffer. The output buffer includes a compressed version of the input data.