Methods, devices and systems for data compression and decompression are disclosed. A collection of data is obtained. The collection of data is sampled to establish, for a plurality of different symbol sizes, relative frequencies of symbols of the respective sizes in the collection of data. A code is generated to contain variable-length codewords by entropy encoding sampled symbols in the collection of data based on a metric which reflects the relative frequencies of the sampled symbols as well as their sizes. Symbols in the collection of data are compressed into compressed representations using the generated code, wherein the compressed representation of a symbol comprises a codeword which represents the symbol as well as metadata for decompressing the compressed representation.

A method for encoding and compressing a bit stream is provided. The method includes: receiving a bit stream; determining whether a first number of bits that are consecutive and identical in the bit stream is greater than or equal to a first preset value; and when the first number is greater than or equal to the first preset value, the first number of bits are encoded as a first code in a first encoding way, wherein the first code is composed of a first prefix and a first suffix, and the first prefix represents what the consecutive bits are and the first suffix represents the first number.

Compressed domain processors configured to perform operations on data compressed in a format that preserves order. The Compressed domain processors may include operations such as addition, subtraction, multiplication, division, sorting, and searching. In some cases, compression engines for compressing the data into the desired formats are provided.

Methods, devices and systems for data compression and decompression are disclosed. A collection of data is obtained. The collection of data is sampled to establish, for a plurality of different symbol sizes, relative frequencies of symbols of the respective sizes in the collection of data. A code is generated to contain variable-length codewords by entropy encoding sampled symbols in the collection of data based on a metric which reflects the relative frequencies of the sampled symbols as well as their sizes. Symbols in the collection of data are compressed into compressed representations using the generated code, wherein the compressed representation of a symbol comprises a codeword which represents the symbol as well as metadata for decompressing the compressed representation.

A highly programmable data processing unit includes multiple processing units for processing streams of information, such as network packets or storage packets. The data processing unit includes one or more specialized hardware accelerators configured to perform acceleration for various data-processing functions. The data processing unit is configured to retrieve speculative probability values for range coding a plurality of bits with a single read instruction to an on-chip memory that stores a table of probability values. The data processing unit is configured to store state information used for context-coding packets of a data stream so that the state information is available after switching between data streams.

Compressed domain processors configured to perform operations on data compressed in a format that preserves order. The Compressed domain processors may include operations such as addition, subtraction, multiplication, division, sorting, and searching. In some cases, compression engines for compressing the data into the desired formats are provided.

A processing device includes compression circuitry to encode an input stream with an encoding that translates multiple symbols of fixed length into multiple codes of variable length between one and a maximum length, to generate a compressed stream. The compression circuitry is to: determine at least a first symbol of the multiple symbols having a first code that exceeds the maximum length; identify a short code of the multiple codes that is to be lengthened to provide an increased encoding capacity for the at least the first symbol; generate multiple code-length converted values including to increase the length of the short code to the maximum length and decrease, to the maximum length, a length of the first code of the at least the first symbol; and generate, with use of the set of code-length converted values, the compressed stream at the output terminal.

Representative embodiments are disclosed for a rapid and highly parallel decompression of compressed executable and other files, such as executable files for operating systems and applications, having compressed blocks including run length encoded (RLE) data having data-dependent references. An exemplary embodiment includes a plurality of processors or processor cores to identify a start or end of each compressed block; to partially decompress, in parallel, a selected compressed block into independent data, dependent (RLE) data, and linked dependent (RLE) data; to sequence the independent data, dependent (RLE) data, and linked dependent (RLE) data from a plurality of partial decompressions of a plurality of compressed blocks, to obtain data specified by the dependent (RLE) data and linked dependent (RLE) data, and to insert the obtained data into a corresponding location in an uncompressed file. The representative embodiments are also applicable to other types of data processing for applications having data dependencies.

A memory device includes a memory; and at least one processor configured to: obtain a symbol stream including a plurality of symbols; determine a Huffman tree corresponding to the symbol stream, wherein each symbol of the plurality of symbols is assigned a corresponding prefix code from among a plurality of prefix codes based on the Huffman tree; generate a prefix length table based on the Huffman tree, wherein the prefix length table indicates a length of the corresponding prefix code for each symbol; generate a logarithm frequency table based on the prefix length table, wherein the logarithm frequency table indicates a logarithm of a frequency count for each symbol, generate a cumulative frequency table which indicates a cumulative frequency count corresponding to each symbol; generate a compressed bitstream by iteratively applying an encoding function to the plurality of symbols based on the logarithm frequency table and the cumulative frequency table; and store the compressed bitstream in the memory.

A highly programmable data processing unit includes multiple processing units for processing streams of information, such as network packets or storage packets. The data processing unit includes one or more specialized hardware accelerators configured to perform acceleration for various data-processing functions. The data processing unit is configured to retrieve speculative probability values for range coding a plurality of bits with a single read instruction to an on-chip memory that stores a table of probability values. The data processing unit is configured to store state information used for context-coding packets of a data stream so that the state information is available after switching between data streams.