A data compressor with a hash computing hardware configured to evaluate the hash value for the current hash key extracted from a source data string, obtain a hash line corresponding to the hash value from a hash table, and perform hash key comparison to find at least one matching hash key. The hash line includes a prefix address column that stores a prefix address. Each entry of the hash line is provided to store a hash key and an offset. The hash computing hardware evaluates an address of the at least one matching hash key by combining the prefix address and an offset of the at least one matching hash key, and the offset of the at least one matching hash key is obtained from an entry storing the at least one matching hash key.

Decomposing a value range of the respective syntax elements into a sequence of n partitions with coding the components of z laying within the respective partitions separately with at least one by VLC coding and with at least one by PIPE or entropy coding is used to greatly increase the compression efficiency at a moderate coding overhead since the coding scheme used may be better adapted to the syntax element statistics. Accordingly, syntax elements are decomposed into a respective number n of source symbols s.sub.i with i=1 . . . n, the respective number n of source symbols depending on as to which of a sequence of n partitions into which a value range of the respective syntax elements is sub-divided, a value z of the respective syntax elements falls into, so that a sum of values of the respective number of source symbols s.sub.i yields z, and, if n>1, for all i=1 . . . n1, the value of s.sub.i corresponds to a range of the i.sup.th partition.

Described embodiments provide systems and methods for stream-based compression. An encoder of a first device receives an input stream of bytes including a first byte preceded by one or more second bytes. The encoder may determine to identify a prefix code for the first byte. The encoder may select a prefix code table using the one or more second bytes. The encoder may identify, from the selected prefix code table, the prefix code of the first byte. The encoder may generate an output stream of bytes by replacing the first byte in the input stream with the prefix code of the first byte. The encoder may transmit the output stream from the encoder of the first device to a decoder of a second device. The output stream may have a fewer number of bits than the input stream.

A highly programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets, is described. The data processing unit includes one or more specialized hardware accelerators configured to perform acceleration for various data-processing functions. This disclosure describes a parallel decoding of codewords within input data stream based on a codeword type and position.

Decomposing a value range of the respective syntax elements into a sequence of n partitions with coding the components of z laying within the respective partitions separately with at least one by VLC coding and with at least one by PIPE or entropy coding is used to greatly increase the compression efficiency at a moderate coding overhead since the coding scheme used may be better adapted to the syntax element statistics. Accordingly, syntax elements are decomposed into a respective number n of source symbols s.sub.i with i=1 . . . n, the respective number n of source symbols depending on as to which of a sequence of n partitions into which a value range of the respective syntax elements is sub-divided, a value z of the respective syntax elements falls into, so that a sum of values of the respective number of source symbols s.sub.i yields z, and, if n>1, for all i=1 . . . n1, the value of s.sub.i corresponds to a range of the i.sup.th partition.

The entropy coding of a current part of a predetermined entropy slice is based on, not only, the respective probability estimations of the predetermined entropy slice as adapted using the previously coded part of the predetermined entropy slice, but also probability estimations as used in the entropy coding of a spatially neighboring, in entropy slice order preceding entropy slice at a neighboring part thereof. Thereby, the probability estimations used in entropy coding are adapted to the actual symbol statistics more closely, thereby lowering the coding efficiency decrease normally caused by lower-delay concepts. Temporal interrelationships are exploited additionally or alternatively.

One embodiment provides a computer implemented method of data compression using a hardware accelerator. The method includes communicating, using a hardware accelerator, to a client device a set of instructions that compressed data be sent in LZ compressed format; receiving LZ compressed data at the hardware accelerator in LZ compressed format; decompressing the LZ compressed data using the hardware accelerator to create decompressed data; recompressing the decompressed data into GZ format, using the hardware accelerator, to create GZ compressed data; and sending the GZ compressed data to a storage device.

Decomposing a value range of the respective syntax elements into a sequence of n partitions with coding the components of z laying within the respective partitions separately with at least one by VLC coding and with at least one by PIPE or entropy coding is used to greatly increase the compression efficiency at a moderate coding overhead since the coding scheme used may be better adapted to the syntax element statistics. Accordingly, syntax elements are decomposed into a respective number n of source symbols s.sub.i with i=1 . . . n, the respective number n of source symbols depending on as to which of a sequence of n partitions into which a value range of the respective syntax elements is sub-divided, a value z of the respective syntax elements falls into, so that a sum of values of the respective number of source symbols s.sub.i yields z, and, if n>1, for all i=1 . . . n1, the value of s.sub.i corresponds to a range of the i.sup.th partition.

Method and apparatus for processing data are disclosed. The method may include: sorting to-be-compressed at least one piece of data in a predetermined order; for a sorted piece of data, in response to the data having a common prefix with a previous piece of data of the piece of data, adding the common prefix to a common prefix set; dividing the sorted at least one piece of data into at least one group based on the common prefix set and determining a common prefix of each group; and storing, for a group in the at least one group, a common prefix of the group and a non-common prefix portion of each piece of data in the group, and recording a common prefix corresponding to each piece of data in the each group and a shared length between the each piece of data and the common prefix.

Embodiments of the invention are directed to a DEFLATE compression accelerator and to a method for verifying the correctness of the DEFLATE compression accelerator. The accelerator includes an input buffer and a Lempel-Ziv 77 (LZ77) compressor communicatively coupled to an output of the input buffer. A switch is communicatively coupled to the output of the input buffer and to the output of the LZ77 compressor. The switch is configured to bypass the LZ77 compressor during a compression test. The accelerator further includes a deflate Huffman encoder communicatively coupled to an output of the switch and an output buffer communicatively coupled to the deflate Huffman encoder. When the switch is not bypassed, the compressor can be modified to produce repeatable results.