A processing device includes compression circuitry to encode an input stream with an encoding that translates multiple symbols of fixed length into multiple codes of variable length between one and a maximum length, to generate a compressed stream. The compression circuitry is to: determine at least a first symbol of the multiple symbols having a first code that exceeds the maximum length; identify a short code of the multiple codes that is to be lengthened to provide an increased encoding capacity for the at least the first symbol; generate multiple code-length converted values including to increase the length of the short code to the maximum length and decrease, to the maximum length, a length of the first code of the at least the first symbol; and generate, with use of the set of code-length converted values, the compressed stream at the output terminal.

A highly programmable data processing unit includes multiple processing units for processing streams of information, such as network packets or storage packets. The data processing unit includes one or more specialized hardware accelerators configured to perform acceleration for various data-processing functions. The data processing unit is configured to retrieve speculative probability values for range coding a plurality of bits with a single read instruction to an on-chip memory that stores a table of probability values. The data processing unit is configured to store state information used for context-coding packets of a data stream so that the state information is available after switching between data streams.

Embodiments of the present disclosure include techniques for compressing data using a tree encoded bit mask that may result in higher compression ratios. In one embodiment, an input vector having a plurality of values is received by a first plurality of switch circuits. Selection of the input values is controlled by sets of bits from the bit mask. The sets of bits specify locations of portions of the input vector where particular value of interest reside. The switch circuits output multiple values of the input vector, which include the particular value of interest. A second stage of switch circuits is controlled by logic circuit that detects values on the outputs of the first stage of switch circuits and outputs the values of interest. In some embodiments, the values of interest may be non-zero values of a sparse input vector, and the switch circuits may be multiplexers.

A lossless data compressor prevents normalization overruns on-the-fly as symbol occurrence counts are rounded to generate symbol frequencies, allowing an encoding table generator to generate encoding table entries without waiting for the symbol frequency table to finish filling. Rounding errors are accumulated as symbols are normalized and compensated for by reducing a symbol frequency when the symbol frequency is at least 2 and the accumulated errors have exceeded a threshold. The symbol frequency is also reduced when the number of remaining states in the encoding table is insufficient for a number of remaining unprocessed symbols and states for a current encoding table entry. Since error compensation occurs as symbols are being normalized, encoding table generation is not forced to wait for all symbols in the block to be processed, reducing latency. Three pipeline stages can operate on three input blocks: symbol counting, normalization/error compensation/encoding table generation, and data encoding.

Compressed domain processors configured to perform operations on data compressed in a format that preserves order. The Compressed domain processors may include operations such as addition, subtraction, multiplication, division, sorting, and searching. In some cases, compression engines for compressing the data into the desired formats are provided.

The present invention relates to a method and apparatus for compression and decompression of a numerical file. The compression method comprises: read a numerical file, convert each numerical element into a 32-bit floating point number; combine all the numbers to form a binary numerical file; group the binary numerical file into a n-bit sequence pattern; generate a Huffman tree based on frequency of occurrences of a plurality of unique bit patterns present in the binary numerical file; generate codewords and replace unique bit patterns with codewords so that a compressed binary numerical file is generated. A method for decompression comprises: read a compressed binary numerical file having codewords; fetch a part or entire compressed binary numerical file using an address dictionary; replace the codewords with unique bit patterns using a Huffman tree such that a decompressed binary numerical file being generated.

This disclosure provides methods, devices, and systems for data compression and decompression. The present implementations more specifically relate to entropy encoding and decoding techniques for keeping a state variable within upper and lower bounds using a noniterative process. The entropy encoding uses a fixed state threshold to determine a number of bits to remove and removes the bits from a current state prior to encoding a symbol with the current state. The entropy decoding decodes encoded data in a bitstream based on a current state to obtain the symbol and a new state and determines a number of bits to read from the bitstream and to add to the new state to update the current state.

Methods are provided for efficiently encoding and decoding multivariate correlated data sequences for transmission over multiple channels of a network. The methods include transforming data vectors from correlated sources into vectors that comprise substantially independent and correlated components, and generating a common information vector based on the correlated components, and two private information vectors. The methods also include computing the amount of information, such as Wyner's lossy common information, in the common information vector, computing rates that lie on the Gray-Wyner rate region, and choosing compression rates based on the amount of common information. The methods may be applicable, in general, to a wide range of communications and/or storage systems and, particularly, to sensor networks and multi-user virtual environments for gaming and other applications.

Example techniques described herein determine a classification of a variable-length source data such as an executable code. A neural network system that includes a convolution filter, a recurrent neural network, and a fully connected layer can be configured in a computing device to classify executable code. The neural network system can receive executable code of variable length and reduce its dimensionality by generating a variable-length sequence of features extracted from the executable code. The sequence of features is filtered, and applied to one or more recurrent neural networks and to a neural network. The output of the neural network classifies the data. Other disclosed systems include a system for reducing the dimensionality of command line input using a recurrent neural network. The reduced dimensionality of command line input may be classified using the disclosed neural network systems.

Example techniques described herein determine a classification of a variable-length source data such as an executable code. A neural network system that includes a convolution filter, a recurrent neural network, and a fully connected layer can be configured in a computing device to classify executable code. The neural network system can receive executable code of variable length and reduce its dimensionality by generating a variable-length sequence of features extracted from the executable code. The sequence of features is filtered, and applied to one or more recurrent neural networks and to a neural network. The output of the neural network classifies the data. Other disclosed systems include a system for reducing the dimensionality of command line input using a recurrent neural network. The reduced dimensionality of command line input may be classified using the disclosed neural network systems.