Compressing files is disclosed. An input file to be compressed is first aligned. Aligning the file includes splitting the file into sequences that can be aligned. When splitting the file into sequences or when performing subsequent recursive splitting, the splitting is based on a longest sequence match. The result is a compression matrix, where each row of the matrix corresponds to part of the file. A consensus sequence is determined from the compression matrix. Using the consensus sequence, pointer pairs are generated. Each pointer pair identifies a subsequence of the consensus matrix. The compressed file includes the pointer pairs and the consensus sequence.

A system comprises an encoder configured to entropy encode a bitstream comprising both compressible and non-compressible symbols. The encoder parses the bitstream into a compressible symbol sub-stream and a non-compressible sub-stream. The non-compressible symbol sub-stream bypass an entropy encoding component of the encoder while the compressible symbol sub-stream is entropy encoded. When a quantity of bytes of entropy encoded symbols and bypass symbols is accumulated a chunk of fixed or known size is formed using the accumulated entropy encoded symbol bytes and the bypass bytes without waiting on the full bitstream to be processed by the encoder. In a complementary manner, a decoder reconstructs the bitstream from the packets or chunks.

A decoder includes circuitry configured to receive a bitstream identify, in the bitstream, a current frame, wherein the current frame includes a first region and a third region, detect, in the bitstream, an indication that the first region is encoded according to a lossless encoding protocol, and decode the current frame, wherein decoding the current frame further comprises decoding the first region using a lossless decoding protocol corresponding to the lossless encoding protocol.

A FRAMEWORK and the associated method, schema and design for processing digital data, whether random or not, through encoding and decoding losslessly and correctly for purposes including the purposes of encryption/decryption or compression/decompression or both. There is no assumption of the digital information to be processed before processing. A Universal Coder is invented and now Pigeonhole meets Blackhole.

Encoding devices, methods and programs that encode with high transmission efficiency by controlling a running disparity are disclosed. In one example, an encoding device includes a scrambling circuit that scrambles an input data string, a calculation circuit that calculates a first running disparity of the scrambled data string, a determination circuit that determines whether or not to invert the scrambled data string on the basis of a first running disparity calculated by the calculation circuit and a second running disparity calculated at a time point before the first running disparity, and an addition circuit that inverts or non-inverts the scrambled data string on the basis of a determination result by the determination circuit, adds a flag indicating the determination result, and outputs the data string. The technology can be applied to devices that perform SLVS-EC standard communication.

A weight data compression method includes: generating a 4-bit data string of 4-bit data items each expressed as any one of nine 4-bit values, by dividing ternary weight data into data items each having 4 bits; and generating first compressed data including a first flag value string and a first non-zero value string by (i) generating the first flag value string by assigning one of 0 and 1 as a first flag value of a 1-bit flag to a 4-bit data item 0000 and assigning an other of 0 and 1 as a second flag value of the 1-bit flag to a 4-bit data item other than 0000 among the 4-bit data items in the 4-bit data string and (ii) generating the first non-zero value string by converting the 4-bit data item other than 0000 into a 3-bit data item having any one of eight 3-bit values.

A system and method to compress application control data, such as weights for a layer of a convolutional neural network, is disclosed. A multi-core system for executing at least one layer of the convolutional neural network includes a storage device storing a compressed weight matrix of a set of weights of the at least one layer of the convolutional network and a decompression matrix. The compressed weight matrix is formed by matrix factorization and quantization of a floating point value of each weight to a floating point format. A decompression module is operable to obtain an approximation of the weight values by decompressing the compressed weight matrix through the decompression matrix. A plurality of cores executes the at least one layer of the convolutional neural network with the approximation of weight values to produce an inference output.

A system and method for encoding data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

A computer-implemented method, computer program product and computing system for: processing an unencoded data file to identify a plurality of file segments, wherein the unencoded data file is a dataset for use with a blockchain process; mapping each of the plurality of file segments to a portion of a dictionary file to generate a plurality of mappings that each include a starting location and a length, thus generating a related encoded data file based, at least in part, upon the plurality of mappings; receiving a request to manipulate the unencoded data file from the blockchain process; and processing the related encoded data file based, at least in part, upon the plurality of mappings and the dictionary file to generate a modified encoded data file that represents the requested manipulations of the unencoded data file.

A processing device for executing a machine learning neural network operation includes memory and a processor. The processor is configured to receive input data at a layer of the machine learning neural network operation, receive a plurality of sorted filters to be applied to the input data, apply the plurality of sorted filters to the input data to produce a plurality of different feature maps, compress the plurality of different feature maps according to a sparsity of the feature maps and store the plurality of different feature maps in the memory.