A system and method for compacting data that uses mismatch probability estimation to improve entropy encoding methods to account for, and efficiently handle, previously-unseen data in data to be compacted. Training data sets are analyzed to determine the frequency of occurrence of each sourceblock in the training data sets. A mismatch probability estimate is calculated comprising an estimated frequency at which any given data sourceblock received during encoding will not have a codeword in the codebook. Entropy encoding is used to generate codebooks comprising codewords for data sourceblocks based on the frequency of occurrence of each sourceblock. A “mismatch codeword” is inserted into the codebook based on the mismatch probability estimate to represent those cases when a block of data to be encoded does not have a codeword in the codebook. During encoding, if a mismatch occurs, a secondary encoding process is used to encode the mismatched sourceblock.

Decoding a bitset, each bit of the bitset corresponding to a respective value in a range of a minimum value to a maximum value, includes decoding, from a compressed bitstream, indexes of bits of the bitset, each bit of the bits having a first value. Decoding the bitset also includes setting all other bits of the bitset not decoded from the compressed bitstream to a second value. Decoding the indexes of bits of the bitset includes decoding a number of the indexes of the bits of the bitset, decoding a first index of the indexes in a first range having a first lower bound and a first upper bound, and decoding a last index of the indexes in a second range having a second lower bound and a second upper bound.

A method for decompressing data may include receiving a first sequence of bits and performing a plurality of iterations. Each of the plurality of iterations may include scanning bits of the first sequence, starting from a starting point, to search for at least one of a variable length codeword or a bypass indicator, the starting point being either a starting point of the first sequence or a starting point defined in a previous iteration. The method also include, for at least one of the plurality of iterations, when a bypass indicator is found, outputting a neural network coefficient related value (NNCRV) that is non-compressed and follows the bypass indicator, and defining a starting point that follows the NNCRV as a starting point for a next iteration.

A system and method for encoding data using a plurality of encoding libraries. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

A system and method for encoding anonymized dataset. A dataset may be pre-processed by dividing into a plurality of sourceblocks at all reasonable sourceblock lengths, and then counting how many times each sourceblock occurs in the dataset, resulting in a tally record of tokens and their count value. This tally record may then be anonymized and transmitted as an anonymized tally record to a data deconstruction engine which combined with a library manager creates a codebook and performs optimization techniques on the codebook. The received anonymized tally record may be parsed into individual tokens by identifying the tokens with the highest count value. The tokens may then be sent, in descending order of count value, to the library manger where each token may be assigned a codeword. Then a half-backed codebook is created using the tokens and each token's unique codeword, before sending the half-backed codebook to a system user.

A processor having a hardware decompressor configured to pad a non-equidistant data set, which is data received at irregular time intervals, with one or more of a predefined value, wherein the data is radar or optical sensor data; and a Fourier transform engine configured to receive the padded non-equidistant data set directly and continuously per data set from the hardware decompressor, and to FFT process the received padded non-equidistant data set.

Circuits, methods, and apparatus for efficiently implementing encoding and decoding between binary and multilevel data.

In various examples, metadata may be generated corresponding to compressed data streams that are compressed according to serial compression algorithms—such as arithmetic encoding, entropy encoding, etc.—in order to allow for parallel decompression of the compressed data. As a result, modification to the compressed data stream itself may not be required, and bandwidth and storage requirements of the system may be minimally impacted. In addition, by parallelizing the decompression, the system may benefit from faster decompression times while also reducing or entirely removing the adoption cycle for systems using the metadata for parallel decompression.

A hardware implementable lossless data compression decompression algorithm is disclosed, where the input data string is described in term of consecutive groups of alternating same type bits, where one of these groups of same type bits is defined as a preferred group with the other groups having either lower or higher number of same type bits, where the data string is partitioned into variable length processing strings where the variable length is determined by the occurrence of the preferred group or of a determined number of bits consisting of groups of lower number of same type bits, where these variable length processing strings are processed function of the configuration and content of each processing string only, where consecutive processing strings are additionally processed based on their content only, where processing is performed in a loop until a certain target performance is achieved, where processing is done without any data analysis, and where no negative compression gain is achieved for any content of an input string.

A multichannel data packer includes a plurality of two-input multiplexers and a controller. The plurality of two-input multiplexers is arranged in 2.sup.N rows and N columns in which N is an integer greater than 1. Each input of a multiplexer in a first column receives a respective bit stream of 2.sup.N channels of bit streams. Each respective bit stream includes a bit-stream length based on data in the bit stream. The multiplexers in a last column output 2.sup.N channels of packed bit streams each having a same bit-stream length. The controller controls the plurality of multiplexers so that the multiplexers in the last column output the 2.sup.N channels of bit streams that each has the same bit-stream length.