The generation of symbol-encoded data from digital data, as part of the compression of the digital data into a compressed digital data, can be performed with reference to multiple alternative alphabets. A selection of a specific alphabet is made based on the digital data being compressed, the compression parameters, or combinations thereof. Information indicative of the selected alphabet is encoded into one or more headers of the resulting compressed digital data. A single alphabet can be selected for all of a set of digital data being compressed, or multiple different alphabets can be selected, with different ones of the multiple different alphabets being utilized to compress different portions of the digital data. Additionally, rather than explicitly specifying a specific selected alphabet, the header information can comprise information from which a same alphabet can be independently selected heuristically by both the compressor and the corresponding decompressor.

A system and method for codebook-based data encoding. Portions of the data are encoded by different encoding libraries, depending on which library provides the greatest compaction for a given portion of the data. This methodology not only provides substantial improvements in data compaction over use of a single data compaction algorithm with the highest average compaction, but provides substantial additional security in that multiple decoding libraries must be used to decode the data. In some embodiments, each portion of data may further be encoded using different sourceblock sizes, providing further security enhancements as decoding requires multiple decoding libraries and knowledge of the sourceblock size used for each portion of the data. In some embodiments, encoding libraries may be randomly or pseudo-randomly rotated to provide additional security.

A system and method for distributed node-based data compaction. The system uses machine learning on data chunks to generate codebooks which compact the data to be stored, processed, or sent with a smaller data profile than uncompacted data. The system uses a data compaction in an existing blockchain fork or implemented in a new blockchain protocol from which nodes that wish to or need to use the blockchain can do so with a reduced storage requirement. The system uses network data compaction across all nodes to increase the speed of and decrease the size of a blockchain's data packets. The system uses data compaction firmware to increase the efficiency at which mining rigs can computationally validate new blocks on the blockchain. The system can be implemented using any combination of the three data compaction services to meet the needs of the desired blockchain technology.

A computer architecture for graph-traversal provides a processor for bottom-up sequencing through the graph data according to vertex degree. This ordered sequencing reduces redundant edge checks. In one embodiment, vertex adjacency data describing the graph may be allocated among different memory structures in the memory hierarchy to provide faster access to vertex data associated with vertices of higher degree reducing data access time. The adjacency data also may be coded to provide higher compression in memory of vertex data having high vertex degree.

Methods and apparatus are described by which data is compressed using semi-dynamic Huffman code generation. Embodiments generate symbol statistics over a portion of data. The symbol statistics are expanded to include all possible literals that could appear within the data. Any literal or reference added to the statistics may be given a frequency of one. The statistics are used to generate a semi-dynamic Huffman code. The entire data is then compressed using the semi-dynamic Huffman code.

An encoder for encoding input data to generate corresponding encoded data is provided. The encoder (10) is operable to process a sequence of elements in the input data from a first element thereof to a last element thereof. The elements have corresponding symbols. The encoder is operable to compute probabilities of symbols present in the sequence, wherein the probabilities of the symbols are computed while disregarding those elements of the sequence that have already been encoded into the encoded data. Moreover, the probabilities of the symbols are adaptively changed as the sequence of elements is progressively encoded into the encoded data. Furthermore, information describing the probabilities is delivered; optionally, the probabilities of the symbols are accompanied with additional information indicating how the probabilities are adaptively changing. There is also provided a decoder for performing an inverse of encoding performed by the encoder.

A system and method for data compression with intrusion detection, that measures in real-time the probability distribution of an encoded data stream, compares the probability distribution to a reference probability distribution, and uses one or more statistical algorithms to determine the divergence between the two sets of probability distributions to determine if an unusual distribution is the result of a data intrusion. The system comprises both encoding and decoding machines, an intrusion detection module, a codebook training module, and various databases which perform various analyses on encoded data streams.

An encoder for encoding input data to generate corresponding encoded data is provided. The encoder (10) is operable to process a sequence of elements in the input data from a first element thereof to a last element thereof. The elements have corresponding symbols. The encoder is operable to compute probabilities of symbols present in the sequence, wherein the probabilities of the symbols are computed whilst disregarding those elements of the sequence that have already been encoded into the encoded data. Moreover, the probabilities of the symbols are adaptively changed as the sequence of elements is progressively encoded into the encoded data. Furthermore, information describing the probabilities is delivered; optionally, the probabilities of the symbols are accompanied with additional information indicating how the probabilities are adaptively changing. There is also provided a decoder for performing an inverse of encoding performed by the encoder.

A data encoding method for encoding an array of data values as data sets and escape codes for values not encoded by the data sets, an escape code including a unary coded portion and a non-unary coded portion, the method including: setting a coding parameter defining a minimum number of bits of a non-unary coded portion; adding an offset value of 1 or more to the coding parameter to define a minimum least significant data portion size; generating one or more data sets indicative of positions, relative to the array of data values, of data values of predetermined magnitude ranges, to encode the value of at least one least significant bit of each data value; generating respective complementary most-significant data portions and least-significant data portions; encoding the data sets; encoding the most significant data portions; and encoding the least-significant portions.

An embodiment of a semiconductor package apparatus may include technology to load compressed symbols in a data stream into a first content accessible memory, break a serial dependency of the compressed symbols in the compressed data stream, and decode more than one symbol per clock. Other embodiments are disclosed and claimed.