Examples described herein relate to performing data compression by performing dictionary matching of data using hardware circuitry to generate dictionary matched results and post-processing of dictionary matched results using software executed by a processor. In some examples, dictionary matching includes LZ77 dictionary matching. In some examples, dictionary matching occurs on multiple segments of data in parallel.

There is provided with a data processing apparatus. An acquisition unit acquires feature plane data of a layer included in a neural network. A control unit outputs a first control signal corresponding to the layer for controlling first compression processing and a second control signal corresponding to the layer for controlling second compression processing. A first compression unit performs the first compression processing corresponding to the first control signal on the feature plane data. A second compression unit performs the second compression processing corresponding to the second control signal on the feature plane data after the first compression processing. A type of processing of the second compression processing is different from the first compression processing.

A method of compressing weights of a neural network includes compressing a weight set including the weights of a the neural network, determining modified weight sets by changing at least one of the weights, calculating compression efficiency values for the determined modified weight sets based on a result of compressing the weight set and results of compressing the determined modified weight sets, determining a target weight of the weights satisfying a compression efficiency condition among the weights based on the calculated compression efficiency values, and determining a final compression result by compressing the weights based on a result of replacing the determined target weight.

A method for compressing columnar data may include generating, for a data column included in a data chunk, a dictionary enumerating, in a sorted order, a first set of unique values included in the first data column. A compression technique for generated a compressed representation of the data column having a fewest quantity of bytes may be identified based at least on the dictionary. The compression technique including a dictionary compression applying the dictionary and/or another compression technique. A compressed data chunk may be generated by applying the compression technique to compress the data column included in the data chunk. The compressed data chunk may be stored at a database in a variable-size persistent page whose size is allocated based on the size of the compressed representation of the data column. Related systems and articles of manufacture are also provided.

A disclosure for lossless data compression can include receiving a data block by a processor, performing, by the processor, a sparse transform extraction on the data block, selecting, by the processor, a transform matrix for the data block, modeling, by the processor, the selected transform matrix for the data block, selecting, by the processor, a transform coefficient model for the data block, modeling, by the processor, the selected transform coefficient model for the data block, compressing, by the processor, the data in the data block using the selected transform matrix and the selected transform coefficient model.

A disclosure for lossless data compression can include receiving a data block by a processor, performing, by the processor, a sparse transform extraction on the data block, selecting, by the processor, a transform matrix for the data block, modeling, by the processor, the selected transform matrix for the data block, selecting, by the processor, a transform coefficient model for the data block, modeling, by the processor, the selected transform coefficient model for the data block, compressing, by the processor, the data in the data block using the selected transform matrix and the selected transform coefficient model.

A weight data compression method includes: generating a 4-bit data string of 4-bit data items each expressed as any one of nine 4-bit values, by dividing ternary weight data into data items each having 4 bits; and generating first compressed data including a first flag value string and a first non-zero value string by (0) generating the first flag value string by assigning one of 0 and 1 as a first flag value of a 1-bit flag to a 4-bit data item 0000 and assigning an other of 0 and 1 as a second flag value of the 1-bit flag to a 4-bit data item other than 0000 among the 4-bit data items in the 4-bit data string and (ii) generating the first non-zero value string by converting the 4-bit data item other than 0000 into a 3-bit data item having any one of eight 3-bit values.

A method of compressing weights of a neural network includes compressing a weight set including the weights of a the neural network, determining modified weight sets by changing at least one of the weights, calculating compression efficiency values for the determined modified weight sets based on a result of compressing the weight set and results of compressing the determined modified weight sets, determining a target weight of the weights satisfying a compression efficiency condition among the weights based on the calculated compression efficiency values, and determining a final compression result by compressing the weights based on a result of replacing the determined target weight.

A method of encoding input data includes dividing the input data into a plurality of data packets, an input packet of the plurality of data packets including a plurality of digits in a first base system, base-converting the input packet from the first base system to generate a base-converted packet including a plurality of converted digits in a second base system, the second base system having a base value lower than that of the first base system, and incrementing the converted digits to generate a coded packet for transmission through a communication channel.

A method of encoding input data includes dividing the input data into a plurality of data packets, an input packet of the plurality of data packets including a plurality of digits in a first base system, base-converting the input packet from the first base system to generate a base-converted packet including a plurality of converted digits in a second base system, the second base system having a base value lower than that of the first base system, and incrementing the converted digits to generate a coded packet for transmission through a communication channel.