Provided is a method and a device for the lossless compression of a data stream which includes a sequence of structured data objects which have a list of properties which each contain a key value pair, the method having the following steps: dividing the structured data objects of the data stream into a constant data object portion which has key value pairs with constant values and into variable data object portions which have key value pairs with variable values; transmitting the constant data object portion of the structured data objects once to a receiver; and transmitting the variable data object portions of the divided data objects of the data stream to the receiver.

Systems and methods are described for avoiding redundant data transfers using delta coding techniques when reliably and opportunistically communicating data to multiple user systems. According to embodiments, user systems track received block sequences for locally stored content blocks. An intermediate server intercepts content requests between user systems and target hosts, and deterministically chucks and fingerprints content data received in response to those requests. A fingerprint of a received content block is communicated to the requesting user system, and the user system determines based on the fingerprint whether the corresponding content block matches a content block that is already locally stored. If so, the user system returns a set of fingerprints representing a sequence of next content blocks that were previously stored after the matching content block. The intermediate server can then send only those content data blocks that are not already locally stored at the user system according to the returned set of fingerprints.

A system and method for global data compression. The method includes splitting a dataset into a plurality of blocks; for each block of the plurality of blocks: computing at least one similarity hash for the block; determining, based on the at least one similarity hash, whether a similar block is found for the block, wherein a similar block for a block has a similarity hash that is similar to one of the computed at least one similarity hash for the block; compressing the block by replacing data of the block with a reference to the similar block and a delta when a similar block is found, wherein the delta is a difference in data between the block and the similar block; and compressing the block independently when a similar block is not found.

A system, a method and a computer program product for storing data, which include receiving a data stream having a plurality of transactions that include at least one portion of data, determining whether at least one portion of data within at least one transaction is substantially similar to at least another portion of data within at least one transaction, clustering together at least one portion of data and at least another portion of data within at least one transaction, selecting one of at least one portion of data and at least another portion of data as a representative of at least one portion of data and at least another portion of data in the received data stream, and storing each representative of a portion of data from each transaction in the plurality of transactions, wherein a plurality of representatives is configured to form a chain representing the received data stream.

According to one aspect of the present application, a method for data compression comprises: creating a first trie for a first set of strings, the first set of strings comprising a plurality of raw data strings, wherein a trie consists of a plurality of nodes linked through parent-child relation, and wherein each edge of the trie is of at least one character and the edge corresponds to a state transition from a parent node of the edge to a child node of the edge; collecting edges of the first trie longer than a predetermined length and making these edges a first subset of strings of the first trie; segmenting a string in the first subset of strings into two or more fragments when the string satisfies a predetermined condition and collecting all segmented fragments and all un-segmented strings in the first subset of strings as a segmented set of strings; and storing the first set of strings using the first trie and the segmented set of strings so as to compress the raw data strings.

Certain aspects of the present disclosure provide techniques for increasing processor caching efficiency by cache data pattern optimization. One embodiment includes a method for managing data in a cache, including: receiving data to be cached at the cache; determining that the data to be cached matches a predefined data pattern; and updating a tag RAM associated with the cache with a pattern tag comprising tag bits and pattern bits, wherein the pattern bits match the predefined data pattern.

A system for compressing an input data stream to create a compressed output data stream is provided. The system comprises a memory storing a hash table comprising hash entries each comprising a hash value of an associated subset of following data items of an input data stream and a pointer to a memory location of the associated subset. A processor coupled to the memory executes operations while instructing an SIMD engine to execute concurrently one or more of the operations for consecutive subsets: calculate the hash value for each subset, search the hash table for a match of each calculated hash value and update the hash table according to the match result. The processor then updates the compressed output data stream according to the match result and a comparison result depending on the match result and operations for the plurality of associated subsets to create the compressed output data stream.

Techniques are provided for implementing additional compression for existing compressed data. Format information stored within a data block is evaluated to determine whether the data block is compressed or uncompressed. In response to the data block being compressed according to a first compression format, the data block is decompressed using the format information. The data block is compressed with one or more other data blocks to create compressed data having a second compression format different than the first compression format.

A communication apparatus includes a memory, and a processor coupled to the memory and the processor configured to receive communication data, measure a usable communication band width for communication with a destination apparatus, determine a parameter relating to a reduction ratio for a data amount of the communication data, based on the measured communication band width, reduce the data amount of the communication data, based on the determined parameter, and transmit the communication data of the reduced data amount to the destination apparatus.

An object of the present invention is to efficiently compress a plurality of kinds of data series with different sampling rates. A data compression device has a grouping unit and a compression unit. The grouping unit groups a plurality of kinds of data series with different sampling rates. The compression unit compresses the data series grouped by the grouping unit.