One example method includes receiving data, partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups, deduplicating the data after the partitioning, packing unique data segments remaining after deduplicating into one or more compression regions, compressing the compression regions, and writing an object, that includes the compression regions, to a durable log. The deduplicating and compressing for a similarity group may be performed by a dedup-compression instances uniquely assigned to that similarity group.

A system and method for differential compression are disclosed. In the disclosure, input data blocks selected for compression are received. For each input data block: the input data block is divided into at least two segments. For each of the at least two segments, a similarity degree between the respective segment and each of the data blocks excluding the respective data block is computed. For each of the at least two segments, the data block which has a biggest similarity degree with the respective segment among the data blocks excluding the respective data block is selected as an optimal reference data block for the respective segment. The differential compression is applied to the input data block and optimal reference blocks in response to determine a differential compression is to be applied based on the similarity degree between the segments of the input data block and the corresponding optimal reference blocks.

A system and method for storing compressed data in a memory system includes identifying user data to be compressed and compressing pages of user data to form data extents that are less than or equal to the uncompressed data. A plurality of compressed pages are combined to a least fill a page of the memory. The data may be stored as sectors of a page, where each sector includes a CRC or error correcting code for the compressed data of that sector. The stored data may also include error correcting code data for the uncompressed page and error correcting code for the compressed page. When data is read in response to a user request, the sector data is validated using the CRC prior to selecting the data from the read sectors for decompression, and the error correcting code for the uncompressed page may be used to validate the decompressed data.

The present invention discloses a data transmission method having data reuse mechanism that includes the steps outlined below. A driver corresponding to a communication circuit is operated as a transmission terminal to analyze under-transmitted data to generate reuse setting information, indication information and packets having a complete packet and incomplete packets so as to be transmitted by the transmission terminal through a transmission interface to be received by the communication circuit as a receiving terminal. The receiving terminal identifies the complete packet and the incomplete packets according to the indication information. A data location that a reusable data section corresponds to is determined according to the reuse information. The complete packet is outputted. A non-reusable data section of each of the incomplete packets and the reusable data section of the complete packet are reconstructed to output reconstructed packets according to the data location.

In one embodiment a system, apparatus, and method for optimizing index value lengths when indexing data items in an array of data items is described, the method including producing, at a first processor, an ordered series of index values, sending the ordered series of index values to an indexing processor, receiving, at the indexing processor, a data object including the array of data items, associating, at the indexing processor, a first part of one of the index values with a first one data item of the array of data items, associating, at the indexing processor, a second part of the one of the index values with a next one data item of the array of data items, repeating the steps of associating a first part of one of the index values and associating a second part of the one of the index values until all of the data items in the array of data items are indexed.

Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each small data chunk, a signature is generated based on a combination of a representation of characters used in selecting data to be deduplicated. A c-spectrum of the small data chunk being a sequence of representations of different characters ordered by a frequency of occurrence in the small data chunk, and an f-spectrum of the small data chunk being a corresponding sequence of frequencies of the different characters in the small data chunk.

A storage aware memory controller and method for managing a physical storage system. A described controller includes: a system for mapping physical memory space into a memory region and a storage region; a system for applying different error protections schemes, in which a fine-grained memory fault tolerance scheme is applied to data in the memory region and a course-grained memory fault tolerance scheme is applied to data in the storage region; and an in-memory storage filesystem that compresses and stores individual pages of data in the storage region, wherein each page of data is compressed into a set of codewords that are codeword aligned such that no codeword shares compressed data from different pages, and wherein the in-memory storage filesystem stores a compression-aware logical block address (CA-LBA) for each page of data.

A computing device may identify a plurality of equally sized data blocks of a first navigation file and may identify a plurality of equally sized data blocks of a second navigation file. The computing device may perform binary difference operations between the data blocks of the first navigation file to corresponding data blocks of the second navigation file The result of the binary difference operations is stored in a plurality of navigation patch files. At least two of the plurality of navigation patch files are merged into a merged navigation patch file.

A system, a method and a computer program product for storing data, which include receiving a data stream having a plurality of transactions that include at least one portion of data, determining whether at least one portion of data within at least one transaction is substantially similar to at least another portion of data within at least one transaction, clustering together at least one portion of data and at least another portion of data within at least one transaction, selecting one of at least one portion of data and at least another portion of data as a representative of at least one portion of data and at least another portion of data in the received data stream, and storing each representative of a portion of data from each transaction in the plurality of transactions, wherein a plurality of representatives is configured to form a chain representing the received data stream.

Preparing data for deduplication including in response to receiving a request to transfer data from a source storage system to a target storage system, accessing, by the source storage system, a compressed data block; generating, by the source storage system, a padded compressed data block by padding the compressed data block to conform to a fixed block size, wherein the fixed block size is greater than a size of the compressed data block; and sending, by the source storage system, the padded compressed data block to the target storage system.