A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.

Techniques are disclosed for compressing data. The techniques include identifying, in data to be compressed, a first set of values, wherein the first set of values include a first number of two or more consecutive identical non-zero values; including, in compressed data, a first control value indicating the first number of non-zero values and a first data item corresponding to the consecutive identical non-zero values; identifying, in the data to be compressed, a second value having an exponent value included in a defined set of exponent values; including, in the compressed data, a second control value indicating the exponent value and a second data item corresponding to a portion of the second value other than the exponent value; and including, in the compressed data, a third control value indicating a third set of one or more consecutive zero values in the data to be compressed.

Techniques for generating data sets may include: receiving an initial buffer that achieves a compression ratio responsive to compression processing using a compression algorithm, the initial buffer including first content located at a first position in the initial buffer and including second content located at a second position in the initial buffer; and generating a data set of buffers using the initial buffer. The data set may be expected to achieve a specified deduplication ratio responsive to deduplication processing and to achieve the compression ratio responsive to compression processing using the compression algorithm. Generating the data set may include generating a first plurality of buffers where each buffer of the first plurality is not a duplicate of another buffer in the first plurality, and generating a second plurality of duplicate buffers. Each duplicate buffer may be a duplicate of a buffer in the first plurality of buffers.

Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for data compression. The method includes: determining an amount of data to be compressed in a storage system; determining, based on the amount of the data to be compressed, a target compression level for compressing the data to be compressed; and compressing the data to be compressed according to the target compression level. In this way, it is possible to compress data to be compressed using a compression level corresponding to the amount of the data to be compressed, thereby improving the efficiency of data compression in the storage system.

A method, system and computer-readable storage medium for transferring data segments from one computer system to a second computing system. Prior to transfer of the data segments, the first system calculates compressibility ratio of each segment and compares the compressibility ratio to a preset threshold. Based on the comparison, the first system assigns a compressibility hint to each segment. The first system transfers the segments to the second system, together with the corresponding compressibility hint. The second system stores each segment in a compressible region or in a non-compressible region based on the hint. Then the second system compresses the compressible region and stores the compressed region in a container, and stores the non-compressible region uncompressed in the container.

Dynamic compression with dynamic multi-stage encryption for a data storage system in accordance with the present description includes, in one aspect of the present description, preserves end-to-end encryption between a host and a storage controller while compressing data which was received from the host in encrypted but uncompressed form, using MIPs and other processing resources of the storage controller instead of the host. In one embodiment, the storage controller decrypts encrypted but uncompressed data received from the host to unencrypted data and compresses the unencrypted data to compressed data. The storage controller then encrypts the compressed data to encrypted, compressed data and stores the encrypted, compressed data in a storage device controlled by the storage controller. Other aspects and advantages may be realized, depending upon the particular application.

One embodiment provides a computer implemented method of data compression including segmenting user data into data segments; deduplicating the data segments to form deduped data segments; compressing the deduped data segments into compression units using a hardware accelerator; packing the compression units into compression regions; and packing the compression regions into one or more containers.

Techniques are provided for implementing additional compression for existing compressed data. Format information stored within a data block is evaluated to determine whether the data block is compressed or uncompressed. In response to the data block being compressed according to a first compression format, the data block is decompressed using the format information. The data block is compressed with one or more other data blocks to create compressed data having a second compression format different than the first compression format.

A compression engine calculates replacement CRC codes, in predetermined data lengths, for DIF-in cleartext data including cleartext data and multiple CRC codes based on the cleartext data. The compression engine generates headered compressed-text data in which a header including the replacement CRC codes is added to compressed-text data in which the cleartext data is compressed, and generates code-in compressed-text data by calculating multiple CRC codes based on the headered compressed-text data to add the calculated CRC codes to the headered compressed-text data.

Systems and methods are described for avoiding redundant data transfers using delta coding techniques when reliably and opportunistically communicating data to multiple user systems. According to embodiments, user systems track received block sequences for locally stored content blocks. An intermediate server intercepts content requests between user systems and target hosts, and deterministically chucks and fingerprints content data received in response to those requests. A fingerprint of a received content block is communicated to the requesting user system, and the user system determines based on the fingerprint whether the corresponding content block matches a content block that is already locally stored. If so, the user system returns a set of fingerprints representing a sequence of next content blocks that were previously stored after the matching content block. The intermediate server can then send only those content data blocks that are not already locally stored at the user system according to the returned set of fingerprints.