The described technology is generally directed towards reducing the amount of data stored in a sequence of data blocks by combining deduplication and compression. According to an embodiment, a system can comprise a memory that can store computer executable components, and a processor that can execute the components stored in the memory. The components can comprise a data block identifier that can identify, for a sequence of data blocks, a first data block that corresponds to a first data, resulting in a first identified data block, and a deduplication component that can identify a second data block that corresponds to the first data, resulting in a second identified data block, wherein the deduplication component can replace the second identified data block with a key value corresponding to the first identified data block. Further, a compression component can compress the first identified data block, resulting in a compressed data block.

One embodiment provides a computer implemented method of data compression including segmenting user data into data segments; deduplicating the data segments to form deduped data segments; compressing the deduped data segments into compression units using a hardware accelerator; packing the compression units into compression regions; and packing the compression regions into one or more containers.

An aspect of managing data block compression in a storage system includes performing, for each block written to the storage system: bit-wise traversing the block, searching the block for a pattern indicating a repeating sequence of bits and, upon determining the pattern exists in the block and the repeating sequence of bits in the pattern exceeds a threshold value, removing the repeating sequence of bits from the block thereby yielding a reduced-size block.

A conditional construction container (CCC) provides seamless restore operation during abort event. During normal restore, the CCC assumes a hold state, wherein data blocks are sent from the backup source to the restore host directly. When interrupt occurs, the CCC assumes compression mode, wherein the CCC compresses the data blocks and stores the compressed blocks in memory allocated to the CCC. When the restore operation can resume, the backup agent instructs the CCC to begin data reconstruction operation and the CCC will assume busy mode no longer accepting new data blocks. During the busy state the CCC decompresses the compressed data blocks from the container and reconstruct the original data onto the restore host. Since the container constructed in the memory allocated to the CCC contain data that was obtained upon the interrupt, the restore operation resumes by reconstructing the data from the point of the interrupt seamlessly.

A method of data management in a data storage system including a plurality of data storage units, the method includes dividing each file in the data storage system into a plurality of blocks having a common size. The method further includes generating a hash value for each block using a common hash algorithm. The method further includes identifying a plurality of similar files in different data storage units, based on a comparison of the hashes for each file. The method further includes copying one or more of the identified similar files, such that similar files are stored in a single data storage unit and for one or more of the copied files, generating a link to a new location of the file. The method provides an efficient, effective, and adequate reduction of the primary storage as well as the secondary storage space.

Method for encoding of quality values of a data structure, whereby said data structure comprises a set of genomic reads, wherein the method comprises the following steps executable by a data processing system: ascertain the quality values of each read covering a certain index locus, determine a codebook identifier identifying a specific codebook from a plurality of codebooks for said certain index locus based on the ascertained quality values of said certain index locus, whereby each code-book provides a mapping from a quality value of said quality value alphabet to a corresponding quantized quality value of a quantized quality value alphabet, quantizing all ascertained quality values at said certain index locus using the specific codebook identified by the codebook identifier at said certain index locus in order to obtain for each quality value at said certain index locus a corresponding quantized quality value, and encode all determined codebook identifiers using a first entropy encoder and encode all quantized quality values using a second entropy encoder or a set of encoders.

In some implementations, a system can optimize offline map data updates. For example, a server device in the system can determine a metric for identifying map data objects based on attributes of the map data objects. The server device can then generate a quadtree that stores the map data objects in nodes of the quadtree based on the metric. When processing an update to the map data stored at the server device, the server device can generate update data describing the updates for each node in the quadtree based on a binary difference algorithm and/or a semantic difference algorithm. The server device can select the algorithm based on which algorithm results in the smallest compressed size of the update data.

A method of and system of for compressing and decompressing a localized software resource is disclosed. The method may include receiving a software resource, the software resource being in a first language, receiving a localized software resource for compression, where the software resource in the first language is a counterpart of the localized software resource in the second language. Upon receiving the software resources creating a first local dictionary for the localized software resource based at least in part on one or more first language words in the software resource and on data from a global dictionary, and compressing the localized software resource based on the local dictionary.

A communication apparatus includes a memory, and a processor coupled to the memory and the processor configured to receive communication data, measure a usable communication band width for communication with a destination apparatus, determine a parameter relating to a reduction ratio for a data amount of the communication data, based on the measured communication band width, reduce the data amount of the communication data, based on the determined parameter, and transmit the communication data of the reduced data amount to the destination apparatus.

Dynamic compression with dynamic multi-stage encryption for a data storage system in accordance with the present description includes, in one aspect of the present description, preserves end-to-end encryption between a host and a storage controller while compressing data which was received from the host in encrypted but uncompressed form, using MIPs and other processing resources of the storage controller instead of the host. In one embodiment, the storage controller decrypts encrypted but uncompressed data received from the host to unencrypted data and compresses the unencrypted data to compressed data. The storage controller then encrypts the compressed data to encrypted, compressed data and stores the encrypted, compressed data in a storage device controlled by the storage controller. Other aspects and advantages may be realized, depending upon the particular application.