Preparing data for deduplication including: generating, by a storage system for a compressed data block, a padded compressed data block by padding the compressed data block to conform to a fixed block size, wherein the fixed block size is greater than a size of the compressed data block; storing, in the storage system, the padded compressed data block beginning at a block boundary of a storage device in the storage system; and performing block-based deduplication on the storage system, wherein the block-based deduplication determines whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system.

A data compression method based on sampling and estimation is provided. The method includes: receiving a piece of data; extracting N data regions from M data regions of the piece of data; examining a data redundancy ratio in the N data regions; and determining, according to a value of the data redundancy ratio, whether to compress the piece of data.

A method and system for improving tape drive memory storage is provided. The method includes receiving, by a storage tape drive, a data stream for storage. The data stream is passed through a non-volatile memory device (NVS2) of the storage tape drive. The data stream is divided into adjacent variable length data chunks and a chunk list file including similarity identifiers for each of the adjacent variable length data chunks is generated and stored within a (non-volatile memory device) NVS1. Duplicate data including duplicated data with respect to a group of data chunks of the adjacent variable length data chunks is identified and deleted from the NVS2 of the storage tape drive such that the group of data chunks remains within NVS2. The group of data chunks is written to a data storage tape cartridge. Pointers identifying each data chunk and an associated storage position are generated and stored.

A data management device includes a persistent storage and a processor. The persistent storage includes an object storage. The processor segments a file into file segments. The processor generates meta-data of the file segments. The processor stores a portion of the file segments in a data object of the object storage. The processor stores a portion of the meta-data of the file segments in a meta-data object of the object storage.

Preparing data for deduplication including: generating, by a storage system for a compressed data block, a padded compressed data block by padding the compressed data block to conform to a fixed block size, wherein the fixed block size is greater than a size of the compressed data block; storing, in the storage system, the padded compressed data block beginning at a block boundary of a storage device in the storage system; and performing block-based deduplication on the storage system, wherein the block-based deduplication determines whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system.

The present invention realizes a storage device that has a high data reduction effect without decreasing I/O performances. The storage device includes a processor, an accelerator, a memory, and a storage medium, the processor specifies data to be compressed that is data stored in the storage medium from data stored in the memory and transmits a compression instruction including information relating to the data to be compressed to the accelerator, and the accelerator reads the plurality of continuous items of data from the memory and compresses the plurality of items of data to be compressed obtained by excluding data that is not to be compressed from the plurality of items of data, based on the information relating to the data to be compressed received from the processor, to generate compressed data stored in the storage device.

Compression and decompression technology within a solid-state device (SSD) is disclosed that provides a good compression ratio while taking up less on-chip area. An input interface receives an input stream to be compressed. An output interface provides a compressed stream. A history buffer is of a fixed size that is a fraction of a size of a data buffer. Processing logic encodes into the compressed stream element types, literals and pointers, the latter which reference copies of data found elsewhere within the history buffer during compression. The history buffer may be multiple banks in width, where the data is loaded from the input stream sequentially across rows of the banks. The decompression side may be similarly designed, optionally with a different number of banks. The pointers may be a fixed two bytes including four bits for length and eleven bits for offset of back reference to a copy (or other combination).

An encoder for encoding input data to generate corresponding encoded data includes data processing hardware which is operable: to determine at least partial reoccurrences of data blocks or data packets within the input data, wherein the data blocks or data packets include a plurality of bytes; to employ at least one reference symbol to relate reoccurrences of mutually similar data blocks or data packets and/or to indicate whether or not there are reoccurrences of mutually similar data blocks or data packets within the input data; to employ a plurality of change symbols, for example a plurality of mask bits, to indicate changed and unchanged data elements of partial reoccurrences of data blocks or data packets within the input data and a change of data values of changed data elements; and to encode the at least one reference symbol and the plurality of change symbols into the encoded data.

A data compression device including a processor to perform a procedure comprising: obtaining data of a predetermined number (Z) of digits in a time series; and performing a compression process on the data. The data is obtained by encoding a vibration state of a measurement target. The compression process includes: deleting upper digits when the upper digits do not include significant information; and adding a unique code to a top of the upper digits when the upper digits include significant information. A digit number (X) of the upper digits is smaller than the predetermined number (Z).

An encoder for encoding input data (D1) to generate corresponding encoded data (E2) includes data processing hardware which is operable: (a) to determine at least partial reoccurrences of data blocks or data packets within the input data (D1), wherein the data blocks or data packets include a plurality of bytes; (b) to employ at least one reference symbol to relate reoccurrences of mutually similar data blocks or data packets and/or to indicate whether or not there are reoccurrences of mutually similar data blocks or data packets within the input data (D1); (c) to employ a plurality of change symbols, for example a plurality of mask bits, to indicate changed and unchanged data elements of partial reoccurrences of data blocks or data packets within the input data (D1) and a change of data values of changed data elements; and (d) to encode the at least one reference symbol and the plurality of change symbols into the encoded data (E2).

There are provided methods of using the encoder to encode input data (D1) to generate the corresponding encoded data (E2). Moreover, there are provided a corresponding decoder, and a corresponding method of decoding the encoded data (E2) to generate corresponding decoded data (D3).