Exemplary surveillance system is provided having a plurality of data generating devices, and nodes for data handling of the data streams generated from the data generating devices. The system may be configured to fragment the data streams and store the fragments among the plurality of nodes of the system. The system may be configured to redundantly transmit data through the nodes so the fragmented data stream arrive at the desired location for storage. The data transmission may permit redirection or retransmission based on node or data transmission failure.

Techniques are provided for implementing intelligent defragmentation in a storage system. A storage control system manages a logical address space of a storage volume. The logical address space is partitioned into a plurality of extents, wherein each extent comprises a contiguous block of logical addresses of the logical address space. The storage control system monitors input/output (I/O) operations for logical addresses associated with the extents, and estimates fragmentation levels of the extents based on metadata associated with the monitored I/O operations. The storage control system identifies one or more extents as candidates for defragmentation based at least on the estimated fragmentation levels of the extents.

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

According to one embodiment, a memory system includes a compressor configured to output second data obtained by compressing input first data and a non-volatile memory to which third data based on the second data output from the compressor is written. The compressor includes a dictionary coding unit configured to perform dictionary coding on the first data, an entropy coding unit configured to perform entropy coding on the result of the dictionary coding, a first calculation unit configured to calculate compression efficiencies of the dictionary coding and the entropy coding, and a first control unit configured to control an operation of at least one of the dictionary coding unit and the entropy coding unit based on the compression efficiencies and a power reduction level.

A computer storage device having a host interface, a controller, non-volatile storage media, and firmware. The firmware instructs the controller to: receive, via the host interface, a request from a host to allocate a namespace of a quantity of non-volatile memory; generate, in response to the request, a namespace map identifying a plurality of blocks of addresses having a same predetermined block size, and a partial block of addresses having a size smaller than the predetermined block size; and convert, using the namespace map, logical addresses in the namespace communicated from the host to physical addresses for the quantity of the non-volatile memory. For example, the request for allocating the namespace can be in accordance with an NVMe protocol.

Systems and methods are described for resource-efficient memory deduplication and write-protection. In an example, a method includes receiving, by a computing device having a processor, a request to assess deduplication for a plurality of candidate files. The computing device may perform one or more iterative steps for deduplication. The iterative steps may include: receiving, from the plurality of candidate files, a candidate file that is not write-protected; determining, based on a predetermined Bernoulli distribution, a decision to write-protect the candidate file; rendering the candidate file as a write-protected candidate file; determining, based on a review of other candidate files from the plurality of candidate files, that the write-protected candidate file can be deduplicated; and deduplicating the write-protected candidate file.

A storage method includes determining layers that are to be merged in a database; determining a data block that is to be modified based on files of the layers that are to be merged in the database; reading a corresponding data block in the memory and reorganizing the read data block, according to a key corresponding to the data block that is to be modified; replacing the data block that is to be modified in the memory with the reorganized data block, when byte size of the reorganized data block does not exceed a first preset value; and storing the key and statistical information of the reorganized data block in a corresponding file of the database.

A data storage system and a global deduplication method thereof are provided. The data storage system includes multiple storage devices and one dispatch device. The dispatch device divides an original data corresponding to a data writing request into at least one data chunk. The dispatch device performs a summary calculation on one data chunk, so as to generate a representative value. The dispatch device performs a first distribution calculation on the representative value, so as to determine a destination location corresponding to the representative value. The dispatch device transmits the data chunk and the representative value to at least one destination storage device among the storage devices through a communication network according to the destination location. The at least one destination storage device checks the representative value, so as to determine whether to store the data chunk in a storage space of the at least one destination storage device.

One example method includes receiving data, partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups, deduplicating the data after the partitioning, packing unique data segments remaining after deduplicating into one or more compression regions, compressing the compression regions, and writing an object, that includes the compression regions, to a durable log. The deduplicating and compressing for a similarity group may be performed by a dedup-compression instances uniquely assigned to that similarity group.

Disclosed herein are system, method, and computer program product embodiments for managing data storage devices. In some embodiments, a server receives a request to store data in a first storage device. The server determines the storage space remaining in the first data storage device based on historical data associated with the first data storage device. The server further determines that the first data storage device will exceed its storage capacity based on a size of the data and the storage space remaining in the first data storage device. The server transfers a set of data stored in the first data storage device to a second storage device. Transferring the set of data causes the storage space remaining in the first storage device to be greater than or equal to the size of the data. The server stores the data in the first data storage device.