In one approach, filesets to be backed up are divided into partitions and snapshots are pulled for each partition. In one architecture, a data management and storage (DMS) cluster includes a plurality of peer DMS nodes and a distributed data store implemented across the peer DMS nodes. One of the peer DMS nodes receives fileset metadata for the fileset and defines a plurality of partitions for the fileset based on the fileset metadata. The peer DMS nodes operate autonomously to execute jobs to pull snapshots for each of the partitions and to store the snapshots of the partitions in the distributed data store.

As the volume of data under management expands rapidly, so do the costs associated with storing and that data on secondary storage devices. The illustrative approach provides an improvement to the information management system by delaying certain tasks that meet a set of criteria until a specified threshold is met. The system receives a request to be performed on a set of data stored on secondary devices. Power management module determines whether the task satisfies a set of criteria for delayed execution, queues the task, and when a specified threshold of the queued tasks is met powers up the necessary components to execute the tasks.

Example object storage systems, bookkeeping engines, and methods provide quota usage monitoring for control entities, such as accounts, users, and buckets. An object data store is configured to enable control entities to access data objects associated with each control entity. Data objects are mapped to the control entities and the data objects are processed to identify object usage values corresponding to each combination of data object and control entity. Total usage values are calculated for each control entity and used to determine a data object access response for a target data object and associated control entities.

Technologies are provided to ensure integrity of erasure coded data that is subject to read and write access from distributed processes. Multiple processes that access erasure coded data can be coordinated in an efficient, scalable and fault-tolerant manner so that integrity of the original data is maintained. The Technologies include a fault-tolerant access coordination protocol that ensures exclusive write access by a client. The coordination protocol achieves scalability by not relying on centralized components, and achieves efficiency and performance by piggy-packing access coordination messages on operations of the underlying erasure coding protocol.

Various embodiments set forth techniques for free space management in a block store. The techniques include receiving a request to allocate one or more blocks in a block store, accessing a sparse hierarchical data structure to identify an allocator page identifying a region of a backing store having a greatest number of free blocks, and allocating the one or more blocks.

A method for synchronous replication of stream data includes receiving a stream of data blocks for storage at a first storage location associated with a first geographical region and at a second storage location associated with a second geographical region. The method also includes synchronously writing the stream of data blocks to the first storage location and to the second storage location. While synchronously writing the stream of data blocks, the method includes determining an unrecoverable failure at the second storage location. The method also includes determining a failure point in the writing of the stream of data blocks that demarcates data blocks that were successfully written and not successfully written to the second storage location. The method also includes synchronously writing, starting at the failure point, the stream of data blocks to the first storage location and to a third storage location associated with a third geographical region.

A method for execution by a dispersed storage network (DSN) managing unit includes receiving access information from a plurality of distributed storage and task (DST) processing units via a network. Cache memory utilization data is generated based on the access information. Configuration instructions are generated for transmission via the network to the plurality of DST processing units based on the cache memory utilization data.

In some examples, fabric driven NVMe subsystem zoning may include receiving, from a non-volatile memory express (NVMe) Name Server (NNS), a zoning specification that includes an indication of a host that is to communicate with a given NVMe subsystem of an NVMe storage domain. Based on the zoning specification, the host may be designated as being permitted to connect to the given NVMe subsystem of the NVMe storage domain. An NVMe connect command may be received from the host. Based on the designation and an analysis of the NVMe connect command, a connection may be established between the given NVMe subsystem of the NVMe storage domain and the host.

Systems and methods for performing data protection operations including garbage collection operations and copy forward operations. For deduplicated data stored in a cloud-based storage or in a cloud tier that stores containers containing dead and live segments or dead and live regions such as compression regions, the dead compression regions are deleted by copying the live compression regions into new containers and then deleting the old containers. The copy forward is based on a recipe from a data protection system and is performed using a serverless approach.

Enabling a protocol for efficiently and reliably using the NVME protocol over a network, referred to as NVME over Network, or NVMEoN, may include an NVMEoN exchange layer for handling exchanges between initiating and target nodes on a network, a burst transmission protocol that provides guaranteed delivery without duplicate retransmission, and an exchange status block approach to manage state information about exchanges.