Systems for accessing client data is described. A request to access a first data block is received. The request indicates a first logical address referencing the first data block. First mapping data is employed to identify a first physical addresses corresponding to the first logical addresses. The first mapping data encodes a first LOM transaction ID and candidate local addresses. The first mapping data is employed to identify the candidate local address and the first LOM transaction ID. A usage table is employed to determine the current status of the first LOM transaction ID. The candidate local address is employed to access the first data block. Second mapping data is employed to identify an updated local address of the set of local addresses. The updated local address currently references the first data block. The updated local address is employed to access the first data block.

One embodiment provides a system which facilitates data movement. The system allocates, in a volatile memory of a first storage drive, a first region to be accessed directly by a second storage drive or a first NIC. The first storage drive, the second storage drive, and the first NIC are associated with a first server. The system stores data in the first region. Responsive to receiving a first request from the second storage drive to read the data, the system transmits, by the first storage drive to the second storage drive, the data stored in the first region while bypassing a system memory of the first server. Responsive to receiving, from a third storage drive associated with a second server, a second request to read the data, the system retrieves, by the first NIC, the data stored in the first region while bypassing the system memory of the first server.

Example apparatus and methods produce a set of rateless erasure codes (e.g., fountain codes) for a file stored in a primary data store (e.g., hard drive) or in an archive system. The archive system may store the file in a redundant array of independent disks (RAID). A first subset of the rateless erasure codes are stored in an object storage using a synchronous protocol. A second subset of rateless erasure codes are stored in the object storage using an asynchronous protocol. The object storage system may inform the archive system when desired redundancy has been achieved or when desired redundancy has been lost. The archive system may buffer rateless erasure codes before providing the codes to the object storage to improve performance. A failure in the archive system or object storage system may be mitigated by retaining the file in the primary data store until the desired redundancy is achieved.

A method begins by a computing device of a dispersed storage network (DSN) sending, to a set of storage units of the DSN, a set of write requests regarding a set of encoded data slices, where each write request includes a write set information table that includes a listing of which storage unit of the set of storage units is being sent which encoded data slice of the set of encoded data slices for storage. The method continues with at least some of the storage units receiving write requests of the set of write requests. The method continues with one of the storage units interpreting the write set information table to determine that a particular encoded data slice assigned to a particular storage unit should be stored by a different storage unit and facilitating storing of the particular encoded data slice in the different storage unit.

Techniques described and suggested herein include systems and methods for minimizing inter-facility data transfer during retrieval of data archives stored on data storage systems using redundancy coding techniques. For example, redundancy coded shards, which may include identity shards that contain unencoded original data of archives, may be configured such that a variable number of the shards can be leveraged to meet performance requirements or time-to-retrieval limitations for retrieval requests associated with the archives stored and/or encoded therein. Under some circumstances, implementing systems may monitor throughput rates, capabilities, and burdens, so as to preferentially retrieve data such that the identity shards are favored and fewer hosting data storage facilities are used for a given retrieval.

method and apparatus for synchronizing replicated data in a storage network. In an embodiment, a method begins by a processing module of a computing device identifying a first storage set and a second storage set for replicated storage of a data object. The processing module initiates storage of the data object in both the first and second storage sets, and further maintains a synchronization status for the data object. The processing module determines, based at least in part on the synchronization status, to resynchronize the first storage set and the second storage set. In response to determining to resynchronize the first storage set and the second storage set, the processing module identifies a latest available revision of the data object, determines that the second storage set requires the latest available revision of the data object to maintain synchronization, and facilitates storage of the identified latest available revision of the data object in the second storage set.

A system and method is disclosed for fast parallel data processing in a distributed storage system. An example method includes receiving at least one digital object; determining, by a processor of a computing device, whether the at least one digital object has a fixed data block structure; in response to determining that the at least one digital object has a fixed data block structure, determining by the processor a size of each fixed data block of the at least one digital object; determining a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block; and partitioning the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.

A method begins by a dispersed storage (DS) processing module receiving one or more pairs of coded values and continues with the DS processing module creating a received coded matrix from the one or more pairs of coded values, followed by the generation of a data matrix. When a decode threshold number of pairs of coded values are included in a first plurality of data blocks corresponding to a data stream from the data matrix and the decode threshold number of pairs of coded values are determined to be valid, the decode threshold number of pairs of coded values are decoded. The method continues with the generation of a second data matrix and the data stream is extracted from the second data matrix.

Embodiments use data shuttle devices to restore erasure-coded data in a distributed storage environment. In some embodiments, a first data shuttle is communicatively coupled to a first node of the storage environment. On the data shuttle, first restoration data is generated from a first erasure-coded data portion stored on the first node. The first data shuttle or a second data shuttle is communicatively coupled to a second node of the storage environment. On the data shuttle at the second node, second restoration data is generated from a second erasure-coded data portion stored on the second node. Subsequent to transporting the first or second data shuttle from at least one of the other nodes to a third node, a third erasure-coded data portion is restored at the third node. The third erasure-coded data portion is generated via an erasure-coding process from one or more of the first or second restoration data.

An encoder receives a sequence of packets. For each packet, the encoder selects a window of at least previous packets in the sequence of packets. The encoder identifies in the window one or more earlier packets among the previous packets and one or more later packets separated from the one or more earlier packets by a gap including gap packets. The encoder encodes the one or more earlier packets and the one or more later packets into a forward error correction (FEC) packet corresponding to the packet, without using any of the gap packets, and transmits the FEC packet and the packet.