Systems and techniques for managing data storage are disclosed. In some aspects, a front-end node responds to a request to write an object by dividing the object into multiple source data segments. The front-end node generates redundancy data for the multiple source data segments using a rateless erasure encoding. The front-end node associates a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment. The rateless erasure encoding further includes defining multiple segment-level fragments within each of the encoded segments. The front-end node transmits each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes. For each of the received encoded segments, the storage node generates one or more protection fragments based on redundancy data generated from the segment-level fragments and stores the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.

A system and method is disclosed for fast parallel data processing in a distributed storage system. An example method includes receiving at least one digital object; determining, by a processor of a computing device, whether the at least one digital object has a fixed data block structure; in response to determining that the at least one digital object has a fixed data block structure, determining by the processor a size of each fixed data block of the at least one digital object; determining a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block; and partitioning the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.

Disclosed embodiments of the present disclosure relate, generally, to systems, methods, and devices for correction of burst-errors induced during transmission of encoded blocks of information. Some embodiments relate to decoders configured to test candidate corrections on a received block of information and select a candidate correction that best fits the characteristics of burst-errors expected for a type of transmission scheme. Such tested candidate corrections may be selected based on characteristics of burst-errors typically induced for a type of transmission scheme. Some embodiments relate to decoders configured to test candidate corrections for correcting burst-errors and perform standard error correcting techniques such as Reed-Solomon forward error correction techniques. Some embodiments relate to systems, such as serial/deserializer interfaces, that incorporate such decoders.

Embodiments use data shuttle devices to restore erasure-coded data in a distributed storage environment. In some embodiments, a first data shuttle is communicatively coupled to a first node of the storage environment. On the data shuttle, first restoration data is generated from a first erasure-coded data portion stored on the first node. The first data shuttle or a second data shuttle is communicatively coupled to a second node of the storage environment. On the data shuttle at the second node, second restoration data is generated from a second erasure-coded data portion stored on the second node. Subsequent to transporting the first or second data shuttle from at least one of the other nodes to a third node, a third erasure-coded data portion is restored at the third node. The third erasure-coded data portion is generated via an erasure-coding process from one or more of the first or second restoration data.

A rebuild node of a storage system can assess risk of the storage system not being able to provide a data object. The rebuild node(s) uses information about data object fragments to determine health of a data object, which relates to the risk assessment. The rebuild node obtains object fragment information from nodes throughout the storage system. With the object fragment information, the rebuild node(s) can assess object risk based, at least in part, on the object fragments indicated as existing by the nodes. To assess object risk, the rebuild node(s) treats absent object fragments (i.e., those for which an indication was not received) as lost. When too many object fragments are lost, an object cannot be rebuilt. The erasure coding technique dictates the threshold number of fragments for rebuilding an object. The risk assessment per object influences rebuild of the objects.

Embodiments of the present invention disclose methods and apparatuses for correcting errors in data stored in a solid state device. The solid state device may have a plurality of bits stored in multi-level memory cells. The method may include identifying one or more errors in a plurality of memory cells. The method may further include converting the erroneous cells to erasures. The method may further include correcting the one or more erasures.

To ensure that there is an elected manager among storage nodes of an erasure coding group (“ECG”), an ECG manager (“ECGM”) election process is periodically performed among available storage nodes that are configured with the software to perform the services of an ECGM. When a storage node is activated, an ECGM process of the storage node begins executing and is assigned a process identifier (“PID”). A storage node can utilize a service query framework to identify other available storage nodes and retrieve their ECGM PIDs. The storage node then selects a PID according to a criterion and elects the storage node corresponding to the selected PID to be the acting ECGM. This process is performed periodically, so even if the acting ECGM storage node fails, a new ECGM is eventually selected from the available storage nodes.

A network system for increasing data throughput and decreasing transmission delay from a source node to a sink node via a relay node. The network system may comprise a source node configured to encode a plurality of data packets using rateless coding and transmit the plurality of data packets; at least one relay node configured to receive at least one of the plurality of data packets from the source node, and if the at least one relay node has received a sufficient quantity of the plurality of data packets, regenerate, re-encode, and relay the plurality of data packets; and a sink node configured to receive one or more of the plurality of data packets from the at least one relay node, and if the sink node has received the sufficient quantity of the plurality of data packets, regenerate and decode the plurality of data packets.

An encoding method includes: receiving configuration data related to encoding with a predetermined encoding mode; determining an encoding strategy based on the configuration data, wherein the encoding strategy includes parameters associated with encoding the data on an entity; and causing the data to be encoded on the entity based on the encoding strategy.

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.