A system and method is disclosed for fast parallel data processing in a distributed storage system. An example method includes receiving at least one digital object; determining, by a processor of a computing device, whether the at least one digital object has a fixed data block structure; in response to determining that the at least one digital object has a fixed data block structure, determining by the processor a size of each fixed data block of the at least one digital object; determining a number of archive chunks for storing the at least one digital object on a selected plurality of storage servers based at least upon the size of each fixed data block; and partitioning the at least one digital object into the number of archive chunks for storing on the selected plurality of storage servers using erasure coding.

A method for generating coded fragments comprises receiving data to be encoded, splitting the data into a plurality of data fragments, identifying a first group of data fragments from among the plurality of data fragments using a coding matrix, summing the data fragments within the first group of data fragments to generate a first group sum, and using the first group sum to calculate at least a portion of two or more coded fragments.

Embodiments use data shuttle devices to restore erasure-coded data in a distributed storage environment. In some embodiments, a first data shuttle is communicatively coupled to a first node of the storage environment. On the data shuttle, first restoration data is generated from a first erasure-coded data portion stored on the first node. The first data shuttle or a second data shuttle is communicatively coupled to a second node of the storage environment. On the data shuttle at the second node, second restoration data is generated from a second erasure-coded data portion stored on the second node. Subsequent to transporting the first or second data shuttle from at least one of the other nodes to a third node, a third erasure-coded data portion is restored at the third node. The third erasure-coded data portion is generated via an erasure-coding process from one or more of the first or second restoration data.

Methods and apparatus for transmitting and receiving broadcast signals are provided. The method for transmitting a broadcast signal includes encoding mobile data for forward error correction (FEC), encoding signaling data, forming data groups including the encoded mobile data and the encoded signaling data and transmitting a signal frame that includes the data groups.

A method for defining an erasure code for system having a predetermined number of data disks is disclosed. The method includes selecting step, constructing step, determining step and repeating step. The selecting step includes selecting a predetermined acceptable number of failures for the system. The constructing step includes constructing a first Tanner graph for two failures acceptable system having predetermined number of data disks. The determining step includes determining erasure code from the first Tanner graph. The repeating step includes repeating the constructing step and the determining step by increasing the acceptable number of failures by one and constructing another Tanner graph in response to the increased acceptable number of failures by increasing number of parity nodes until the predetermined number of failures for the system is reached.

A decoding device includes: a BP decoder that performs BP decoding on an input signal: a maximum likelihood decoder that performs maximum likelihood decoding on a signal subjected to the BP decoding; and a selector that selects one of the input signal, the signal subjected to the BP decoding, and a signal subjected to the maximum likelihood decoding. In a configuration of the decoding device, when a decoder is appropriately operated according to quality of data, a calculation scale can be reduced, and power consumption can be decreased.

A rebuild node of a storage system can assess risk of the storage system not being able to provide a data object. The rebuild node(s) uses information about data object fragments to determine health of a data object, which relates to the risk assessment. The rebuild node obtains object fragment information from nodes throughout the storage system. With the object fragment information, the rebuild node(s) can assess object risk based, at least in part, on the object fragments indicated as existing by the nodes. To assess object risk, the rebuild node(s) treats absent object fragments (i.e., those for which an indication was not received) as lost. When too many object fragments are lost, an object cannot be rebuilt. The erasure coding technique dictates the threshold number of fragments for rebuilding an object. The risk assessment per object influences rebuild of the objects.

Systems, methods, and devices for encoding and decoding data packets for transmission across a data network. To encode, data packets are first subjected to a an outer code process to result in outer coded packets. The outer coded packets are then divided into generations or groups of outer coded packets, each group or generation having an equal number of packets. Output packets are then created by forming random linear combinations of the outer coded packets from a specific generation or group of outer coded packets. The coefficients for the various elements of each linear combination is selected from a Galois field of values. To decode the incoming packets, enough packets are received until an iterative decoding process can be initiated.

A failure resilient distributed replicated data storage system is described herein. The storage system includes zones that are independent, and autonomous from each other. The zones include nodes that are independent and autonomous. The nodes include storage devices. When a data item is stored, it is partitioned into a plurality of data objects and a plurality of parity objects calculated. Reassembly instructions are created for the data item. The data objects and parity objects are spread across all nodes and zones in the storage system. Reassembly instructions are also spread across the zones. When a read request is received, the data item is prepared from the lowest latency nodes according to the reassembly instructions. This provides for data resiliency while keeping the amount of storage space required relatively low.

Example apparatus and methods provide improved reclamation, garbage collection (GC) and defragmentation (defrag) for data storage devices including solid state drives (SSD) or shingled magnetic recording (SMR) drives. An erasure code (EC) layer that facilitates logically or physically erasing data from the SSD or SMR as a comprehensive GC or defrag is added to the SSD or SMR. Erased data may be selectively recreated from the EC layer as needed. Pre-planned EC write zones may be established to further optimize GC and defrag. Recreated data may be written to selected locations to further optimize SSD and SMR performance. Erasure code data may be distributed to co-operating devices to further improve GC or defrag. Example apparatus and methods may also facilitate writing data to an SMR drive using tape or VTL applications or processes and providing a pseudo virtual tape library on the SMR drive.