A system for software error-correcting code (ECC) protection or compression of original data using ECC data in a first memory is provided. The system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The software ECC protection or compression includes: a data matrix for holding the original data in the first memory; a check matrix for holding the ECC data in the first memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the ECC data; and a thread for executing on the processing core. The thread includes a Galois Field multiplier for multiplying entries of the data matrix by an entry of the encoding matrix, and a sequencer for ordering operations using the Galois Field multiplier to generate the ECC data.

A method of data recovery for a distributed storage system is a method of recovering multiple failed nodes concurrently with the minimum feasible bandwidth when failed nodes exist in a distributed storage system. By means of selecting assistant nodes, obtaining helper data sub-blocks through computing the selected assistant nodes, then computing a repair matrix and finally multiple the repair matrix and the helper data sub-blocks, the missing data blocks are reconstructed; or the missing data blocks are reconstructed by decoding. The method is applicable to data recovery in the case of any number of failed nodes and any reasonable combinations of coding parameters. The data recovery herein can reach the theoretical lower limit of the minimum recovery bandwidth.

A method begins by identifying a plurality of encoded data slices requiring rebuilding. The method continues by determining an amount of memory required for rebuilding the plurality of encoded data slices and allocating memory in one or more storage units for the rebuilding the plurality of encoded data slices as reserve memory. The method continues by obtaining a plurality of rebuilt encoded data slices associated with the plurality of encoded data slices requiring rebuilding and storing the plurality of rebuilt encoded data slices in the reserve memory.

A method for adjustable error correction in a storage cluster is provided. The method includes determining health of a non-volatile memory of a non-volatile solid-state storage unit of each of a plurality of storage nodes in a storage cluster on a basis of per flash package, per flash die, per flash plane, per flash block, or per flash page. The determining is performed by the storage cluster. The plurality of storage nodes is housed within a chassis that couples the storage nodes as the storage cluster. The method includes adjusting erasure coding across the plurality of storage nodes based on the health of the non-volatile memory and distributing user data throughout the plurality of storage nodes through the erasure coding. The user data is accessible via the erasure coding from a remainder of the plurality of storage nodes if any of the plurality of storage nodes are unreachable.

Identifying, by a sender and for each frame i of a plurality of frames of a video stream, a partition of a set of video data symbols D[i] into a first set of video data symbols U[i] and a second set of video data symbols V[i]. Generating, by the sender and for each frame i, a set of one or more streaming forward error correction (FEC) code parity symbols P[i] based on the symbols: V[i−τ] through V[i−1], U[i−τ], and the symbols D[i], wherein τ is a function of a maximum tolerable latency of the video stream expressed as a whole number of frames. Encoding, by the sender and for each frame i, packets carrying the symbols D[i], and P[i]. Transmitting, by the sender, each frame i of encoded packets in frame order to one or more receivers.

Methods and apparatus are provided for controlling wireless signal transmissions, wherein problematic symbol patterns are relocated to an erasure region of a data packet prior to erasure encoding and transmission. Relocating the problematic symbol patterns is done so that, when the resulting erasure codeword is punctured and transmitted, the problematic patterns are not transmitted. Yet, those patterns can be restored by the decoder at the receiving device using an erasure decoder in accordance with erasure decoding techniques, e.g., punctured low-density parity-check (LDPC) decoding techniques. In this manner, problematic symbol patterns that may be corrupting during transmission due to noise are removed (punctured) prior to transmission, then restored by the decoder during decoding.

A method includes acquiring, by a managing unit of a dispersed storage network (DSN), storage unit status information and data object storage status information from a plurality of storage units of DSN memory of the DSN. The method further includes determining, by the managing unit, DSN status information of the DSN memory based on the storage unit status information and the data object storage status information. The method further includes identifying, by the managing unit, DSN memory issues within the DSN memory. The method further includes prioritizing, by the managing unit, corrective remedies for the DSN memory issues based on the status information of the DSN memory. The method further includes facilitating, by the managing unit, the execution of the prioritized corrective remedies to correct the DSN memory issues.

One embodiment provides a system which facilitates data management. During operation, the system receives, by a first memory device, data to be written to a first non-volatile memory of the first memory device and to a second non-volatile memory of a storage drive distinct from the first memory device. The system performs, by the first memory device on the received data, a compression operation and erasure code (EC)-encoding to obtain a compressed EC codeword. The system initiates a first write operation and a second write operation in parallel, wherein the first write operation comprises writing a first part of the compressed EC codeword to the first non-volatile memory, and wherein the second write operation comprises writing the first part of the compressed EC codeword to the second non-volatile memory.

Computer implemented systems and methods for migrating datacenter data include providing a quantity of carriers having a data storage capacity, receiving, by the quantity of carriers, a quantity of data stored in a first data storage system having a first location and including erasure coded data blocks. The quantity of carriers migrates to a second data storage system having a second location; and transmits the quantity of data to the second data storage system.

A stripe merging method and system based on erasure codes are provided. A StripeMerge-P algorithm is used first to determine alignment information of parity chunks of erasure code stripes based on a preprocessed hash table. Through a greedy strategy, erasure code stripe pairs to be merged are selected for merging. Through the hash table, location information of the parity chunks is directly looked up, so that no additional computing overhead is required, and the overhead of selecting and merging the erasure code stripe pairs is further reduced through the combination with the greedy strategy.