An encoding method includes: receiving configuration data related to encoding with a predetermined encoding mode; determining an encoding strategy based on the configuration data, wherein the encoding strategy includes parameters associated with encoding the data on an entity; and causing the data to be encoded on the entity based on the encoding strategy.

Techniques are described for achieving durability of a data object stored in a network storage system. In some embodiments, erasure coding is applied to break a data object into fragments wherein the original data object can be recovered with fewer than all of the fragments. These fragments are stored on multiple storage nodes in a distributed storage cluster of a network storage system. So that individual storage nodes have knowledge of the state of the stored data object, a proxy server acing as a central agent can wait for acknowledgments indicating that the fragments have been successfully stored at the storage nodes. If the proxy server receives successful write responses from a sufficient number of the storage nodes, the proxy server can report that the data object is durably stored by placing markers on the storage nodes.

A method for hierarchical correction coding includes converting data for a storage system into w storage device arrays, each storage device array including n storage devices, and each storage device divided into m sectors or pages. The n storage devices are grouped into l groups of t storage devices each. Erasures in the w storage device arrays are corrected based on protecting each row and column in each m×n array by an erasure-correcting code. Each group of t storage devices contains extra parities to correct extra erasures in addition to erasures corrected by vertical parities in each m×t subarray, and w, n, m, l and t are positive integers.

One embodiment provides a method comprising, for at least one data block, selecting an erasure code from a plurality of erasure codes based on at least one property of the at least one data block and information relating to a data cache, and encoding, utilizing at least one hardware processor, the at least one data block with the selected erasure code. The information relating to the data cache includes cache space usage of the data cache.

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.

Example apparatus and methods monitor conditions in an object storage system. The conditions monitored may include a load balance measure in the system, a capacity balance measure in the system, a fault tolerance measure in the system, or a usage pattern measure in the system. A distribution plan or redistribution plan for storing or moving erasure codes in the object storage system may be determined based on the conditions. The distribution plan or the redistribution plan for the erasure codes may be updated dynamically in response to changing conditions in the object storage system. The distribution or redistribution may depend on a weighted combination of the load balance measure, the capacity balance measure, the fault tolerance measure, or the usage pattern measure so that responding to one sub-optimal condition (e.g., load imbalance) does not create a different sub-optimal condition (e.g., unacceptable fault tolerance).

An arrangement for decoding a data word using a Reed-Muller code, has: (1) N input terminals, (2) a first level of E>>D summing modules, each summing module being linked with F different input terminals and each input terminal being linked with E summing modules, (3) a first level of E decision modules, each of the D inputs of each decision module being linked respectively with an output from D different summing modules, (4) a second level of H summing modules, (5) a second level of G decision modules, (6) a third level of G summing modules, and (7) G output terminals. N signifies the code length and D signifies the minimum spacing of the code, E is equal to D-2, F is equal to N/D, G is the number of symbols of the data word that need to be corrected and is a natural number between 1 and E<<D.

This invention gives a coding method of MBR (Minimum Bandwidth Regenerating) codes. The related method includes the following steps: equally divide the original file of size B into k(k+1)/2 blocks, obtaining the first packets; construct a symmetrical k×k system matrix S with these first packets; generate k ID codes, wherein each ID code contains k elements; obtain the coded packet through operations between one column of the system matrix and the ID code; repeat the above steps with (n−k) different columns of the system matrix separately to get the (n−k) coded packets; construct the (n−k)×k check matrix P with the column number g which is the serial number of the ID codes in the coded packet set P.sub.g; store the rows of the system matrix and coded matrix to n nodes, each node stores one row. The present invention also involves a method to repair the failed nodes of the above coding scheme.

Technology is disclosed for a data storage architecture for providing enhanced storage resiliency for a data object. The data storage architecture can be implemented in a single-tier configuration and/or a multi-tier configuration. In the single-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data fragments, which are stored across many storage devices. In the multi-tier configuration, a data object is encoded, e.g., based on an erasure coding method, to generate many data segments, which are sent to one or more tiers of storage nodes and at least one latent storage. Each of the storage nodes further encodes the data segment to generate many data fragments representing the data segment, which are stored across many storage devices associated with the storage node. The I/O operations for rebuilding the data in case of device failures is spread across many storage devices, which minimizes the wear of a given storage device.

Every time a node computer receives multidimensional data from a data source, the node computer which has received the multidimensional data in a computer system: writes the multidimensional data; reads the multidimensional data; analyzes the read multidimensional data; and outputs a result of the analysis. Such writing of the multidimensional data is data locality EC processing (Erasure Coding with data locality). The data locality EC processing is to: write all of one or more data chunks constituting the multidimensional data to the node computer; and write a parity of the data chunk, with respect to each of the one or more data chunks, to a node computer(s) other than the node computer.