The present disclosure is based on erasure coding, information dispersal, secret sharing and ramp schemes to assure reliability and security. More precisely, the present disclosure combines ramp threshold secret sharing and systematic erasure coding.

A system is provided. The system includes a data storage system and a client device communicatively coupled to the data storage device. The client device includes a processing device to receive a data request directed to the data storage system, translate the data request to a backend protocol of the data storage system, and retrieve one or more portions of data from the data storage system based on the translated data request. In some embodiments, the processing device is a data processing unit of the client device dedicated to executing a protocol endpoint of the data storage system.

Provided is a method for sending digital data over a number of channels wherein a sender performs the following steps: encoding source data having a first number of source symbols, the encoding being such that an error correction code is generated from the source data, the error correction code comprising a second number of repair symbols higher than the first number as well as identifiers where each identifier is assigned to a corresponding repair symbol, the error correction code adding redundancy to the source data; encrypting each repair symbol by an encryption process which is based on a shared secret between the sender and a receiver, where the encryption process for a respective repair symbol depends on the identifier assigned to the respective repair symbol; feeding pairs of the encrypted repair symbols and the assigned identifiers to the number of channels which are connected to the receiver.

Data storage method, apparatus, and system are disclosed. The data storage method includes: encoding a data block using an erasure coding to obtain corresponding multiple data fragments; separately writing the multiple data fragments to corresponding allocated storage nodes; and determining that writing of the data block is successful if a number of storage nodes to which data fragments is successfully written within a first defined time duration is greater than or equal to a defined value, wherein the defined value is greater than or equal to a number of data block fragments in the multiple data fragments and is less than or equal to a number of the multiple data fragments. As such, the latency of a data write can be reduced, and the performance stability can be improved when a storage node encounters a failure.

Systems for accessing client data is described. A request to access a first data block is received. The request indicates a first logical address referencing the first data block. First mapping data is employed to identify a first physical addresses corresponding to the first logical addresses. The first mapping data encodes a first LOM transaction ID and candidate local addresses. The first mapping data is employed to identify the candidate local address and the first LOM transaction ID. A usage table is employed to determine the current status of the first LOM transaction ID. The candidate local address is employed to access the first data block. Second mapping data is employed to identify an updated local address of the set of local addresses. The updated local address currently references the first data block. The updated local address is employed to access the first data block.

Client data is structured as a set of data blocks. A first subset of data blocks is stored on a current segment of a plurality of disks. A second subset of data blocks is stored on a previous segment. A request to clean client data is received. The request includes a request to update the current segment to include the second subset of data blocks. The second subset of data blocks is accessed and transmitted from a lower layer to a higher layer of the system. Parity data is generated at the higher layer. The parity data is transmitted to the lower layer. The lower layer is employed to generate a local copy of the second subset of data blocks. Each local address that references the local copy of the second subset of data blocks is included in the current segment. The parity data is written in the current segment.

Distributed storage systems frequently use a centralized metadata repository that stores metadata in an eventually consistent distributed database. However, a metadata repository cannot be relied upon for determining which erasure coded fragments are lost because of a storage node(s) failures. Instead, when recovering a failed storage node, a list of missing fragments is generated based on fragments stored in storage devices of available storage nodes. A storage node performing the recovery sends a request to one or more of the available storage nodes for a fragment list. The fragment list is generated, not based on a metadata database, but on scanning storage devices for fragments related to the failed storage node. The storage node performing the recovery merges retrieved lists to create a master list indicating fragments that should be regenerated for recovery of the failed storage node(s).

Various embodiments set forth techniques for erasure coding of replicated data blocks. The techniques include receiving, by a pre-designated node, data associated with an erasure coded strip from a first node; receiving, by the pre-designated node, a replica for a first data block; saving the replica in an erasure coded strip; and in response to a trigger condition, replacing, by the pre-designated node, the replica and at least one replica of a second data block with an error correction block.

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.

A computer implemented method for recovering erased entries within a system of arrays includes identifying a system consisting of a plurality of arrays, wherein each array consists of m rows and n columns of entries, each entry is divided into p symbols consisting of a plurality of bits, protecting the m rows and n columns of entries in the system with an erasure-correcting code allowing the recovery of a number of erased entries in such rows and columns, detecting an erasure corresponding to an entry in the identified system, and, responsive to detecting an erasure, determining the value of the erased entry according to the p symbols of one or more non-erased entries.