A system identifies a data object, stored in a client, for replication. The system identifies a redundancy number associated with a protected amount of failed storage nodes. The system determines whether the total count of available storage nodes is at least two greater than the redundancy number. The system determines a distribution number that is at most the total count minus the redundancy number, in response to a determination that the total count of available storage nodes is at least two greater than the redundancy number. The system creates erasure code for the data object. The system allocates the distribution number of sections of a replica of the data object to a corresponding distribution number of available storage nodes and the redundancy number of sections of the erasure code to the redundancy number of available storage nodes, which excludes the corresponding distribution number of available storage nodes.

Disclosed is an erasure-based Reed-Solomon code soft-decision decoding method and device, capable of reducing a decoding time while minimizing the effect on error correction performance. The Reed-Solomon code soft-decision decoding device includes an erasure control circuit configured to determine whether a number of errors in a codeword is odd or even, and to provide a key equation solver circuit with a first erasure pattern or a second erasure pattern according to a result of the determining when a decoding failure is detected by a decoding error detection circuit, the first erasure pattern being provided when the number of errors is odd, the second erasure pattern being provided when the number of errors is even.

A storage system is provided. The storage system includes a plurality of storage units, each having a controller and solid-state storage memory. The storage system further includes one or more first pathways that couple processing devices of a plurality of storage nodes and is configured to couple to a network external to the storage system and one or more second pathways that couple the plurality of storage nodes to the plurality of storage units, wherein the one or more second pathways enable multiprocessing applications.

A method for managing data includes obtaining data from a host, performing an erasure coding procedure to the data to obtain a plurality of slices, wherein each slice in the plurality of slices comprises a plurality of data chunks and at least one parity chunk, generating a plurality of segment entries, wherein each segment entry in the plurality of segment entries specifies a segment, generating metadata slice entries, wherein each metadata slice entry is associated with a slice in the plurality of slices, storing the plurality of segment entries and the metadata slice entries in an accelerator pool in a first data cluster, and storing, across a plurality of fault domains in the first data cluster, the data chunks and the parity chunk of each slice in the plurality of slices based on the plurality of segment entries.

A method and device for improving data storage security, related to the technical field of cloud storage is disclosed. The method includes: detecting access counts of user data sets in a cloud storage resource pool, where the user data sets in the cloud storage resource pool are data obtained by encoding with an erasure coding algorithm of a first security level; determining a first user data set whose access count in the cloud storage resource pool meets a preset condition; obtaining a second security level according to attribute information of the cloud storage resource pool, wherein the attribute information includes the first security level and an increment for security step; and re-encoding the first user data set with an erasure coding algorithm of the second security level to obtain re-encoded first user data sets.

Techniques for using erasure coding in a single region to reduce the likelihood of losing objects in a cloud object storage platform are provided. In one set of embodiments, a computer system can upload a plurality of data objects to a region of a cloud object storage platform, where the plurality of data objects including modifications to a data set. The computer system can further compute a parity object based on the plurality of data objects, where the parity object encodes parity information for the plurality of data objects. The computer system can then upload the parity object to the same region where the plurality of data objects was uploaded.

A method includes receiving, at a first computing device, a first input/output (IO) command from a first artificial intelligence processing unit (AI PU), the first IO command associated with a first AI model training operation. The method further includes receiving, at the first computing device, a second IO command from a second AI PU, the second IO command associated with a second AI model training operation. The method further includes assigning a first timestamp to the first IO command based on a first bandwidth assigned to the first AI model training operation. The method further includes assigning a second timestamp to the second IO command based on a second bandwidth assigned to the second AI model training operation.

A data storage system includes a plurality of data blocks. A set of data blocks are protected by an erasure correcting code and each of the data blocks in the set of data blocks includes block identification information. The data storage system includes a processor and logic integrated with the processor, executable by the processor, or integrated with and executable by the processor. The logic is configured to verify the block identification information for each of the data blocks in the set of data blocks at the time of read and, as part of reconstructing a data block, reconstruct the block identification information for the reconstructed data block, and verify the block identification information.

A fault-tolerant data storage system associates durability requirements of service level agreements (SLAs) for volumes stored in the fault-tolerant data storage system with volume partitions stored in the fault-tolerant data storage system. For a given volume partition, volume data is stored in two or more replicas on two or more different system components and/or erasure encoded across multiple other system components. The fault-tolerant data storage system uses the respective durability requirements of the SLAs and failure statistics of the system components to allocate bandwidth for replacing lost instances of redundantly stored volume data such that the lost data is replaced within a target time calculated to guarantee the durability requirements of the SLAs are satisfied.

Methods and devices are provided for error correction of distributed data in distributed systems using Reed-Solomon codes. In one embodiment, processes are provided for error correction that include receiving a first correction code for data fragments stored in storage nodes, constructing a second correction code responsive to an unavailable storage node of the storage nodes, performing erasure repair of the unavailable storage node, and outputting a corrected data fragment. The first correction code is a Reed-Solomon code represented as a polynomial and the second correction code is represented as a second polynomial with an increased subpacketization size. Processes are configured to account for repair bandwidth and sub-packetization size. Code constructions and repair schemes accommodate different sizes of evaluation points and provide a flexible tradeoff between the subpacketization size repair bandwidth of codes. In addition, schemes are provided to manage a single node failure and multiple node failures.