A storage system comprises a plurality of enclosures and a storage controller. Each enclosure comprises at least one processing device and a plurality of drives configured in accordance with a redundant array of independent disks (RAID) arrangement. The storage controller obtains data pages associated with at least one input-output request and provides the obtained data pages to the at least one processing device of a given enclosure. The storage controller issues a command to the at least one processing device of the given enclosure to perform at least one operation based at least in part on the obtained data pages. The at least one processing device of the given enclosure receives the obtained data pages from the storage controller, calculates RAID parities based at least in part on the received data pages, and stores the data pages and RAID parities on the plurality of drives according to the RAID arrangement.

An apparatus comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to select a stripe column size for stripes of a data storage system, to determine a first compress block size for a first one of the stripes based on compressibility of data to be stored, to select a first prime number for computing parity blocks for the first stripe and a first number of sub-stripes for splitting stripe columns of the first stripe, to generate metadata specifying the first compress block size, the first prime number and the first number of sub-stripes for the first stripe, and to store data compressed using the first compress block size in the first stripe. The first prime number and first number of sub-stripes for the first stripe is different than a second prime number and second number of sub-stripes for a second stripe.

An apparatus comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to select a stripe column size for stripes of a data storage system, to determine a first compress block size for a first one of the stripes based on compressibility of data to be stored, to select a first prime number for computing parity blocks for the first stripe and a first number of sub-stripes for splitting stripe columns of the first stripe, to generate metadata specifying the first compress block size, the first prime number and the first number of sub-stripes for the first stripe, and to store data compressed using the first compress block size in the first stripe. The first prime number and first number of sub-stripes for the first stripe is different than a second prime number and second number of sub-stripes for a second stripe.

A system for software error-correcting code (ECC) protection or compression of original data using ECC data in a first memory is provided. The system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The software ECC protection or compression includes: a data matrix for holding the original data in the first memory; a check matrix for holding the ECC data in the first memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the ECC data; and a thread for executing on the processing core. The thread includes a Galois Field multiplier for multiplying entries of the data matrix by an entry of the encoding matrix, and a sequencer for ordering operations using the Galois Field multiplier to generate the ECC data.

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.

A method for calculating a plurality (M) of redundancy blocks for multiple (N) data blocks of a plurality (D) of words each, the method comprises: receiving the number (M) of redundancy blocks by a calculator that comprises multiple (R) calculation units; configuring the calculator according to M and R; concurrently calculating, if M equals R, by the multiple (R) calculation units of the calculator, R sets of parity vectors, each set includes a plurality (D) of parity vectors; and calculating the plurality (M) of the redundancy blocks based on the R sets of parity vectors.

An apparatus comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured to select a stripe column size for stripes of a data storage system, to determine a first compress block size for a first one of the stripes based on compressibility of data to be stored, to select a first prime number for computing parity blocks for the first stripe and a first number of sub-stripes for splitting stripe columns of the first stripe, to generate metadata specifying the first compress block size, the first prime number and the first number of sub-stripes for the first stripe, and to store data compressed using the first compress block size in the first stripe. The first prime number and first number of sub-stripes for the first stripe is different than a second prime number and second number of sub-stripes for a second stripe.

The disclosure provides a very flexible mechanism for a storage controller to create RAID stripes and to re-create corrupted stripes when necessary using the erasure coding scheme. Typically, this is known as a RAID 6 implementation/feature. The erasure code calculations are generated using the Galois Multiplication hardware and the system controller can pass any polynomial into the hardware on a per stripe calculation basis. The polynomial value is passed to the hardware via an input descriptor field. The descriptor controls the entire computation process.

Balanced Reed-Solomon codes in accordance with embodiments of the invention enable balanced load in distributed storage systems. One embodiment includes a storage controller, wherein the storage processor is configured by the controller Reed-Solomon application to: receive a data segment; partition the data segment into a first block of data and a second block of data; transmit the first block of data to a first node controller in the plurality of node controllers; transmit the second block of data to a second node controller in the plurality of node controllers; wherein the node processor in the first node controller is configured by the node Reed-Solomon application to: receive the first block of data; encode the first block of data using a balanced and sparsest error-correcting code; and store the encoded first block of data in the node memory of the first node controller.

An accelerated erasure coding system includes a processing core for executing computer instructions and accessing data from a main memory, and a non-volatile storage medium for storing the computer instructions. The processing core, storage medium, and computer instructions are configured to implement an erasure coding system, which includes: a data matrix for holding original data in the main memory; a check matrix for holding check data in the main memory; an encoding matrix for holding first factors in the main memory, the first factors being for encoding the original data into the check data; and a thread for executing on the processing core. The thread includes: a parallel multiplier for concurrently multiplying multiple entries of the data matrix by a single entry of the encoding matrix; and a first sequencer for ordering operations through the data matrix and the encoding matrix using the parallel multiplier to generate the check data.