An encryption module encrypts starting data using a random key to produce encrypted data. A hash module performs a secure hash function on the encrypted data using a secret key to produce a hash value. Processing circuitry masks the random key using the hash value to produce a masked random key, and combines the encrypted data and the masked random key to produce a secure package. A distributed storage and task module encodes the secure package to produce a set of encoded data slices. The secret key and a decode threshold number of the encoded data slices included in the set of encoded data slices are sufficient to recover the secure package and the starting data. The set of encoded data slices is stored in a set of storage units.

Systems, methods, and devices for encoding and decoding data packets for transmission across a data network. To encode, data packets are first subjected to a an outer code process to result in outer coded packets. The outer coded packets are then divided into generations or groups of outer coded packets, each group or generation having an equal number of packets. Output packets are then created by forming random linear combinations of the outer coded packets from a specific generation or group of outer coded packets. The coefficients for the various elements of each linear combination is selected from a Galois field of values. To decode the incoming packets, enough packets are received until an iterative decoding process can be initiated.

A detection circuit performs a turbo detection process to recover a frame of data symbols from a received signal, the data symbols of the frame having been effected, during transmission, by a Markov process with the effect that the data symbols of the frame in the received signal are dependent one or more preceding data symbols which can be represented as a trellis having a plurality of trellis stages. The detection circuit comprises a plurality of processing elements, each of the processing elements is associated with one of the trellis stages representing the dependency of the data symbols of the frame according to the Markov process and each of the processing elements is configured to receive one or more soft decision values corresponding to one or more data symbols associated with the trellis stage, and each of one or more of the processing elements is configured, in one clock cycle to receive fixed point data representing a priori forward state metrics a priori backward state metrics, and fixed point data representing a priori soft decision values for the one or more data symbols being detected for the trellis stage. For each of a plurality of clock cycles of the turbo detection process, the detection circuit is configured to process, for each of the processing elements representing the trellis stages, the a priori information for the one or more data symbols being detected for the trellis stage associated with the processing element, and to provide the extrinsic soft decision values corresponding to the one or more data symbols for a next clock cycle of the turbo detection process.

Methods and apparatus for selection of memory devices in a distributed storage network. In an example, a computing device receives a data object for storage and selects a set of storage nodes of a plurality of sets of storage nodes for storing the data object. Selection of the set of storage nodes includes determining storage attributes associated with each set of storage nodes of the plurality of sets of storage nodes. Selection of the set of storage nodes additionally includes determining a storage preference associated with the data object, and comparing the storage preference with the storage attributes of the plurality of sets of storage nodes to determine a best match. Following selection of a set of storage nodes, the computing device facilitates storage of the data object in the selected set of storage nodes.

Example apparatus and methods provide improved reclamation, garbage collection (GC) and defragmentation (defrag) for data storage devices including solid state drives (SSD) or shingled magnetic recording (SMR) drives. An erasure code (EC) layer that facilitates logically or physically erasing data from the SSD or SMR as a comprehensive GC or defrag is added to the SSD or SMR. Erased data may be selectively recreated from the EC layer as needed. Pre-planned EC write zones may be established to further optimize GC and defrag. Recreated data may be written to selected locations to further optimize SSD and SMR performance. Erasure code data may be distributed to co-operating devices to further improve GC or defrag. Example apparatus and methods may also facilitate writing data to an SMR drive using tape or VTL applications or processes and providing a pseudo virtual tape library on the SMR drive.

A network system for increasing data throughput and decreasing transmission delay from a source node to a sink node via a relay node. The network system may comprise a source node configured to encode a plurality of data packets using rateless coding and transmit the plurality of data packets; at least one relay node configured to receive at least one of the plurality of data packets from the source node, and if the at least one relay node has received a sufficient quantity of the plurality of data packets, regenerate, re-encode, and relay the plurality of data packets; and a sink node configured to receive one or more of the plurality of data packets from the at least one relay node, and if the sink node has received the sufficient quantity of the plurality of data packets, regenerate and decode the plurality of data packets.

Techniques are described for achieving durability of a data object stored in a network storage system. In some embodiments, erasure coding is applied to break a data object into fragments wherein the original data object can be recovered with fewer than all of the fragments. These fragments are stored on multiple storage nodes in a distributed storage cluster of a network storage system. So that individual storage nodes have knowledge of the state of the stored data object, a proxy server acing as a central agent can wait for acknowledgments indicating that the fragments have been successfully stored at the storage nodes. If the proxy server receives successful write responses from a sufficient number of the storage nodes, the proxy server can report that the data object is durably stored by placing markers on the storage nodes.

A method for hierarchical correction coding includes converting data for a storage system into w storage device arrays, each storage device array including n storage devices, and each storage device divided into m sectors or pages. The n storage devices are grouped into l groups of t storage devices each. Erasures in the w storage device arrays are corrected based on protecting each row and column in each m×n array by an erasure-correcting code. Each group of t storage devices contains extra parities to correct extra erasures in addition to erasures corrected by vertical parities in each m×t subarray, and w, n, m, l and t are positive integers.

One embodiment provides a method comprising, for at least one data block, selecting an erasure code from a plurality of erasure codes based on at least one property of the at least one data block and information relating to a data cache, and encoding, utilizing at least one hardware processor, the at least one data block with the selected erasure code. The information relating to the data cache includes cache space usage of the data cache.

The invention provides methods of encoding content for distribution over a network and methods for decoding encoded content which has been distributed over the network. In a first example in which the content is divided into a plurality of segments and each segment comprising a plurality of blocks of data, the method comprises selecting a segment from the plurality of segments and selecting at least two blocks of the selected segment from a store of blocks. A new encoded block is created from a linear combination of the selected blocks.