Rebalancing as a result of re-encoding a code chunk in response to scaling out of a geographically diverse storage system employing erasure coding technology is disclosed. After a scaling out event, a new erasure coding scheme can be selected. An old coding chunk generated according to an old erasure coding scheme can be re-encoded into a new coding chunk according to the new erasure coding scheme and based on a data chunk not previously protected by the old coding chunk. The re-encoding can be selected to diversify distribution of chunks, resulting in rebalancing occurring as part of re-encoding. In an embodiment, the new coding chunk can be generated in a new zone from the scaling out event. In another embodiment, the data chunk can be moved to the new zone from the scaling out event.

A remote vehicle control system includes a vehicle mounted sensor system including a video camera system for producing video data and a distance mapping sensor system for producing distance map data. A data handling system is used to compress and transmit both the video and distance map data over a cellular network using feed forward correction. A virtual control system acts to receive the video and distance map data, while providing a user with a live video stream supported by distance map data. Based on user actions, control instructions can be sent to the vehicle mounted sensor system and the remote vehicle over the cellular network.

Disclosed herein are methods, systems, and apparatus, including computer programs encoded on computer storage media, for storing blockchain data. One of the methods includes determining, by a blockchain node, one or more blocks that are infrequently visited; performing error correction coding of the one or more blocks to generate one or more encoded blocks; dividing, based on one or more predetermined rules, each of the one or more encoded blocks to a plurality of data sets; selecting one or more data sets from the plurality of data sets of each of the one or more encoded blocks based on the one or more predetermined rules; hashing the one or more data sets to generate one or more hash values corresponding to the one or more data sets; storing the one or more hash values; and deleting the one or more data sets.

A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

A computer-implemented method, according to one embodiment, includes: selecting strips from each storage unit for a given erasure code stripe such that the given erasure code stripe includes at most one strip from a high failure rate region of the respective storage unit, where each of the storage units include high and low failure rate regions. The selected strips are organized such that a number of each strip in the given erasure code stripe is offset from the remaining strips by an amount that is greater than a total number of strips in the high failure rate regions. The organized selected strips are further mapped to form the given erasure code stripe such that the high failure rate regions on each storage unit are mapped to one or more sequentially numbered strips, and the low failure rate regions are mapped to additional sequentially numbered strips.

A remote vehicle control system includes a vehicle mounted sensor system including a video camera system for producing video data and a distance mapping sensor system for producing distance map data. A data handling system is used to compress and transmit both the video and distance map data over a cellular network using feed forward correction. A virtual control system acts to receive the video and distance map data, while providing a user with a live video stream supported by distance map data. Based on user actions, control instructions can be sent to the vehicle mounted sensor system and the remote vehicle over the cellular network.

According to one general aspect, a system may include a data interface circuit configured to receive data access requests, wherein each data access request is associated with a data set. The system may include a plurality of storage devices each storage device configured to store data, and wherein the plurality of storage devices has an amount of available storage space, an amount of used storage space, and an amount of redundant storage space. The system may include a data management circuit configured to: monitor the amount of available storage space in the in the plurality of storage devices, determine, for one or more data set, a level of redundancy to be associated with the respective data set, generate, for a first data set of the data sets, a redundant data portion to be associated with the first data set, and dynamically adjust the level of redundancy associated with the first data set, based, at least in part, upon the amount of available storage space.

One embodiment facilitates data placement in a storage device. During operation, the system receives a request indicating first data to be written to a non-volatile memory which includes a plurality of dies, wherein a plurality of error correction code (ECC) codec modules reside on the non-volatile memory. The system receives, by a first codec module residing on a first die, the first data. The system encodes, by the first codec module operating on the first die, the first data based on an error correction code (ECC) to obtain first ECC-encoded data which includes a first set of ECC parity bits. The system writes the first ECC-encoded data to the first die.

A system is provided for performing erasure coding (EC) in a distributed storage system. During operation, the system can perform a partial encoding of a received first set of data fragments and second set of data fragments using EC to generate a first and a second EC codeword, respectively. The system can then distribute the first and the second set of data fragments among a set of storage nodes within the distributed storage system. The system can also distribute a first and the second set of intermediate parity fragments in the first and second EC codeword, respectively, among a subset of the storage nodes with alignments. The system can then merge the first and the second set of intermediate parity fragments to generate an overall parity for both the first and the second set of data fragments. The system can store, based on the alignments, each overall parity fragment in the overall parity in the corresponding subset of storage nodes.

The present disclosure relates to a system and a method for data protection. In some embodiments, an exemplary method for data encoding includes: receiving a data bulk; performing an erasure coding (EC) encoding on the data bulk to generate one or more EC codewords; distributing a plurality of portions of each EC codeword of the one or more EC codewords across a plurality of solid-state drives (SSDs); performing, at each SSD of the plurality of SSDs, an error correction coding (ECC) encoding on portions of the one or more EC codewords distributed to the SSD to generate an ECC codeword; and storing, in each SSD of the plurality of SSDs, the ECC codeword.