Method and apparatus for managing data in a storage device, such as a solid-state drive (SSD). A non-volatile memory (NVM) is arranged into multiple garbage collection units (GCUs) each separately erasable and allocatable as a unit. Read circuitry applies read voltages to memory cells in the GCUs to sense a programmed state of the memory cells. Calibration circuitry groups different memory cells from different GCUs into calibration groups that share a selected set of read voltages. A read command queue accumulates pending read commands to transfer data from the NVM to a local read buffer. Read command coalescing circuitry coalesces selected read commands from the queue into a combined command for execution as a single batch command. The combined batch command may include read voltages for use in retrieval of the requested data.

A distributed storage system node is disclosed. The distributed storage system node may include at least one storage device, which may act as the primary replica for data subject to an Input/Output (I/O) request. A cost analyzer may calculate a local estimated time required to complete the I/O request at the primary replica, and a remote estimated time required to complete the I/O request at a secondary replica of the data. An I/O redirector may direct the I/O request to either the primary replica or the secondary replica based on the local estimated time required and the one remote estimated time required.

Systems, methods, and computer readable storage mediums for optimistically managing compressed data in a storage system. When possible, multiple input blocks are compressed into a buffer and stored in a single physical block on a storage device. The metadata of the multiple input blocks can be managed separately. A fingerprint of the compressed buffer can be generated and used as an index to the single physical block. Alternatively, fingerprints of the uncompressed input blocks can be generated, and reference counts can be maintained to track the number of input blocks which reference the compressed buffer. In some embodiments the physical block is associated with a zone represented by a virtual construct, wherein the zone is dynamically mapped to underlying storage of the zoned storage system.

A key-value storage architecture with data compression is shown. During the garbage collection, the controller compresses valid pieces of key-value data to generate a piece of compressed data. Each piece of key-value data is in key-value format. The controller codes the piece of compressed data to generate a first piece of compressed key-value data that is also in key-value format, and programs the first piece of compressed key-value data into the non-volatile memory.

During execution of garbage collection, an application receives a first request to overwrite a reference field of an object, the object comprising a first reference and the first request comprising a memory address at which the reference field is stored, and a second reference to be written to the reference field. Responsive to receiving the first request, the system determines a current remembered set phase, and loads the first reference. The application determines that remembered set metadata of the first reference does not match the current remembered set phase. Responsive to that determination, the application adds an entry to a remembered set data structure, modifies the second reference to include the current remembered set phase as the remembered set metadata, and stores the modified second reference to the reference field. In subsequent writes to the reference field, the application refrains from adding to the remembered set data structure.

A system and method for distributing data over a plurality of remote storage nodes. Data are split into segments and each segment is encoded into a number of codeword chunks. None of the codeword chunks contains any of the segments. Each codeword chunk is packaged with at least one encoding parameter and identifier, and metadata are generated for at least one file and for related segments of the at least one file. The metadata contains information to reconstruct from the segments, and information for reconstructing from corresponding packages. Further, metadata are encoded into package(s), and correspond to a respective security level and a protection against storage node failure. A plurality of packages are assigned to remote storage nodes to optimize workload distribution. Each package is transmitted to at least one respective storage node as a function iteratively accessing and retrieving the packages of metadata and file data.

Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.

Garbage collection (GC) states are stored within references stored on a heap memory to track a progress of GC operations with respect to the references. GC state may be stored in a non-addressable portion of references. Based on the GC state of a particular reference, a set of GC operations are selected and performed for the reference. However, references stored on a call stack do not include any indication of GC state. Hence, loading a reference from heap to call stack involves removing the indication of GC state. Writing a reference to heap involves adding the indication of GC state. References embedded within a compiled method also do not indicate any GC state. Metadata of the compiled method indicate a GC state, which is implicated to the embedded references. GC operations are selected and performed for each embedded reference based on the GC state of the compiled method.

A processing system executes a specialized wavefront, referred to as a “garbage collecting wavefront” or GCWF, to identify and deallocate resources such as, for example, scalar registers, vector registers, and local data share space, that are no longer being used by wavefronts of a workgroup executing at the processing system (i.e., dead resources). In some embodiments, the GCWF is programmed to have compiler information regarding the resource requirements of the other wavefronts of the workgroup and specifies the program counter after which there will be a permanent drop in resource requirements for the other wavefronts. In other embodiments, the standard compute wavefronts signal the GCWF when they have completed using resources. The GCWF sends a command to deallocate the dead resources so the dead resources can be made available for additional wavefronts.

Systems, methods, and computer readable storage mediums for discovering volumes which are good candidates for space reclamation. A storage subsystem identifies the file system storage capacity for a given volume from the file system metadata of the given volume. Then, the storage subsystem compares the file system capacity of the given volume to the allocated capacity on the storage subsystem. If the allocated capacity is greater than the file system capacity by a given threshold, the storage subsystem marks the given volume as a candidate for space reclamation and generates an alert to the user to reclaim the space of the given volume.