Flow state information that is stored in a first memory among a plurality of memories for maintaining flow state information at a network device is updated based on packets ingressing the network device. The memories are arranged in a hierarchical arrangement in which memories at progressively higher levels of hierarchy are configured to maintain flow state information corresponding to progressively larger sets of flows processed by the network device. When it is determined that a fullness level of the first memory exceeds a first threshold, flow state information associated with at least one flow, among a first set of flows for which flow state information is currently being maintained in the first memory, is transferred from the first memory to a second memory, the second memory being at a higher hierarchical level than the first memory. A new flow is instantiated in space freed up in the first memory.

Secure distributed storage and transmission of electronic content is provided over at least one communication network. At least one data file is received and parsed into a plurality of segments, wherein each one of the segments has a respective size. Thereafter, each of the plurality of segments is divided into a plurality of slices, wherein each one of the slices has a respective size. A plurality of data chunks are encoded, each data chunk comprising a portion of at least two of the slices, wherein no portion comprises an entire slice. The data chunks are packaged with at least metadata, and each of the packages is assigned to respective remote storage nodes. Each of the packages is transmitted to the respectively assigned remote storage node.

A request is received, from a mutator thread, to load a first reference to a first object from a heap memory onto a call stack of the application thread. Responsive to receiving the request, a system retrieves the first reference from the heap memory. The system executes a bitwise shift operation that (a) removes one or more bits representing the first garbage collection state and (b) generates a second reference from the first reference. Based on a particular bit, of the one or more bits removed from the first reference by the shift operation, the system determines whether to perform a set of garbage collection operations on the first reference to bring the first reference to a good state. The second reference, without any indication of any of the plurality of garbage collection states, is stored to the call stack.

Systems, methods, and computer readable storage mediums for discovering volumes which are good candidates for space reclamation. A storage subsystem identifies the file system storage capacity for a given volume from the file system metadata of the given volume. Then, the storage subsystem compares the file system capacity of the given volume to the allocated capacity on the storage subsystem. If the allocated capacity is greater than the file system capacity by a given threshold, the storage subsystem marks the given volume as a candidate for space reclamation and generates an alert to the user to reclaim the space of the given volume.

Techniques are provided for volume group backup, volume group restore, and volume group garbage collection for volume groups backed up to an object store. A volume group workflow is implemented to orchestrate individual consistent volume workflows that are separately and individually implemented by nodes hosting constituent volumes of a volume group. The volume group workflow and the individual consistent volume workflows are performed to back up the volume group to the object store, restore a volume group backup from the object store to a restore destination, and/or perform garbage collection on slots of objects storing data unique to a volume group backup to delete.

A technique improves storage efficiency of an object store configured to maintain numerous snapshots for long-term storage in an archival storage system by efficiently determining data that is exclusively owned by an expiring snapshot to allow deletion of the expiring snapshot from the object store. The technique involves managing index data structures to enable efficient garbage collection across a very large number of data objects. When a snapshot expires, the technique obviates the need to scan the numerous snapshot data objects to determine which index structures are no longer needed and can be reclaimed (garbage collected). The technique is directed to management of underlying storage based on different sets of policies. When certain snapshots expire and are ready for deletion, the technique is directed to finding those data blocks that are no longer referenced (used) by any valid snapshots.

Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.

A request is received, from a mutator thread, to load a first reference to a first object from a heap memory onto a call stack of the application thread. Responsive to receiving the request, a system retrieves the first reference from the heap memory. The system executes a bitwise shift operation that (a) removes one or more bits representing the first garbage collection state and (b) generates a second reference from the first reference. Based on a particular bit, of the one or more bits removed from the first reference by the shift operation, the system determines whether to perform a set of garbage collection operations on the first reference to bring the first reference to a good state. The second reference, without any indication of any of the plurality of garbage collection states, is stored to the call stack.

During execution of garbage collection marking, an application thread receives a first request to overwrite a reference field of an object, the object comprising at least a first reference and the first request comprising a second reference to be written to the reference field. Responsive to receiving the first request, the application thread determines a marking parity for objects being traversed by the garbage collection marking process and loads the first reference from the heap. The application thread determines that marking metadata of the first reference does not match the marking parity. Responsive to that determination, the application thread adds the first reference to a marking list, modifies the second reference to include the current marking parity as the marking metadata, and stores the modified second reference to the first reference field. In subsequent writes to the reference field, the application thread refrains from adding to the marking list.

During execution of garbage collection, an application receives a first request to overwrite a reference field of an object, the object comprising a first reference and the first request comprising a memory address at which the reference field is stored, and a second reference to be written to the reference field. Responsive to receiving the first request, the system determines a current remembered set phase, and loads the first reference. The application determines that remembered set metadata of the first reference does not match the current remembered set phase. Responsive to that determination, the application adds an entry to a remembered set data structure, modifies the second reference to include the current remembered set phase as the remembered set metadata, and stores the modified second reference to the reference field. In subsequent writes to the reference field, the application refrains from adding to the remembered set data structure.