A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

An input switching circuit dynamically connects, based on an input mapping table, input streams to inputs of a wavefront pre-transform circuit. An output switching circuit dynamically connects, based on an output mapping table, output data at outputs of the wavefront pre-transform circuit to transport streams. A controller controls, based on a wiping command, at least one of the input and output switching circuits to alter at least one of the input and output mapping tables such that the at least one of the input and output switching circuits is disabled for connection. A first subset of the transport streams operates in a foreground mode available to a user and is transported for storage in remote storage sites at a network and a second subset of the transport streams operates in a background mode available to an administrator and is not transported for storage in the remote storage sites.

Solutions for supporting storage using a multi-writer log-structured file system (LFS) are disclosed that include receiving incoming data from an object of a plurality of objects that are configured to simultaneously write to the LFS from different nodes; based at least on receiving the incoming data, determining whether sufficient free segments are available in a local segment usage table (SUT) for writing the incoming data; based at least on determining that insufficient free segments are available, requesting allocation of new free segments; writing the incoming data to a log; acknowledging the writing to the object; determining whether the log has accumulated a full segment of data; based at least on determining that the log has accumulated a full segment of data, writing the full segment of data to a first segment of the free segments; and updating the local SUT to mark the first segment as no longer free.

Intelligent management of stub files in hierarchical storage is provided by: in response to identifying a file to migrate from a file system to offline storage, providing metadata for the file to a machine learning engine; receiving a stub profile for the file from the machine learning engine that indicates an offset from a beginning of the file and a length from the offset for previewing the file; and migrating the portion of the file from the file system to an offline storage based on the stub profile. In some embodiments this further comprises: monitoring file system operations; in response to detecting a read operation of the portion of the file: determining a file type; providing file data to the machine learning engine; and performing a supervised learning operation based on the file type and the file data to update the machine learning engine.

Improving performance of garbage collection (GC) processes in a deduplicated file system having a layered processing architecture that maintains a log structured file system storing data and metadata in an append-only log arranged as a monotonically increasing log data structure of a plurality of data blocks wherein a head of the log increases in chronological order and no allocated data block is overwritten. The storage layer reserves a set of data block IDs within the log specifically for the garbage collection process, and assigns data blocks from the reserved set to GC I/O processes requiring acknowledgment in a possible out-of-order manner relative to an order of data blocks in the log. It strictly imposes using in-order I/O acknowledgement for other non-GC processes using the storage layer, where these processes may be deduplication backup processes using a segment store layer at the same protocol level as the GC layer.

A method of managing storage space of a storage device, wherein the storage device includes a plurality of snapshots of a file, includes the steps of: in response to a request to delete a first snapshot, determining a first amount of time that elapsed between a creation of the first snapshot and a creation of a second snapshot that is a child snapshot of the first snapshot: and after determining the first amount of time, executing a first process to delete the first snapshot over a first time interval, wherein the first time interval is based on the first amount of time.

Automated methods and systems for compressing log messages stored in a log message databased are described herein. The automated methods and systems perform lossy compression of an original set of log messages by identifying log messages that represent each of the various types of events recorded in the original set. The log messages in the original set are overwritten by corresponding representative log messages. Source coding is used to construct a source coding scheme and variable length binary codewords for each of the representative log messages. The representative log messages are replaced by the codewords, which occupies significantly less storage space than the original set. The lossy compressed set of log messages can be decompressed to obtain the representative log messages using the source coding scheme.

An embodiment may involve persistent storage including a parent filesystem and a pre-configured amount of free space within the parent filesystem that is dedicated for shared use. The embodiment may also involve one or more processors configured to, for each of a plurality of child filesystems: create a sparse file with an apparent size equivalent to the pre-configured amount of free space; create a virtual mapped device associated with the sparse file; establish one or more cryptographic keys for the virtual mapped device; create an encrypted virtual filesystem for the virtual mapped device and within the sparse file, wherein the encrypted virtual filesystem uses the cryptographic keys for application-transparent encryption and decryption of data stored by way of the encrypted virtual filesystem; and mount the encrypted virtual filesystem within the parent filesystem as one of the child filesystems.

A method of synchronously executing input/output operations (IOs) for a plurality of applications using a storage device with a file system includes the steps of: receiving a first write IO including an instruction to write first data at a first address of the file system; determining that, within a first range of the file system comprising the first address, there are no pending unmap IOs for deallocating storage space of the storage device from files of the plurality of applications; after determining that there are no pending unmap IOs within the first range, locking the first range to prevent incoming unmap IOs from deallocating storage space within the first range from the files of the plurality of applications; after locking the first range, writing the first data to the storage device at the first address; and after writing the first data, unlocking the first range.

Systems and methods for distributed file processing are disclosed. In one embodiment, a computer-implemented method for distributed file processing in a distributed network may include: (1) receiving, at a first distributed host in a network of a plurality of distributed hosts, an archive command; (2) the first distributed host identifying a plurality of files in a shared file system to archive in response to the archive command; (3) the first distributed host splitting at least one of the plurality of files that is above a predetermined size into a plurality of file chunks; (4) the first distributed host instructing a second distributed host to archive one of the plurality of files or plurality of file chunks to a store; and (5) the first distributed host and the second distributed host archiving the plurality of files and the plurality of file chunks to the store.