Systems and methods for an interconnect layer send queue reservation system are provided. In one example, a method involves performing a transfer of data (e.g., an NVLog) from a storage system to a secondary storage system. A send queue having a fixed number of slots is maintained within an interconnect layer interposed between a file system and a Remote Direct Memory Access (RDMA) layer of the storage system. The interconnect layer implements an application programming interface (API) for the reservation system. A deadlock situation is avoided by, during a suspendable phase of a write transaction, making a reservation for slots within the send queue via the reservation system for the transfer of data. When the reservation is successful, the write transaction proceeds with a modify phase, during which the reservation is consumed and the interconnect layer is caused to perform an RDMA operation to carry out the transfer of data.

A memory image can be captured by generating metadata indicative of a state of volatile memory and/or byte-addressable PMEM at a particular time during execution of a process by an application. This memory image can be persisted without copying the in-memory data into a separate persistent storage by storing the metadata and safekeeping the in-memory data in the volatile memory and/or PMEM. Metadata associated with multiple time-evolved memory images captured can be stored and managed using a linked index scheme. A linked index scheme can be configured in various ways including a full index and a difference-only index. The memory images can be used for various purposes including suspending and later resuming execution of the application process, restoring a failed application to a previous point in time, cloning an application, and recovering an application process to a most recent state in an application log.

In order to quickly recover a management information file, an Edge file storage stores user files and a management information file that manages management states of the user files in the Edge file storage. The Edge file storage manages operation logs indicating operation contents concerning user files accepted by nodes in association with the respective nodes. The Edge file storage extracts operation logs for a user file associated with a targeted management information file as a management information file stored in a failed node from the operation logs corresponding to the nodes. The Edge file storage aggregates operation logs that are extracted from the operation logs corresponding to the nodes and are used for the user file associated with the targeted management information file. The Edge file storage recovers the targeted management information file based on the aggregated operation log.

One example method includes performing a recovery operation. A recovery operation is performed using streams rather than volumes in the cloud and without using compute instances or servers for do data. Do data is written to a do stream. Occasionally, a compute instance power on reads data from the do stream. The do data ready from the do stream is applied to a cloud volume and a snapshot of the cloud volume is performed. The backups include discrete PiT backups and recovery can be performed to any of the discrete PiT backups.

One example method includes performing a recovery operation. A recovery operation is performed using streams rather than volumes in the cloud. Do data is written to a do stream. When the do data is applied to a cloud volume, which is a copy or replica of a production volume, data being replaced is saved to an undo stream to ensure that any PiT can be recovered using one or more of the cloud volume, the do stream and/or the undo stream.

One example method includes performing a recovery operation. A recovery operation is performed using streams rather than volumes in the cloud and without using compute instances or servers for do data or undo data. Do data is written to a do stream. Occasionally, a compute instance power on reads data from the do stream. After reading the data, chunks are read from cloud storage and updated. Data overwritten in the chunks are saved to an undo stream. A snapshot of the updated chunks and the associated undo stream is stored in the cloud storage.

Methods and apparatus for protecting data. Backup copies of data are created in real time and restoration of this backup data is enabled. For example, backup repositories of files stored in a primary storage device of a computer system may be created by examining information concerning the files to determine critical fields therein, and storage of the critical fields to a critical storage device and of non-critical fields and tags that are substituted for the critical fields to a context storage device effected. Following compromise of the files stored in the primary storage device, accesses by applications may be directed to the context storage device, e.g., as a means of rapid failover, and/or for each file stored in the context storage device, record-by-record copying of such files to the primary storage device may be effected to restore the contents of the primary storage device.

Techniques described herein relate to a method for managing data protection services for data clusters. The method includes identifying, by a data protection manager, a migration event associated with a first data cluster; in response to identifying the migration event: identifying a second data cluster capable of hosting the first data cluster using data cluster topology information and data cluster component inventory information associated with the first data cluster; generating a dependency map associated with the second data cluster based on the first data cluster; identifying a migration sequence based on the dependency map; initiating data cluster components of the second data cluster to perform migration of the first data cluster; and migrating the first data cluster to the second data cluster.

A method may include performing, at multiple data partitions, a point-in-time recovery to a specified time by applying transactions that have committed at each data partition up to the specified time. Open transactions that have not been committed at each data partition at the specified time may be identified. A corresponding transaction coordinator may be queried for an outcome of the open transactions. Open transactions that affect a single data partition may be aborted. The point-in-time recovery may be performed at the transaction coordinator partitions by determining the outcome of the open transactions. The transaction coordinator partitions may abort any transaction that remains open subsequent to the point-in-time recovery. If the specified time may be greater than a safety threshold, data up to an earlier point-in-time than specified may be recovered depending on whether there are missing portions of transaction logs at the data partitions or the transaction coordinator partitions.

A key-value engine of a storage system may perform a restart recovery after a system failure. The key-value engine may read a metadata log to locate a latest system checkpoint, and load a page mapping table from the latest system checkpoint. The key-value engine may replay to apply changes to the page mapping table from a system transaction log starting from a system transaction replay starting point. The key-value engine may further form one or more read-only replicas using an underlying file stream opened in a read-only mode during the recovery after the system failure to further facilitate fast recovery and provide fast response to user transactions that conduct read only transactions after the system failure.