The interface is to receive an indication to execute an optimize command. The processor is to receive a file name; determine whether adding a file of the file name to a current bin causes the current bin to exceed a threshold; associate the file with the current bin in response to determining that adding the file does not cause the current bin to exceed the bin threshold; in response to determining that adding the file to the current bin causes the current bin to exceed the bin threshold: associate the file with a next bin, indicate that the current bin is closed, and add the current bin to a batch of bins; determine whether a measure of the batch of bins exceeds a batch threshold; and in response to determining that the measure exceeds the batch threshold, provide the batch of bins for processing.

An identification of a primary snapshot created for a primary storage system is received. A first request for a first metadata of a first file directory structure object associated with the primary snapshot is issued. A second request for data content of the first file directory structure object associated with the primary snapshot is determined to be sent to a recipient device based on a received response to the first request. A third request for a second metadata of a second file directory structure object associated with the primary snapshot is determined to be sent to the recipient device. Timing and ordering of issuance of a plurality of requests that at least includes the second request and the third request to the recipient device are managed based on a determined performance metric of the recipient device and corresponding relative impact to the performance metric of the recipient device.

A computer program product, system, and computer implemented method for rapid database restoration using a database restore and recovery process that leverages one or more sparse data files and/or blocks by restoring one or more sparse data files and/or blocks and providing a mechanism to redirect requests to the one or more sparse data files and/or blocks to a backup copy of the actual data files and/or blocks and a process to populate the one or more sparse data files and/or blocks while the database is operational for servicing user requests. The approach includes the creation and population of one or more sparse data files and/or blocks, a redirection mechanism to service read operations where necessary, and a process to restore the data to one or more sparse data files and/or blocks over time, while the database maintains operability.

This patent document describes failure recovery technologies for the processing of streaming data, also referred to as pipelined data. The technologies described herein have particular applicability in distributed computing systems that are required to process streams of data and provide at-most-once and/or exactly-once service levels. In a preferred embodiment, a system comprises many nodes configured in a network topology, such as a hierarchical tree structure. Data is generated at leaf nodes. Intermediate nodes process the streaming data in a pipelined fashion, sending towards the root aggregated or otherwise combined data from the source data streams towards. To reduce overhead and provide locally handled failure recovery, system nodes transfer data using a protocol that controls which node owns the data for purposes of failure recovery as it moves through the network.

A technique for correcting errors in a data storage system operates while the data storage system remains online. The technique includes identifying an object for validation, scanning a plurality of pointers, and counting a number of pointers that point to the object. The technique further includes repairing a discrepancy between the count of pointers and a reference count stored in connection with the object.

There are provided methods and systems for correcting an error from a memory. For example, there is provided a system for mitigating an error in a memory. The system can include a memory controller communicatively coupled to a host. The memory controller may be configured to receive information associated with a memory location. The information can indicate the error at the memory location. The controller may be configured to perform, upon receiving the information, certain operations. The operations can include copying data around the memory location, placing the copied data in a reserved area. And the operations can further include outputting, to a central controller, a set of physical addresses associated with the reserved area, wherein the central controller is configured to modify the set of physical address to conduct a data recovery off-line.

Techniques for providing direct access to backup data can include: receiving a selection of a backup of a file system, wherein the backup is stored on a backup system connected to a data storage system; creating, in accordance with said selection of the backup, a remote copy of the backup, wherein the remote copy is stored on the backup system; generating a proxy file system for the remote copy on the backup system, wherein the proxy file system is mapped by the data storage system to the remote copy on the backup system; and providing a host with direct access to the remote copy using the proxy file system.

A reference snapshot of a storage is stored. Data changes that modify the storage are received. The data changes are captured by a write filter of the storage. The received data changes are logged. The data changes occurring after an instance time of the reference snapshot are applied to the reference snapshot to generate a first incremental snapshot corresponding to a first intermediate reference restoration point. The data changes occurring after an instance time of the first incremental snapshot are applied to the first incremental snapshot to generate a second incremental snapshot corresponding to a second intermediate reference restoration point.

Methods, computer program products, computer systems, and the like for efficient metadata management are disclosed, which can include receiving a subunit of storage, storing a first metadata portion of the subunit of storage in a first unit of storage, and storing a second metadata portion of the subunit of storage in a second unit of storage.

Staging data on a storage element integrating fast durable storage and bulk durable storage, including: receiving, at a storage element integrating fast durable storage and bulk durable storage, a data storage operation from a host computer; storing data corresponding to the data storage operation within fast durable storage in accordance with a first data resiliency technique; and responsive to detecting a condition for transferring data between fast durable storage and bulk durable storage, transferring the data from fast durable storage to bulk durable storage in accordance with a second data resiliency technique.