A hybrid data storage device disclosed herein includes a main data store, one or more data storage caches, and a data storage cache management sub-system. The data storage cache may be divided into an over-provisioned portion and an advertised space portion. Clusters of data on the data storage cache corresponding to the over-provisioned portion are marked as pending eviction rather than actually evicted when the data storage cache management sub-system receives a write request, thereby increasing the effective capacity and reducing write amplification of the data storage cache.

A hybrid data storage device disclosed herein includes a main data store, one or more data storage caches, and a data storage cache management sub-system. The data storage cache may be divided into an over-provisioned portion and an advertised space portion. Clusters of data on the data storage cache corresponding to the over-provisioned portion are marked as pending eviction rather than actually evicted when the data storage cache management sub-system receives a write request, thereby increasing the effective capacity and reducing write amplification of the data storage cache.

A dual-server based storage system maintains a first cache and a first non-volatile storage (NVS) in a first server, and a second cache and a second NVS in a second server, where data in the first cache is also written in the second NVS and data in the second cache is also written in the first NVS. In response to a failure of the first server, a determination is made as to whether space exists in the second NVS to accommodate the data stored in the second cache. In response to determining that space exists in the second NVS to accommodate the data stored in the second cache, the data is transferred from the second cache to the second NVS.

A dual-server based storage system maintains a first cache and a first non-volatile storage (NVS) in a first server, and a second cache and a second NVS in a second server, where data in the first cache is also written in the second NVS and data in the second cache is also written in the first NVS. In response to a failure of the first server, a determination is made as to whether space exists in the second NVS to accommodate the data stored in the second cache. In response to determining that space exists in the second NVS to accommodate the data stored in the second cache, the data is transferred from the second cache to the second NVS.

Provided are a computer program product, system, and method for invoking demote threads on processors to demote tracks from a cache. A plurality of demote ready lists indicate tracks eligible to demote from the cache. In response to determining that a number of free cache segments in the cache is below a free cache segment threshold, a determination is made of a number of demote threads to invoke on processors based on the number of free cache segments and the free cache segment threshold. The determined number of demote threads are invoked to demote tracks in the cache indicated in the demote ready lists, wherein each invoked demote thread processes one of the demote ready lists to select tracks to demote from the cache to free cache segments in the cache.

Provided are a computer program product, system, and method for invoking demote threads on processors to demote tracks from a cache. A plurality of demote ready lists indicate tracks eligible to demote from the cache. In response to determining that a number of free cache segments in the cache is below a free cache segment threshold, a determination is made of a number of demote threads to invoke on processors based on the number of free cache segments and the free cache segment threshold. The determined number of demote threads are invoked to demote tracks in the cache indicated in the demote ready lists, wherein each invoked demote thread processes one of the demote ready lists to select tracks to demote from the cache to free cache segments in the cache.

In described examples, a processor system includes a processor core generating memory transactions, a lower level cache memory with a lower memory controller, and a higher level cache memory with a higher memory controller having a memory pipeline. The higher memory controller is connected to the lower memory controller by a bypass path that skips the memory pipeline. The higher memory controller: determines whether a memory transaction is a bypass write, which is a memory write request indicated not to result in a corresponding write being directed to the higher level cache memory; if the memory transaction is determined a bypass write, determines whether a memory transaction that prevents passing is in the memory pipeline; and if no transaction that prevents passing is determined to be in the memory pipeline, sends the memory transaction to the lower memory controller using the bypass path.

The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

The present invention is generally directed to a caching framework that provides a common abstraction across one or more big data engines, comprising a cache filesystem including a cache filesystem interface used by applications to access cloud storage through a cache subsystem, the cache filesystem interface in communication with a big data engine extension and a cache manager; the big data engine extension, providing cluster information to the cache filesystem and working with the cache filesystem interface to determine which nodes cache which part of a file; and a cache manager for maintaining metadata about the cache, the metadata comprising the status of blocks for each file. The invention may provide common abstraction across big data engines that does not require changes to the setup of infrastructure or user workloads, allows sharing of cached data and caching only the parts of files that are required, can process columnar format.

A synchronization capability to synchronize updates to page tables by forcing updates in cached entries to be made visible in memory (i.e., in in-memory page table entries). A synchronization instruction is used that ensures after the instruction has completed that updates to the cached entries that occurred prior to the synchronization instruction are made visible in memory. Synchronization may be used to facilitate memory management operations, such as bulk operations used to change a large section of memory to read-only, operations to manage a free list of memory pages, and/or operations associated with terminating processes.