System and method of handling journal space in a storage cluster with multiple delta log instances

Abstract

Techniques for handling journal space in a storage cluster with multiple delta log instances. The techniques include writing delta updates for a respective metadata type to an “active” set of data containers in a delta log instance and raw delta updates to a raw delta log, switching a designation of the “active” set of data containers from “active” to “de-staging” once one or more of the “active” set of data containers has been filled, writing a bookmark for the respective metadata type to the raw delta log and a bookmark list, determining that a de-staging operation has been completed for writing the delta updates from the “de-staging” set of data containers to a storage array, determining that the bookmark for the respective metadata type is the oldest bookmark in the list, and reclaiming space between a tail of the raw delta log and the bookmark written to the raw delta log.

Claims

1. A method of handling journal space in a storage cluster with multiple delta log instances, comprising: writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log; in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list; and in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log.

2. The method of claim 1 further comprising: in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, removing the first bookmark for the first metadata type from the bookmark list.

3. The method of claim 1 further comprising: writing delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log; and in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, writing a second bookmark for the second metadata type to both the raw delta log and the bookmark list.

4. The method of claim 3 further comprising: in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array: determining that the second bookmark is not the oldest bookmark in the bookmark list; and deferring reclaiming second space in the raw delta log.

5. The method of claim 4 further comprising: in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array: determining that the first bookmark is the oldest bookmark in the bookmark list; and reclaiming at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.

6. The method of claim 1 further comprising: setting a threshold specifying a minimum allowed free space in the raw delta log.

7. The method of claim 6 further comprising: determining that free space in the raw delta log is below the set threshold; and forcing at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array.

8. A storage node of a storage cluster, comprising: a volatile memory including a first delta log instance and a bookmark list, the first delta log instance having a respective set of data containers; a persistent memory including a raw delta log; and processing circuitry configured to execute program instructions to: write delta updates of a first metadata type to both the respective set of data containers in the first delta log instance and the raw delta log; in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, write a first bookmark for the first metadata type to both the raw delta log and the bookmark list; and in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaim at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and if the first bookmark is determined not to be the oldest bookmark in the bookmark list, defer the reclaiming of the first space in the raw delta log.

9. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions, in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, to remove the first bookmark for the first metadata type from the bookmark list.

10. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions to: write delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log; and in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, write a second bookmark for the second metadata type to both the raw delta log and the bookmark list.

11. The storage node of claim 10 wherein the processing circuitry is further configured to execute the program instructions to: in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array: determine that the second bookmark is not the oldest bookmark in the bookmark list; and defer reclaiming second space in the raw delta log.

12. The storage node of claim 11 wherein the processing circuitry is further configured to execute the program instructions to: in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array: determine that the first bookmark is the oldest bookmark in the bookmark list; and reclaim at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.

13. The storage node of claim 8 wherein the raw delta log is configured as a ring buffer having a head and a tail, and wherein the processing circuitry is further configured to execute the program instructions to write the delta updates of the first metadata type at the head of the raw delta log.

14. The storage node of claim 13 wherein the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the raw delta log.

15. The storage node of claim 8 wherein the bookmark list is configured as a linked list having a head and a tail, and wherein the processing circuitry is further configured to execute the program instructions to write the first bookmark for the first metadata type at the head of the linked list.

16. The storage node of claim 8 wherein the processing circuitry is further configured to execute the program instructions to set a threshold specifying a minimum allowed free space in the raw delta log.

17. The storage node of claim 16 wherein the processing circuitry is further configured to execute the program instructions to determine that free space in the raw delta log is below the set threshold, and force at least the first de-staging operation to be performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array.

18. A computer program product including a set of non-transitory, computer-readable media having instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method comprising: writing delta updates of a first metadata type to both a respective set of data containers in a first delta log instance and a raw delta log; in response to one or more of the respective set of data containers in the first delta log instance being filled with at least some of the delta updates, writing a first bookmark for the first metadata type to both the raw delta log and a bookmark list; and in response to a first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to a storage array: if the first bookmark is determined to be an oldest bookmark in the bookmark list, reclaiming at least first space between a tail of the raw delta log and the first bookmark written to the raw delta log; and if the first bookmark is determined not to be the oldest bookmark in the bookmark list, deferring the reclaiming of the first space in the raw delta log.

19. The computer program product of claim 18 wherein the method further comprises: in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array, removing the first bookmark for the first metadata type from the bookmark list.

20. The computer program product of claim 18 wherein the method further comprises: writing delta updates of a second metadata type to both a respective set of data containers in a second delta log instance and the raw delta log; in response to one or more of the respective set of data containers in the second delta log instance being filled with at least some of the delta updates, writing a second bookmark for the second metadata type to both the raw delta log and the bookmark list; in response to a second de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the second delta log instance to the storage array: determining that the second bookmark is not the oldest bookmark in the bookmark list; and deferring reclaiming second space in the raw delta log; and in further response to the first de-staging operation being performed for writing the delta updates filling one or more of the respective set of data containers in the first delta log instance to the storage array: determining that the first bookmark is the oldest bookmark in the bookmark list; and reclaiming at least the first space and the second space between the tail of the raw delta log and the second bookmark written to the raw delta log.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The foregoing and other objects, features, and advantages will be apparent from the following description of particular embodiments of the present disclosure, as illustrated in the accompanying drawings, in which like reference characters refer to the same parts throughout the different views.

(2) FIG. 1 is a block diagram of an exemplary data storage environment, in which techniques can be practiced for handling journal space in a storage cluster with multiple delta log instances;

(3) FIG. 2 is a block diagram of volatile memory, persistent memory, and storage array components of a storage node in the data storage environment of FIG. 1;

(4) FIGS. 3a-3g are block diagrams describing an illustrative example of handling journal space in a storage cluster with multiple delta log instances; and

(5) FIG. 4 is a flow diagram of an exemplary method of handling journal space in a storage cluster with multiple delta log instances.

DETAILED DESCRIPTION

(6) Techniques are disclosed herein for handling journal space in a storage cluster with multiple delta log instances. The disclosed techniques can include writing delta updates for a respective type of metadata to a current designated “active” set of data containers in a metadata delta log instance and raw delta updates for respective metadata types to a raw delta log, switching a designation of the current designated “active” set of data containers from “active” to “de-staging” once one or more of the current designated “active” set of data containers has been filled, and writing a bookmark for the respective metadata type to both the raw delta log and a bookmark list. The disclosed techniques can further include determining that a de-staging operation has been performed in a background process for writing the delta updates from the current designated “de-staging” set of data containers to a storage array, removing the bookmark for the respective metadata type from the bookmark list, determining that the removed bookmark was the “oldest” bookmark in the list, and reclaiming the space between a tail of the raw delta log and the bookmark written to the raw delta log. In this way, memory space reclamation can be more safely performed in a storage node that includes multiple delta log instances and a single raw delta log.

(7) FIG. 1 depicts an illustrative embodiment of an exemplary data storage environment 100, in which techniques can be practiced for handling journal space in a storage cluster with multiple delta log instances. As shown in FIG. 1, the data storage environment 100 can include a plurality of host computers 102.1, 102.2, . . . , 102.n, at least one data storage processor 104 (also referred to herein as a “storage node”), a storage array 106, and a communications medium 103 that includes at least one network 108. For example, each of the plurality of host computers 102.1, . . . , 102.n can be configured as a web server computer, a file server computer, an email server computer, an enterprise server computer, and/or any other suitable client/server computer or computerized device. The plurality of host computers 102.1, . . . , 102.n can be configured to provide, over the network 108, storage input/output (IO) requests (e.g., small computer system interface (SCSI) commands, network file system (NFS) commands) to the storage node 104. Such storage IO requests (e.g., write IO requests, read IO requests) can direct the storage node 104 to write and/or read data blocks, data pages, data files, or any other suitable data elements to/from volumes (VOLs), logical units (LUs), file systems, and/or any other suitable storage targets, such as a plurality of storage targets 118.1, 118.2, . . . , 118.m maintained in the storage array 106.

(8) The communications medium 103 can be configured to interconnect the plurality of host computers 102.1, . . . , 102.n with the storage node 104 to enable them to communicate and exchange data and/or control signaling. As shown in FIG. 1, the communications medium 103 can be illustrated as a “cloud” to represent different communications topologies such as a backbone topology, a hub-and-spoke topology, a loop topology, an irregular topology, and so on, or any suitable combination thereof. As such, the communications medium 103 can include copper-based data communications devices and cabling, fiber optic devices and cabling, wireless devices, and so on, or any suitable combination thereof. The communications medium 103 can be further configured to support storage area network (SAN)-based communications, network attached storage (NAS)-based communications, local area network (LAN)-based communications, metropolitan area network (MAN)-based communications, wide area network (WAN)-based communications, wireless communications, distributed infrastructure communications, and/or any other suitable communications.

(9) The storage node 104 can be connected directly to the storage array 106 or by an optional network infrastructure 110, which can include an Ethernet network, an InfiniBand network, a fiber channel network, and/or any other suitable network(s). As shown in FIG. 1, the storage node 104 can include a communications interface 112, processing circuitry 114, and a memory 116. The communications interface 112 can include one or more of an Ethernet interface, an InfiniBand interface, a fiber channel interface, and/or any other suitable communications interface. The communications interface 112 can further include SCSI target adapters, network interface adapters, and/or any other suitable adapters for converting electronic, optical, and/or wireless signals received over the network 108 to a form suitable for use by the processing circuitry 114.

(10) The memory 116 can include volatile memory such as a random-access memory (RAM) 122 or any other suitable volatile memory, as well as persistent memory such as a nonvolatile random-access memory (NVRAM) 124 or any other suitable persistent memory. The memory 116 can also store a variety of software constructs realized in the form of specialized code and data 128 (e.g., program instructions) that can be executed by the processing circuitry 114 to carry out the techniques and/or methods disclosed herein. The memory 116 can further include an operating system 126, such as a Linux operating system (OS), a Unix OS, a Windows OS, or any other suitable operating system.

(11) The processing circuitry 114 can include one or more physical storage processors and/or engines configured to execute the specialized code and data 128, as well as data movers, director boards, blades, IO modules, storage drive controllers, switches, and/or any other suitable computer hardware or combination thereof. For example, the processing circuitry 114 can execute the specialized code and data 128 as program instructions out of the memory 116, process storage IO requests (e.g., write IO requests, read IO requests) issued by the respective host computers 102.1, . . . , 102.n, and/or store data and/or metadata to the storage array 106 in the data storage environment 100, which can be a clustered RAID environment.

(12) As shown in FIG. 1, the storage array 106 can include a coded target 120, which can be configured to store coded data for use in regenerating lost or corrupted data on one or more of the storage targets 118.1, 118.2, . . . , 118.m. In certain implementations, the coded target 120 can be stored to a hard disk drive (HDD) configured to store parity data in a RAID array. Alternatively (or in addition), the storage targets 118.1, 118.2, . . . , 118.m and the coded target 120 can be stored to one or more HDDs, solid state drives (SSDs), flash devices, and/or any other suitable storage device(s). It is noted that the storage node 104 can include a keyboard, a mouse, and/or any other suitable 10 device(s), an uninterruptable power supply (UPS), and/or any other suitable storage node component(s).

(13) In the context of the processing circuitry 114 being implemented using one or more processors executing the specialized code and data 128, a computer program product can be configured to deliver all or a portion of the specialized code and data 128 to the respective processor(s). Such a computer program product can include one or more non-transient computer-readable storage media, such as a magnetic disk, a magnetic tape, a compact disk (CD), a digital versatile disk (DVD), an optical disk, a flash drive, a solid state drive (SSD), a secure digital (SD) chip or device, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and so on. Further, the non-transient computer-readable storage media can be encoded with sets of program instructions for performing, when executed by the respective processor(s), the various techniques and/or methods disclosed herein.

(14) FIG. 2 depicts volatile memory, persistent memory, and storage array components that can be included in the storage node 104 of FIG. 1. As shown in FIG. 2, the volatile memory components can be embodied in the RAM 122, and include multiple instances of metadata delta logs 204.1, 204.2, . . . , 204.p. The metadata delta log 204.1 can include a first set 210 of data buckets (also referred to herein as data “containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N (in which “N” is any suitable positive integer) and an associated bloom filter (BF) 206, as well as a second set 212 of data buckets (“containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N and an associated bloom filter (BF) 208. Likewise, the metadata delta log 204.2 can include a first set 218 of data buckets (“containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N and an associated bloom filter (BF) 214, as well as a second set 220 of data buckets (“containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N and an associated bloom filter (BF) 216. In general, the metadata delta log 204.p (in which “p” is any suitable positive integer) can include a first set 226 of data buckets (“containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N and an associated bloom filter (BF) 222, as well as a second set 228 of data buckets (“containers”) H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N and an associated bloom filter (BF) 224. Each of the first and second sets of data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N included in the respective metadata delta logs 204.1, . . . , 204.p can be configured to store delta updates to metadata pages. For example, each of the data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N can be configured as a binary tree (“b-tree”) structure or any other suitable data structure.

(15) As described herein, the processing circuitry 114 of the storage node 104 can execute the specialized code and data 128 as program instructions out of the memory 116, and process storage IO requests (e.g., write IO requests, read IO requests) issued by the respective host computers 102.1, . . . , 102.n to write/read data and/or metadata to/from the storage array 106. For example, each portion of metadata (e.g., each metadata delta update) can be stored to a metadata page on the storage array 106. Further, each metadata delta update can specify a change to a metadata page on the storage array 106.

(16) In certain implementations, the storage node 104 can convert a plurality of metadata delta updates into a plurality of delta update tuples 202, respectively. As shown in FIG. 2, each of the plurality of delta update tuples 202 can include several tuple parameters, such as a logical index “Li” of a corresponding metadata page “i,” an entry index “Ei” that refers to a specific delta update entry or offset within the metadata page “i,” a record or delta type “Tp” that defines the size or type “p” of the delta update entry, and a payload or value “V” of the delta update entry. The storage node 104 can also write delta update tuples for particular delta types T.sub.1, T.sub.2, . . . , T.sub.p to the metadata delta logs 204.1, . . . , 204.p, respectively, in the volatile memory (i.e., the RAM 122). For example, the storage node 104 can write delta update tuples “Li:Ei:T.sub.1:V” to the metadata delta log 204.1 (as illustrated by a path 236), as well as write delta update tuples “Li:Ei:T.sub.2:V” to the metadata delta log 204.2 (as illustrated by a path 238). In general, the storage node 104 can write delta update tuples “Li:Ei:T.sub.p:V” to the metadata delta log 204.p (as illustrated by a path 240), in which “p” is any suitable positive integer. The storage node 104 can subsequently store the delta update tuples written to the metadata delta logs 204.1, 204.2, . . . , 204.p to a metadata page store 234 (as illustrated by paths 244, 246, . . . , 248, respectively) included in the storage array 106.

(17) In certain implementations, the storage node 104 can determine the target data containers H.sub.1, H.sub.2, H.sub.3, . . . , or H.sub.N for the respective delta update tuples Li:Ei:T.sub.1:V in the first set 210 of data containers included in the metadata delta log 204.1 based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. Likewise, the storage node 104 can determine the target data containers H.sub.1, H.sub.2, H.sub.3, . . . , or H.sub.N for the respective delta update tuples Li:Ei:T.sub.2:V in the first set 218 of data containers included in the metadata delta log 204.2 based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. In general, the storage node 104 can determine the target data containers H.sub.1, H.sub.2, H.sub.3, . . . , or H.sub.N for the respective delta update tuples Li:Ei:T.sub.p:V in the first set 226 of data containers included in the metadata delta log 204.p (in which “p” is any suitable positive integer) based on a predetermined hash function of the logical index “Li” of the corresponding metadata page. In such implementations, the various data containers H.sub.1, . . . , H.sub.N included in the respective metadata delta logs 204.1, . . . , 204.p can be referred to as hash-based sorted buckets (HBSBs).

(18) As shown in FIG. 2, the storage node 104 can write copies of the delta update tuples Li:Ei:T.sub.p:V (also referred to herein as “raw delta updates”) for the particular delta types T.sub.1, T.sub.2, . . . , T.sub.p to a single journal (also referred to herein as the “raw delta log”) 205 (as illustrated by a path 242) included in the persistent memory (i.e., the NVRAM 124). In the event of a disaster, data loss, and/or data corruption, the storage node 104 can replay the raw delta log 205 to apply the delta update tuples written thereto to the metadata delta logs 204.1, 204.2, . . . , 204.p included in the volatile memory (i.e., the RAM 122), thereby recovering the metadata delta logs 204.1, 204.2, . . . , 204.p to a consistent state. In certain implementations, the raw delta log 205 can be configured as a ring buffer having a “head” 230 and a “tail” 232. Further, the storage node 104 can write each respective tuple Li:Ei:T.sub.p:V for a particular delta type T.sub.1, T.sub.2, . . . , or T.sub.p to the head 230 of the ring buffer, and subsequently release allocated space for the respective tuple Li:Ei:T.sub.p:V from the tail 232 of the ring buffer. As such, the raw delta log 205 can store the respective tuples Li:Ei:T.sub.p:V in time order (e.g., from oldest to newest). In certain implementations, the storage node 104 can determine that the raw delta log 205 is full, initiate one or more de-staging operations to store at least some of the delta update tuples written to the respective metadata delta logs 204.1, 204.2, . . . , 204.p to the metadata page store 234 (as illustrated by the paths 244, 246, . . . , 248, respectively) included in the storage array 106, and subsequently release the space for the delta update tuples from the tail 232 of the ring buffer.

(19) During operation, the storage node 104 can receive a plurality of write IO requests for one or more of the storage targets 118.1, . . . , 118.m from one or more of the host computers 102.1, . . . , 102.n. The plurality of write IO requests can direct the storage node 104 to write pending changes (“deltas,” “delta updates”) for one or more different metadata types to the respective metadata delta logs 204.1, . . . , 204.p included in the RAM 122. In response to receipt of the plurality of write IO requests, the storage node 104 can write delta updates for a respective metadata type to a metadata delta log instance optimized for handling the respective metadata type. For example, the metadata delta log 204.1 can be optimized for handling the metadata type “T.sub.1,” and the metadata delta log 204.2 can be optimized for handling the metadata type “T.sub.2.” In general, the metadata delta log 204.p (in which “p” is any suitable positive integer) can be optimized for handling the metadata type “T.sub.p.”

(20) With regard to the functionality of the metadata delta log 204.1 included in the RAM 122, the storage node 104 can write delta updates for the metadata type “T.sub.1” to fill one or more of the first set 210 of data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N with the metadata delta updates. It is noted that the first set 210 of data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N included in the metadata delta log 204.1 can initially be designated as the “active” set, while the second set 212 of data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N included in the metadata delta log 204.1 can initially be designated as the “de-staging” set. The storage node 104 can also write copies of delta updates (“raw delta updates”) to the raw delta log 205 included in the NVRAM 124. Once one or more of the first set 210 of data containers H.sub.1, H.sub.2, H.sub.3, . . . , H.sub.N have been filled, the storage node 104 can switch the designation of the first set 210 of data containers from “active” to “de-staging,” as well as switch the designation of the second set 212 of data containers from “de-staging” to “active.”

(21) In response to the switching of the “active” and “de-staging” designations for the first and second sets 210, 212 of data containers of the metadata delta log 204.1, the storage node 104 can write a first bookmark to the raw delta log 205. In certain implementations, such a bookmark can be configured as a record with several attributes, including a “type” attribute “T.sub.p” specifying the metadata type handled by the metadata delta log 204.p, an “identifier” attribute “I” specifying the current designated “active” set of data containers included in the metadata delta log 204.p, and an “offset” attribute “A.sub.p” specifying the current allocation offset in the raw delta log 205 (in which “p” is any suitable positive integer). The storage node 104 can also write the bookmark record having the attributes T.sub.p, I, A.sub.p to a bookmark list in the RAM 122. In certain implementations, the bookmark list can be configured as a linked list (or any other suitable memory structure) having a head and a tail, and the bookmark record having the attributes T.sub.p, I, A.sub.p can be written at the head of the bookmark list. The storage node 104 can then initiate a transaction commit operation (also referred to herein as a “de-staging operation”) to be performed in a background process, de-staging or otherwise writing the delta updates for the respective metadata type (e.g., the metadata type “T.sub.1”) from the current designated “de-staging” set (e.g., the first set 210 or the second set 212) of data containers to an associated metadata page in one of the plurality of storage targets 118.1, . . . , 118.m maintained in the storage array 106.

(22) It is noted that each of the remaining metadata delta logs 204.2, . . . , 204.p included in the RAM 122 can have the same functionality as the metadata delta log 204.1 described hereinabove. It is further noted that when one or more of the “active” set of data containers for a respective metadata delta log 204.1, . . . , or 204.p is full, the designation of the set of data containers can be switched from “active” to “de-staging,” and a de-staging operation can be initiated for a respective metadata type T.sub.1, T.sub.2, . . . , or T.sub.p from the full set of data containers. The order of the de-staging operations for the metadata delta logs 204.1, . . . , 204.p can therefore be unpredictable. Such de-staging operations can also be relatively long processes. Most of the time, de-staging operations are performed for all of the metadata types T.sub.1, T.sub.2, . . . , T.sub.p to assure that the overall de-staging operation is a smooth process. In addition, for each metadata delta log 204.1, . . . , 204.p, while the “active” set of data containers are being filled, the “de-staging” set of data containers can be de-staged. Once the “de-staging” set of data containers have completely de-staged, the next switch of the “active” and “de-staging” designations can occur.

(23) Having written raw delta updates to the raw delta log 205 for the different metadata types T.sub.1, T.sub.2, . . . , T.sub.p, as well as written bookmarks specifying the respective metadata types T.sub.1, T.sub.2, . . . , T.sub.p, to both the raw delta log 205 and the bookmark list, the storage node 104 can determine whether at least a first de-staging operation for delta updates of a first metadata type (e.g., the metadata type “T.sub.1”) and a second de-staging operation for delta updates of a second metadata type (e.g., the metadata type “T.sub.2”) have been performed in the background process. If the storage node 104 determines that the first de-staging operation has been performed before the second de-staging operation, the storage node 104 can remove the first bookmark having the attributes T.sub.1, I, A.sub.1 from the bookmark list, checking to see whether the first bookmark was removed from the tail of the bookmark list. In other words, the storage node 104 can check to see whether the first bookmark was the “oldest” bookmark in the bookmark list. If the first bookmark was indeed removed from the tail of the bookmark list, then the storage node 104 can safely perform a memory space reclaim operation to reclaim the space between a tail of the raw delta log and the first bookmark written to the raw delta log.

(24) If the storage node 104 determines that the second de-staging operation has been performed before the first de-staging operation, then the storage node 104 can remove a second bookmark having the attributes T.sub.2, I, A.sub.2 from the bookmark list, checking to see whether the second bookmark was removed from the tail of the bookmark list. In other words, the storage node 104 can check to see whether the second bookmark having the attributes T.sub.2, I, A.sub.2 was the “oldest” bookmark in the bookmark list. If the second bookmark having the attributes T.sub.2, I, A.sub.2 was not removed from the tail of the bookmark list (i.e., the first bookmark remains at the tail of the bookmark list), then the storage node 104 can defer performing a memory space reclaim operation on the raw delta log 205 until the first de-staging operation for the metadata type T.sub.1 has been performed in the background process.

(25) The disclosed techniques for handling journal space in a storage cluster with multiple delta log instances will be further understood with reference to the following illustrative example, and FIGS. 3a-3g. In this example, a storage node of a storage cluster writes copies of delta updates (“raw delta updates”) for different metadata types T.sub.1, T.sub.2, T.sub.3 to a single raw delta log 302 (see, e.g., FIG. 3a) included in a persistent memory (e.g., NVRAM), and determines whether to perform a memory space reclaim operation (or defer performing a memory space reclaim operation) on the raw delta log 302. The raw delta log 302 is configured as a ring buffer having a “head” 306 and a “tail” 308. In addition, the storage node 104 writes bookmarks having attributes T.sub.p, I, A.sub.p (e.g., p=1, 2, or 3) to a bookmark list 304 (see, e.g., FIG. 3d) in a volatile memory (e.g., RAM). The bookmark list 304 is configured as a linked list having a “head” 310, a “tail” 312, and a terminator node 310.

(26) In this example, the storage node writes delta updates for the metadata type “T.sub.1” to a first metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also writes copies of delta updates (“raw delta updates”) to space specified by an allocation offset “A.sub.1” in the raw delta log 302 (see FIG. 3a). Once one or more of the current designated “active” set of data containers in the first metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B.sub.1” at the head 306 of the raw delta log 302 (i.e., at the end of the space specified by “A.sub.1”) (see FIG. 3a). The storage node also writes the bookmark “B.sub.1” having attributes T.sub.1, 1, A.sub.1 at the head 310 of the bookmark list 304 (see FIG. 3d). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B.sub.1” is indicated as “1” for purposes of illustration.

(27) Further in this example, the storage node writes delta updates for the metadata type “T.sub.2” to a second metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also continues to write raw delta updates to space specified by an allocation offset “A.sub.2” in the raw delta log 302 (see FIG. 3b). Once one or more of the current designated “active” set of data containers in the second metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B.sub.2” at the head 306 of the raw delta log 302 (i.e., at the end of the space specified by “A.sub.2”) (see FIG. 3b). The storage node also writes the bookmark “B.sub.2” having attributes T.sub.2, 1, A.sub.2 at the head 310 of the bookmark list 304 (see FIG. 3d). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B.sub.2” is indicated as “1” for purposes of illustration.

(28) Still further in this example, the storage node writes delta updates for the metadata type “T.sub.3” to a third metadata delta log, filling one or more of a current designated “active” set of data containers with the metadata delta updates. The storage node also continues to write raw delta updates to space specified by an allocation offset “A.sub.3” in the raw delta log 302 (see FIG. 3c). Once one or more of the current designated “active” set of data containers in the third metadata delta log have been filled, the storage node switches the designation of the current designated “active” set from “active” to “de-staging.” Having switched the designation for the set of data containers from “active” to “de-staging,” the storage node writes a bookmark “B.sub.3” at the head 306 of the raw delta log 302 (i.e., at the end of the space specified by “A.sub.3”) (see FIG. 3c). The storage node also writes the bookmark “B.sub.3” having attributes T.sub.3, 1, A.sub.3 at the head 310 of the bookmark list 304 (see FIG. 3d). It is noted that the identifier attribute “I” that specifies the current designated “active” set of data containers in the bookmark “B.sub.3” is indicated as “1” for purposes of illustration.

(29) In addition, the storage node initiates one or more transaction commit operations (also referred to herein as “de-staging operations”) to be performed in one or more background processes, de-staging or otherwise writing the delta updates for a respective metadata type (e.g., the metadata type “T.sub.1,” “T.sub.2,” or “T.sub.3”) from a current designated “de-staging” set of data containers to an associated metadata page in a storage target maintained in a storage array. Further, the storage node determines whether at least a first de-staging operation for delta updates of the metadata type “T.sub.1,” a second de-staging operation for delta updates of the metadata type “T.sub.2,” and a third de-staging operation for delta updates of the metadata type “T.sub.3” have been performed in the background process(es).

(30) In this example, the storage node determines that the first de-staging operation for delta updates of the metadata type “T.sub.1” has been performed before both the second de-staging operation and the third de-staging operation. In this case, the storage node removes the bookmark “B.sub.1” having the attributes T.sub.1, I, A.sub.1 from the bookmark list 304, and checks to see whether the bookmark “B.sub.1” was removed from the tail 312 of the bookmark list 304. In other words, the storage node checks to see whether the bookmark “B.sub.1” was the “oldest” bookmark in the bookmark list 304. If the bookmark “B.sub.1” was indeed removed from the tail 312 of the bookmark list 304 (i.e., the bookmark “B.sub.1” was the “oldest” bookmark in the bookmark list 304), then the storage node can safely perform a memory space reclaim operation to reclaim the space specified by “A.sub.1” (as well as the space occupied by the bookmark “B.sub.1”) in the raw delta log 302. As shown in FIG. 3e, once the bookmark “B.sub.1” is removed from the bookmark list 304, the bookmarks “B.sub.2” and “B.sub.3” remain in the bookmark list 304. As further shown in FIG. 3e, the memory space reclaim operation can be safely performed by advancing the tail 308 of the raw delta log 302 to the start of the space specified by “A.sub.2.”

(31) If, however, the storage node determines that the second de-staging operation for delta updates of the metadata type “T.sub.2” has been performed before both the first de-staging operation and the third de-staging operation, then the storage node removes the bookmark “B.sub.2” having the attributes T.sub.2, I, A.sub.2 from the bookmark list 304, and determines that the bookmark “B.sub.2” was not removed from the tail 312 of the bookmark list 304. In other words, the storage node determines that the bookmark “B.sub.2” was not the “oldest” bookmark in the bookmark list 304. As shown in FIG. 3f, once the bookmark “B.sub.2” is removed from the bookmark list 304, the bookmarks “B.sub.1” and “B.sub.3” remain in the bookmark list 304. The storage node therefore defers performing a memory space reclaim operation on the raw delta log 302 until at least the first de-staging operation for delta updates of the metadata type “T.sub.1” has been performed in the background process(es). As further shown in FIG. 3f, because the memory space reclaim operation cannot be safely performed by the storage node at this time, the space specified by “A.sub.1,” “A.sub.2,” and “A.sub.3,” as well as their corresponding bookmarks “B.sub.1,” “B.sub.2,” and “B.sub.3,” remain in the raw delta log 302.

(32) The storage node then determines that the first de-staging operation for delta updates of the metadata type “T.sub.1” has been performed in the background process(es). Further, the storage node removes the bookmark “B.sub.1” having the attributes T.sub.1, I, A.sub.1 from the bookmark list 304, and determines that the bookmark “B.sub.1” was removed from the tail 312 of the bookmark list 304. In other words, the storage node determines that the bookmark “B.sub.1” was the “oldest” bookmark in the bookmark list 304. As shown in FIG. 3g, once the bookmark “B.sub.1” is removed from the bookmark list 304, only the bookmark “B.sub.3” remains in the bookmark list 304. Because the bookmark “B.sub.1” was indeed removed from the tail 312 of the bookmark list 304 (i.e., the bookmark “B.sub.1” was the “oldest” bookmark in the bookmark list 304), the storage node can safely perform a memory space reclaim operation to reclaim the adjacent space specified by “A.sub.1” and “A.sub.2” (as well as the space occupied by the bookmarks “B.sub.1” and “B.sub.2”) in the raw delta log 302. As described hereinabove, the storage node previously determined that the second de-staging operation for delta updates of the metadata type “T.sub.2” has been performed in the background process(es). As further shown in FIG. 3g, the memory space reclaim operation can be safely performed by advancing the tail 308 of the raw delta log 302 to the start of the space specified by “A.sub.3.” Once the storage node subsequently determines that the third de-staging operation for delta updates of the metadata type “T.sub.3” has been performed, the sole bookmark “B.sub.3” having the attributes T.sub.3, I, A.sub.3 can be removed from the bookmark list 304, and a memory space reclaim operation can be safely performed to reclaim the space specified by “A.sub.3” (as well as the space occupied by the bookmark “B.sub.3”) in the raw delta log 302.

(33) A method of handling journal space in a storage cluster with multiple delta log instances is described below with reference to FIG. 4. As depicted in block 402, delta updates of a metadata type are written to both a set of data containers in a delta log instance and a raw delta log. The term “active” corresponds to a designation initially applied to the set of data containers in the delta log instance. As depicted in block 404, in response to the designation of the set of data containers in the delta log instance being switched from “active” to “de-staging,” a bookmark for the metadata type is written to both the raw delta log and a bookmark list. As depicted in block 406, in response to a de-staging operation being performed for writing the delta updates of the metadata type from the set of data containers in the delta log instance to a storage array: the bookmark for the metadata type is removed from the bookmark list; if the bookmark was an oldest bookmark in the bookmark list, space is reclaimed in the raw delta log; and, if the bookmark was not the oldest bookmark in the bookmark list, the reclaiming of the space in the raw delta log is deferred.

(34) Having described the above illustrative embodiments, other alternative embodiments or variations can be made and/or practiced. For example, it was described herein with reference to the illustrative example that, if the storage node determines that the bookmark “B.sub.2” for the metadata type “T.sub.2” was not removed from the tail 312 of the bookmark list 304 (i.e., the bookmark “B.sub.2” was not the “oldest” bookmark, but, rather, the bookmark “B.sub.1” was the “oldest” bookmark in the bookmark list 304), then the storage node can defer performing a memory space reclaim operation on the raw delta log 302 until an additional de-staging operation (e.g., a de-staging operation for the metadata type “T.sub.1”) has been performed in a background process. As an alternative (or addition) to the disclosed techniques, in order to address the case where such a de-staging operation for the metadata type “T.sub.1” is not performed for an extended time (e.g., the metadata type “T.sub.1” may have become idle), the storage node can set a minimum allowed free space threshold for the raw delta log, and monitor the free space in the raw delta log 302 to determine whether or not it is below the set threshold. If it is determined that the free space in the raw delta log 302 is below the set threshold, then the storage node can force the additional de-staging operation (e.g., the de-staging operation for the metadata type “T.sub.1”) to be performed, thereby assuring that at least the minimum allowed free space in the raw delta log is maintained.

(35) It was further described herein that, in the event of a disaster, data loss, and/or data corruption, a storage node can replay the raw delta log to apply the delta update tuples written thereto to the multiple instances of metadata delta logs in the volatile memory, thereby recovering the metadata delta logs to a consistent state. As an alternative (or addition) to the disclosed techniques, the storage node can further replay the raw delta log to apply the several bookmarks written thereto to the bookmark list in the volatile memory, thereby recovering the bookmark list to a consistent state.

(36) Several definitions of terms are provided below for the purpose of aiding the understanding of the foregoing description, as well as the claims set forth herein.

(37) As employed herein, the term “storage system” is intended to be broadly construed to encompass, for example, private or public cloud computing systems for storing data, as well as systems for storing data comprising virtual infrastructure and those not comprising virtual infrastructure.

(38) As employed herein, the terms “client,” “host,” and “user” refer, interchangeably, to any person, system, or other entity that uses a storage system to read/write data.

(39) As employed herein, the term “storage device” may refer to a storage array including multiple storage devices. Such a storage device may refer to any non-volatile memory (NVM) device, including hard disk drives (HDDs), solid state drives (SSDs), flash devices (e.g., NAND flash devices, NOR flash devices), and/or similar devices that may be accessed locally and/or remotely (e.g., via a storage attached network (SAN)). A storage array (drive array, disk array) may refer to a data storage system used for block-based, file-based, or object storage. Storage arrays can include, for example, dedicated storage hardware containing HDDs, SSDs, and/or all-flash drives. A data storage entity may be any one or more of a file system, object storage, a virtualized device, a logical unit (LU), a logical unit number (LUN), a logical volume, a logical device, a physical device, and/or a storage medium. An LU may be a logical entity provided by a storage system for accessing data from the storage system and may be used interchangeably with a logical volume. An LU or LUN may be used interchangeably with each other. A LUN may be a logical unit number for identifying an LU and may also refer to one or more virtual disks or virtual LUNs, which may correspond to one or more virtual machines. A physical storage unit may be a physical entity such as a drive or disk or an array of drives or disks for storing data in storage locations that can be accessed by address. A physical storage unit may be used interchangeably with a physical volume.

(40) As employed herein, the term “storage medium” may refer to one or more storage media such as a hard drive, a combination of hard drives, flash storage, a combination of flash storage, a combination of hard drives, flash storage, and other storage devices, and/or any other suitable types or combinations of computer readable storage media. A storage medium may also refer to both physical and logical storage media, include multiple levels of virtual-to-physical mappings, and include an image or disk image. A storage medium may be computer-readable and may be referred to as a computer-readable program medium.

(41) As employed herein, the term “IO request” or simply “IO” may be used to refer to an input or output request such as a data read request or data write request.

(42) As employed herein, the terms, “such as,” “for example,” “e.g.,” “exemplary,” and variants thereof describe non-limiting embodiments and mean “serving as an example, instance, or illustration.” Any embodiments described herein using such phrases and/or variants are not necessarily to be construed as preferred or more advantageous over other embodiments, and/or to exclude the incorporation of features from other embodiments. In addition, the term “optionally” is employed herein to mean that a feature or process, etc., is provided in certain embodiments and not provided in other certain embodiments. Any particular embodiment of the present disclosure may include a plurality of “optional” features unless such features conflict with one another.

(43) While various embodiments of the present disclosure have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure, as defined by the appended claims.