Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

Techniques described herein relate to systems and methods of data storage, and more particularly to providing layering of file system functionality on an object interface. In certain embodiments, file system functionality may be layered on cloud object interfaces to provide cloud-based storage while allowing for functionality expected from a legacy applications. For instance, POSIX interfaces and semantics may be layered on cloud-based storage, while providing access to data in a manner consistent with file-based access with data organization in name hierarchies. Various embodiments also may provide for memory mapping of data so that memory map changes are reflected in persistent storage while ensuring consistency between memory map changes and writes. For example, by transforming a ZFS file system disk-based storage into ZFS cloud-based storage, the ZFS file system gains the elastic nature of cloud storage.

Methods, systems, apparatuses, and computer program products are described herein for the generation and traversal of a hierarchical index structure. The structure indexes search keys from data ingested from different data sources and enables efficient retrieval of the keys. As data is ingested, index nodes are generated at the lowest level of the structure. The nodes are analyzed to determine whether such nodes comprise duplicate keys. Responsive to doing so, a new index node is generated located at a higher level of the structure. This process results in a DAG comprising orphan nodes including different search keys. When processing a query for search keys, the orphan index nodes are initially analyzed for the keys. Upon finding a search key, its child nodes are recursively searched until location information specifying the location of ingested data in which the search key is located is found.

Methods, systems, apparatuses, and computer program products are described herein for the generation and traversal of a hierarchical index structure. The structure indexes search keys from data ingested from different data sources and enables efficient retrieval of the keys. As data is ingested, index nodes are generated at the lowest level of the structure. The nodes are analyzed to determine whether such nodes comprise duplicate keys. Responsive to doing so, a new index node is generated located at a higher level of the structure. This process results in a DAG comprising orphan nodes including different search keys. When processing a query for search keys, the orphan index nodes are initially analyzed for the keys. Upon finding a search key, its child nodes are recursively searched until location information specifying the location of ingested data in which the search key is located is found.

A file system manager implemented at a provider network identifies a storage device of a first group of storage devices of a provider network as an initial location of a file system object. Based on an access metric associated with the object, the file system manager initiates a transfer of contents of the object to a second storage device of a different storage device group, without receiving a client request specifying the transfer. In response to an access request received via a file system programmatic interface, contents of the object are provided from the second storage device. Based on a second access metric, the object is transferred back to the first group of storage devices.

A file system manager implemented at a provider network identifies a storage device of a first group of storage devices of a provider network as an initial location of a file system object. Based on an access metric associated with the object, the file system manager initiates a transfer of contents of the object to a second storage device of a different storage device group, without receiving a client request specifying the transfer. In response to an access request received via a file system programmatic interface, contents of the object are provided from the second storage device. Based on a second access metric, the object is transferred back to the first group of storage devices.

A stackable filesystem architecture that curtails data theft and ensures file integrity protection. In this architecture, processes are grouped into ranked filesystem views, or “security domains.” Preferably, an order theory algorithm is utilized to determine a proper domain in which an application is run. In particular, a root domain provides a single view of the filesystem enabling transparent filesystem operations. Each security domain transparently creates multiple levels of stacking to protect the base filesystem, and to monitor file accesses without incurring significant performance overhead. By combining its layered architecture with view separation via security domains, the filesystem maintains data integrity and confidentiality.

A stackable filesystem architecture that curtails data theft and ensures file integrity protection. In this architecture, processes are grouped into ranked filesystem views, or “security domains.” Preferably, an order theory algorithm is utilized to determine a proper domain in which an application is run. In particular, a root domain provides a single view of the filesystem enabling transparent filesystem operations. Each security domain transparently creates multiple levels of stacking to protect the base filesystem, and to monitor file accesses without incurring significant performance overhead. By combining its layered architecture with view separation via security domains, the filesystem maintains data integrity and confidentiality.

In some aspects, a computing system may generate a content-defined tree. A content-defined tree may be a tree of cryptographic hashes where each leaf is a hash of a chunk (e.g., data chunk) of a data object, and each parent node (e.g., interior node) is the hash of a concatenation of the hashes of the parent's children nodes. To create parent nodes for the leaf nodes, a computing system may group leaf nodes together based on a rolling hash (e.g., a rolling hash of the hashes of the leaf nodes) satisfying a condition. Each parent node may include a hash that represents the concatenation of the hashes of the leaf nodes that fall under the corresponding parent node.

Example methods and systems are directed to a multitenant application server using a union file system. Each tenant has one or more users and a tenant layer in the union file system. Each user has a user layer in the union file system. The union file system provides a logical file system to each user based on the user layer, the tenant layer, and a base layer comprising a set of application layers. A first user shares an application template file with other users of the same tenant by moving the file from the first user's user layer to the tenant layer. After the file is moved, all users of the tenant have access to the application defined by the application template file. The moving of the file is achieved by modifying metadata for the file.