Embodiments described herein provide improved methods and systems for generating metadata for media objects at a computational engine (such as an artificial intelligence engine) within the storage edge controller, and for storing and using such metadata, in data processing systems.

Metadata computation apparatus includes a host interface, a storage interface and one or more processors. The host interface is configured to communicate over a computer network with one or more remote hosts. The storage interface is configured to communicate with one or more non-volatile memories of one or more storage devices. The processors are configured to manage local storage or retrieval of media objects in the non-volatile memories, to compute metadata for a plurality of media objects that are stored, or are en-route for storage, on the storage devices, wherein the media objects are of multiple media types, wherein the computed metadata tags a target feature in the media objects of at least two different media types among the multiple media types, and to store, in the non-volatile memories, the metadata tagging the target feature found in the at least two different media types, for use by the hosts.

A controller, for use in a storage device of a data processing system, includes a host interface, a memory interface and one or more processors. The host interface is configured to communicate over a computer network with one or more remote hosts of a data processing system. The memory interface is configured to communicate locally with a non-volatile memory of the storage device. The one or more processors are configured to manage local storage or retrieval of media objects at the non-volatile memory, and to selectively compute metadata that defines content characteristics of media objects that are stored, or that are to be stored, in the non-volatile memory.

A storage control device coupled to a storage device and located remotely from a host device receives media object data from the host device. The storage control device identifies a type of the media object data and select, based on the identified type, a computational model from among a plurality of computational models for use by a computational engine of the storage control device. The computational engine uses the selected computational model to generate metadata describing the media object data. The metadata is stored in the storage device so as to be selectively retrievable from the storage device separately from the media object data.

This disclosure describes a storage aggregator controller with metadata computation control. The storage aggregator controller communicates, via a host interface, over a computer network with one or more remote hosts, and also communicates, via a storage device interface, with a plurality of local storage devices, which are separate from the remote host(s) and which have respective non-volatile memories. The storage aggregator controller manages the local storage devices for storage or retrieval of media objects. The storage aggregator controller also governs a selective computation, at aggregator control circuitry or at a storage device controller of one or more of the storage devices, of metadata that defines content characteristics of the media objects that are retrieved from the plurality of storage devices or that are received from the one or more hosts over the computer network for storage in the plurality of storage devices.

Disclosed herein is a persistent memory (PMEM)-based distributed memory object system, referred to as the PMEM DMO system, that provides affordable means of integrating low-latency PMEM spaces with other devices, including servers that do not directly support PMEM. One embodiment comprises providing a cluster of servers with PMEM storage (PMEM servers) and connecting the PMEM servers to a plurality of applications servers using a low-latency network, such as a remote direct memory access; background processes on each of the application servers are tasked to perform input/output operations for the application servers to locally materialize objects from and synchronize/persist objects to the remote PMEM spaces on the PMEM servers. Data materialized from the PMEM servers is stored to the local cache of the application server for use. Also disclosed are data eviction policies for clearing the local cache of the application servers to make space for new data read.

A new snapshot of a storage volume is created by suppressing write requests. Once pending write requests from the computing nodes are completed, storage nodes create a new snapshot for the storage volume by allocating a new segment to the new snapshot. Subsequent write requests to the storage volume are then performed on the segments allocated to the new snapshot. A block map records segments where current data for an LBA of a slice of a storage volume is stored. Block maps may be written to a storage device in order to free memory. Block maps may be read back into memory when needed. Writing and reading of block maps may be performed upon fragments of block maps. On restarting of the storage node, block maps may be restored from block maps stored in the storage device.

Emulating a remote direct memory access (RDMA) link between controllers in a storage array, including: inserting, into a buffer utilized by a direct memory access (DMA) engine of a first storage array controller, a data transfer descriptor describing data stored in memory of the first storage array controller and a location to write the data to memory of the second storage array controller; retrieving, in dependence upon the data transfer descriptor, the data stored in memory of the first storage array controller; and writing the data into the memory of the second storage array controller in dependence upon the data transfer descriptor.

A multi-platform data storage system configured to accessing a plurality of storage platforms that use different storage access and/or storage management protocols. The multi-platform data storage system can, for example, include a storage mobility and management layer providing virtual management of data stored in the plurality of storage platforms, and a storage protocol converter operatively coupled between the storage mobility and management layer and the plurality of storage platforms. During access and/or management communication from the storage mobility and management layer to a particular one of the storage platforms, the storage protocol converter can operate to convert the access and/or management communication from a layer protocol used by the storage mobility and management layer to the storage access protocol used by the particular one of the storage platforms.

A multi-platform data storage system configured to maintain containers including one or more virtual storage resources. The multi-platform data storage system can, for example, include a storage interface configured to enable access to a plurality of storage platforms that use different storage access and/or management protocols, the plurality of storage platforms storing data objects in physical data storage; and a storage mobility and management layer providing virtual management of virtual storage resources corresponding to one or more data objects stored in the plurality of storage platforms, the storage mobility and management layer including at least a container management sub-system that manages logical containers that contain one or more of the virtual storage resources.