Systems and methods include obtaining statistics based on monitoring in a cloud-based system for a given time period; and, responsive to determining an arrangement of counters for N counters, storing each of M counters for the given time period as a plurality of records with each record including a record type, a possible offset to a next record in terms of a counter identifier (ID), and a counter value, wherein N and M are integers and M<<N, and wherein the arrangement is determined such that most frequently used counters occupy lower counter IDs. The systems and methods can further include updating the arrangement of the counters for the N counters, to perform an optimization such that the most frequently used counters occupy lower counter IDs.

Space efficient distributed storage systems are disclosed. For example, A system comprising a distributed storage volume (DSV) deployed on a plurality of hosts, the DSV comprising logical volumes, the logical volumes deployed on physical storage devices; and a first host of the plurality of hosts with a local cache, and a storage controller, the storage controller executing on a processor to receive a request relating to a first file; query the DSV to determine whether a second file that is a copy of the first file is stored in the DSV; and based on determining from the querying that the second file resides in a logical volume of the logical volumes in the DSV, store a separate reference to the second file in at least one logical volume of the DSV, wherein the separate reference is a virtual reference or link to the second file.

The storage device includes a first memory, a process device that stores data in the first memory and reads the data from the first memory, and an accelerator that includes a second memory different from the first memory. The accelerator stores compressed data stored in one or more storage drives storing data, in the second memory, decompresses the compressed data stored in the second memory to generate plaintext data, extracts data designated in the process device from the plaintext data, and transmits the extracted designated data to the first memory.

Systems, methods, and computer programs are disclosed for selectively compressing/decompressing flash storage data. An embodiment of a system comprises a compression/decompression component, a flash memory device, a flash controller in communication with the flash memory device, and a storage driver in communication with the compression/decompression component and the flash controller. The storage driver is configured to selectively control compression and decompression of data stored in the flash memory device, via the compression/decompression component, according to a storage usage collar comprising an upper usage threshold and a lower usage threshold.

Methods, systems, and computer-readable media are disclosed for dynamic partitioning in distributed computing environments. One method includes: receiving a first data set and a second data set; mapping the first data set into a first set of key-value pairs; mapping the second data set into a second set of key-value pairs; estimating, using a sketch, a frequency count for each key based on the first set of key-value pairs and the second set of key-value pairs; determining whether the estimated frequency count for each key is greater than or equal to a predetermined threshold; and partitioning the key when the estimated frequency count for the key is greater than or equal to the predetermined threshold.

Objects stored in a zip archive may be extracted in random-access fashion (without involving other objects stored in the zip archive) using the addresses of the objects stored in the central directory of the zip archive. However, zip archives often provide insufficient information to enable random access to the data within an object. This capability may be provided by segmenting the object into sections of a section size, and including in the zip archive a block table specifying, for respective sections, the block size of the corresponding block. A zip archive extractor may achieve random access to the object by using the block table to computing the addresses of blocks comprising the selected portion and extracting only those blocks. Backwards compatibility of the zip archive with other zip archive extractors may be preserved by including the block table within a zip extension of the central directory of the zip archive.

A file management system includes a communication device and a control device. The control device acts as a file manager, a file outputter, an information file generator, and a file merger. The file manager manages various files. The file outputter outputs a file based on a target file among the various files, to outside of the computer via the communication device. The information file generator generates an information file which is a document file containing information linked with the target file. The file merger generates a merged target file, by merging the information file with the target file, when the target file is of a specific file format. The file outputter outputs, when the file merger generates the merged target file, the merged target file to outside of the computer via the communication device, as the file based on the target file.

Disclosed is a method for identifying a group for digital content items. The method includes (a) retrieving one or more digital content item, (b) determining at least one characteristic of each digital content item, (c) transforming, with a first transformation method, the determined one or more digital content item to form one or more transformed digital content item, (d) determining at least one characteristic of the one or more digital content item, (e) assessing the determined at least one characteristic of the one or more digital content item with respective at least one characteristic of the one or more transformed digital content item to form a first transformation assessment result, and (f) using the first transformation assessment result to determine a group for the one or more digital content item.

A method for selectively compressing unstructured data includes preparing, at a host system, a data blob made up of unstructured data, for storage on a storage system. The method encodes a first portion of the data blob with a first color and a second portion of the data blob with a second color. The first color indicates that the associated portion should remain uncompressed, while the second color indicates that the associated portion should be compressed. The method transmits the encoded data blob to a storage virtualization layer associated with the storage system. In response, the storage virtualization layer stores the first portion in an uncompressed format, and compresses the second portion and stores the second portion in a compressed format. A corresponding system and computer program product are also disclosed herein.

A method, apparatus, and system for interposed file system driver is provided, which provides a logical file system on top of an existing base file system. One such interposed file system driver is a compression and deduplication layered driver (“COLD driver”). File system operations are intercepted from the operating system through the COLD driver, which is provided as an upper-level operating system driver that operates on top of an existing base file system. By processing file data through various modules, the existing base file system can be extended as a logical file system with compression, deduplication, indexing, and other functionality. The COLD driver can be implemented without requiring modifications to existing base file system structures or base file system drivers. Server deployments may thus leverage the additional file system functionality provided by the COLD driver without having to migrate to another file system.