Space efficient distributed storage systems are disclosed. For example, A system comprising a distributed storage volume (DSV) deployed on a plurality of hosts, the DSV comprising logical volumes, the logical volumes deployed on physical storage devices; and a first host of the plurality of hosts with a local cache, and a storage controller, the storage controller executing on a processor to receive a request relating to a first file; query the DSV to determine whether a second file that is a copy of the first file is stored in the DSV; and based on determining from the querying that the second file resides in a logical volume of the logical volumes in the DSV, store a separate reference to the second file in at least one logical volume of the DSV, wherein the separate reference is a virtual reference or link to the second file.

Cache line data and metadata are compressed and stored in first and, optionally, second memory regions, the metadata including an address tag When the compressed data fit entirely within a primary block in the first memory region, both data and metadata are retrieved in a single memory access. Otherwise, overflow data is stored in an overflow block in the second memory region. The first and second memory regions may be located in the same row of a DRAM, for example, or in different regions of a DRAM and may be configured to enable standard DRAM components to be used. Compression and decompression logic circuits may be included in a memory controller.

In an aspect, high priority lines are stored starting at an address aligned to a cache line size for instance 64 bytes, and low priority lines are stored in memory space left by the compression of high priority lines. The space left by the high priority lines and hence the low priority lines themselves are managed through pointers also stored in memory. In this manner, low priority lines contents can be moved to different memory locations as needed. The efficiency of higher priority compressed memory accesses is improved by removing the need for indirection otherwise required to find and access compressed memory lines, this is especially advantageous for immutable compressed contents. The use of pointers for low priority is advantageous due to the full flexibility of placement, especially for mutable compressed contents that may need movement within memory for instance as it changes in size over time

Systems, methods, and computer programs are disclosed for selectively compressing/decompressing flash storage data. An embodiment of a system comprises a compression/decompression component, a flash memory device, a flash controller in communication with the flash memory device, and a storage driver in communication with the compression/decompression component and the flash controller. The storage driver is configured to selectively control compression and decompression of data stored in the flash memory device, via the compression/decompression component, according to a storage usage collar comprising an upper usage threshold and a lower usage threshold.

Systems, methods, and computer programs are disclosed for scheduling decompression of an application from flash storage. One embodiment of a system comprises a flash memory device and a preemptive decompression scheduler component. The preemptive decompression scheduler component comprises logic configured to generate and store metadata defining one or more dependent objects associated with the compressed application in response to an application installer component installing a compressed application to the flash memory device. In response to a launch of the compressed application by an application launcher component, the preemptive decompression scheduler component determines from the stored metadata the one or more dependent objects associated with the compressed application to be launched. The preemptive decompression scheduler component preemptively schedules decompression of the one or more dependent objects based on the stored metadata.

Some aspects of the disclosure relate to a pre-fetch mechanism for a cache line compression system that increases RAM capacity and optimizes overflow area reads. For example, a pre-fetch mechanism may allow the memory controller to pipeline the reads from an area with fixed size slots (main compressed area) and the reads from an overflow area. The overflow area is arranged so that a cache line most likely containing the overflow data for a particular line may be calculated by a decompression engine. In this manner, the cache line decompression engine may fetch, in advance, the overflow area before finding the actual location of the overflow data.

A method includes inputting a decompressing compressed image in a computing device. The method also includes applying one or more delta images by a processor to reduce a transfer time for the inputted and decompressed patch image. The method also includes performing one or more calls to a system of memory caches for reads and writes of the decompressed patch image and additional input and output due to a shortage of space in an internal random-access memory (RAM). The method also includes locating arbitrary storage to redirect the decompressed patch image and the additional input and output. The method also includes redirecting the inputted decompressed patch image and the additional input and output to the arbitrary storage.

A system includes a compression engine that stores the compression format information embedded in the compressed data. The compression format information can be included in a header that includes compression control surface (CCS) information. The system includes a shared memory to store compressed data for multiple hardware pipelines, where blocks of the compressed data have a common memory footprint and the compression header. The compression engine can compress data to store in the shared memory including generation of the header. The compression engine can decompress data read from the shared memory, including identification of the compression format from the header.

According to one embodiment of the present disclosure, an approach is provided in which a processor selects a page of data that is compressed by a first compression algorithm and stored in a memory block. The processor identifies a utilization amount of the compressed page of data and determines whether the utilization amount meets a utilization threshold. When the utilization amount fails to meet the utilization threshold, the processor uses a second compression algorithm to recompresses the page of data.

A graphics processor includes multiple levels of memory units, including a memory device and a cache device located near a graphics component. The graphics processor includes distributed compression/decompression, including a module between the cache device and the memory device. The module can perform compression of write data when the write data is moved from the cache device to the memory device, and perform decompression of read data when the read data is moved from the memory device to the cache device. The graphics processor can include a second level of cache with another compression module between the first level of cache and the second level of cache.