In various examples, metadata may be generated corresponding to compressed data streams that are compressed according to serial compression algorithms—such as arithmetic encoding, entropy encoding, etc.—in order to allow for parallel decompression of the compressed data. As a result, modification to the compressed data stream itself may not be required, and bandwidth and storage requirements of the system may be minimally impacted. In addition, by parallelizing the decompression, the system may benefit from faster decompression times while also reducing or entirely removing the adoption cycle for systems using the metadata for parallel decompression.

Methods and systems for providing dynamic dictionary-based compression and decompression are described herein. A computing device may receive, during a currently running session with a client device, a plurality of messages. The computing device may determine, based on the plurality of messages, one or more frames. The computing device may determine, based on the one or more frames, data samples. The computing device may compress the one or more frames based on a compression dictionary. The computing device may train, during the currently running session, the compression dictionary based on the determined data samples, to create a new compression dictionary. The computing device may determine, during the currently running session and based on receiving additional messages, one or more additional frames. In addition, the computing device may compress the one or more additional frames based on the new compression dictionary.

According to one embodiment, a buffer stores first hash values and first complementary data. A first conversion unit converts consecutive characters in a second character string into second hash values and second complementary data. A search unit searches for consecutive first hash values from the buffer, and output a pointer. A selection unit selects consecutive first hash values and pieces of first complementary data from the buffer. A second conversion unit converts the consecutive first hash values into a third character string using the pieces of first complementary data. A comparison unit compares the second character string with the third character string to acquire a matching length. An output unit output the matching length with the pointer.

According to one embodiment, a memory system includes a compressor configured to output second data obtained by compressing input first data and a non-volatile memory to which third data based on the second data output from the compressor is written. The compressor includes a dictionary coding unit configured to perform dictionary coding on the first data, an entropy coding unit configured to perform entropy coding on the result of the dictionary coding, a first calculation unit configured to calculate compression efficiencies of the dictionary coding and the entropy coding, and a first control unit configured to control an operation of at least one of the dictionary coding unit and the entropy coding unit based on the compression efficiencies and a power reduction level.

Storage arrays in a mirror relationship use dynamic compression dictionary size adjustment and data buffer merging to more efficiently compress data associated with maintenance of consistency of a replicated storage object. A compression dictionary size selector selects a supported compression dictionary size based on the aggregate size of the updates enqueued for transmission in the data buffers, e.g., a compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the enqueued updates. A combiner selects enqueued updates based on the selected compression dictionary size, e.g., aggregate enqueued update size less than or equal to, and as close as possible to, the selected compression dictionary size. The selected updates are coalesced and compressed using the selected compression dictionary size.

Storage arrays in a mirror relationship use dynamic compression dictionary size adjustment and data buffer merging to more efficiently compress data associated with maintenance of consistency of a replicated storage object. A compression dictionary size selector selects a supported compression dictionary size based on the aggregate size of the updates enqueued for transmission in the data buffers, e.g., a compression dictionary size that is greater than or equal to, and as close as possible to, the aggregate size of the enqueued updates. A combiner selects enqueued updates based on the selected compression dictionary size, e.g., aggregate enqueued update size less than or equal to, and as close as possible to, the selected compression dictionary size. The selected updates are coalesced and compressed using the selected compression dictionary size.

Systems and methods are provided for encoding a multi-pixel caching scheme for lossless encoders. The systems and methods can include obtaining a sequence of pixels, determining repeating sub-sequences of the sequence of pixels consisting of a single repeated pixel and non-repeating sub-sequences of the sequence of pixels, responsive to the determination, encoding the repeating sub-sequences using a run-length of the repeated pixel and encoding the non-repeating sub-sequences using a multi-pixel cache, wherein the encoding using a multi-pixel cache comprises, encoding non-repeating sub-sequences stored in the multi-pixel cache as the location of the non-repeating sub-sequences in the multi-pixel cache, and encoding non-repeating sub-sequences not stored in the multi-pixel cache using the value of the pixels in the non-repeating sub-sequences.

Broadly speaking, the present techniques relate to a machine-implemented method for registering a device with a server, the method performed at the device comprising: applying a data-reducing function to at least one object, object instance, resource and/or resource instance at the device to generate resource data comprising compressed data representative of the at least one object object instance, resource and/or resource instance; transmitting a registration message comprising said resource data to register said device with server.

A computer memory compression method involves analyzing computer memory content with respect to occurrence of duplicate memory objects as well as value redundancy of data values in unique memory objects. The computer memory content is encoded by eliminating the duplicate memory objects and compressing each remaining unique memory object by exploiting data value locality of the data values thereof. Metadata is provided to represent the memory objects of the encoded computer memory content. The metadata reflects eliminated duplicate memory objects, remaining unique memory objects as well as a type of compression used for compressing each remaining unique memory object. A memory object in the encoded computer memory content is located using the metadata.

A highly programmable device, referred to generally as a data processing unit, having multiple processing units for processing streams of information, such as network packets or storage packets, is described. The data processing unit includes one or more specialized hardware accelerators configured to perform acceleration for various data processing functions. This disclosure describes a programmable hardware-based data compression accelerator that includes a pipeline for performing static dictionary-based and dynamic history-based compression on streams of information, such as network packets. The search block may support single and multi-thread processing, and multiple levels of compression effort. To achieve high-compression, the search block may operate at a high level of effort that supports a single thread and use of both a dynamic history of the input data stream and a static dictionary of common words. The static dictionary may be useful in achieving high-compression where the input data stream is relatively small.