An approach for fingerprinting large data objects at the wire speed has been disclosed. The techniques include Fresh/Shift pipelining, split Fresh, optimization, online channel sampling, and pipelined selection. The architecture can also be replicated to work in parallel for higher system throughput. Fingerprinting may provide an efficient mechanism for identifying duplication in a data stream, and deduplication based on the identified fingerprints may provide reduced storage costs, reduced network bandwidth consumption, reduced processing time and other benefits. In some embodiments, fingerprinting may be used to ensure or verify data integrity and may facilitate detection of corruption or tampering. An efficient manner of generating fingerprints (either via hardware, software, or a combination) may reduce a computation load and/or time required to generate fingerprints.

Preparing data for deduplication including: generating, by a storage system for a compressed data block, a padded compressed data block by padding the compressed data block to conform to a fixed block size, wherein the fixed block size is greater than a size of the compressed data block; storing, in the storage system, the padded compressed data block beginning at a block boundary of a storage device in the storage system; and performing block-based deduplication on the storage system, wherein the block-based deduplication determines whether the padded compressed data block matches one or more other padded compressed data blocks stored in the storage system.

An approach for fingerprinting large data objects at the wire speed has been disclosed. The techniques include Fresh/Shift pipelining, split Fresh, optimization, online channel sampling, and pipelined selection. The architecture can also be replicated to work in parallel for higher system throughput. Fingerprinting may provide an efficient mechanism for identifying duplication in a data stream, and deduplication based on the identified fingerprints may provide reduced storage costs, reduced network bandwidth consumption, reduced processing time and other benefits. In some embodiments, fingerprinting may be used to ensure or verify data integrity and may facilitate detection of corruption or tampering. An efficient manner of generating fingerprints (either via hardware, software, or a combination) may reduce a computation load and/or time required to generate fingerprints.

A method of compression is disclosed in which an input sequence of bits is divided into a plurality of portions. Each portion is sub-divided into a plurality of sub-divisions. Frequency analysis is performed to determine the number of occurrences of each sub-division permutation and new values are assigned, based on the frequency analysis, to each of the sub-division permutations. For each portion a label representing the permutation of bits in that portion is assigned. The label comprises a representation of a combined value resulting from combining the new values associated with the sub-division permutations of that portion. A processed sequence of bits is generated by replacing, within the input sequence of bits, bit portions with the respective label representing the permutation of bits in that portion.

A computing device may identify a plurality of equally sized data blocks of a first navigation file and may identify a plurality of equally sized data blocks of a second navigation file. The computing device may perform binary difference operations between the data blocks of the first navigation file to corresponding data blocks of the second navigation file The result of the binary difference operations is stored in a plurality of navigation patch files. At least two of the plurality of navigation patch files are merged into a merged navigation patch file.

In one embodiment a system, apparatus, and method for optimizing index value lengths when indexing data items in an array of data items is described, the method including producing, at a first processor, an ordered series of index values, sending the ordered series of index values to an indexing processor, receiving, at the indexing processor, a data object including the array of data items, associating, at the indexing processor, a first part of one of the index values with a first one data item of the array of data items, associating, at the indexing processor, a second part of the one of the index values with a next one data item of the array of data items, repeating the steps of associating a first part of one of the index values and associating a second part of the one of the index values until all of the data items in the array of data items are indexed.

A storage aware memory controller and method for managing a physical storage system. A described controller includes: a system for mapping physical memory space into a memory region and a storage region; a system for applying different error protections schemes, in which a fine-grained memory fault tolerance scheme is applied to data in the memory region and a course-grained memory fault tolerance scheme is applied to data in the storage region; and an in-memory storage filesystem that compresses and stores individual pages of data in the storage region, wherein each page of data is compressed into a set of codewords that are codeword aligned such that no codeword shares compressed data from different pages, and wherein the in-memory storage filesystem stores a compression-aware logical block address (CA-LBA) for each page of data.

A computing device may identify a series of bits representative of a first binary large object (BLOB) for navigation data including road segments and road attributes. The computing device duplicates each bit of the series of bits a predetermined number of times to form a first bit string. The first bit string is larger than the series of bits by a factor of the predetermined number. The computing device performs a binary difference of the first bit string to a second bit string representative of a second BLOB. A result of the binary difference is stored in a navigation patch file.

Through use of the technologies of the present invention, one is able to store and to retrieve data efficiently. One may realize these efficiencies by coding the data and storing coded data that is of a smaller size than original data.

A computing device may identify a series of bits representative of a first binary large object (BLOB) for navigation data including road segments and road attributes. The computing device duplicates each bit of the series of bits a predetermined number of times to form a first bit string. The first bit string is larger than the series of bits by a factor of the predetermined number. The computing device performs a binary difference of the first bit string to a second bit string representative of a second BLOB. A result of the binary difference is stored in a navigation patch file.