Distribution of data in a neural network data set is used to determine an optimal compressor configuration for compressing the neural network data set and/or the underlying data type of the neural network data set. By using a generalizable optimization of examining the data prior to compressor invocation, the example non-limiting technology herein makes it possible to tune a compressor to better target the incoming data. For sparse data compression, this step may involve examining the distribution of data (e.g., in one example, zeros in the data). For other algorithms, it may involve other types of inspection. This changes the fundamental behavior of the compressor itself. By inspecting the distribution of data (e.g., zeros in the data), it also possible to very accurately predict the data width of the underlying data. This is useful because this data type is not always known a priori, and lossy compression algorithms useful for deep learning depend on knowing the true data type to achieve good compression rates.

Lossless compression of a content item using a neural network trained on content item cohorts. A computing system includes a neural network that is used to train a plurality of symbol prediction models. Each symbol prediction model is trained based on a corresponding cohort of content items. A particular symbol prediction model of the models trained is selected based on an intrinsic characteristic of a particular content item to be losslessly compressed such as, for example, the type or file extension of the content item. The content item is then losslessly compressed based on a set of symbol predictions fed to an arithmetic coder that are generated using the particular symbol prediction model selected.

A method for lossy data compression, the method including receiving raw data at a storage device, receiving a request to compress flag, accessing an onboard data compression algorithm library containing various data compression algorithms respectively corresponding to lossy data compression schemes, selecting one of the data compression algorithms based on a number of parameters, running the selected data compression algorithm either online such that the raw data is compressed by the storage device when it is received, and is then stored on the storage device as compressed data, or offline such that the raw data is stored at the storage device, is later compressed by the storage device according to the selected data compression algorithm, and is resaved at the storage device as compressed data.

Data compression techniques are provided that remove redundancy across the boundary of compression search engines. An illustrative method comprises splitting the data frame into a plurality of sub-chunks; comparing at least two of the plurality of sub-chunks to one another to remove at least one sub-chunk from the plurality of sub-chunks that substantially matches at least one other sub-chunk to generate a remaining plurality of sub-chunks; generating matching sub-chunk information for data reconstruction identifying the at least one removed sub-chunk and the corresponding substantially matched at least one other sub-chunk; grouping the remaining plurality of sub-chunks into sub-units; removing substantially repeated patterns within the sub-units to generate corresponding compressed sub-units; and combining the compressed sub-units with the matching sub-chunk information to generate a compressed data frame. The data frame optionally comprises one or more host pages compressed substantially simultaneously, and the compressed data frame for a plurality of host pages compressed substantially simultaneously comprises a host page address for each host page.

Embodiments of the present invention relate to the database field, and in particular, to a data transmission method and apparatus, so as to reduce overheads for data transmission between nodes in a distributed database and lighten network load. In the embodiments of the present invention, a DN determines a column that has a distribution rule of parameters in the column; the DN determines, according to a data type of the parameters of the column and the distribution rule of the parameters in the column, a compression algorithm corresponding to the column; the DN compresses the column using the compression algorithm; and the DN sends a compressed column to a target node. In this way, the overheads for the data transmission between the nodes are reduced, and the network load is lightened.

Data may be efficiently analyzed and compressed as part of a data compression service. A data compression request may be received from a client indicating data to be compressed. An analysis of the data or metadata associated with the data may be performed. In at least some embodiments, this analysis may be a rules-based analysis. Some embodiments may employ one or more machine learning techniques to historical compression data to update the rules-based analysis. One or more compression techniques may be selected out of a plurality of compression techniques to be applied to the data. Data compression candidates may then be generated according to the selected compression techniques. In some embodiments, a compression service restriction may be enforced. One of the data compression candidates may be selected and sent in a response.

A system is disclosed comprising a transceiver, transcoder, memory, and a processor for receiving raw data, partitioning the raw data into substrings of predetermined length, assigning each substring to a corresponding predetermined frequency based upon a data set or first lookup table based on the substring's given pattern, and transmitting said frequency using an antenna. Embodiments include a compression component for receiving raw data as input, breaking the raw data into subsets of predetermined length, comparing the raw data to a second lookup table, the second lookup table comprising all possible bit patterns for a file of the length of the raw data, wherein the possible bit patterns are partitioned in n-bit partitions, the n-bit partitions having a corresponding assigned value, the values of which are assembled by a given function so as to produce a code for each possible bit pattern.

Video coding systems or apparatus utilizing context-based adaptive binary arithmetic coding (CABAC) during encoding and/or decoding, are configured according to the invention with an enhanced binarization of non-zero Delta-QP (dQP). During binarization the value of dQP and the sign are separately encoded using unary coding and then combined into a binary string which also contains the dQP non-zero flag. This invention capitalizes on the statistical symmetry of positive and negative values of dQP and results in saving bits and thus a higher coding efficiency.

Methods, apparatuses, and storage media associated with compaction of data from one or more computing devices are disclosed. In various embodiments, one or more Internet of Things (IoT) devices may transmit information to a computing system. The computing system may group together raw data received from these one or more IoT devices based on a shared attribute. The computing system may select a compaction scheme to represent the knowledge conveyed by a group of the raw data. The computing system may apply this compaction scheme to the group of raw data to generate data that is representative of the group of raw data. Other embodiments may be disclosed or claimed.

Methods, devices and systems enhance compression and decompression of data values when they comprise a plurality of semantically meaningful data fields. According to a first inventive concept of the present invention disclosure, compression is not applied to each data value as a whole, but instead to at least one of the semantically meaningful data fields of each data value, and in isolation from the other ones. A second inventive concept organizes the data fields that share the same semantic meaning together to accelerate compression and decompression as multiple compressors and decompressors can be used in parallel. A third inventive concept is a system where methods and devices are tailored to perform compression and decompression of the semantically meaningful data fields of floating-point numbers after first partitioning further at least one of said data fields into two or a plurality of sub-fields to increase the degree of value locality and improve compressibility of floating-point values.