One example method includes receiving data, partitioning the data according to their respective similarity groups, and the similarity groups collectively define a range of similarity groups, deduplicating the data after the partitioning, packing unique data segments remaining after deduplicating into one or more compression regions, compressing the compression regions, and writing an object, that includes the compression regions, to a durable log. The deduplicating and compressing for a similarity group may be performed by a dedup-compression instances uniquely assigned to that similarity group.

Embodiments for enabling granular migration of data with high efficiency. A defined metadata element, a tag, is assigned to each file, and then tag filtering is used to direct the data to the proper location. Files with different tags can be selected for transfer, and such a group of tags is referred to as a tag set. Embodiments can be used with a defined backup system file migration process, such as present in the Data Domain File System. By using snapshots, incoming new data (ingested file) is allowed to continue while the migration is in process and maintaining data consistency at the same time. This is achieved by performing operations on B+ Tree snapshots in conjunction with tag filtering on keys present in the leaf pages of these structures. This method is efficient became it makes a single pass walk of a B+ Tree in contrast with previous methods that look up files one-by-one via their pathname.

A client application of a content management system provides instructions for synchronizing content items and placeholder items using a local file journal and updated file journal. The client application compares entries in the updated file journal to entries in the local file journal to determine modifications to make to content items or placeholder items stored in a shared content storage directory on the client device. Based on the result of the comparison, the client application replaces placeholder items with content items or replaces content items with placeholder items.

A digital security system can store data associated with entities in resolver trees. If the digital security system determines that two resolver trees are likely representing the same entity, the digital security system can use a merge operation to merge the resolver trees into a single resolver tree that represents the entity. The single resolver tree can include a merge node indicating a merge identifier of the merge operation. Nodes containing information merged into the resolver tree from another resolver tree during the merge operation can be tagged with the corresponding merge identifier. Accordingly, if the merge operation is to be undone, for instance if subsequent information indicates that the entries are likely separate entities, the resolver tree can be unmerged and the nodes tagged with the merge identifier can be restored to a separate resolver tree.

One or more buckets of key-value pairs of a first node of a distributed storage system are selected to be migrated to a second node of the distributed storage system. One or more underlying database files corresponding to the one or more selected buckets are identified. The one or more identified underlying database files are directly copied from a storage of a first node to a storage of the second node. The copied underlying database files are linked in a database of the second node to implement the one or more selected buckets in the second node.

A method for updating a log structured merged (LSM) tree, the method includes (a) performing preemptive full merge operations at first LSM tree levels; and (b) performing capacity triggered merge operations at second LSM tree levels while imposing one or more restrictions; wherein the second LSM tree levels comprise a largest LSM tree level and one or more other second LSM tree levels that are larger from each first LSM tree level; wherein files of the one or more other second LSM tree levels are aligned with files of the largest LSM tree level.

A method, apparatus, and computer program product for communicating real-time updates among web clients. A structured data object is composed from a set of data nodes according to a domain specific language in response to requesting access to a data within a human resources database. The structured data object is interpreted within a context of a record ID/form to identify content for the web page and to determine whether real-time updates are enabled for the web page. Interpreting the structured data object, displays the form with the content. An event queue is created and a connection to the server hub is established when real-time updates are enabled. In response to receiving an update event in the event queue, a notification of the update event is displayed when the update event matches the record ID/form that a first user has opened.

An information processing device includes a memory, and a processor coupled to the memory and configured to acquire a plurality of data from first databases, where the data are stored in the first databases in a first period, and identifiers are stored in the first databases in association with the data, where the first databases correspond to servers that perform manipulations on a plurality of records containing the data stored in a second database, and the identifiers indicate order of the manipulations, and store the data in a third database in order indicated by the identifiers.

Garbage collection processing in a distributed shared log system includes a client identifying obsoleted log entries for a shared data object. The client sends information associated with the identified obsoleted log entries to a shared log server. The shared log server receives information associated with obsoleted log entries from all clients in the distributed shared log system and uses the information to delete the obsoleted log entries. The shared log server can update a snapshot mark to indicate the earliest time that a snapshot of the shared log can be taken. The snapshot mark can be updated based on the information associated with obsoleted log entries.

Dynamic generation of data catalogs may be implemented for accessing data sets in different storage locations. Data sets may be accessed in order to extract portions of data. Structure recognition techniques may be applied to the extracted data in order to determine structural information for the data sets. The structural information may then be stored as part of a data catalog for the data sets. Requests to access the data catalog from different clients may be received and the requested structural data supplied so that the clients may access different data sets utilizing the supplied structural data. Data catalogs may be updated as changes to data sets are made.