A device, system and method for synchronizing databases changes is provided. A change-indication bit-vector indicates a position of a change to a stored data-set and/or a respective key position defined by a data-set representation (DSR) that defines a data-set structure. When the change comprises a data-field deletion, an indication of the change-indication bit-vector is provided to a replication node managing a replica database storing a replica data-set; the replication node updates the replica and a respective DSR to show the deletion at the indicated position. When the change comprises a data-field addition, the indication of the change-indication bit-vector and a record of a key value of a new data field and a data value of the new data field, is provided to the replication database; the replication node updates the replica and the respective DSR to show the addition at the indicated position.

Systems and methods for a bouncing replication protocol are described herein. The system can include a replication cluster including a plurality of servers and a master controller that can receive a first request to replicate a first transaction and execute a batching process to replicate the first transaction. The batching process can include selecting a first server for replication of the first transaction, determining that a pending acknowledgement from the selected first server has not been identified, adding the first transaction to a first batch for the first server, and sending the first batch including only the first transaction. The master controller can receive a second request to replicate a second transaction, execute the batching process with the master controller to replicate the second transaction, which executing of the batching process includes adding the second transaction to a second batch including a plurality of transactions, receive an acknowledgement indicative of completion of replication, and send the second batch upon receipt of the acknowledgement indicative of completion of replication.

A file access event relating to a file may be detected. A local file value rule may be applied to modify a local value of the file in response to the file access event. A local file replication rule may be applied using the modified local value to determine whether to replicate the file.

Dynamic application instance discovery and state management within a distributed system. A distributed system may implement application instances configured to perform one or more application functions within the distributed system, and discovery and failure detection daemon (DFDD) instances, each configured to store an indication of a respective operational state of each member of a respective group of the number of application instances. Each of the DFDD instances may repeatedly execute a gossip-based synchronization protocol with another one of the DFDD instances, where execution of the protocol between DFDD instances includes reconciling differences among membership of the respective groups of application instances. A new application instance may be configured to notify a particular DFDD instance of its availability to perform an application function. The particular DFDD instance may be configured to propagate the new instance's availability to other DFDD instances via execution of the synchronization protocol, without intervention on the part of the new application instance.

A system for data object replication includes at least one hardware processor and at least one memory storing instructions. The instructions cause the at least one hardware processor to perform operations including decoding a replication request to obtain a data object. Object dependencies associated with the data object and a plurality of additional data objects are determined. A replication sequence of the data object and the plurality of additional data objects is determined based on the object dependencies. A replication of the data object and at least one of the plurality of additional data objects is performed according to the replication sequence. A notification of a successful completion of the replication is generated.

Exemplary methods, apparatuses, and systems include a controller node receiving a request to perform a consistency check of a distributed file system. The controller node transmits, to each of a plurality of nodes, a request for the node to use logical metadata of the distributed file system owned by the node to construct an expected copy of physical metadata mapped to the logical metadata, determine which of the plurality of nodes own actual portions of the physical metadata, transmit corresponding portions of the expected copy of the physical metadata to each of the nodes determined to own actual portions of the physical metadata, and compare expected copies of the physical metadata received from other nodes to the actual physical metadata owned by the node. The controller node receives a result of the comparison from each of the nodes, aggregates the received results, and generates an error report.

A system and method for implementing optimized data replication across cloud storage nodes, the system comprising a cluster of computer system devices. The system comprises one or more memory devices and a plurality of processors. The one or more memory devices, stores a set of program modules. A processor among the plurality of processor executes the set of program modules. The set of program modules comprises an input module and a data transfer module. The input module receives a first instruction to add a first computer system device to the cluster, wherein the first computer system device comprises a first memory device. The data transfer module copies data in at least one memory device in the cluster of computer system devices, to the first memory device, based on number of computer system devices in the cluster being lesser than a predefined number.

The disclosed technology relates to a system configured to identify a first operation in a first set of operations configured to converge a server state and a file system state, wherein the first operation is not in a second set of operations generated in response to an change to at least one of the server state and the file system state. The system is further configured to cancel the first operation, identify a second operation in both the first set of operations and the second set of operations, and initiate execution of the second operation concurrently with the canceling of the first operation.

A method and apparatus for distribution of applications amongst a number of servers, ensuring that changes to application data on a master for that application are asynchronously replicated to a number of slaves for that application. Servers may be located in geographically diverse locations; the invention permits data replication over high-latency and lossy network connections and failure-tolerance under hardware and network failure conditions. Access to applications is mediated by a distributed protocol handler which allows any request for any application to be addressed to any server, and which, when working in tandem with the replication system, pauses connections momentarily to allow seamless, consistent live-migration of applications and their state between servers. Additionally, a system which controls the aforementioned live-migration based on dynamic measurement of load generated by each application and the topological preferences of each application, in order to automatically keep servers at an optimum utilisation level.

A distributed, web-services based storage system. A system may include a web services interface configured to receive, according to a web services protocol, a given client request for access to a given data object, the request including a key value corresponding to the object. The system may also include storage nodes configured to store replicas of the objects, where each replica is accessible via a respective unique locator value, and a keymap instance configured to store a respective keymap entry for each object. For the given object, the respective keymap entry includes the key value and each locator value corresponding to replicas of the object. A coordinator may receive the given client request from the web services interface, responsively access the keymap instance to identify locator values corresponding to the key value and, for a particular locator value, retrieve a corresponding replica from a corresponding storage node.