Generic multi-source heterogeneous large-scale data synchronization client-server method

Abstract

The present disclosure provides a generic multi-source heterogeneous large-scale data system, including a sync node config unit, an install & deployment unit, a block & pipelining unit, a unilateral sync unit, a bilateral sync unit and a correctness guarantee unit. The system operates on a middle layer which is above a node database layer and beneath an application logic layer. In a data synchronization process, a client end transmits captured local change information to a server end in accordance with a synchronization task plan; the server end receives and transfers the change information to an asynchronous parallel message processing mechanism to be stored in a corresponding message queue; the server end polls a local message queue for reading to-be-processed change information, and then performs subsequent data change in accordance with rules for heterogeneous data mapping, so as to maintain consistence of synchronized data objects between a source end and a target end. The system of the present disclosure operates independently in a manner parallel to local applications of a synchronous node, and provides a guarantee mechanism of relaxed transaction for Internet distributed multi-source heterogeneous data synchronization through collaboration of loose coupling.

Claims

1. A generic multi-source heterogeneous large-scale data synchronization method, comprising: constructing a synchronization topology of a tree structure, the synchronization topology comprising plural synchronization pairs composed of a pair of adjacent nodes, the synchronization pair comprising a client end and a server end; executing installation scripts at the server end, so as to create a heterogeneous data mapping table, a change log, a number of message queues, a number of message queue monitoring threads and a data renewal thread pool, and for executing installation scripts at the client end, so as to create a change log, a change polling thread and a sync thread pool; grouping change records to be sync at a source data end into sync blocks, and transmitting the sync blocks sequentially to the server end in accordance with network conditions and processing capacity of the server end, and allow the server end to store the received sync blocks in corresponding message queues; wherein, a pipeline type of non-stop transmission is adopted for the client end transmitting the change records within a sync block, and after a sync block is sent the transmission process is locked until the server end returns a receipt acknowledgment, then the transmission of the subsequent sync blocks is resumed, which is repeated until all changes are sent; scheduling several sync threads of the client end to initiate data synchronization operation when a change polling thread of the client end monitors that the change log is not blank; and scheduling several data renewal threads of the server end to perform data renewal operation when a message queue monitoring thread of the server end monitors that there are non-blank message queues; sequentially executing two unilateral syncs in opposite directions to accomplish bilateral sync; and recording data change events of the source data end sequentially as they occur in the change log at the client end in an unrepeated manner, for sequentially receiving data change of the source data end at the server end, and for implementing each data change respectively in the original order and recording the data change in the change log.

2. The genetic multi-source heterogeneous large-scale data synchronization method of claim 1, wherein, the client end comprises a middle layer software component for realizing synchronization of total/incremental data; the server end comprises a software component for receiving data change of the source data end transmitted by the client end, and renewing the received data change into a target data end; the source data end comprises a database or a file directory serving as a source of the data change during data synchronization and where data replicates are located; the target data end comprises a database or a file directory, where data replicates, in response to data change from the source data end during the data synchronization, are located.

3. The generic multi-source heterogeneous large-scale data synchronization method of claim 1, wherein, intermediate node in the synchronization topology of the tree structure is provided with both a client end and a server end that respectively belong to different synchronization pairs.

4. The generic multi-source heterogeneous large-scale data synchronization method of claim 2, wherein, the change log of the source data end is configured to store data change information.

5. The generic multi-source heterogeneous large-scale data synchronization method of claim 2, further comprising: controlling the client end to acquire metadata information of the source data end involved in the synchronization and to transmit the metadata information to the server end, controlling the server end to construct and store rules for heterogeneous data mapping in accordance with the metadata information, controlling the client end to acquire the rules for heterogeneous data mapping from the server end to construct three change capture triggers, which are Insert, Deletion, and Update triggers at the source data end for each data table involved in the synchronization, and controlling the server end to provide a visual configuration tool for maintaining or adjusting the rules for heterogeneous data mapping.

6. The generic multi-source heterogeneous large-scale data synchronization method of claim 2, wherein, a data renewal thread pool pre-created at the server end comprises data renewal threads in a preset number, and the data renewal in each data object of the target data end is in the charge of one data renewal thread; a sync thread pool pre-created at the client end comprises data sync threads in a preset number created in accordance with the metadata information, and the incremental or total synchronization of each data object at the source data end is in the charge of only one sync thread.

7. The generic multi-source heterogeneous large-scale data synchronization method of claim 5, further comprising: controlling the change capture triggers to capture a data change event, and storing corresponding data change information in the change log of the client end each time data change occurs at the source data end; wherein, the data change information comprises the name of a data table undergoing data change, primary key values of the data record and the type of the change operation.

8. The generic multi-source heterogeneous large-scale data synchronization method of claim 7, further comprising: controlling the sync threads of the client end to group records of corresponding change data or synchronization operations into sync blocks in accordance with sync presets of each data table undergoing data change, and encapsulate each of the sync blocks into a SyncML message packet and transmit the packet sequentially to the server end.

9. The generic multi-source heterogeneous large-scale data synchronization method of claim 8, further comprising: in the data synchronization process, controlling the server end to assign a data receiving thread to a session request from the client end, so as to receive the SyncML message packets uploaded by the client end; wherein, the data receiving thread is configured to receive SyncML message packets for resolving and restoring each sync block from its SyncML message packet, to store the resolved sync block in a designated message queue and feedback information for successful synchronization to the client end after successful storage.

10. The generic multi-source heterogeneous large-scale data synchronization method of claim 9, further comprising: controlling the sync threads of the client end to associate the change log with a data table undergoing data change, to order the change log sequentially onward as insertion thereof are recorded and read all data records of changes from the change log; and controlling the sync threads to delete the records in the change log corresponding to the sync block after the server end receives the sync block and returns an acknowledgment message.

11. The generic multi-source heterogeneous large-scale data synchronization method of claim 9, further comprising: controlling the data renewal thread of the server end to read sync blocks from the message queue, and perform local data renewals in accordance with the rules for heterogeneous data mapping, so as to allow target data replicates at the target data end to be consistent with synchronous data of source data replicates at the source data end.

12. The generic multi-source heterogeneous large-scale data synchronization method of claim 9, further comprising, controlling the sync threads of the client end to conduct Hash calculation to a sync block to be transmitted, and encapsulate the Hash values and the sync block into a SyncML message packet; and controlling the data receiving thread of the server end to verify resolved the sync block after receipt of the SyncML message packet, and store the sync block in the message queue in case of successful verification, and return a message about failed synchronization otherwise.

13. The generic multi-source heterogeneous large-scale data synchronization method of claim 9, further comprising: controlling size of the sync block, and controlling the sync threads of the client end to enter a locked waiting state after transmission of a sync block until the server end returns an acknowledge message or time is run out.

14. The generic multi-source heterogeneous large-scale data synchronization method of claim 1, wherein, rules for heterogeneous data mapping of the heterogeneous data mapping table created at the server end comprises a data table name, a primary key or a virtual primary key, a field name, a field data type, a field data length and mapping relations between the fields of the synchronized source data end and target data end.

15. The generic multi-source heterogeneous large-scale data synchronization method of claim 14, further comprising: constructing the virtual primary key in the rules for heterogeneous data mapping; controlling the server end, when the data table in the source data end fails to define the primary key, to construct a virtual primary key capable of uniquely identifying its data records for the data table, and store the construction rules of the virtual primary key in the server end in accordance with field information in the metadata information.

16. The generic multi-source heterogeneous large-scale data synchronization method of claim 1, further comprising: checking whether a message queue corresponding to the client end is present at the server end; if not, controlling the server end to construct a corresponding message queue for the client end, and the message queue is configured to temporarily store the sync blocks of the corresponding client end received at the server end.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) In order to make a clearer description of technical solutions in specific implementations of the present disclosure or prior art, figures involved in description for the specific implementations or the prior art will be briefly introduced, and obviously, the figures in the following description illustrate some implementations of the present disclosure, for one with ordinary skill in the art, other figures can also be obtained in accordance with these figures without delivering creative efforts.

(2) FIG. 1 is a structure diagram for generic multi-source heterogeneous large-scale data synchronization system;

(3) FIG. 2 is a flowchart for data synchronization processing by a generic multi-source heterogeneous large-scale data synchronization system;

(4) FIG. 3 is a flowchart for a heterogeneous data synchronization method;

(5) FIG. 4 is a flowchart for a heterogeneous data synchronization method.

(6) FIG. 5 is a structural diagram for a client end in an embodiment of the present disclosure.

(7) FIG. 6 is a structural diagram for a server in an embodiment of the present disclosure.

DETAILED DESCRIPTION

(8) Technical solutions of the present disclosure will be described clearly and completely as follows in conjunction with the figures, apparently, the described embodiments are just part rather than all embodiments of the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by one with ordinary skill in the art without delivering creative efforts shall fall into the protection scope of the present disclosure.

(9) It should be noted that, in the description of the present disclosure, orientation or position relationships indicated by terms such as “center”, “above”, “beneath”, “left”, “right”, “vertical”, “horizontal”, “inside” and “outside” are based on figure illustrations, and are merely intended to facilitate and simplify description of the present disclosure, rather than indicating or implying that the device or element concerned must be specifically oriented or constructed or operated in a specific manner, and thus shall not be construed as limitations on the present disclosure. In addition, terms such as “first”, “second”, “third” are intended for the purpose of description, and shall not be interpreted to indicate or imply relative importance.

(10) In the description of the present disclosure, it should be noted that, unless specified and defined otherwise, the terms of “installation”, “interconnection” and “connection” shall be understood in a broad sense, for example, a fixed connection, a removable connection, an integral connection, a mechanical connection, an electrical connection, a direct connection, an indirect connection via intermediate medium, or further, internal communication between two elements, a wireless connection, or a wired connection. Case-by-case interpretation can be made to the above terms in the present disclosure by one with ordinary skill in the art.

(11) Furthermore, technical features involved in the described different implementations of the present disclosure can be combined with each other in the case of no contradictions.

Embodiment 1

(12) The embodiment provides a generic multi-source heterogeneous large-scale data synchronization system, as shown in FIG. 1, including: a sync node config unit 11, an install & deployment unit 12, a block & pipelining unit 13, a unilateral sync unit 14, a bilateral sync unit 15 and a correctness guarantee unit 16.

(13) The sync node config unit 11, for constructing a synchronization topology of a tree structure, the synchronization topology comprising plural synchronization pairs composed of a pair of adjacent nodes, the synchronization pair comprising a client end (relative low-level node) and a server end (relative high-level node), an intermediate node of the synchronization tree (a node other than the nodes as leaves and roots) can be provided with both a client end and a server end fulfilling different roles yet belonging to different synchronization pairs. The client end includes a middle layer software component for realizing total/incremental data synchronization; the server end includes a software component for receiving data change of the source data end transmitted by the client end and for renewing the received data change into the target data end; the client end and the server end can be installed on the same computer, each delivering their functions during synchronization operation of a synchronization pair where they are located; the source data end includes a database or a file directory serving as a source of the data change during data synchronization and where data replicates are located; and the target data end includes a database or a file directory, where data replicates, in response to data change from the source data end during the data synchronization, are located.

(14) The system provides propagation and non-propagation configuration strategies for unidirectional data synchronization of the synchronization tree (client end.fwdarw.server end). The propagation strategy records the data change information into a change log of the target data end (a server end of the current synchronization pair), while the non-propagation strategy does not perform the recording operation.

(15) The install & deployment unit 12 is configured to execute installation scripts at the server end, so as to create a heterogeneous data mapping table, and execute installation scripts at the client end, so as to create a synchronization configuration table, a change log, a change polling thread and a sync thread pool.

(16) The rules for heterogeneous data mapping created at the server end includes a data table name, a primary key or a virtual primary key, a field name, a field data type, a field data length and mapping relations between the fields of the synced source data end and target data end. During construction of the rules for heterogeneous data mapping at server end, if the data table at the source data end fails to define the primary key, server end constructs a “virtual primary key” capable of uniquely identifying its data record for the data table, and stores the construction rules of the “virtual primary key” in the server end in accordance with field information in the metadata information. The “virtual primary key” algorithm constructs the “virtual primary key” by screening the candidate primary key fields of the data table. In addition, the server end further checks whether a message queue corresponding to the client end is present, and if not, constructs a corresponding message queue for the client end. The message queue is configured to temporarily store the sync blocks of the corresponding client end received at the server end.

(17) The client end acquires metadata information of the source data end involved in the synchronization (structure data for all data tables involved in the synchronization at the source data end) and transmits the metadata information to the server end; the server end autonomously constructs and store rules for heterogeneous data mapping, i.e., “source data end”“target data end” in accordance with the metadata information. The client end acquires the rules for heterogeneous data mapping from the server end to construct three change capture triggers, i.e., “Insert”, “Delete”, And “Update” triggers at the source data end for each data table involved in the synchronization, in accordance with the rules and the change capture triggers. For the sake of checking and modifying the rules for heterogeneous data mapping, the embodiment provides a visual configuration tool at the server end for maintaining or adjusting the rules for heterogeneous data mapping.

(18) Specifically, the client end follows a specific method in pre-creating a sync thread pool, i.e., creating several data sync threads (i.e., in a preset number) in accordance with the metadata information. The Incremental or total synchronization of one data table is in the charge of only one sync thread. The server end pre-creates a renewal thread pool following a specific method, i.e., create several data renewal threads (i.e., in a preset number). Data renewal of one data table is in the charge of only one data renewal thread.

(19) The block & pipelining unit 13 is configured to group the data changes into sync blocks at the client end sync, and transmit the sync blocks sequentially to the server end in accordance with network conditions and processing capacity of the server end, and allow the server end to store the received sync blocks in corresponding message queues. Where, a pipeline type of non-stop transmission is adopted for the client end transmitting the change records within a sync block, and after a sync block is sent the transmission process is locked until the server end returns a receipt acknowledgment, then the transmission of the subsequent sync blocks is resumed, which is repeated until all changes are sent.

(20) The unilateral sync unit 14 is configured to schedule several sync threads of the client end to initiate data synchronization operation when a change polling thread of the client end monitors that a change log is not blank; and schedule several data renewal threads of the server end to perform data renewal operation when a message queue monitoring thread of the server end monitors that the message queue is not blank. Collaboration between the block & pipelining unit 13 and the unilateral sync unit 14 can accomplish one data synchronization, with its data processing procedures shown in FIG. 2.

(21) Each time data change occurs at the source data end, a trigger captures the data change event, and stores data change information in a change log in the client end. The data change information includes a table name, a data-recorded primary key value, a change operation type of a corresponding data table undergoing the data change. The change polling thread of the client end discovers data change at the source data end by monitoring the change log, and the client end further initiates data synchronization operation through the sync thread. The sync thread divides all the change data or synchronization operation record into plural sync blocks in accordance with a synchronous preset of each data table, encapsulates each sync block into a SyncML message packet, and sequentially transmits the packets to the server end. In the data synchronization process, the server end assigns a data receiving thread to a session request from each SyncML message packet, and is in charge of receiving a SyncML message packet uploaded by the client end. The data receiving thread receives each SyncML message packet and from which resolves out and stores the sync block into a designated message queue through a message processing mechanism, and then feedbacks acknowledgement for successful synchronization to the client end after successful storage. The message queue monitoring thread of the server end monitor whether the message queue is blank, if not, notifies the server end to schedule several data renewal threads to perform data renewal operation. The data renewal threads of the server end perform subsequent processing in an asynchronous parallel manner, i.e., read sync blocks from the message queue, and perform local data change in accordance with the rules for heterogeneous data mapping, so as to allow the consistency between synchronization data of the target data (local data) replicates and the source data replicates.

(22) When reading change data from a data table a sync thread of the client end firstly associates the change log with the data table, orders the change log sequentially onward as Inserts thereof are recorded and reads all data records of changes to the change log; secondly, the data sync threads transmits the SyncML message packet to the server end under an SyncML generic protocol; and finally, the data sync thread deletes the records of the change log corresponding to the sync block in the SyncML message packet, after the server end returns an acknowledgment (of receipt) message.

(23) The heterogeneous data synchronization system in the embodiment is applicable to incremental synchronization and total synchronization. Wherein the incremental synchronization refers to synchronization of local data change newly generated since successful synchronization last time to the target data end. During implementation of the incremental synchronization, the client end polls the local change log for transmitting to-be-synchronized data change to the server end, and deletes corresponding local change log records after the server end returns a synchronization acknowledge message. Total synchronization refers to replicate all current data objects involved in the data synchronization from the source data end onto the target data end. The total synchronization in this embodiment includes two types, namely merging synchronization and refreshing synchronization. The refreshing synchronization completely erases the data table at the target data end before starting replicating the source data object to the target data object, and it is usually used for resetting a synchronization task of the synchronization pair. The merging synchronization executes an Insert operation on the target data object if the corresponding data records have not existed there, otherwise executes an Update operation. To be noted, during implementation of the total synchronization, a trigger already activated at the source data end can capture and record all local changes during the period, so as to facilitate subsequent incremental synchronization operation.

(24) The bilateral sync unit 15 is configured to sequentially execute two unilateral syncs in opposite directions to accomplish bilateral sync, i.e., CS, equivalent to CS and SC, wherein C and S respectively represent the client end and the server end, the arrow .fwdarw. or represents the synchronization direction (transmission direction of the sync blocks), the same usage is applicable in the followings.

(25) The correctness guarantee unit 16 is configured to record data change events of the source data end sequentially as they occur in the change log at the client end in an unrepeated manner, and sequentially receive data change of the source data end at the server end, and implement each data change respectively in the original order and record the data change in the change log. In this embodiment, data synchronizations, C.fwdarw.S or S.fwdarw.C, are both initiated at the client end, thus in this embodiment, the client end checks and manages the triggers of data table involved in the synchronization, for example, the client end is in charge of management of constructing, modifying and activating the triggers and consistency check. For further guaranteeing correctness in the data synchronization, incremental synchronization in this embodiment forbids single-point multi-process synchronization, i.e., forbids multiple client end processes performing data synchronization on the same source data object, so as to avoid mis-sequence in synchronization of data change of the same data object.

(26) In addition, the system in this embodiment further provides generic “multi-attribute primary keysingle-attribute primary key” mapping rules, to record data change events for single and non-single attribute primary key data tables I in a change log at the client end.

(27) Due to the fact that data synchronizations in this embodiment are initiated by the client end, in order to realize unified supervision of the data synchronization, this embodiment further provides a synchronization scheduling and a synchronization state real-time monitoring unit for each client end.

(28) For improving reliability and security of the data synchronization, this embodiment provides two synchronization performance options, “optimistic” and “cautious”. Under the “cautious” option, the sync blocks to be transmitted are required to undergo Hash calculation, and the Hash value is encapsulated into a SyncML message packet along with a sync block; after receiving the SyncML message packet, the receiver performs Hash verification to a interpreted sync block, i.e., re-calculates the Hash value of the sync block, and compares the value to a transmitted Hash value, and the verification is passed if the two are identical; in this case, the sync block stores the message queue, otherwise the receiver returns a synchronization failing message. Under the “optimistic” option, the above processing involving the Hash value is omitted. The “optimistic” option is applicable to an application scenario where there is an optimistic expectation for the reliability of the synchronization transmission and which tends to seek for higher synchronization performance, for example, conditions with a desirable network quality for synchronization communication; on the contrary, the “cautious” option is applicable to an application scenario where there is a lower expectation for the reliability of the synchronization transmission and which tends to verify the correctness of the synchronization transmission results more cautiously, for example, conditions with a not so desirable network quality for synchronization communication.

(29) In this embodiment, the transmission is performed with a sync block as a synchronization message unit, and adopts a block-lock transmission mechanism, i.e., enters a locked waiting state after transmission of each sync block until the server end returns an acknowledge message or time is run out. The larger the sync block, the higher the transmission efficiency, and conversely the higher the cost for retransmission of the entire sync block once transmission failure occurs. This embodiment provides a function of adjusting the size of the sync block (i.e., how many of change data or change operation records are included therein), so as to debug efficiency and reliability of the large-scale data synchronization. If a network environment, including the end system environment, has high communication reliability, then the size of sync block in the synchronization is adjusted upward, otherwise when this environment has a low communication reliability the size of sync block in the synchronization should be adjusted downward. Lock-free transmission mechanism can be regarded as a special case of the lock-step transmission mechanism, equivalent to adjusting the sync block to maximum, in another word, the entire transmitted contents are deemed as one sync block.

Embodiment 2

(30) This embodiment provides a heterogeneous data synchronization method, which is applicable to the generic multi-source heterogeneous large-scale data synchronization system in embodiment, and which will be introduced using a client end as an example, as illustrated in FIG. 3, including the following steps:

(31) S11: initialization stage for synchronization configuration. The client end acquires metadata information of data tables involved in synchronization at a source data end, and transmits the metadata information to a server end. Then in accordance with the heterogeneous data mapping rules and change capture rule template obtained from the server, the client end constructs three change capture triggers, i.e., Insert, Delete, and Update triggers at the source data end.

(32) S12: change event triggering stage. As soon as a change event happens the corresponding trigger shall be triggered, then the trigger stores change event information into the local change log. The change event information includes a name of the involved data table, a primary key value of the data record, the change type, as well as the change time and other information.

(33) S13: data synchronization stage. A change polling thread polls the change log to check whether the change log is blank, and if not, notifies the client end to schedule several data sync threads to perform data synchronization operation of the involved data tables. The data sync threads groups change sets of at least one data table into sync blocks respectively in accordance with the rules for heterogeneous data mapping acquired from the server end and a preset value. Accidental abortion of mass heterogeneous data synchronization between loose coupling autonomous applications can not be neglected in a scenario of an unreliable Internet communication environment, thus it is necessary to provide a synchronization mechanism for resuming-transmission at abort point, i.e., divides the synchronization process into several sections, creating a save point at the end of each section to save the current position of synchronization progress, so as to determine the start position of a recovery synchronization process resumed after the synchronization abortion.

(34) Prior to each round of data synchronization, the sync thread counts a total record number of the data tables to be synchronized, and informs the server end of the name of each data table and the total record number of those data tables to be synchronized, and then it starts a new round of synchronization session. When transmitting the last sync block in a round of synchronization, the sync thread informs the server end that this is the last sync block transmitted in this round of synchronization session; the server end receives the last sync block and then ends the current round of synchronization session.

(35) Preferably, step S11 includes the following sub-steps:

(36) S111: creating a data change log table (referred to as change log in short). The change log includes the name of a data table undergoing change, primary key value of the data record, the type of the change event, and change time and other information. Specifically, the change log is shown in 1 (applicable to the client end and the server end):

(37) TABLE-US-00001 TABLE 1 Field description Field name Field type Record identification number CHG_SEQ_NUM INTEGER Time of Change CHG_TIMESTAMP TIMESTAMP URI of Client SOURCE_URI VARCHAR(256) Name of Source Data table SOURCE_TABLE VARCHAR(128) Primary key value SOURCE_KEY_VALUE VARCHAR(128) Change type CHG_TYPE CHAR(1)

(38) S112: capturing the data change event via the triggers, wherein operation events such as Update (change), Insert (insert) and Delete (delete) of each row of data records are captured. S113: recording the data change events in the change log. The preset change log and the triggers enable the implementation of incremental synchronization of data without interfering with any upper layer application program or affecting normal database throughput.

(39) Hereafter the server end will be taken as an example to introduce the heterogeneous data synchronization method in embodiment 2, as illustrated in FIG. 4, including the following steps:

(40) S21: initialization stage for synchronization configuration. The server end automatically constructs and stores rules for heterogeneous data mapping between the source data end and the target data end in accordance with the metadata information uploaded by the client end; the server end constructs several data renewal threads (in a preset value).

(41) This embodiment adopts a middle layer system written in Java languages, uses a JDBC (Java Data Base Connectivity) standard to realize a unified database access interface. Specifically, herein a method for establishing data types conversion from data types of various heterogeneous relational database to the unified Java data types (as an intermediate data type) is adopted, to establish a mapping relation for data types between heterogeneous mainstream database products such as Oracle, MySQL, MSSQL Server and Sybase, that is, the preset mapping rules. Adopting Java data type as an intermediate database type all makes sure that errors are unlikely to occur in transmission resolution between the client end and the server end.

(42) Partial mapping between Java data type and common heterogeneous data types are illustrated as table 2.

(43) TABLE-US-00002 TABLE 2 Microsoft JDBC data type Oracle9i SQLServer2000 Sybase ASE15 Java data type BIT NUMBER(1) BIT BIT boolean TINYINT NUMBER(3) TINYINT TINYINT byte SMALLINT SMALLINT, SMALLINT SMALLINT, short NUMBER UNSIGNEDSMALLINT INTEGER INTEGER, INT INT, UNSIGNEDINT int NUMBER BIGINT NUMBER BIGINT BIGINT, long UNSIGNEDBIGINT REAL REAL, NUMBER REAL REAL float FLOAT FLOAT, NUMBER FLOAT FLOAT double

(44) During practical construction of rules for heterogeneous data mapping, mapping relations between the above various heterogeneous data type and Java data type acts as medium, for determining correspondence between specific data types of the two parties in the synchronization.

(45) Preferably, during the processing of constructing the heterogeneous mapping rules in step S21, the process further includes: determining whether the involved data table defines a primary key, if not, constructing a virtual primary key for uniquely identifying its data records, so as to implement a synchronization guarantee process.

(46) Specifically, the algorithm for constructing the virtual primary key is as follows: virtual primary key=str(F1)+ASII(11)+str(F2)+ . . . +ASII(11)+str(Fk), wherein F1, F2, . . . and Fk are the fields capable of jointly constituting a primary key, a str(X) function calculates a string value of a variable X, “+” is the operator of string concatenation, ASII (11) represents a character with an ASCII (American Standard Code for Information Interchange) decimal coding value of 11, which is a non-displayed character (referred to as a “vertical tab”) and is unlikely to appear in the text contents.

(47) S22: data receiving stage. For each session request from the client end for uploading SyncML message packets, the server end assigns a data receiving thread to take charge of receipt and resolution of SyncML message packets, and hands over the resolved sync block to the message processing mechanism to store in the corresponding message queue of the client. In order to improve the whole efficiency of the synchronization process, the present disclosure inserts an asynchronous parallel message processing mechanism at the server end. After receiving SyncML message packet from the client end, a data receiving thread of the server end performs simple resolution and processing only, i.e., schedules the message processing mechanism to put the resolved sync block into the queue, and returns an acknowledge message to the client end to signal that the client end can proceed with transmission of other SyncML message packets without waiting for the target data end to complete the synchronization renewal. And the sync blocks waited in the message queue will be picked up by an asynchronous processing module of the server end for parallel processing, in other words, multiple-channel parallel processing. In this way, transmission progress of the message from the client end is improved, and furthermore, that it is not necessary for the server end to complete the whole process immediately eases the overall processing pressure of the server. And in applications of large-scale data synchronization, the server end usually encounters a bottleneck of performance.

(48) S23: data renewal stage. The message queue monitoring thread monitors whether the message queues are blank, if not blank, then the server end is notified to schedule corresponding data renewal threads to carry out data renewal onto the corresponding synchronization data objects of the queues. A data renewal thread is configured to read sync block from its message queue to renew the target data end. A data renewal thread reads the sync blocks from its message queue, and then renews the sync blocks into the corresponding synchronization data object of the target data end in accordance with the rules for heterogeneous data mapping.

(49) Preferably, step S22 includes the following sub-steps:

(50) S221: receiving a SyncML message packet uploaded by the client end, the SyncML message packet containing one sync block.

(51) S222: resolving a SyncML message packet into a sync block. To find broader applications, the present disclosure performs data synchronization based on a generic SyncML standard protocol in the IT industry. SyncML, initially brought up as an open standard protocol for information synchronization independent of platforms, now belongs to the Data Synchronization and Device Management protocol family of the OMA (Open Mobile Alliance) Organization, and can perform data synchronization on compatible devices, programs or networks, enabling any device or program to obtain consistent data. However, the SyncML only provides a basic communication framework, which is far from satisfying the demand for products of practical large-scale data synchronization applications, for example, it fail to provide a reliability guarantee mechanism, which is critical in practical applications. In order to fill these gaps, the generic multi-source heterogeneous large-scale data synchronization system in embodiment 1 is provided with a correctness guarantee unit 16, for guaranteeing reliability of the heterogeneous data synchronization.

(52) The embodiments of the disclosure further provides a client end, as illustrated in FIG. 5, comprising, at least one processor; and a memory in communication connection with the at least one processor; wherein, the memory stores instruction capable of being executed by the at least one processor, the instructions are executed by the at least one processor to cause the at least one processor to execute the steps:

(53) acquiring heterogeneous data to be synchronized;

(54) acquiring at least one data table, i.e., table including identification information of the heterogeneous data created in accordance with the mapping rules, in accordance with the heterogeneous data; respectively creating a data sync thread in accordance with each data table; and

(55) grouping, by the data sync thread, change records into sync blocks in accordance with a preset value, respectively.

(56) In the client end, wherein, the heterogeneous data comprises incremental heterogeneous data, the acquiring heterogeneous data to be synchronized includes: creating a data change log table (referred to as change log in short) which includes field description, a field name and a field type; capturing change event information of the data through triggers; and recording change event information of the data in the change log.

(57) The embodiments of the disclosure further provides a server end, as illustrated in FIG. 6, comprising a wireless network interface, a processor and a memory which are interconnected with each other via a bus 80; wherein, the memory stores computer instructions to be executed to cause the processor to perform the steps:

(58) acquiring a sync block;

(59) placing the sync block in a message queue; withdrawing the sync block from the message queue and renewed the sync block in a target database.

(60) In the server end, the acquiring of the sync block comprises receiving a SyncML message packet uploaded by the client end, the SyncML message packet containing one sync block; and resolving the SyncML message packet into a sync block.

(61) Obviously, the above embodiments are merely intended to clearly illustrate rather than limit the numerated implementations. For one with ordinary skill in the art, other different forms of modifications or changes may further be made on the basis of the aforementioned descriptions. It is unnecessary and impossible to exhaust all implementations. And modifications or changes derived herefrom obviously fall into the protection scope of the present disclosure.