METHODS AND SYSTEMS FOR STORING DATA IN A DATABASE
20230008700 · 2023-01-12
Assignee
Inventors
Cpc classification
International classification
Abstract
A method comprising, by a processor and memory circuitry, obtaining a plurality of data comprising one or more groups of data, obtaining a data structure usable to determine, for at least a first data type and a second data type, a given data type which is adapted to represent at least both data of the first and second data types for their storage, for at least one given group of data which comprises one or more subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determining a data type which is adapted to represent said subset of data for its storage, and using the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type which is adapted to represent data belonging to subsets of data S.sub.1 to S.sub.N for their storage.
Claims
1. A method comprising, by a processor and memory circuitry (PMC): obtaining a plurality of data to be stored in a database, wherein the plurality of data comprises one or more groups of data, obtaining a data structure informative of a plurality of data types according to which data can be represented for their storage, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of the first data type and data of the second data type for their storage, for at least one given group of data of the plurality of data, which comprises one or more subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determining a given data type which is adapted to represent said subset of data for its storage, and using the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type which is adapted to represent all data, or at least part thereof, belonging to subsets of data S.sub.1 to S.sub.N, for their storage, thereby facilitating storing the given group of data in a database according to this common data type.
2. The method of claim 1, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of the first data type and data of the second data type for their storage according to an efficiency criterion, wherein the efficiency criterion takes into account at least one of: a storage size for representing both data of the first data type and data of the second data type in a database, and a query speed for searching data of the first data type and data of the second data type in a database.
3. The method of claim 1, comprising storing data of the at least one given group of data according to the common data type.
4. The method of claim 1, comprising, for at least one group of data which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset S.sub.i, with i from 1 to N, performing a data type determination comprising: for subset S.sub.1, determining a data type adapted to represent data of subset S.sub.1 for its storage, for i>1, determining a data type adapted to represent data of subset S.sub.i for its storage, wherein, when this data type is not adapted to represent data of at least one subset S.sub.j, with j<i, using the data structure to determine a new data type which is adapted to represent both data of subsets S.sub.i and S.sub.j for their storage, based on the data type determination, providing a common data type which is adapted to represent all data of subsets S.sub.1 to S.sub.N for their storage.
5. The method of claim 1, comprising, by the PMC: obtaining a database DB comprising at least one group of data G.sub.DB, wherein a plurality of data DS comprises at least one group of data G.sub.DS which has to be stored together with data of the group of data G.sub.DB, for the at least one group of data G.sub.DS of the plurality of data DS, which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determining a given data type which is adapted to represent said subset of data, and using the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type DT′.sub.DB which is adapted to represent group of data G.sub.DS and group of data G.sub.DB, for storing group of data G.sub.DS and group of data G.sub.DB according to the common data type DT′.sub.DB.
6. The method of claim 5, comprising: storing group of data G.sub.DS and group of data G.sub.DB according to the common data type DT′.sub.DB in a same group of data of the database DB.
7. The method of claim 1, comprising, by the PMC: obtaining a database DB comprising at least one group of data G.sub.DB associated with a data type DT.sub.DB, wherein a plurality of data DS comprises at least one group of data G.sub.DS which has to be stored together with group of data G.sub.DB, and for the at least one group of data G.sub.DS which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset S.sub.i, with i from 1 to N: determining a data type adapted to represent data of subset S.sub.i for its storage, wherein, when this data type is not adapted to represent data of a current common data type DT.sub.C, using the data structure to determine a data type which is adapted to represent both data of subset S.sub.i and data of the current common data type DT.sub.C, and assigning this data type as current common data type DT.sub.C, wherein for i=1, the current common data type DT.sub.C is equal to DT.sub.DB, and providing the current common data type DT.sub.C determined for i=N, which is adapted to represent group of data G.sub.DB and group of data G.sub.DS, for their storage according to said current common data type DT.sub.C.
8. The method of claim 7, comprising storing group of data G.sub.DS and group of data G.sub.DB according to the current common data type DT.sub.C determined for i=N, thereby providing a database storing both group of data G.sub.DS and group of data G.sub.DB in a same group of data.
9. The method of claim 1, wherein the plurality of data types comprises at least one of: Integer; Float; Date; Boolean; Time; Date and Time; String; String with a predefined number of characters.
10. A method comprising, by a processor and memory circuitry (PMC): obtaining a plurality of data, wherein the plurality of data comprises one or more groups of data, obtaining a data structure informative of a plurality of data types according to which data can be represented for their storage, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of the first data type and data of the second data type for their storage, for at least one group of data G.sub.DS of the plurality of data, which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for subset S.sub.1, determining a current common data type D.sub.TC adapted to represent data of subset S.sub.1 for its storage, repeatedly performing, for each subset S.sub.i, with i from 2 to N: determining a data type adapted to represent data of subset S.sub.i for its storage, wherein, when this data type is not adapted to represent data of the current common data type DT.sub.C, using the data structure to determine a given data type which is adapted to represent both data of subset S.sub.i and data of the current common data type DT.sub.C, and assigning this given data type as current common data type DT.sub.C, providing the current common data type DT.sub.C determined for i=N, which is adapted to represent data of group G.sub.DS for their storage, thereby facilitating storing the group of data G.sub.DS in a database according to this common data type, or repeatedly performing, for each subset S.sub.i, with i from 1 to N, determining a data type adapted to represent data of subset S.sub.i for its storage, wherein when this data type is not adapted to represent data of a current common data type DT.sub.C, using the data structure to determine a given data type which is adapted to represent both data of subset S.sub.i and data of the current common data type DT.sub.C, and assigning this given data type as current common data type DT.sub.C, wherein for i=1, the current common data type DT.sub.C corresponds to a data type DT.sub.DB of a group of data G.sub.DB of an existing database DB, providing the current common data type DT.sub.C determined for i=N, which is adapted to represent said group of data G.sub.DB and said group of data G.sub.DB for their storage according to said current common data type DT.sub.C, thereby facilitating storing the group of data G.sub.DB with the group of data G.sub.DB of the database DB according to this common data type.
11. The method of claim 10, wherein the data structure comprises a tree comprising a plurality of nodes each associated with a data type, wherein a parent node of a given node corresponds to a data type which is broader than a data type of said given node, wherein each time the tree is searched or queried, a given node associated with a given data type output by the search is stored, wherein a subsequent search or query in the tree is allowed only in this given node or in parent nodes of this given node.
12. A system comprising a processor and memory circuitry (PMC) configured to: obtain a plurality of data to be stored in a database, wherein the plurality of data comprises one or more groups of data, obtain a data structure informative of a plurality of data types according to which data can be represented for their storage, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of the first data type and data of the second data type for their storage, for at least one given group of data of the plurality of data, which comprises one or more subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determine a given data type which is adapted to represent said subset of data for its storage, and use the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type which is adapted to represent all data, or at least part thereof, belonging to subsets of data S.sub.1 to S.sub.N for their storage, thereby facilitating storing the given group of data in a database according to this common data type.
13. The system of claim 12, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of the first data type and data of the second data type for their storage according to an efficiency criterion, wherein the efficiency criterion takes into account at least one of: a storage size for representing both data of the first data type and data of the second data type in a database, and a query speed for searching data of the first data type and data of the second data type in a database.
14. The system of claim 12, configured to store data of the at least one given group of data according to the common data type.
15. The system of claim 12, configured to, for at least one group of data of the data structure which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset S.sub.i, with i from 1 to N, perform a data type determination comprising: for subset S.sub.1, determining a data type adapted to represent data of subset S.sub.1 for its storage, for i>1, determining a data type adapted to represent data of subset S.sub.i for its storage, wherein, when this data type is not adapted to represent data of at least one subset S.sub.j, with j<i, using the data structure to determine a new data type which is adapted to represent both data of subsets S.sub.i and S.sub.j for their storage, based on the data type determination, provide a common data type which is adapted to represent all data of subsets S.sub.1 to S.sub.N for their storage.
16. The system of claim 12, configured to: obtain a database DB comprising at least one group of data G.sub.DB, wherein a plurality of data DS comprises at least one group of data G.sub.DS which has to be stored together with data of the group of data G.sub.DB, for the at least one group of data G.sub.DS of the plurality of data DS, which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determine a given data type which is adapted to represent said subset of data, and use the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type DT′.sub.DB which is adapted to represent group of data G.sub.DS and group of data G.sub.DB, for storing data of G.sub.DB and data of G.sub.DS according to common data type DT′.sub.DB.
17. The system of claim 16, configured to: store group of data G.sub.DS and group of data G.sub.DB according to the common data type DT′.sub.DB in a same group of data of the database DB.
18. The system of claim 12, configured to: obtain a database DB comprising at least one group of data G.sub.DB associated with a data type DT.sub.DB, wherein a plurality of data DS comprises at least one group of data G.sub.DS which has to be stored together with group of data G.sub.DB, and for the at least one group of data G.sub.DS which comprises a plurality of subsets of data S.sub.1 to S.sub.N: for each subset S.sub.i, with i from 1 to N: determine a data type adapted to represent data of subset S.sub.i for its storage, wherein, when this data type is not adapted to represent data of a current common data type DT.sub.C, use the data structure to determine a data type which is adapted to represent both data of subset S.sub.i and data of the current common data type DT.sub.C, and assign this data type as current common data type DT.sub.C, wherein for i=1, the current common data type DT.sub.C is equal to DT.sub.DB, and provide the current common data type DT.sub.C determined for i=N, which is adapted to represent group of data G.sub.DB and group of data G.sub.DS, for their storage according to said current common data type DT.sub.C.
19. The system of claim 18, configured to store group of data G.sub.DS and group of data G.sub.DB according to the current common data type DT.sub.C determined for i=N, thereby providing a database storing both group of data G.sub.DS and group of data G.sub.DB in a same group of data.
20. A non-transitory storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform: obtaining a plurality of data to be stored in a database, wherein the plurality of data comprises one or more groups of data, obtaining a data structure informative of a plurality of data types according to which data can be represented for their storage, wherein the data structure is usable to determine, for at least a first data type and a second data type among said plurality of data types, a given data type which is adapted to represent at least both data of at least the first data type and data of the second data type for their storage, for at least one given group of data of the plurality of data, which comprises one or more subsets of data S.sub.1 to S.sub.N: for each subset of data S.sub.1 to S.sub.N, determining a given data type which is adapted to represent said subset of data for its storage, and using the given data type of each subset of data S.sub.1 to S.sub.N and the data structure to determine a common data type which is adapted to represent all data, or at least part thereof, belonging to subsets of data S.sub.1 to S.sub.N for their storage, thereby facilitating storing the given group of data in a database according to this common data type.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0033] In order to understand the invention and to see how it can be carried out in practice, embodiments will be described, by way of non-limiting examples, with reference to the accompanying drawings, in which:
[0034]
[0035]
[0036]
[0037]
[0038]
[0039]
[0040]
[0041]
[0042]
[0043]
[0044]
[0045]
[0046]
[0047]
[0048]
[0049]
[0050]
[0051]
[0052]
DETAILED DESCRIPTION
[0053] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the presently disclosed subject matter may be practiced without these specific details. In other instances, well-known methods have not been described in detail so as not to obscure the presently disclosed subject matter.
[0054] Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the specification discussions utilizing terms such as “obtaining”, “providing”, “searching”, “determining”, “representing”, “associating”, “storing” or the like, refer to the action(s) and/or process(es) of a processor and memory circuitry (PMC) that manipulate and/or transform data into other data, said data represented as physical, such as electronic, quantities and/or said data representing the physical objects.
[0055] The term “processor and memory circuitry” (PMC) as disclosed herein should be broadly construed to include any kind of electronic device with data processing circuitry, which includes for example a computer processing device operatively connected to a computer memory (e.g. digital signal processor (DSP), a microcontroller, a field programmable gate array (FPGA), and an application specific integrated circuit (ASIC), etc.) capable of executing various data processing operations.
[0056] It can encompass a single processor or multiple processors, which may be located in the same geographical zone or may, at least partially, be located in different zones and may be able to communicate together.
[0057] The term “non-transitory memory” as used herein should be expansively construed to cover any volatile or non-volatile computer memory suitable to the presently disclosed subject matter.
[0058]
[0059] According to some embodiments, memory 120 can store a data structure 180 informative of a link between various data types. As explained in the various embodiments hereinafter, the data structure can be used e.g. for the purpose of storing data provided by the user in an existing database and/or in a new database, while complying with differences which can occur in data types.
[0060] A user (or a device) can provide, as an input to the system 100, a plurality of data 130. The plurality of data 130 can be divided into one or more groups of data.
[0061] In some embodiments, the plurality of data 130 comprises semi-structured data.
[0062] Each group of data can be identified for example because all data of this group share a common key or marker associated with this data.
[0063] In a non-limitative example, the plurality of data 130 comprises a table, and each group of data is a line or a column of the table. In the example of
[0064] In some embodiments, the plurality of data 130 comprises structured data.
[0065] Generally, each group of data comprises data which share similarity, which is why this data is stored in the same group by the user. Examples of data similarity will be provided hereinafter.
[0066] As explained hereinafter, the plurality of data 130 is provided to the system 100 because it is desired e.g. to store the plurality of data 130 into one or more databases 160.
[0067] In some embodiments, a new database is created based on the plurality of data 130 provided by the user.
[0068] In other embodiments, a database already exists, and it is attempted to store the plurality of data 130 together with data of the existing database (e.g. in this existing database and/or in a new common database).
[0069] According to some embodiments, database 160 can be stored e.g. in a cloud 140. According to some embodiments, database 160 can be stored e.g. in a memory 150 (which can be e.g. a local memory or a remote memory accessible using known communication techniques, such as a network, wire/wireless communication, etc.). This is, however, not limitative.
[0070] Attention is now drawn to
[0071] The method can comprise obtaining (operation 210) a data structure informative of different data types. Non-limitative examples of this data structure are provided in
[0072] The data structure can store a plurality of data types, or can be informative of the plurality of data types without necessarily storing them. A data type defines a type of data according to which data can be represented or encoded, in particular for the purpose of storing this data according to this data type.
[0073] Examples of data types include (this list is not limitative): [0074] String; [0075] Number (integer, float, Boolean, etc.); [0076] Date; [0077] Time; [0078] Date and time; [0079] Percentage; [0080] Currency; [0081] Etc.
[0082] According to some embodiments, each data type can be defined in a more accurate way (with one or more sub-categories—for example, sub-categories can define e.g. different values representative of maximal size or length for each data type).
[0083] Non-limitative examples of data types are provided hereinafter.
[0084] According to some embodiments, a number can be defined as an “integer”.
[0085] In some embodiments, “integer” can be defined according to a maximal value that can be represented or encoded.
[0086] For example, data type “integer (X)” can represent any integer which value is equal to or less than “X”.
[0087] According to some embodiments, a number can be defined as a “float”.
[0088] Data type “float (Y)” can represent any float which value is equal to or less than “Y”.
[0089] Each value of X or Y can therefore define a different data type in the data structure. If no value X or Y is defined, then “integer” or “float” are not limited to a predefined maximal value (up to a limit inherent to a storage capacity of the database, or up to a predefined storage maximal threshold of the database).
[0090] According to some embodiments, a “float” can be defined according to the number of decimal places.
[0091] For example, “float (Z)” defines a number which comprises at most Z decimal places (but there is no limitation on the integer part of the number, except a limit inherent to the storage capacity of the database or up to a predefined storage maximal threshold of the database).
[0092] Each value of Z can therefore define a different data type in the data structure.
[0093] According to some embodiments, a number can be represented as a “Boolean”. The data type “Boolean” can comprise either value “1” or value “0”. This option is represented in
[0094] According to some embodiments, a number can be defined as a “fraction”.
[0095] Data type “fraction (1/X)” can define different numbers that can be encoded, wherein X can have different integer values (e.g. when X=2, data is encoded according to halves, etc.). Each value of X can therefore define a different data type in the data structure.
[0096] According to some embodiments, a number can be defined as a “percentage”.
[0097] According to some embodiments, a percentage can be defined according to the number of decimal places. For example, “percentage (Z)” defines a Z decimal places percentage and not more. Each value of Z can therefore define a different data type in the data structure.
[0098] A string can be defined according to a number of characters (which can comprise e.g. either letters, numbers, or other characters). For example, “varchar (X)” (for variable characters) can encode a string which comprises, at most, X characters, but not more.
[0099] Each value of X can therefore define a different data type in the data structure.
[0100] If no value is defined for X, then “varchar” can correspond to a string of any size (up to a limit inherent to the storage capacity of the database or up to a predefined storage maximal threshold of the database).
[0101] According to some embodiments, a particular category of strings can be defined as a possible data type. For example, a data type can be defined as “Boolean” and comprises two possible values: “true” or “false”. This option is represented in
[0102] According to some embodiments, data type “date” can be defined differently depending on the format of the date. Examples of format can comprise (this is not limitative): [0103] Day/month/year, wherein day, month and year are numbers; [0104] Day of the week/day in the month/month/year, wherein day in the month, month and year are numbers, and day of the week is a string; [0105] Day/month/year, wherein day and year are numbers, and month is a string; [0106] Etc.
[0107] According to some embodiments, data type “time” can be defined differently depending on the format of the time. Examples of format can comprise (this is not limitative): [0108] Hour/minutes; [0109] Hour/minutes/seconds; [0110] Hour/minutes with parameter AM or PM; [0111] Etc.
[0112] A combination of different data types can provide another data type. For example, data type “date and time” defines date and also time. It can combine one of the definitions provided above for date and one of the definitions provided above for time.
[0113] In other embodiments, the data type “date and time” can be defined using international standards, such as ISO-8601.
[0114] Examples of data types provided above are not limitative and other examples can be used.
[0115] The data structure can be stored in various ways. In some embodiments, the data structure is stored as a tree (e.g. “decision tree”) comprising a plurality of nodes (see example of
[0116] In particular, each node is representative of a different data type. The decision tree can comprise various layers or levels. Each node N.sub.P (which is not an end node of the decision tree) is connected to at least one other node N.sub.C of the decision tree. N.sub.C is a children node of parent node N.sub.P.
[0117] This connection means that any data which is of data type of a children node N.sub.C can be also represented by data type of a parent node N.sub.P. In other words, a parent node N.sub.P represents a data type which is broader than data type of the children node N.sub.C (therefore data type of children node is included in data type of parent node).
[0118] For example, assume parent node N.sub.P represents a “float with two decimal places” and children node N.sub.C represents an “integer”. Any integer can be also represented by a float with two decimal places equal to zero.
[0119] If a children node N.sub.1 is connected to another parent node N.sub.3 through at least one other node N.sub.2 (in other words, N.sub.1 is connected indirectly to N.sub.3 and N.sub.2 ensures junction between N1 and N.sub.3), then both N.sub.2 and N.sub.3 correspond to data types which can also represent data type associated with node N.sub.1.
[0120]
[0121] As shown, a node is associated with data type “Boolean”, which can have two values “1” or “0”.
[0122] Node “Boolean” is connected to parent node “integer”, which is a broader data type.
[0123] Node “integer” is itself connected to parent node “float”. Indeed, any integer can be also represented by a float.
[0124] The node “float” is connected to parent node “varchar(25)”. Indeed, most databases use a scientific notation to represent large numbers. According to IEEE-754 standard, the largest float is 1.7976931348623157e+308, therefore a string with at most 25 characters is sufficient to represent a float up to this number.
[0125] The decision tree further comprises a node “Date and time” (which corresponds to a data type representing both date and time). The actual representation which is used to represent date and time (see above various examples) can depend e.g. on parameters of the database itself which is used to store data.
[0126] As shown, node “date and time” is also connected to parent node “varchar(25)” (in
[0127] The decision tree comprises a plurality of nodes “varchar(X)”, from X=0 (empty string) to X=N (largest supported string). In order to simplify representation of the decision tree, not all nodes “varchar(X)” are represented (for example, it is not represented that varchar(0) is connected to varchar(1), which is itself connected to varchar(2), etc.). In practice, each node “varchar(X)” can be connected to a parent node “varchar(X+1)”, up to the root of the decision tree (which comprises node “varchar(N)”). It is however not mandatory to have a node for each integer from X=0 to X=N, and in a more simplified implementation, only some values from X=0 to X=N are associated with a node varchar(X).
[0128] It has to be noted that the representation of
[0129]
[0130] In the example of
[0131] Indeed, any Boolean (“true” or “false”) can be represented by a broader data type varchar(5) (which covers any string comprising at most 5 characters).
[0132] The representation of the data structure as a decision tree is not limitative. Other representations can be used, such as, a table, a graph, a list, a hashmap, etc. In some embodiments, the data structure can be stored as a piece of software which can be queried, and which returns an output based on the query.
[0133] As shown in operations 210 and 220 of
[0134] In other words, the data type which is output based on the data structure is a broader data type which includes both “data type 1” and “data type 2”.
[0135] For example, assume a first data type is “Boolean” (which can have value “true” or “false”) and another data type is “varchar(3)” (which corresponds to any string that comprises at most three characters), then a common data type can be output as “varchar(5)” (which corresponds to any string that comprises at most five characters).
[0136] In the representation of
[0137] According to some embodiments, the data structure is usable so as to determine, for at least a first data type and a second data type among the plurality of data types, a given data type which is adapted to represent both data of at least the first data type and the second data type for their storage according to an efficiency criterion. According to some embodiments, and as explained hereinafter, since the data structure can be used iteratively, a common data type can be determined for a number of different data types which is larger than two.
[0138] According to some embodiments, the efficiency criterion can ensure an optimization (which is not necessarily the most optimized one) of the storage which is required to store both data belonging to the first data type and data belonging to the second data type according to a common data type.
[0139] Assume the decision tree of
[0140] Since each parent node in the decision tree represents a broader data type of its children nodes, storage (size of memory) required to store any data according to the representation of the corresponding broader data type is higher than storage required to store this data according to the representation of the data type of a children node.
[0141] This can be understood e.g. on a simple example. Assume “data type 6” is varchar(8) (any string which comprises at most eight characters), “data type 7” is varchar(9) (any string which comprises at most nine characters) and “data type 8” is varchar(10) (any string which comprises at most ten characters).
[0142] Any string which has a length which is equal to or less than eight characters can be represented either by varchar(8), varchar(9) or varchar(10).
[0143] However, for all strings which have a length that is equal to or less than eight characters, varchar(10) will require more storage than varchar(9), and varchar(9) will require more storage than varchar(8).
[0144] This is because the representation of varchar(10) will systematically comprise e.g. ten bits (if each character is coded using only one bit, this is however not limitative), wherein for all strings of eight characters two of these bits will be useless and will systematically comprise a zero.
[0145] The same applies to varchar(9) which will always comprise an additional bit which is useless. In other words, varchar(9) and varchar(10) are too broad data types for representing strings of eight characters, and therefore are not the most optimized for data storage in this example.
[0146] Although an example has been provided which pertains to strings, this applies similarly to other data types (see other non-limitative examples of data types above).
[0147] According to some embodiments, the efficiency criterion is met for a given data type when the given data type requires the lowest storage for representing data of at least the first data type and the second data type among all data types of the data structure.
[0148] This is however not limitative and, in some embodiments, it can be defined that the given data type is not necessarily the data type which requires the lowest storage among all data types of the data structure, but one of the lowest (e.g. the second, or third—this is not limitative).
[0149] According to some embodiments, the efficiency criterion can take into account other parameters, such as query speed for querying the data stored using the common data type in the database. This can be illustrated on a simple example.
[0150] Assume that first data includes a string with 1000 chars, all with the value of ‘A’: “AAAAAA . . . ” (1000 times), and second data includes a string with 1000 chars, all with the string ‘B’: “BBBBBB . . . ” (1000 times).
[0151] If only storage size is considered, a common data type can be varchar(6), since the first data can be represented as “Ax1000” and the second data can be represented as “Bx1000”. However, if the first data and the second data are stored using varchar(6), when the user queries his data and transmits a request “val>=“B””, it will be required to decompress the stored value “Ax1000” back to its original form (“AAAAAA . . . ” repeated 1000 times) and then determine whether it is larger than “B” (answer is FALSE). Then, it will be required to decompress the stored value “Bx1000” back to its original form (“BBBBBB . . . ” repeated 1000 times) and determine whether it is larger than “B” (answer is TRUE).
[0152] To the contrary, if the first data (“AAAAAA . . . ” repeated 1000 times) and the second data are each stored using the common data type “varchar(1000)”, when the user transmits the request “val>=“B””, the answer can be immediately found without requiring decompressing each stored value. The query speed is therefore optimized, although the storage size is less optimal. This example is however not limitative.
[0153] The data structure can therefore store and/or output for each pair of data types of the structure (or more generally for each group of data types) a common data type which is both adapted to represent the pair of data types. In some embodiments, the common data type is selected to meet the efficiency criterion, as mentioned above.
[0154] In particular, the data structure can store (in some hierarchical fashion): [0155] information indicating which data types can represent other data types of the data structure; and [0156] information indicating which data types require more storage than the others.
[0157] If the data structure is a decision tree, the data type which is adapted to represent both “data type 1” and “data type 2” and which meets the efficiency criterion can be found e.g. by searching for the first parent node in the decision tree which is connected to both node of data type 1 and node of data type 2 (because the tree was built so that the first parent node connected to both node of data type 1 and data type 2 is the most optimized with respect to the efficiency criterion). This is however not limitative.
[0158] In some examples, if the query pertains to “data type 1” and “data type 4” (see
[0159] For example, in
[0160] In some embodiments, the data structure is a tree which can be searched only in one “direction”. This means that that for any query in which a common data type has to be found for a first data type and a second data type, the decision tree will search only among: [0161] parent nodes of the node associated with the first data type and of the node associated with the second data type, and [0162] if the first data type is a parent node of the second data type (or conversely), among the first data type (or the second data type).
[0163] In some embodiments, and as explained hereinafter, the data structure is used several times in a process in which several data of different data types are aggregated into a common database.
[0164] In some embodiments, in this process, each time the data structure is searched (or queried), the current position in the data structure can be stored, and when the data structure is subsequently searched (or queried), it will allow only a search (or a query): [0165] in the direction of the decision tree in which parent nodes of the current position are located, or [0166] in the current position, in case the current position is a parent node of all other nodes of the query.
[0167] This ensures that the data type which is output will comply with all data types that have been searched/queried until the current search/query, since the decision tree always outputs a data type which is the same or broader than the previous data type.
[0168] Specific examples will be provided hereinafter.
[0169] Attention is now drawn to
[0170] As shown, the method comprises obtaining (operation 500) a plurality of data. The plurality of data can be provided to the system 100 by a user using any adapted means, such as a network, a storage device, another computer, a wire communication, a wireless communication, etc.
[0171] As explained above, the plurality of data is divided into one or more groups of data (for example, each group is a different column, or is associated with a different label which can be identified). Pointers or labels or any adapted representation can indicate to which group each data belongs.
[0172] In some embodiments, only one group of data is present and therefore all data is stored into one unique group.
[0173] Assume at least one group of data G.sub.1 of plurality of data can be divided into one or more subsets S.sub.1 to S.sub.N (e.g. by a processor and memory circuitry). For example, if the plurality of data comprises a table, and G.sub.1 is a column, then each subset S.sub.i can comprise one or more lines which belong to column G.sub.1. If a single subset is present, then the group is processed as a whole.
[0174] A non-limitative example is provided in
[0175] Another non-limitative example is provided in
[0176] The subsets of a given group of data are not necessarily of the same size. In addition, the subsets of data are necessarily of the same size between two different groups of data. In some embodiments, predefined rules indicate how to divide a given group of data into one or more subsets (e.g. the rules indicate the size of each subset). In some embodiments, the data type of the data is already known (e.g. provided by the user) and each subset can be selected to include data of the same data type.
[0177] The method can comprise, for each subset S.sub.i, determining (operation 510) a data type which is adapted to represent data belonging to this subset of data S.sub.i.
[0178] Therefore, for each subset of data S.sub.i, a data type DT.sub.Si is obtained. Embodiments for determining data type of a subset of data comprising a plurality of data will be described hereinafter (see
[0179] According to some embodiments, the data type of each data is already available when each group of data is obtained. For example, the user can specify the data type of each data. A label (e.g. reflecting the data type indicated by the user) can be associated with each data, which stores the data type.
[0180] According to some embodiments, the method can include determining the data type. Determination of data type of a given piece of data can involve using a computer-implemented function (software) which receives, as an input, data and can analyse characters stored in the data in order to output a data type of the data (among the predefined number of data types which are stored in the data structure).
[0181] The function can attempt to find the data type which meets an efficiency criterion among all possible data types of the data structure. The efficiency criterion can define that the selected data type ensures a low memory storage size of the data (e.g. the lowest memory storage size) and/or that query speed is minimized, as mentioned above.
[0182] For example, if the data is “false”, data type “Boolean” ensures a lower memory storage than “varchar(5)”.
[0183] For example, the function can perform the following operations: [0184] extracting characters of the data; [0185] comparing with predefined patterns of data; and [0186] identifying data type of data.
[0187] Assume for example that the data comprises “Wednesday March 10”. Assume that multiple predefined patterns for day/time are stored. The piece of data can be compared to these patterns, and it can be identified that the data corresponds to a date. This piece of data can be converted into another format (e.g. Jason format), which better fits to the storage in a database.
[0188] In some embodiments, the function can perform the following operations: [0189] extracting characters of the data; [0190] analysing whether characters are strings and/or numbers; [0191] if the characters are only numbers, identifying the correct data type (integer, float, etc.) with the correct size (size of the integer, or number of decimal places, etc.), and outputting the relevant data type among all pre-stored data types; [0192] if the characters comprise numbers and additional characters which are not letters, identifying the correct data type accordingly. For example, if the additional characters comprise sign “%”, identifying data type “percentage”; if the additional characters comprise a sign representative of currency, such as “$”, identifying data type “currency”; if the additional characters comprise at least two times character “/” between two numbers, identifying data type “date”, etc.; [0193] if the characters comprise only letters, identifying a data type such as “Boolean” or “varchar” (with a length of the string); and [0194] if the characters comprise numbers and letters, attempting to identify a data type such as “date”, “date and time”, or, if this is not adequate, “varchar”.
[0195] Operations described above are not limitative and other operations can be performed depending on the data types that are to be identified.
[0196] The method can comprise, based on the data types DT.sub.Si that have been identified, and on the plurality of data, an operation 520 which comprises determining a common data type DT.sub.com which is adapted to represent all data belonging to subsets of data S.sub.1 to S.sub.N of a group of data.
[0197] If a plurality of groups of data are present in the data structure, a common data type can be output which is specific to each group of data.
[0198] As explained hereinafter, this common data type can be used to store all data of the group of data in a database, wherein all data are represented/encoded using this common data type.
[0199] In particular, the common data type can be a data type which: [0200] is adapted to represent all data of a given group of data, and [0201] enables storage of the data of the given group of data according to an efficiency criterion (as mentioned above, this can include the fact that the common data type requires the lowest storage size (or, in some embodiments, one of the lowest) among all possible data types stored in the data structure, for storing data of the given group of data, and/or enables to optimize query speed of the data stored according to the common data type in a database).
[0202] It has been mentioned above that operation 510 comprises determining data type DT.sub.Si of a subset S.sub.i. Embodiments for determining data type which is adapted to a whole subset S.sub.i are provided below.
[0203] Assume subset S.sub.i comprises “Data j” with j from 1 to p.
[0204] If subset S.sub.i comprises only one data (p=1), then data type DT.sub.Si can be determined using the function mentioned above, which detects data type of “data 1” stored in the subset S.sub.i. Since a single piece of data is present, DT.sub.Si is determined and is adapted for the whole subset S.sub.i.
[0205] If subset S.sub.i comprises a plurality of data (that is to say that p>1—for example, subset S.sub.i corresponds to a plurality of lines of a given column), then data type DT.sub.Si of subset S.sub.i has to be determined such that it is adapted for all data of this subset S.sub.i. An example is shown in
[0206] Assume data type DT.sub.Si,j is determined for each “Data j” of subset S.sub.i (with j from 1 to p), using the function mentioned above.
[0207] According to some embodiments, operation 510 can include using all data types DT.sub.Si,j (with j from 1 to p) determined for data of subset S.sub.i to search (or query) the data structure. A common data type DT.sub.Si is output, which is adapted for this subset S.sub.i. In some embodiments, the common data type DT.sub.Si meets an efficiency criterion, that is to say that it requires the lowest storage (or one of the lowest storage requirement) among all possible data types stored in the data structure, for storing data of the subset S.sub.i, and/or enables to optimize query speed for data of subset S.sub.i stored using the common data type DT.sub.Si.
[0208] According to some other embodiments, common data type DT.sub.Si of subset S.sub.i can be determined by progressively scanning data of subset S.sub.i.
[0209] This can comprise, for j from 1 to p: [0210] for j=1, DT.sub.Si=DT.sub.Si,1; [0211] for j>1, comparing DT.sub.Si,j with DT.sub.Si; [0212] if DT.sub.Si,j is equal to DT.sub.Si, then incrementing j by one; [0213] if DT.sub.Si,j is different from DT.sub.Si: [0214] using the data structure to determine a common data type to DT.sub.Si,j and DT.sub.Si (e.g. which meets the storage criterion), and [0215] assigning DT.sub.Si equal to this common data type and incrementing j by one.
[0216] A non-limitative example of this second method is provided in
[0217] A user stores in a table data representative of an address, which can comprise numbers, numbers with characters, or numbers with expression such as “Street” or “Avenue” (signification of the data does not need to be known by the system and is provided for illustration only).
[0218] A first subset S.sub.1 comprises three data: “28”, 254” and “28Street”.
[0219] A common data type of “28” and “254” is “integer(3)” (meaning integer with at most three digits). Therefore, at this stage, DT.sub.S1 is temporarily equal to integer(3).
[0220] This data type DT.sub.S1 is compared to data type of “28Street” which is “varchar(8)” (meaning a string with at most eight characters). Since the two data types are different, a new common data type DT.sub.S1 is determined using the data structure, which corresponds to DT.sub.S1=“varchar(8)” for subset S.sub.1.
[0221] Attention is now drawn to
[0222] The method comprises obtaining a plurality of data (operation 600, similar to operation 500 above), determining data type of one or more subsets of each group of data of the plurality of data (operation 610, similar to operation 510 above) and determining a common data type for each group based on the data structure (operation 620, similar to operation 520 above).
[0223] Assume the plurality of data has to be stored in a database to be created. The method can comprise creating a database comprising the same number of groups (G′.sub.1 to G′.sub.M) as the plurality of data (G.sub.1 to G.sub.M). Each group G′.sub.i of the database is assigned the common data type DT.sub.Gi that was identified for group G.sub.i of the plurality of data. Each data of group G.sub.i is represented and stored in corresponding group G′.sub.i according to the common data type DT.sub.Gi (operation 630).
[0224] A non-limitative example is provided in the upper part of
[0225] Attention is now drawn to
[0226] Assume that a second plurality of data (see e.g. reference 682 in the bottom part of
[0227] This second plurality of data 682 comprises also data divided into a plurality of groups G.sub.i. This can be identified by determining labels associated with the data and/or columns or lines of the plurality of data. It is assumed for example that the first column (or row) of the second plurality of data 682 corresponds to the same group as the first column (or row) of the database. This applies also the other columns (or rows).
[0228] Once the second plurality of data 682 has been obtained by the system (operation 640), a method can comprise determining data type of one or more subsets of each group of data of the second plurality of data 682 (operation 650, similar to operation 510 above).
[0229] It is desired to store data of each group G.sub.i of the second plurality of data 682 with data already stored in the corresponding group G′.sub.i into a common database (e.g. by updating the existing database 681).
[0230] Therefore, the method can comprise determining (operation 660), for each group G.sub.i of the second plurality of data 682, a new common data type DT′.sub.Gi which is adapted both for data of the second plurality of data 682 and for data type DT.sub.Gi of the existing database 681. In some examples, the new common data type DT′.sub.Gi remains equal to DT.sub.Gi since data of the second plurality of data 682 is compliant with the “old” common data type DT.sub.Gi.
[0231] The method can comprise, for each group G.sub.i of the second plurality of data 682, encoding data according to the new common data type DT′.sub.Gi for their storage (operation 670).
[0232] If a new common data type DT′G.sub.i different from DT.sub.Gi is obtained, then the method can comprise (operation 670), for each group G′.sub.i of the database 681, re-encoding existing data of the database 681 according to the new common data type DT′.sub.Gi.
[0233] In some embodiments, this can comprise changing in the existing database 681 the data type associated to each group G.sub.i.
[0234] In other embodiments (e.g. for a database which does not allow changing data type of the existing groups), the method can comprise duplicating data of the existing database 681 and creating a new database with the new common data type assigned to each group of the new database. The new database will store both data of database 681 and data of the second plurality of data 682 according to the new common data type DT′.sub.Gi.
[0235] All this data is then stored by group into a common database (see reference 683), in which each group is assigned with the new common data type DT′.sub.Gi.
[0236] These operations can be repeated each time new data has to be inserted in the database.
[0237] If the second plurality of data 682 comprises more groups than the existing database, then the additional group of the second plurality of data 682 will cause creation of a new group in the database.
[0238] Creation of a new group can be performed similarly to any method described in the present application in which a new database is created (the difference is that instead of creating a new database, a new group is created in addition to existing group(s) of the database).
[0239] For example, the method of
[0240] If the second plurality of data comprises less groups than the existing database, then only the groups which are common to the second plurality of data and to the existing database will be handled e.g. according to the method of
[0241] Attention is drawn to
[0242] Assume a data type DT.sub.Si has been obtained for each subset of data S.sub.i (with i from 1 to N, as explained above in operation 510) of a group of data (operation 710).
[0243] The method can comprise (operation 720) using the data structure in order to find a common data type for all these subsets, and therefore for the whole group. For example, if the data structure is a decision tree, then each data type DT.sub.Si corresponds to a node in the decision tree (a plurality of nodes is thus identified, one for each data type DT.sub.Si). The common data type can be identified in the decision tree as a parent node which is connected (directly or indirectly) to all of this plurality of nodes.
[0244] The efficiency criterion can be used to select the parent node which optimizes storage and/or query speed among all parent nodes which are representative of a common data type to all data types D.sub.S1 to D.sub.SN.
[0245] For example, assume a first data type of subset S.sub.1 is “float”, a second data type of subset S.sub.2 is “varchar(10)” (string comprising at most ten characters) and a third data type of subset S.sub.3 is “date and time”. The common data type can be identified as varchar(25) (string comprising at most twenty five characters) which is adapted to represent these three data types, and is also optimized in terms of storage requirement.
[0246] If the plurality of data comprises a plurality of groups of data G.sub.1 to G.sub.M (e.g. a plurality of columns), the method can comprise performing operations 710 and 720 for each group of data G.sub.i, in order to find a common data type DT.sub.Gi specific to each group of data G.sub.i.
[0247] Attention is drawn to
[0248] Assume a plurality of data is provided to the system 100 e.g. by a user, in order to be stored in database (for example the database can be stored remotely on a cloud 140, this is however not limitative). The plurality of data comprises one or more groups of data G.sub.1 to G.sub.M.
[0249] The method can comprise performing operations 710 and 720 (see
[0250] Since a common data type DT.sub.Gi has been identified for each group of data G.sub.i, the method can comprise creating a database comprising the same number of groups of data G′.sub.i (e.g. columns), with i from 1 to M, wherein each group of data G′.sub.i is assigned with the common data type DT.sub.Gi that has been identified at operation 720. This assignment represents the fact that any data that is stored in group G′.sub.i of the database will be encoded according to the common data type DT.sub.Gi (as mentioned in the embodiments hereinafter, this common data type can evolve over time).
[0251] The method can comprise encoding data of group G.sub.i according to the common data type DT.sub.Gi, and storing the encoded data in group G′.sub.i of the database (operation 730).
[0252] A non-limitative example is provided in
[0253] A plurality of data 750 is provided by a user (e.g. a government). The plurality of data comprises two groups of data G.sub.1 and G.sub.2.
[0254] In the first group of data G.sub.1, identity number of people is stored. After a period of time, the user has decided to add a new character in the data which can comprise either “M” (for male) and “F” (for female).
[0255] In the second group of data G.sub.2, the user stores data which represents whether each person is a citizen of the country (“true” corresponds to “citizen of the country” and “false” corresponds to “non-citizen”).
[0256] Using the method of
[0257] Using the method of
[0258] Attention is now drawn to
[0259] Assume an existing database has already been created (see e.g. 760 in
[0260] The method can comprise obtaining (operation 770) a second plurality of data (which comprises also data divided into a plurality of groups G′.sub.i). It is assumed that data of group G′.sub.i is to be stored with data of group G.sub.i (because this corresponds to the same group, for example because this is the same i.sup.th column or row in the second second plurality of data and in the database—see below the case in which the number of groups in not the same).
[0261] The method can comprise, for each group G′.sub.i of the second second plurality of data, determining (operation 775—similar to operation 510 of
[0262] The method can comprise, for each group G′.sub.i, inputting all data types of all subsets of the group in the data structure to find a common data type DT.sub.G′i for each group G′.sub.i of the second plurality of data (operation 780 similar to operation 720).
[0263] The method can comprise comparing (operation 785) common data type DT.sub.G′i of each group G′.sub.i of the second plurality of data with current data type DT.sub.Gi of each group G.sub.i of the existing database.
[0264] If this comparison shows a difference, the method can comprise determining (operation 790) a new common data type DT′.sub.Gi which is adapted both for data type DT.sub.Gi and for data type DT.sub.G′i.
[0265] This determination can be performed using the method described in
[0266] The method can comprise (operation 795) storing (for each group) data of the database and data of the second plurality of data according to the new common data type DT′.sub.Gi in the same group of a database (which can be the same database in which an update has been performed, or a new database).
[0267] If the second plurality of data comprises more groups than the existing database, then the additional group of the second plurality of data will cause creation of a new group in the database.
[0268] Creation of a new group can be carried out similarly to any method described in the present application in which a new database is created.
[0269] If the second plurality of data comprises less groups than the existing database, then only the groups which are common to the second plurality of data and to the existing database will be handled e.g. according to the method of
[0270] Attention is drawn to
[0271] The plurality of data comprises one or more groups G.sub.1 to G.sub.N. The system can divide each group into one or more subsets.
[0272] The method comprises determining (operation 800) data type DT.sub.Si for each subset of data S.sub.i of a group of data G.sub.i of a plurality of data (embodiments for performing this determination have been described with reference to operation 510 of
[0273] The method can comprise attempting to determine a common data type of the group G.sub.i by analysing, progressively, data types of the subsets of data S.sub.i of this group G.sub.i.
[0274] Operations will be described for one group, but each group can be treated independently according to this method (groups can be processed in parallel, or one after the other).
[0275] At the first iteration of the method (in which i=1), data type DT.sub.S1 is temporarily considered as the current common data type DT.sub.Ci (in other words, DT.sub.Ci=DT.sub.S1).
[0276] Then the value of i is incremented by one.
[0277] The method comprises (operation 810), for current value of i, comparing data type DT.sub.Si with current common data type DT.sub.Ci-1. Current common data type DT.sub.Ci-1 is valid to represent all data of subsets S.sub.1 to S.sub.i-1.
[0278] Based on this comparison, the method can comprise: [0279] If this comparison shows that DT.sub.Si and DT.sub.Ci-1 are the same, then the current common data type DT.sub.Ci (which will be valid to represent subsets S.sub.1 to S.sub.i) is set equal to DT.sub.Ci-1 (in other words the value of the current common data type remains unchanged). If i is not equal to N, the value of i is then incremented by one and operations 800 and 810 are repeated. In some embodiments, the method returns to operation 810 because data type of each of subsets S.sub.1 to S.sub.N has all been determined beforehand (that is to say that operation 800 was already performed N times before operations 810 to 840 are performed); [0280] If this comparison shows that DT.sub.Si and DT.sub.Ci-1 are not the same, then the method comprises using the data structure (operation 820) to find a new current common data type DT.sub.Ci (which will be valid for subsets S.sub.1 to S.sub.i). Operation 820 can comprise providing to the data structure as an input data type DT.sub.Si and data type DT.sub.Ci-1. A new current common data type DT.sub.Ci (valid for both DT.sub.Si and DT.sub.Ci-1) can be found using the method described with reference to
[0281] If i is equal to N (operation 830), current common data type DT.sub.CN is provided (operation 840). This common data type DT.sub.CN is adapted to represent all data of all subsets S.sub.1 to S.sub.N. In some embodiments, this common data type is the data type of the data structure which: [0282] is adapted to represent all data of all subsets S.sub.1 to S.sub.N; and [0283] meets the efficiency criterion (e.g. is a data type which is optimized for storage of all data of all subsets S.sub.1 to S.sub.N, among all data types of the data structure and/or optimizes query speed of the data in the database).
[0284] The method of
[0285] Attention is now drawn to
[0286] Assume a plurality of data is obtained comprising data divided into one or more groups. The method of
[0287] If a database does not yet exist, the method can comprise (operation 920) creating a new database, with groups similar to the plurality of data. Each group of the database is assigned with the common data type determined for the corresponding group of plurality of data. The plurality of data can thus be stored in the database according to the common data type relevant to its group (operation 930).
[0288] If a database already exists, which already stores data into one or more groups G′.sub.1 to G′.sub.N according to a common data type (DT.sub.G′1 to DT.sub.G′N) specific to each group, the method can comprise: [0289] for each group G.sub.i of the plurality of data, comparing the common data type DT.sub.Gi of this group G.sub.i with the common data type of the corresponding group G′.sub.i of the database; [0290] if there is a match, then data of the group G.sub.i of the plurality of data can be stored in group G′.sub.i of the database according to the common data type DT.sub.G′i=DT.sub.Gi; [0291] if there is a difference, then a new common data type DT′.sub.G′i can be determined using the data structure which fits both data type DT.sub.Gi of the group G.sub.i of the plurality of data and common data type DT.sub.G′i of the group G′.sub.i of the database. The method of
[0292] Attention is now drawn to
[0293] Assume a database already exists in which one or more groups G′.sub.1 to G′.sub.M are present. Each group G′.sub.j stores data according to data type DT.sub.G′j.
[0294] Assume a plurality of data is obtained by the system. The plurality of data has to be stored together with data of the existing database in a common database (operation 1000). The plurality of data is also divided into one or more groups G.sub.1 to G.sub.M.
[0295] In some embodiments, it is possible that the plurality of data comprises less groups than the database—in this case only the common groups will be handled as explained hereinafter.
[0296] In other embodiments, it is possible that the plurality of data comprises more groups than the database—in this case the common groups will be handled as explained hereinafter, and the additional groups of the plurality of data can yield to the creation of new groups in the database, using one or more methods described above (see e.g.
[0297] Each group G.sub.j of the data structure can be divided into a plurality of subsets S.sub.i,j (as already explained above) by the system.
[0298] The method can comprise determining (operation 1010) data type of each subset S.sub.i,j (see e.g. operation 510 in
[0299] A basic principle of the method is to attempt to determine, in a progressive way, a common data type which is valid both for the subsets of data of group G.sub.j, and for data of the corresponding group G′.sub.j in the database (currently stored according to data type DT.sub.G′j).
[0300] The method can comprise, for each group G.sub.j of the plurality of data, comparing: [0301] data type DT.sub.Si,j of the subset S.sub.i,j, [0302] with [0303] a current common data type DT.sub.Ci-1,j, which is adapted to represent both data of subsets S.sub.1,j to S.sub.i-1,j and data of data type D.sub.G′j.
[0304] In other words, DT.sub.Ci-1,j is the current estimation of the common data type (for both data of group G.sub.j of the data structure and for data of group G′.sub.j of the database) when only subsets S.sub.1,j to S.sub.i-1,j of group G.sub.j of the data structure have been processed. At the initialization of the method, that is to say for i=1, DT.sub.C0,j is set equal to DT.sub.G′J (since no subset of group G.sub.j of the data structure has been yet analysed).
[0305] If there is a match of the comparison mentioned above, then the current common data type DT.sub.Ci-1,j does not need to be modified, and therefore DT.sub.Ci,j=DT.sub.Ci-1,j. The value of i is incremented by one and the method returns to operation 1010 or to operation 1020 (the method returns to operation 1020 in case the data type of the subsets of the plurality of data has been already determined beforehand).
[0306] If there is a difference in the comparison mentioned above, the method can comprise using (1030) the data structure to determine a new common data type DT.sub.Ci,j which complies both with the data type DT.sub.Si,j of the subset S.sub.i,j, and with the previous estimation of the common data type DT.sub.Ci-1,j. This new common data type can meet the efficiency criterion as mentioned above. The method of
[0307] The value of i is incremented by one and the method returns to operation 1010 or to operation 1020 (the method returns to operation 1020 in case the data type of the subsets of the plurality of data has been already determined beforehand).
[0308] Once all the subsets of the group G.sub.j have been processed (which means that i=N), a common data type DT.sub.CN,j is obtained, which is adapted to represent both all existing data of the group G′.sub.j of the database and all data of the group G.sub.j of the plurality of data.
[0309] Therefore, all this data of group G′.sub.j and of group G.sub.j can be processed to be encoded (or re-encoded) according to this common data type DT.sub.CN,j, and can be stored in the same database.
[0310] As already mentioned above, the database can be the existing database which is updated, or a new database which comprises both data of the old database, and data of the data structure.
[0311] It has to be noted that the operations described in
[0312] For example, in some embodiments, all the groups of the plurality of data can be processed simultaneously (operations 1010 to 1050 can be performed therefore in parallel for the different groups of the data structure).
[0313] In other embodiments, each group of the plurality of data is processed one after the other according to the method of
[0314] Attention is drawn to
[0315] Assume an existing database 1100 stores data into a single group G′.sub.1. The data type of this group G′.sub.1 is an integer. For example, a user stores data representative of a weight (in Kg) of a person.
[0316] Assume a plurality of data 1110 is provided by the user, which comprises one group G.sub.1. Instead of storing only integer for the weight, the user now stores at least some of the data according to a float representation (that is to say that he stores not only the weight in Kg but also the weight in grams).
[0317] Using the method of
[0320] Since all subsets of group G.sub.1 have been processed, it appears that data type “DT.sub.C2,1=float” is adapted both for group G.sub.1 and group G′.sub.1.
[0321] Therefore, both data of group G.sub.1 and both data of group G′.sub.1 can be represented according to this data type and can be stored in a new or updated group G′.sub.1 of a database 1120 (this new group G′.sub.1 can correspond to group G′.sub.1 of the existing database which was updated, or to a new group G′.sub.1 of a new database which was created).
[0322] Therefore, the method offers automatic storage of the new data which is transparent to the user and is efficient (for storage and/or query speed).
[0323] Assume a plurality of data 1130 is provided by the user. Instead of storing only an integer for the weight, the user now stores a character representative of the identity: “M” for male, “F” for female, and “K” for kids.
[0324] One can understand that this plurality of data 1130 cannot be stored as such in the existing database 1120, since group G′.sub.i only stores data according to a float representation. As explained hereinafter, the method is able to automatically cope with this problem without requiring the intervention of the user.
[0325] Using the method of
[0328] Since all subsets of group G.sub.1 have been processed, it appears that data type “DT.sub.C2,1=varchar(25)” is adapted both for group G.sub.1 and group G′.sub.1. Therefore, both data of group G.sub.1 and both data of group G′.sub.1 can be represented according to this data type and can be stored in a new group G′.sub.1 of a database 1140 (this new group G′.sub.1 can correspond to group G′.sub.1 of the existing database 1120 which was updated, or to a new group G′.sub.1 of a new database which was created).
[0329] As shown in this example, the method allows repetitively updating (if necessary) the data type of the database, without requiring checking at each time all the previous data types that were inserted in the database. This is in particular due to the fact that the data structure is built to provide, for any query of two data types which are not the same, a broader data type. Therefore, this ensures that over time, any update of the data type associated with the database will comply with all previous data already inserted in the database.
[0330] In a particular example, the data structure is a tree (e.g decision tree). In particular, it is a decision tree which can be searched only according to one direction.
[0331] This is shown in the non-limitative example of
[0332] The method comprises storing the current node in the decision tree (operation 1200), which corresponds to the current data type which is identified at this stage as compliant for storing data that have been analyzed.
[0333] Each time an update of the database has to be performed (and therefore the decision tree has to be searched or queried), the decision tree allows a search or a query only according to one direction, that it to say only in the current node, or towards the parent nodes of the current node (operation 1210).
[0334] Since the decision tree is built such that parent nodes represent broader data type category than children nodes, the data type output by the decision in response to a query can be either the same data type, or a broader one. As a consequence, the data type output by the decision tree will always comply with all data processed until now.
[0335] An example is shown in
[0336] Embodiments of the presently disclosed subject matter are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the presently disclosed subject matter as described herein.
[0337] The invention contemplates a computer program being readable by a computer for executing one or more methods of the invention. The invention further contemplates a machine-readable memory tangibly embodying a program of instructions executable by the machine for executing one or more methods of the invention.
[0338] It is to be noted that the various features described in the various embodiments may be combined according to all possible technical combinations.
[0339] It is to be understood that the invention is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Hence, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting. As such, those skilled in the art will appreciate that the conception upon which this disclosure is based may readily be utilized as a basis for designing other structures, methods, and systems for carrying out the several purposes of the presently disclosed subject matter.
[0340] Those skilled in the art will readily appreciate that various modifications and changes can be applied to the embodiments of the invention as hereinbefore described without departing from its scope, defined in and by the appended claims.