MERGING DATA BASED ON PROXIMITY AND VALIDATION
20190179925 ยท 2019-06-13
Inventors
Cpc classification
G06F16/1737
PHYSICS
G06F16/2379
PHYSICS
International classification
Abstract
Various systems, mediums, and methods herein describe aspects of a proximity and validation based data merging system. The system may access a first data set and a plurality of second data sets that correspond to physical locations in a location space. The system my segment the location space into a plurality of location bins each having one or more nodes. The system may generate two or more maps based on two or more mapping rule and map the first data set and the plurality of second data sets into the nodes of the location bins. The system may determine one or more target nodes in the maps that include the first data set and at least one of the second data sets. The system may determine a minimum distance between the first data set and one of the second data sets in a target node.
Claims
1. A method comprising: accessing, by one or more hardware processors, a first data set comprising first location information corresponding to a first physical location in a location space; accessing, by the one or more hardware processors, a plurality of second data sets, wherein each one of the second data sets comprises second location information corresponding to a second physical location in the location space; segmenting the location space, by the one or more hardware processors, into a plurality of location bins based at least on a location bin size, wherein each one of the plurality of location bins comprises one or more nodes; determining, by the one or more hardware processors, two or more mapping rules, wherein each one of the mapping rules is configured to map the first data set based on the first location information and each one of the second data sets based on the second location information to a node of a location bin; generating, by the one or more hardware processors, two or more maps based on the two or more mapping rules; determining, by the one or more hardware processors in the two or more maps, one or more target nodes that comprise the first data set and at least one of the second data sets; determining, by the one or more hardware processors, one or more target distances in the one or more target nodes between the first data set and the at least one of the second data sets of a same target node based on the first location information and the second location information; determining, by the one or more hardware processors, a target second data set that corresponds to a minimum of the one or more target distances; and merging the first data set into the plurality of the second data sets after a validation of the first data set.
2. The method of claim 1, further comprising determining the location bin size prior to the segmenting.
3. The method of claim 2, further comprising iteratively adjusting the location bin size until at least one target node is determined in the two or more maps.
4. The method of claim 1, wherein the first data set comprises first additional information and each one of the second data sets comprises second additional information, the method further comprising validating the first data set based on the first additional information of the first data set and the second additional information of the target second data set.
5. The method of claim 1, wherein the generating the two or more maps comprises: quantizing the second location information of the plurality of second data sets; and quantizing the first location information of the first data set.
6. The method of claim 5, wherein generating at least one of the two or more maps further comprises: translating the first location information of the first data set in a predetermined direction by a predetermined amount prior to the quantizing; and translating the second location information of the plurality of second data sets in the predetermined direction by the predetermined amount prior to the quantizing.
7. The method of claim 1, further comprising determining a metric for the determining the one or more target distances in the one or more target nodes.
8. A system comprising: a non-transitory memory; and one or more hardware processors coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations comprising: accessing a first data set comprising first location information corresponding to a first physical location in a location space; accessing a plurality of second data sets, wherein each one of the second data sets comprises second location information corresponding to a second physical location in the location space; segmenting the location space into a plurality of location bins based at least on a location bin size, wherein each one of the plurality of location bins comprises one or more nodes; determining two or more mapping rules, wherein each one of the mapping rules is configured to map the first data set based on the first location information and each one of the second data sets based on the second location information to a node of a location bin; generating two or more maps based on the two or more mapping rules; determining, in the two or more maps, one or more target nodes that comprise the first data set and at least one of the second data sets; determining one or more target distances in the one or more target nodes between the first data set and the at least one of the second data sets of a same target node based on the first location information and the second location information; determining a target second data set that corresponds to a minimum of the one or more target distances; and merging the first data set into the plurality of the second data sets after a validation of the first data set.
9. The system of claim 8, wherein the first data set comprises first additional information and each one of the second data sets a second additional information, wherein the operations further comprise validating the first data set based on the first additional information of the first data set and the second additional information of the target second data set.
10. The system of claim 9, wherein the first additional information comprises a first Internet Protocol (IP) address and the second additional information comprises a second IP address, and wherein the operations further comprise detecting a fraudulent IP address of the first data set based on a comparison of the first IP address of the first data set with the second IP address of the target second data set.
11. The system of claim 9, wherein the first data set is included in a first database and the plurality of the second data sets are included in a second database, wherein the accessing the first data set comprises accessing the first database and the accessing the plurality of second data sets comprises accessing the second database, and wherein the operations further comprise in response to the validating, merging the first data set into the second database and removing the first data set from the first database.
12. The system of claim 8, wherein the plurality of location bins are configured to cover the location space.
13. The system of claim 8, wherein each one of the plurality of location bins comprises two or more nodes, and wherein each two adjacent location bins from the plurality of location bins are configured to share one or more nodes from the two or more nodes.
14. The system of claim 8, wherein each one of the two or more maps are generated by mapping the first data set and the plurality of second data sets into the one or more nodes of the plurality of location bins using one of the two or more mapping rules.
15. The system of claim 8, wherein the minimum of the one or more target distances corresponds to a least distance between the first data set and the plurality of second data sets.
16. The system of claim 8, wherein the generating the two or more maps comprises: quantizing the second location information of the plurality of second data sets; and quantizing the first location information of the first data set, wherein the quantizing comprises one of a floor, a ceiling, or a rounding operation.
17. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising: accessing a first data set comprising first location information corresponding to a first physical location in a location space and first additional information; accessing a plurality of second data sets, wherein each one of the second data sets comprises second location information corresponding to a second physical location in the location space and second additional information; segmenting the location space into a plurality of location bins based at least on a location bin size, wherein each one of the plurality of location bins comprises one or more nodes; determining two or more mapping rules, wherein each one of the mapping rules is configured to map the first data set based on the first location information and each one of the second data sets based on the second location information to a node of a location bin; generating two or more maps based on the two or more mapping rules; determining, in the two or more maps, one or more target nodes that comprise the first data set and at least one of the second data sets; determining one or more target distances in the one or more target nodes between the first data set and the at least one of the second data sets of a same target node based on the first location information and the second location information; determining a target second data set that corresponds to a minimum of the one or more target distances; validating the first data set based on the first additional information of the first data set and the second additional information of the target second data set; and in response to the validating, merging the first data set into the plurality of the second data sets.
18. The non-transitory machine-readable medium of claim 17, wherein each two adjacent location bins are connected and share one or more nodes.
19. The non-transitory machine-readable medium of claim 17, wherein the location space is a two-dimensional location space that comprises a longitude and a latitude.
20. The non-transitory machine-readable medium of claim 17, wherein the location space is one of a district in a city, one or more cities, one or more states in a country, one or more countries, one or more continents, or an entire globe.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0004]
[0005]
[0006]
[0007]
[0008]
[0009]
[0010]
[0011]
[0012]
[0013]
[0014]
[0015]
[0016]
[0017]
[0018] Embodiments of the present disclosure and their advantages may be understood by referring to the detailed description provided herein. It should be appreciated that reference numerals may be used to illustrate various elements and/or features provided in the figures. Further, the figures may illustrate various examples for purposes of illustration and explanation related to the embodiments of the present disclosure and not for purposes of any limitation.
DETAILED DESCRIPTION
[0019] In the following description specific details are set forth describing certain embodiments. It will be apparent, however, to one skilled in the art that the disclosed embodiments may be practiced without some or all of these specific details. The specific embodiments presented are meant to be illustrative, but not limiting. One skilled in the art may realize other material that, although not specifically described herein, is within the scope and spirit of this disclosure.
[0020] In some embodiments, one or more hardware processors of a system may execute a data access module, a segmentation module, a mapping module, an examination module, a determination module, a bin size updating module, and a database updating module. The data access module obtains a first data set from a first database and also obtains a plurality of second data sets from a second database. Both the first data set and the plurality of second data sets include location information that corresponds to physical locations in a location space. The segmentation module divides the location space into a plurality of location bins based on a location bin size. Each location bin includes at least two nodes that may connect to the adjacent bins and the nodes are shared between the adjacent bins. The mapping module maps the first data set of the first database and the plurality of the second data sets of the second database to the nodes of the plurality of the location bins of the location space. The mapping module uses two or more mapping rules for mapping the first data set and the plurality of the second data sets into the nodes of the plurality of location bins and creates two or maps. The mappings to the nodes of the plurality of the location bins of the location space are based on the location information of the first data set and on the location information of the plurality of the second data sets. The examination module searches the nodes of the plurality of location bins in each of the two or more maps to find target nodes that include the mapped first data set and at least one mapped second data set. If any such target node is found in at least one of the maps, then the determination module determines the physical distance between the first data set and each of the second data sets in the target node of each map. The determination module determines the least physical distance between the first data set and the plurality of the second data sets by determining the least physical distance between the first data set and the second data sets in each target node. The determination module further determines the second data set having the least physical distance to the first data set as the target second data set. In some embodiments, the first data set includes first additional information and the second data set includes second addition information. The determination module further validates the first data set based on the first additional information of the first data set the second additional information of the target second data set. In some examples, the validation is performed based on a first Internet Protocol (IP) address included in the first additional information of the first data set and a second IP address included in the second additional information of the target second data set. After validation, a database updating module updates the first database and the second database by merging the validated first data set into the second data sets of the second database and removing the first data set from the first database.
[0021] Distance calculations are usually invoked to determine proximity of a first data set to a plurality of second data sets. The first data set and the plurality of second data sets may be associated with physical locations in a two-dimensional space and the distance calculations may involve using the physical locations to determine physical distances between the first data set and the plurality of second data sets in a two-dimensional space, such as using L.sup.2 norms, between the physical location of the first data set and physical locations of the elements of the plurality of second data sets. Note that any distance in any matrix space maybe determined (e.g., L.sup.1, L.sup.2, etc. norms and Geodesic distances, etc.). A minimum of the calculated distances is then determined as the least distance between the first data set and the plurality of second data sets. In some examples, the plurality of second data sets comprises a database with a large number of elements and calculating the distances between the first data set and the elements of the plurality of second data sets may be time consuming and require significant processing power. Additionally, it may be required to determine the distances between a number of first data sets with the plurality of second data sets having the large number of elements, which can become very time consuming.
[0022] In some embodiments, a method for addressing the above by merging data sets comprises accessing, by one or more hardware processors, a first data set comprising first location information corresponding to a first physical location in a location space; accessing, by the one or more hardware processors, a plurality of second data sets, wherein each one of the second data sets comprises second location information corresponding to a second physical location in the location space; segmenting the location space, by the one or more hardware processors, into a plurality of location bins based at least on a location bin size, wherein each one of the plurality of location bins comprises one or more nodes; determining, by the one or more hardware processors, two or more mapping rules, wherein each one of the mapping rules is configured to map the first data set based on the first location information and each one of the second data sets based on the second location information to a node of a location bin; generating, by the one or more hardware processors, two or more maps based on the two or more mapping rules; determining, by the one or more hardware processors in the two or more maps, one or more target nodes that comprise the first data set and at least one of the second data sets; determining, by the one or more hardware processors, one or more target distances in the one or more target nodes between the first data set and the at least one of the second data sets of a same target node based on the first location information and the second location information; determining, by the one or more hardware processors, a target second data set that corresponds to a minimum of the one or more target distances; and merging the first data set into the plurality of the second data sets after a validation of the first data set.
[0023] In various circumstances, the example embodiments described herein may resolve various challenges in determining the proximity of a first data set to very large second data sets and improving the processing delay and computational time/power in the determination.
[0024]
[0025] The system 100 may include a mapping module 130 that receives the data sets 112 and 114, the plurality of location bins, and two or more mapping rules. The mapping module 130 may use the two or more mapping rules to map the data sets 112 and 114 into the nodes of the location bins and generate two or more maps. The mapping module 130 is described in more detail with respect to
[0026] The system 100 may include an examination module 140 that receives the two or more maps and searches the two or maps to find and identify one or more target nodes in the two or more maps such that in each target node, the data set 112 and at least one data set 114 coincide in a same node. The examination module 140 is described in more detail with respect to
[0027] The system 100 may include a determination module 160 that is executed when at least one target node is found by the zero target node module 150. The determination module 160 that is described in more detail with respect to
[0028] In some examples, the zero target node module 150 may not find any target nodes in the two or more maps such that the data set 112 and at least one data set 114 may not coincide in a same node. The system 100 may further include a bin size updating module 180. If no target node is found, the bin size updating module 180 may adjust, e.g., increase, the location bin size of the segmentation module 120 and the described steps may be repeated. In some examples, the bin size updating module 180 may iteratively increase the location bin size until at least one target node is found in the two or more maps. In some embodiments, the location bin size may grow linearly or exponentially with each iteration.
[0029] In some examples, the location information may correspond to a physical location, e.g., a physical location in a two-dimensional location space. The location information of data set may include a longitude and/or latitude of the physical location and the longitude may vary from 180 degrees to +180 degrees and the latitude may vary from 90 degrees to +90 degrees. In some examples, the location information may be normalized into a real number between zero and a number less than or equal to 100, e.g., normalized between zero and 1.
[0030] In some examples, each one of the first data set or the second data set may also include additional information that can be used to determine other attributes of the data set. The additional information and/or the location information also includes one or more of a media access control address, a country name, a name of a state, a city name, a zip code, or a street address. In some examples, the system 100 may perform operations of a service provider, such as PayPal, Inc. of San Jose, Calif., USA.
[0031] In some embodiments, the bin size updating module 180 adjusts the location bin size based on attributes of the physical location and increases the location bin size in rural zones and decreases the location bin size in densely populated zones. In some examples, the location space may include both rural and densely populated zones and thus the bin size updating module 180 adjusts the location bin size differently in different zones of the location space. Note that operations performed by the modules as described herein may be combined into other modules or performed by one or more hardware processors.
[0032]
[0033] The segmentation module 120 may also include a location bin determination module 124 that receives the location space and the location information of the data sets 112 and 114 from the location space determination module 122 and determines a location bin size. In some examples, the location bin determination module 124 may use the location information of the plurality of data sets 114 to determine the location bin size such that in each one of the plurality of location bins, at least one data set 114 exists. The segmentation module 120 may also include a dividing module 126 that, based on the location bin size, divides the location space into the plurality of location bins. The location space and the location bins are described in more detail with respect to
[0034]
[0035] The mapping module 130 may include a location translation module 134 that receives the data sets 112 and 114, the location space, the plurality of location bins, and the mapping rules. The location translation module 134 may further modify, based on a mapping rule, the location information of the data sets 112 and 114 for each mapping rule. In some examples, modifying the location information of the data sets 112 and 114 results in translating the data sets 112 and 114 in the location space and may move the data sets 112 and 114 from one location bin to another location bin. In some examples, each map rule incorporates a different modification to the location information of the data sets 112 and 114 and may translate the data sets 112 and 114 differently in the location space. In some examples, at least one mapping rule may not modify the location information of the data sets 112 and 114.
[0036] The mapping module 130 may include a location quantization module 136 that receives the translated data sets 112 and 114, the location space, the plurality of location bins, and the mapping rules, and quantizes the location information of the data sets 112 and 114. In some examples, as noted above, the plurality of location bins include one or more nodes, and the quantization of the location information of the data sets 112 and 114 transfers the data sets 112 and 114 to the nodes of the plurality of location bins such that after the quantization the data sets 112 and 114 are mapped to the nodes of the plurality of location bins.
[0037] In some embodiments, generating the maps includes translating the first location information of the first data set in a predetermined direction by a predetermined amount prior to the quantizing and translating the second location information of the plurality of second data sets in the predetermined direction by the predetermined amount prior to the quantizing.
[0038] In some embodiments quantizing transfers a data point in a location bin to a node of the same location bin. In some examples, the quantization operators include one or more of a floor, a ceiling, or rounding.
[0039]
[0040] The determination module 160 may include distance determination module 164 that receives the location information of the data sets 112 and 114 in the target nodes and determines one or more target distances between data set 112 and each data set 114 in the target nodes based on the selected metric and using the location information, e.g., using the translated location information, of data sets 112 and 114. The distance determination module 164 then determines a minimum of the one or more target distances as the least distance between the first data set 112 and the plurality of second data sets 114. In some examples, the second data set 114 corresponding to minimum target distance is a target second data set.
[0041] In some examples, there is a tradeoff between increasing the bin size and the processing power used by the distance determination module 164. For example, although there may be a single first data set 112 in each target node, if the bin size is large there may be a plurality of the second data set 114 in each target node and thus increasing the processing power used by the distance determination module 164 to determine the target distances. The determination module 160 may include a validation module 166 that receives the target second data set as the closest point to the first data set and validates the first data set based on the additional information of the first data set and the additional information of the target second data set. In some examples, the first data set is a fraudulent data set and thus may not be validated. In some examples, the additional information of the first data set and the target second data set are Internet Protocol (IP) addresses. By examining the IP addresses, the validation module 166 may determine that the first data set is fraudulent. For example, a data set A may contain IP addresses and their geo coordinates (e.g., latitude and longitude) and a data set B may contain city-centers, in the format: state, country, city, region and geo-coordinates. A city center may be assigned for each IP address by finding the nearest neighbor over latitude and longitude.
[0042]
[0043] In some embodiments, as shown in system 200, one or more software modules that include a data access module 210, a segmentation module 220, a mapping module 230, an examination module 240, a zero target node module 250, a determination module 260, a database updating module 270, and a bin size updating module 280 that are consistent with the respective modules 110, 120, 130, 140, 150, 160, 170, and 180 of
[0044] In some examples, the data engine 222 may be implemented using hardware components, such as a processor, an application specific integrated circuit (ASIC), a programmable system-on-chip (SOC), a field-programmable gate array (FPGA), and/or programmable logic devices (PLDs), among other possibilities.
[0045]
[0046] In some examples, each one of the plurality of location bins 330 may include two or more nodes 340, and each two adjacent location bins from the plurality of location bins may connect and share one or more nodes from the two or more nodes.
[0047] As described, in some embodiments, the first data set and the plurality of second data sets include location information such that the location information may correspond to physical locations in a two-dimensional location space. In some examples, the physical locations that correspond to the first data set and the plurality of second data sets are included in the location space 310 of diagram 300 and the mesh 320 segments the location space 310 into a plurality of location bins 330 such that, based on their corresponding physical locations, the first data set the plurality of second data sets are distributed, e.g., mapped, into the location bins 330 of the location space 310. In some examples, the location bins may not be uniformly distributed in the location space 310. In some examples, in a first zone of the location space, the location bins are smaller than a second zone of the location space. In some examples, the location bins may have different shapes with different number of sides and nodes in different zones.
[0048] In some examples, x-direction 316 is a longitudinal direction and y-direction 318 is a latitude direction, the mesh is an arc mesh, and the first size 324 in x-direction 316 and the second size 322 in y-direction 318 are represented as angles. In some examples, the first size 324 and the second size 322 are represented as distances. The two by three bin section 360 of
[0049]
[0050]
[0051] In some embodiments, the location information of the data sets are normalized in a predetermined scheme and the location bins are defined in a predetermined arrangement such that quantization of the location information of the data sets maps the data sets to the nodes of the location bins. In some examples, the nodes of the location bins are distributed such that they coincide at values that may have one-to-one correspondence to integer values such that the nodes may be located at integer values.
[0052] In some examples as shown in table 470 of
[0053]
[0054] In some examples,
[0055]
[0056]
[0057] In some examples as shown in table 670 of
[0058] As described above, when zero target node module 150 determines that no target node is found, the bin size updating module 180 may adjust, e.g., increase, the location bin size of the segmentation module 120. In some examples the location bin size may be doubled. In some examples, the nodes may be kept the same even after changing the bin size and thus instead of increasing the bin size, the location information of the data sets are proportionally reduced. For examples, instead of doubling the bin size, the location information of the data sets are divided in half as shown in the tables of
[0059]
[0060]
[0061] In some examples as shown in table 770 of
[0062]
[0063] One or more of the processes 802-810 of method 800 may be implemented, at least in part, in the form of executable code stored on non-transient, tangible, machine readable media that when run by one or more processors (e.g., the processor 202 in system 200) may cause the one or more processors to perform one or more of the processes 802-810. In some embodiments, method 800 may be performed by one or more modules, such as modules 210, 220, 230, 240, 250, 260, 270, and 280 of
[0064] As shown in
[0065] At step 804, the method 800 may include segmenting the location space into a plurality of location bins. For example, referring back to
[0066] At step 806, the method 800 may include determining two or more mapping rules and generating two or more maps. For example, referring back to
[0067] In some examples, the two or more mapping rules may be selected, e.g., retrieved from memory 204, and the mapping module 130 may use the mapping rules to map the data sets 112 and 114 to the location bins. In some embodiments, the location quantization module 136 of the mapping module 130 maps the data sets 112 and 114 into the nodes of the location bins that are consistent with the nodes 340 of
[0068] At step 808, the method 800 may include determining one or more target nodes. For example, referring back to
[0069] At step 810, the method 800 may include determining a least distance and updating the data sets. For example, referring back to
[0070] In some examples, the additional information of the first data set and the second data sets include an Internet Protocol (IP) address. In some examples, a fraudulent IP address of the first data set may be detected based on a comparison of the IP address of the first data set with the IP address of the target second data set.
[0071] Thus, using embodiments described herein, computer processing power and/or time may be reduced.
[0072] The present disclosure, the accompanying figures, and the claims are not intended to limit the present disclosure to the example embodiments disclosed. As such, it is contemplated that various alternate embodiments and/or modifications to the embodiment disclosed, whether explicitly described or implied herein, are possible in light of the disclosure and/or the figures. Having thus described embodiments of the present disclosure, persons of ordinary skill in the art will recognize that changes may be made to the embodiments disclosed without departing from the scope of the present disclosure.