Determining usage predictions and detecting anomalous user activity through traffic patterns
09774460 ยท 2017-09-26
Assignee
Inventors
Cpc classification
H04L12/14
ELECTRICITY
International classification
H04M15/00
ELECTRICITY
Abstract
A system for selecting and monitoring data plans for telecommunications systems, and methods of determining, learning and estimating usage patterns in-order to predict usage and tie this to cost and the overlaying of plan selection for cost optimization. Further, additional methods are applied to determine anomalous user behaviors and raise security and data leakage related events.
Claims
1. A method of using a computer to detect data traffic of a mobile device, the computer performing the steps of: collecting usage data from the mobile device for a time period; setting threshold values based on a mobile device data plan and an amount of data used by the mobile device within the time period and on historical usage data of the mobile device; comparing the usage data against the threshold values; identifying an anomaly if a statistic of the usage data exceeds a corresponding threshold of the threshold values; generating a graphical user interface including at least one statistic of the usage data; overlaying at least one statistic of the existing usage data or predicted data or both onto the at least one statistic of the usage data; updating the threshold values based on at least one statistic of the usage data.
2. The method of claim 1, wherein at least one statistic is given a weighting based on the historical usage data and the weighting is related to an amount of stored history.
3. The method of claim 2, wherein the weighting afforded the historical usage data increases as the amount of stored history increases up to a maximum weighted percentage for the historical usage data.
4. The method of claim 1, wherein the time period corresponds to a data plan period.
5. The method of claim 1, wherein the collecting step comprises: reading, at a first interval, data traffic statistics from a network interface on the mobile device; adding the data traffic statistics to running totals on the mobile device.
6. The method of claim 5, wherein the collecting step further comprises: when the running totals exceed a delivery threshold, sending the running totals to the server; and adjusting the running totals.
7. The method of claim 1, wherein the collecting step further comprises: when a delivery timer expires, sending the running totals to the server; and adjusting the running totals.
8. The method of claim 1, wherein threshold values include a data limit for the given time period.
9. The method of claim 1, the computer further performing the steps of: extrapolating data traffic statistics to the end of the given time period; if a statistic of the extrapolated data traffic statistics exceeds a corresponding threshold value of the threshold values, flagging an anomaly.
10. A method of using a computer to detect data traffic of a mobile device, the computer performing the steps of: collecting from the mobile device, usage data relating to data usage for the mobile device for a given time period; setting threshold values based at least in part on a mobile device data plan and an amount of data used by the mobile device within a set time period; comparing the usage data against the threshold values; identifying a statistic of the usage data exceeding a corresponding threshold of the threshold values; generating report based on the identified statistic wherein the report includes a comparison between historical usage data associated with a second given time period to the usage data, the second given time period prior to the given time period; presenting the report on a Graphical User Interface (GUI) so that a user of the mobile device can assess changes in their data usage in order to avoid overages; and updating the threshold values based on a running average of the statistic of the usage data.
11. The method of claim 10 wherein the threshold values are further set based on historical usage data of the mobile device, where weighting afforded the historical usage data in comparison to the amount of data used by the mobile device within the set time period is related to the amount of stored history.
12. The method of claim 11, wherein the weighting afforded the historical usage data increases as the amount of stored history increases up to a maximum weighted percentage for the historical usage data.
13. The method of claim 10 wherein the identified statistic is average data usage for the mobile device over more than one set time period.
14. The method of claim 10, wherein the updating of the threshold values is based on formula I,
15. The method of claim 10, wherein the given time period corresponds to a data plan period.
16. The method of claim 10, wherein the collecting step comprises: reading, at a first interval, data traffic statistics from a network interface on the mobile device; adding the data traffic statistics to running totals on the mobile device.
17. The method of claim 16, wherein the collecting step further comprises: when a delivery timer expires, sending running totals of data usage to a server wherein the collecting step is performed at the server; and adjusting the running totals.
18. The method of claim 10, wherein the collecting step further comprises: when a delivery timer expires, sending the running totals to the server; and adjusting the running totals.
19. The method of claim 10 wherein threshold values include a data limit for the given time period.
20. The method of claim 10, the computer further performing the steps of: extrapolating data traffic statistics to the end of the given time period; if a statistic of the extrapolated data traffic statistics exceeds a corresponding threshold value of the threshold values, flagging an anomaly and presenting the anomaly on the Graphical User Interface (GUI).
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The present invention will be better understood after reading the following description of preferred embodiments thereof, made in reference to the appended drawings in which:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
DETAILED DESCRIPTION OF THE INVENTION
(9) The present invention relates to a system for creating, distributing, and monitoring telecommunications data plans. Further, it relates to a process for detecting anomalous user activity through traffic patterns based on data usage and patterns and relating this to security and cost data to determine optimal cost through learned behaviour of past history and usage trends. Additionally, it relates to the determination of anomalies as possible security threats while optimizing costs, minimizing the overhead required to provide such data, and minimizing the volume of data created through this type of monitoring.
(10) The invention, broadly stated, comprises a configuration method and graphical user interface for configuring carrier plans and learned traffic usage related to plans and anomalies derived from this data.
(11) Since telecommunication carriers are competing with complex plans that allow the automatic upgrades based on usage, or the pooling of data across multiple users, the present invention's ability to generate pool plan remainders and to predict overages and calculate next upgrade plan costs into the expense calculations is essential.
(12) The aforementioned link between the plans and usage data is also tied to the predictive capabilities of the current invention in that these same algorithms and techniques/methods are used to determine optimal plans for the users.
(13) The Agent portion of the Invention, depicted in
(14) (1) Initialization: a. The Agent determines, through publicly available system calls and device specific SDKs the available interfaces on the system which are to be monitored for data traffic and usage patterns. This is typically available through an interface table from most operating systems but requires device specific knowledge and proprietary methods for some devices. b. The available interfaces are matched for the availability of a plan, which consists of user, or server specified parameters stores in non-volatile storage on the Agent. c. The establishment of a server communications channel (using any available communication method, although a TCP socket will be used for illustration) and report active status to the server. d. Determine through plans the accessibility and visibility of GUI options for the agent users. e. Start two timers which will loop with the following functionality
(15) (2) Timer (Fast). This timer, runs a quick interval, 1 second will be used for illustration, although this timer can be much faster on capable hardware platforms, and can be much slower, although slowing this timer will affect real time accuracy of data stored on the server. a. On Each Active Interface, this timer will read statistics from the interface using interface specific means (snmp statistics, RAS counters, device specific sent/received bytes, or packet counts) b. Update and add these raw counts to a running total that is maintained for the plan period that survives resets and device additions/removals c. Update and Refresh GUI values/displays d. Check for user specific inputs and configuration changes or requested actions.
(16) (3) Timer (Slow). This timer runs at a slower pace, generally two minutes, although this time can also be configurable. a. Refresh existing interface lists to monitor for mobile devices which may have been inserted or removed. b. Check running counters, updated with the first timer for warning threshold crossings and display/update appropriate warnings. c. Check threshold settings for server communication updates. Since the server can specify a time (or data size) based transfer rate, intermediate counters are kept to monitor when the server should be send updates with this timer. i. Should these thresholds be crossed, the agent will establish the communication link with the server ii. Send the updates from each active interface which requires sending refreshed values to the server iii. Receive (and update) any server plan updates, messages, pooled plan remainder values, or flex plan update/upgrade plans. d. Check for plan period expires and refresh/reset the counters as necessary when plans are updated or expire (resetting warnings appropriately).
(17) The Server Portion of the invention provides a number of functions which run asynchronously and in separate threads, tasks and applications. The state machine for the server portion is illustrated in
(18) (1) The primary data gathering function of the server is performed in a system service but, must be an always on service component which automatically re-establishes itself after power outages or maintenance. This system performs three major tasks. a. It monitors the communications channel(s) for incoming updates from the remote agents, and updates the contact data, the running totals. These functions are performed in an event based fashion when agent data is received on the monitored channel. b. Once data is received, and the running totals are updated, the new running totals, input events are stored in a local database and triggers cause the new data to run through an anomaly detection systems which will look for predefined (or learned) trends and thresholds which may trigger the creation of anomaly alarms. The anomaly determination methods and the definition of the anomalies themselves form a large part of this invention. i. Anomalous usage thresholds are predefined with set limits for a given time period (per plan). When these thresholds are crossed, anomaly alarms are created for large transfer activities. ii. Office hours are validated against preset values in the plan and off hours anomalies are triggered when outside of this range. iii. Projected usage is calculated based on historical averages (using weights equivalent to the amount of stored history) and the current monthly trend, and if projected overages are calculated to occur, this anomaly alarm is raised. iv. Office/Non-Office traffic is determined through discovery of local office resources on the agent, and reported with the data. When non-office thresholds are exceeded, this anomaly alarm is raised v. Anomalies are fed into an email distribution system, stored in a database (or other non-volatile memory). c. Events queued up for the Agent through the Anomaly Server Console application, described below, are then transmitted using the same communications channel. These can include updated plans, messages, pool plan remainder updates, flex plan updates, etc.
(19) (2) The Server Administrative Console provides the user interface to the administrator to administer the remote agents, associated plans, and the devices and users in the system. While all the physical data is transferred between the system service (described above), the administrator queues actions and affects changes to locally stored configurations (plans, reports, user/device parameters) using this interface. a. Plan Creation: Plans, which are a collection of data which includes cost, limits, and time related parameters which establish the data plan with the carrier for a user, a device, or a group of users/devices, are created using a GUI. b. User Group Creation, which are collections of users who share a plan, or are reported in a shared function can be creates and administered c. Plan Distribution: Plans can be queued up for distribution using the service component above using this GUI to users or groups. d. Reporting: Reports graphically or textually showing usage patterns, cost projections, and historical data are generated using this GUI.
(20) Unique features of the claim are the filtering and presentation of the collected data into graphical and concise charts which will show the operator in a single view, problem areas extracted from potentially millions of data points. The sorting of projected usage by overage and cost estimation draws attention immediately to the most critical data.
(21) An algorithm to learn and adjust anomaly detection of usage patterns with respect to transfer sizes with dynamic thresholds used for abnormal transfers is depicted below, in which the variables are defined as follows.
(22) T.sub.0: The Original transfer size criteria, whereby a max value is set to specify an anomaly threshold in a given transfer period. This is initially set to a predetermined threshold and varies according to the formula below. (i.e. if a transfer larger than T.sub.0 occurs, it is flagged to be an anomaly)
(23) T1-T10: Are the actual max readings which may or may not have exceeded this setting in a given plan period. If less that 10, (which is an arbitrary number of readings which may be adjusted to provide greater or less hysteresis causing the learning algorithm to adjust closer to peak values or closer to average values) reading have occurred, the remaining readings are set to be equal to T0. We will refer to this arbitrary number 10 as Tn1 in further descriptions.
(24) Pn: Is the number of plan periods recorded. The formula below uses a max of 12 periods (which again is an arbitrary number of periods which may be adjusted to provide greater or less memory of history causing the learning behavior to learn faster or slower). We will refer to this arbitrary number 12 as Tn2 in further descriptions.
(25) T.sub.0/10: This value is provided as a Max value in the formula to never allow more that a 10% adjustment in a given plan period. Again, this number can be tuned in the algorithm to allow for a faster or slower learning adjustment rate. This 10% value will be referred to as Tn3 in further sections.
(26) Max [T1 . . . Tz] causes the subtraction of the largest of the anomalous transfers from the plan period again to avoid skewing average values by the largest of the discrepancies to average or typical activities. The Algorithm can be adapted to allow for more than one item to be removed, or to have all items greater than a given percentage removed.
T.sub.0=T.sub.0+/Max(T.sub.0/10,T.sub.0{T1/z . . . Tz/zMax[T1 . . . TZ]}/Max[12,Pn])
(27) To is adjusted for the next plan period dynamically for the number of periods on record or past year (12 intervals shown above for illustration, but this can be varied to have a variable hysteresis for memory and variability as described in the parameters above).
(28) An algorithm to learn and adjust anomaly detection of usage patterns with respect to time through the hours of usage is also used. Abnormal usage is detected through a variance of initial ranges of times set as Tmino . . . Tmaxo but with a fixed maximum amount of variability to prevent large shifts based on off hours work anomalies (in the example below, a cap of 1 hour variance per plan period is used, however this amount can be tuned as needed to allow for greater or less learned variability per plan period as needed.
(29) Tmin: The acceptable minimum range of starting work, (i.e. weekdays at 8 am) before which an OFF-HOURS anomaly will be raised.
(30) Tmax: The acceptable minimum range of ending work, (i.e. weekdays at 6 pm) after which an OFF-HOURS anomaly will be raised.
(31) Tmino: The original setting for normal start time (i.e. 8 am), set by the user before learning begins.
(32) Tmaxo: The original setting for normal stop time (i.e. 6 pm), set by the user before learning begins.
(33) Tmin1 . . . N: The N earliest snapshot readings from the days in the plan period. These may be earlier or later than the Tmino value. N can be set to 0 to never change, or the number of working days in the plan period to calculate the overall average of ALL activity.
(34) Tmax . . . N: The N latest snapshot readings from the days in the plan period. These may be earlier or later than the Tmino value. N can be set to 0 to never change, or the number of working days in the plan period to calculate the overall average of ALL activity.
Example 1
(35) After a 31 day plan period with an initial configuration of Tmino=8 am, and N=5 (to take the 5 earliest periods), the system captures the earliest activity periods as (7:01, 7:10, 7:11, 7:13, 7:15) giving variance values from the Tmino value of (59, 50, 49, 47, 45 minutes respectively, and an average variance of 50 minutes. In this case: The Tmin value for the next plan period will be 7:10 instead of 8:00 am (in this example, each of the out of range, or before 8 am events will trigger an anomaly)
Example 2
(36) After a 30 day plan period with an initial configuration of Tmin0=9 am, and N=30 the system reads starting times between 9:00 and 9:30 every day with an average (of the 30 readings) being +15 Minutes Variance. The Tmin value for the next plan period will start at 9:15. (in this example, since each of the Tmin1 . . . 30 events are after the Tmino setting of 9:00, there will NOT be any anomalies triggered.
(37) Note a separate method for weekend (Sat/Sun work is performed) whereby abnormal weekend work does not skew working times during the week.
(38) Further, the Min/Max algorithm can also be used for both startMin and StartMax and StopMin/StopMax values as a variant. In this case, starting work either before startMin or after startMax will be considered an anomaly and the times will adjust using the same learning algorithm.
Tmin=Tmino+/Max(1,{Avg(Tmin1 . . . TminN)})
Tmax=Tmaxo+/Max(1,{Avg(Tmax1 . . . TmaxN)})
(39) Further learning of repetitive atypical events is achieved through the storage of a history of top anomalous events using the same formula and allowing for exceptions in normal anomaly reporting. This is best described with an example.
(40) Assuming working hours are determined to be 9 to 5, the system will detect an anomaly when a user works until 6 pm. However, if after N plan periods, it is detected that the user works later every Friday, (or every last Friday of the month) say, until 9, these events will be matched against known anomalies and no longer reported as such.
(41) Unique features of the invention which are claimed in the patent include the algorithm to provide historical user data with minimal overhead.
(42) This algorithm is based on the parameter configuration of the transfer size (TSZ) and the time limit (TLR) for reporting granularity with last send stamp of (LSS) and CurTime being the current time and MinTLR being the don't send faster than this value
TimeToSend=((LSSCurTime>TLR)(LSSCurtime>MinTLR && TSZ>ADS))
(43) It is time to send data to the server only if (the time limit TLR is passed, or more data has been sent since the last time server reporting was done than the TSZ desired notification size (and at least the minimal time between reporting periods MinTLR has been exhausted)
(44) In order to capture the relevant and timely data to be sent to the centralized server system, the following data must be captured or derived from what is available from the device deploying the communications services:
(45) Machine Name: The PC, Device, or Modem/Card/Phone on which data is collected. To allow for shared devices between users (i.e. a shared laptop computer).
(46) User name: The Logged in User, if applicable to allow for users that use multiple devices, or log into multiple machines. This allows for reporting by user.
(47) Actual Up: The actual current up reading, although derivations such as delta up, or plan up, or incremental up would also be construed as being applicable and within the scope of the claims.
(48) Actual Down: The actual current down reading, although derivations such as delta up, or plan up, or incremental up would also be construed as being applicable and within the scope of the claims.
(49) Office: A determination of whether or not the data being send was of a personal or business nature. Multiple means of detecting whether or not we are in an office environment exists, and any of these methods would be within the scope of the claim.
(50) Plan: The Plan being used on the device at the remote Agent. When roaming, or when using a temporary overlay plan when traveling would also be reported here.
(51) Alarm Status: An alarm status, equivalent to the predetermined alarm states as set on the server (Yellow, Red. Limit), however additional alarm states or a subset of alarm states would also be within the claim.
(52) Device ID: The unique device identifier (phone number, ESN, Mac Address) would be reported here, This also allows for shared phones or data modems across a number of user ids and machines ids.
(53) Additional parameters can be captured for reporting capabilities depending on configuration and include flags available from the device such as
(54) Roaming: When a user is outside of his/her home network, using devices on partner networks, or roaming, has a distinct, usually higher, cost associated with it.
(55) Other parameters could be user role, interface used and application used.
(56) Unique features of the claim are based on the ability of the system to determine the optimal plan based on the current history and anomalies detected and stored and the available plans with associated parameters and costs.
(57) PlanOptimal=Min Cost . . . applying each plan stored in the system with the historical and current data stored for the user and device
(58) Unique features of the claim are based on algorithms and method of determining projected overages based on historical averages and current usage trends within a plan period where P is the plan period, T1 is the current number of used days in the plan, and T2 is the remaining days. U1 is the usage average per same plan period collected historically over time, and U2 is the current usage so far in the plan.
Projected Use=U2+(U1/P*T2)
(59) While all of the algorithms described above have applications for the field of invention narrowly defined as plan management and anomaly detection based on usage data, these algorithms can also be used in other related fields of activity such as network planning, network build out and design, etc.
(60) The algorithms and methods described above have been for illustration purposes only, and are not intended to limit the scope of the appended claims.
(61) Although the present invention has been explained hereinabove by way of a preferred embodiment thereof, it should be pointed out that any modifications to this preferred embodiment within the scope of the appended claims is not deemed to alter or change the nature and scope of the present invention.