A Method And Unit For Adaptive Creation Of Network Traffic Filtering Rules On A Network Device That Autonomously Detects Anomalies And Automatically Mitigates Volumetric (DDOS) Attacks

Abstract

The subject of the invention is the method of adaptive creating network traffic filtering rules on a network device that autonomously detects anomalies and adaptively mitigates volumetric (DDoS) attacks on at least one network device (4) based on actual network flows (3) and after separating them into isolated packet flows (9), recognizes potentially harmful network flows, and then configures or tunes the network filters (19) and packet policing means (17), wherein filtering rules (18) can be propagated to other network devices (27) and selects for further analysis the isolated packet flows (9) associated with at least one configured or tuned network filter (19).

Claims

1. The method of adaptive creating network traffic filtering rules on a network device that autonomously detects anomalies and mitigates volumetric (DDoS) attacks on at least one network device (4) based on actual network flows (3) and after separating them into isolated packet flows (9), recognizes potentially harmful network flows, and then adaptively configures or tunes the network filters (19) and packet policing means (17), wherein filtering rules (18) can be propagated to other network devices (27), and selects for further analysis the isolated packet flows (9) associated with at least one configured or tuned network filter (19), characterized in that within given sample periods (o1) cyclically: a) based on data from the input (1) and output (2) interfaces, packets from actual network flows (3) are sampled on at least one network device (4) using at least one network traffic sampling unit (5), and packet samples (a1) are stored in the sample buffer (6), b) based on packet samples (a1) from the sample buffer (6), using the D0 detector (7) the definitions (8) of the network filters that isolate the packet flows (9) are set and stored in memory (10), c) for each isolated packet flow (9), threshold values (12) are defined in the flow observer unit (11) and stored in memory (10), d) for each isolated packet flow (9), the predicted flow characteristics (14) are determined on the set prediction horizon in the automatic flow controller unit (13) or in the flow observer unit (11) and stored in memory (10), wherein each new isolated packet flow (9) is predicted using default set values, e) for each isolated packet flow (9), it is determined in the flow observer unit (11) whether the previous threshold values (121), computed in the previous sampling period (o1) have been exceeded, by comparing the parameters of the isolated packet flow (9) with the previous threshold values (121) stored in memory (10) and if the previous threshold values (121) have been exceeded, alarm signals (15) are generated and stored in memory (10), f) the actual network flows (3) that correspond with the isolated packet flows (9) that do not meet the threshold values (12) undergo a process of adaptive control and dynamics shaping by the automatic flow control unit (13) as well as by packet policing means (17) and network filters (19).

2. Method of claim 1 characterized in that at step b), packet flows (9) are isolated from any of the available attributes (x1) of packet samples (a1) stored in the sample buffer (6).

3. Method of claim 1 or 2 characterized in that at step f), based on packet samples from the input interface (1) and output interface (2) taken from the sample buffer (6), on threshold values (12), on alarm signals (15), on predicted flow characteristics (14), on definition (8) of network filters as well as on current control signals (16), new control signals (161) for packet policing means are determined by the adaptive control method and stored in memory (10),

4. Method of any of claims 1-3 characterized in that step f) is followed by step g), wherein for isolated packet flows (9) processed in the automatic flow control unit (13) based on the control signals (16), filter rules (18) are created that are: performed on at least one network device (4) using network filters (19) or packet policing means (17).

5. Method of claim 4 characterized in that at step g), the automatic flow control unit (13) propagates the created filter rules (18) to other network devices (27).

6. Method of any of claims 1-5 characterized in that the D0 detector (7) at step b) defines the network filters, through: determination of at least one order list (20) corresponding to the selected attribute (x1) of packets in the set of packet samples available in the sample buffer (6) in descending order due to the selected flow parameters that constitute the set evaluation criteria, for each attribute (x1) of the packet, the summary list (21) is determined by aggregating the ordered lists and stored in the detector memory (22), for each attribute (x1) of a packet placed in the summary lists (21), a base term (23) is created and stored in memory (10), the activity counter (24) is increased for each base term (23) stored in memory (10), the activity counter (24) value is checked in memory (10), and if at least one counter exceeds the set activity value (25), a definition (8) of the network filter is created that corresponds to the base term (23), the counter of which exceeded the activity value (25),

7. Method of any of claims 1-6 characterized in that the D0 detector is run cyclically at the set sampling intervals (o1).

8. Method of any of claims 1-7 characterized in that the given evaluation criteria are the number of bytes, the number of packets, the number of bits per second (bps), the number of packets per second (pps), the number of flows or the average number of bytes in the packet (bpp) recorded every set sampling period (o1).

9. Method of any of claims 1-3 characterized in that: at steps c), d), e), f), either an observer unit (11) or an automatic flow control unit (13) creates a time series (26) that corresponds to the isolated packet flows (9) by the definitions (8) of network filters stored in memory (10) that describe changes in attributes (x1) of packets over time, wherein each time series (26) contains information about changes to any available attribute (x1) of packet samples (a1) stored in the packet buffer (6) for a set sampling period (o1) and stored in memory (10) and used to determine flow characteristics (14) as well as used at steps c), d), e) and f).

10. Method of any of claims 1-9 characterized in that step b) is followed by step b1) in which the packet flow (9) isolated by the D0 detector (7) is divided by the first order D1 detector (28) into packet subflows (29) so that: for each isolated packet flow (9), the corresponding time series (26) are fetched from memory (10), the time series separated by the set characteristics are selected and the combination of separated time series corresponding to the packet flows (9) isolated in the D0 detector (7) is determined, based on the combination of extracted time series, the composite terms (32) composed of base terms (23) are determined and the corresponding packet subflows (29) are searched for in the packet samples (a1) stored in the sample buffer (6); if the packet subflow (29) described by the composite term (32) exists in the packet samples (a1) stored in the buffer (6), a new network filter definition (8) is created that isolates the identified packet flow (9), and is stored in memory (10).

11. Method of any of claims 1-10 characterized in that at least two time series (26) separated from each other are selected either because of maximum linear (orthogonal) independence or because of minimal correlation (stochastic independence) in the signal space in at least two time series (26) in the time domain or frequency domain.

12. Method of any of claims 1-11 characterized in that step b1) is followed by step b2), wherein isolated packet flows (9) are split, using a higher order Di detector (30), by a combination of lower order network filter definitions (8), creating new definitions (8) of higher order network filters that isolate further packet subflows (29): for each isolated packet flow (9), the corresponding time series (26) are fetched from memory (10), time series (26) are selected which are correlated or similar according to another specified criterion, a new composite term (32) is built from the product of the terms building definitions (8) of network filters isolating packet flows (9), which are matched by detected correlated time series or similar time series, and then flows corresponding to the new composite term (32) are searched for in packet samples (a1) stored in the sample buffer (6), if the subflow (29) described by composite term (32) exists in the collected packet samples (a1), a network filter definition (8) that isolates the identified packet flow (9) is created and stored in memory (10).

13. Method of any of claims 1-12 characterized in that at least two time series (26) are qualified as similar because of their correlation with each other or because they are similar in their waveform.

14. Method of any of claims 4-13, characterized in that at step g) the according mitigation of the network packet flow (3) isolated by the network filter (19) performed by the packet policing means (17) includes the rejection of the packet.

15. Method of any of claims 4-14, characterized in that at step g) according mitigation of the network packet flow (3) isolated by the network filter (19) performed by the packet policing means (17) includes packet buffering.

16. Method of any of claims 1-11, characterized in that the individual steps of the process are performed on different network devices or using at least one computer connected to the network via an according network interface.

17. Method characterized in that the flow state (s1) is strictly defined by the values of the network filters stored in memory (10) of the definition (8) and based on the current control signals (16), the values of the alarm signals (15) or their absence, the threshold values (12) and the predicted flow characteristics (14) and is controlled by an observer unit (11), starting with the isolated state (sw) of the flow by means of an appropriate detector, through the monitored state (so) of the flow, operated by the observer unit (11), to the controlled state (sk) of the flow, operated by the automatic flow control unit (13), or to the expiry state (se) of the flow, handled in the flow observation unit (11), a specific value of the filtration rules (18) implemented on the network device by the network filters (19) and appropriate packet policing means (17) shall apply to each of the listed flow states (s1).

18. Adaptive network traffic filtering rules creating unit for one or more network device that autonomously detects anomalies and automatically mitigates volumetric attacks (DDoS), containing a memory, processor or controller and at least one network interface characterized in that: it contains a network traffic sampling unit for the network input (1) interface and network output (2) interface, which performs the function of reading and collecting packet samples (a1) of actual network flow (3) and recording them in a dedicated sample buffer (6) connected with: a detector unit (D0) (7) that isolates packet flows (9) by reading packet samples (a1) stored in the sample buffer (6) and generating definitions (8) of network filters and storing them in memory (10), wherein the memory (10) is connected to the observer unit (11), which reads from memory (10) information about the definitions (8) of network filters and about the corresponding packet samples (a1) stored in the sample buffer (6) and then sets the threshold values (12) and stores them in memory (10), and determines whether the determined threshold values (12) have been exceeded by comparing the parameters of the isolated packet flow (9) with the previous threshold values (121) stored in memory (10) and if the previous threshold values (121) have been exceeded, alarm signals (15) are generated and stored in memory (10); in addition, the observer unit (11) predicts flow characteristics (14) on the set prediction horizon and stores them in memory (10), at the same time, memory (10) is connected to an automatic flow control unit (13) that controls and shapes the dynamics of the actual network flows (3) that correspond to isolated packet flows (9) that do not meet the set thresholds (12), controlling the packet policing means (17) and network filters (19), simultaneously, the automatic flow control unit (13) is connected to at least one other network device (27) to which filtering rules (18) are transmitted, created based on the definitions (8) of network filters and definitions of control signals (16) for packet policing means, read from memory (10), wherein: definitions (8) of network filters and control signals (16) of packet policing means are translated into packet filtering rules (18) and transmitted to packet policing means (17) and network filters (19), thus isolating packet flows of network traffic (3) and shaping their dynamics in the protected area of the network.

19. Unit of claim 18, characterized in that the automatic flow control unit (13) reads from memory (10) alarm signals (15), (predicted and observed) flow characteristics (14), definitions (8) of network filters, current control signals (16) of packet policing means and fetches threshold values (12), wherein automatic flow control unit (13) is also connected to the sample buffer (6) from which it reads the packet samples (a1) from the input interface (1) and the output interface (2) and adapts the new control signals (16) to the packet policing means for the corresponding isolated packet flows (9) and stores them in memory (10).

20. Unit of any of the claims 18-19 characterized in that between the sample buffer (6) and the flow observer unit (11), there is a first order D1 detector unit (28), parallelly connected with D0 detector (7), that is connected to memory (10).

21. Unit of any of the claims 18-20 characterized in that between the sample buffer (6) and the flow observer unit (11) there is a higher order Di detector unit (30), parallelly connected with D0 detector (7), that is connected to memory (10).

22. Unit of any of the claims 18-21 characterized in that between the sample buffer (6) and the flow observer unit (11) there is a first order D1 detector unit (28), serially connected with D0 detector (7), that is connected to memory (10).

23. Unit of any of the claims 18-22 characterized in that between the sample buffer (6) and the flow observer unit (11) there is a higher order Di detector unit (30), serially connected with a first order D1 detector unit (28), that is connected to memory (10).

24. Unit of any of the claims 18-23 characterized in that the observer unit (11), based on the definitions (8) of network filters, alarm signals (15) and set threshold values (12), run a first order D1 detector (28) or higher order Di (30) detector that isolates new flows (29) and stores new definitions (8) of network filters in memory (10).

25. Unit of any of the claims 18-24 characterized in the fact that the observer unit (11), based on the definitions (8) of network filters, alarm signals (15) and threshold values (12), is configured to run the automatic flow control unit (13), which performs the function of control and shaping the dynamics of the isolated packet flow (9) indicated by the observer unit (11).

26. Unit of claim 12 characterized in that the automatic flow control unit (13) is configured to communicate with other network devices (27).

27. Unit of claim 12 characterized in that the individual elements of an integrated unit may be implemented as application specific integrated circuits (ASICs) or as modules of a single integrated unit or as field-programmable gate arrays (FPGAs) or as a computer comprising at least a processor, memory, mass memory and associated network interface and connected to at least one networked device.

28. Computer program product with program coding means which are written on a computer readable medium, for the implementation of a method according to one of claims 1 to 12 when the computer program product is executed on a computer or networked device.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0141] The object of the invention in one embodiment is shown in a series of figures, wherein:

[0142] FIG. 1 shows a diagram of an exemplary installation of an adaptive flow controller unit—an embodiment of a unit implementing a link protection method using an adaptive control system,

[0143] FIG. 2 shows the embodiment of a system,

[0144] FIG. 3 shows the method defined by the diagram of states of observed flows,

[0145] FIG. 4 shows the system using a properly configured database and virtualized services, i.e. independent of the protected network area and/or network device,

[0146] FIG. 5 shows a layout of dirty-clean interfaces with a pair of internal interfaces performing tasks of shaping flow dynamics using packet policing means,

[0147] FIG. 6 shows the concept of a mechanism for increasing anomaly detection accuracy,

[0148] FIG. 7 shows an exemplary implementation of iterative mechanism for increasing anomaly detection accuracy,

[0149] FIG. 8 shows an exemplary implementation of mechanism for isolation and policing flows.

DETAILED DESCRIPTION OF THE INVENTION

[0150] Method autonomously detects actual network flows 3 that pose a potential threat to the protected network on at least one network device 4, isolates packet flows 9 using a set of autonomously generated and distributed definitions 8 of network filters. The network filter definitions 8 are used to configure the network filters 19 on at least one network device and can be distributed to other network devices 27 using dedicated control protocols such as BGP (Border Gateway Protocol). The system generates definitions 8 of network filters and configures network filters 19 with increasing resolution over time, identifies signal dynamics models of representation of observed packet flows, and based on predicted flow characteristics, 14, i.e. identified dynamics models, generates and adapts control signals 16 shaping the dynamics of isolated flows using packet policing means 17, and calculates safe shares of isolated flows in the protected network bandwidth by setting thresholds 12. Control signals 16 shaping the dynamics of isolated flows and definitions 8 of network filters isolating network flows 9 constitute together the filtration rules 18. The diagram of method is shown in FIG. 2.

[0151] According to one embodiment, method starts with sampling the packets. Based on data from input interface 1 and output interface 2, packets from actual network flows 3 are sampled. Traffic data comes from a sampling unit 5 of network device 4, but may also come from multiple devices and be transmitted to a selected network device with a unit according to the invention or a computer with a network interface adapted to perform the various steps of the process according to the invention. At least one sampling unit 5 of network traffic on at least one network device 4 records packet samples a1 in sample buffer 6.

[0152] The range of packet sampling data stored in sample buffer 6 depends on the packet sampling standards used (e.g. sFlow or netFlow). Any of the available x1 attributes can be applied to isolate packets, e.g. t-start, t-end, duration, src address, dst address, src port, dst port, protocol, flag, src as, dst as, tos, mac address, packet samples a1 stored in the sample buffer 6.

[0153] Based on packet a1 samples from sample buffer 6, the detector D0 7 determines the definition 8 of network filters isolating packet flows 9, which are stored in memory 10. The D0 detector isolates from the observed set of packet a1 samples of network traffic flows distinguished by the given traffic statistics, e.g. bps, pps, flows, bpp. Isolated packet flows 9 are described by the definition of network filter 8, which in the case of detector D0 7, distinguishes one single packet attribute x1, e.g. IP address or protocol type.

[0154] For each isolated packet flow 9 through any detector D0, D1, Di, a process is created in the automatic flow control unit 13. The automatic flow control unit 13 is the controller responsible for the control process whose task is to shape the dynamics of actual network flows 3 between the input and output interfaces (in the dirty-clean unit) by updating the control signals 16, and, consequently, updating the filter rules 18. The diagram of the flow insulation unit, in which, on the one hand, the result of the detectors' operation provides the definition 8 of network filters, and, on the other hand, the automatic flow control unit is used to adjust control signals 16, is showed in FIG. 8. The operating diagram of the automatic flow control unit is showed in FIG. 1.

[0155] According to one embodiment, in each iteration of the adaptive control process performed, the model of flow dynamics is identified every of sampling period. The sampling period can be from 0.1 s to 15 s, preferably 5 seconds in this example.

[0156] In the flow observer unit 11, for each isolated packet flow 9, threshold values 12 are determined and stored in memory 10. In the flow observer unit 11 or in the automatic flow control unit 13, predicted flow characteristics 14 are determined within a given prediction horizon. Designated flow features are then stored in memory 10, while each new isolated packet flow 9 is predicted by default set values.

[0157] If the previous thresholds 121 determined in the previous sampling period of have been exceeded, after comparing the parameters of the isolated packet flow 9 with the previous thresholds 121 stored in memory 10, the alarm signals 15 stored in memory 10 are generated.

[0158] Then, for the identified dynamic model, i.e. the predicted flow characteristics 14 of isolated packet flows 9, a synthesis of algorithms for determining control signals 16 and algorithms for predicting flow characteristics 14, e.g. network traffic intensity, is carried out. The appropriate configuration of the network device 4 allowing for control (mitigation) is designed autonomously, taking into account the specification set by the unit administrator. Determination of control signals 16 is a pattern for shaping the dynamics of packet flow and defines the basic parameters of the process of attenuation of anomalies or attacks with the use of packet policing means 17. FIG. 1 shows a block diagram of the solution according to the invention.

[0159] Actual network flows 3, which correspond to isolated packet flows 9, which do not meet the threshold 12, i.e. generate alarm signals, are autonomously adjusted and have dynamics shaped using an automatic flow control unit 13 and packets policing means 17 and network filters 19.

[0160] The automatic flow control unit 13 contains control mechanisms that cyclically, every set sampling period o1 of network traffic, determine successive values of control signals controlling for packet policing means 17, i.e. parameters of a policer or shaper (e.g. token-bucket algorithms), based on the predicted and observed responses of packet sources to imposed rate limits.

[0161] The purpose of the disclosed control process is to shape the flow dynamics in accordance with the specifications set by the system operator, e.g. attenuation of components with a given signal band associated with a selected packet flow. The solution allows for prediction and early mitigation of DDoS attacks, correcting the work of packets policing means 17 or otherwise mechanisms of sending packets of network devices (policers and shapers), as well as conducting advanced diagnostics of network traffic dynamics based on the analysis of predicted flow characteristics and dynamics models obtained based on them. Controlling the work of the packets policing means 17 (police/shapers) is a key element in the process of mitigating an attack. It allows obtaining the required speed and accuracy of attenuation, as well as shaping the (time and frequency) characteristics of the flow according to the accepted assumptions, defined by the operator.

[0162] The adaptive regulator in the form of an automatic flow control unit, by selecting control signals 16 for packets policing means 17, a policer or a shaper, for example, parameters of token-bucket algorithms, is designed to protect the link covered by the unit, e.g. maintaining the flow speeds at safe levels. The task of determining safe levels (upper limits) of the flow rate is performed autonomously by mechanism of reckoning the set threshold values 12, which are then used to perform the control. Threshold values 12 are computed in the flow observer unit 11 and in the automatic flow control unit 13 based on the solution of the non-linear optimization task, which determines the fair distribution of resources in the protected network. The resulting upper limits on the speed of packet flows ensure that the protected link is not saturated and its utility is maximized from the perspective of each flow. They are also used in the process of automatic flow control. Exceeding the flow rate upper limit triggers alarm signals 15 that affect the unit's decision to subject the flow to automatic dynamics shaping in the automatic flow control unit 13.

[0163] The automatic flow control unit taking into account packet samples from input interface 1 and output interface 2 taken from sample buffer 6, threshold values 12, alarm signals 15, predicted flow characteristics 14, definition 8 of network filters and current control signals 16, determines by adaptive control methods, new control signals 161 for packet policing means that are stored in the memory 10.

[0164] Selected packet samples from input 1 and output 2 taken from sample buffer 6 corresponding to isolated packet flows 9, threshold values 12, alarm signals 15, predicted flow characteristics 14, definitions network filters 8 and current control signals 16 are also stored in memory 10. Memory 10 can, in particular, be organized as a flow (time series) database. The automatic flow control unit uses a flow database containing data on medium term distribution of traffic on the link. Thanks to the applied solution, the network administrator does not have to calculate independently the orders limiting the speed of each monitored flow, the invented unit performs this task autonomously while working, and access to data is provided by a database.

[0165] The automatic flow control unit 13 is responsible, in particular, for creating a hardware abstraction layer and provides translation functions for logical formulas in the form of control signals 16 and the definition 8 of network filters. The abstraction layer mediates communication between master control layer and the network infrastructure layer of the unit. It is responsible for mapping the state of network switches and other network devices in the control layer and for transmitting control signals between layers. The automatic flow control unit 13 is also responsible for converting filter rules expressed in the language of reading traffic samples (e.g. pcap-filter) into filter rules for the network device operating system. The applied solution allows for automatic configuration of filtration tools and flow shaping tools on network devices (e.g. network client), as well as for implementation of advanced network control concepts using machine learning tools and artificial intelligence (SI). The automatic flow control unit 13 can also propagate the created filter rules 18 to other network devices.

[0166] For isolated packet flows 9 processed in the automatic flow control unit 13, control signals 16 are used to create filter rules 18, which are executed on at least one network device 4 using network filters 19 or packet policing means 17. By executing on a network device 4, it is understood that the control signals 16 and network filters 8 definitions are used to appropriately configure the available packet policing means 17 and network filters 19 so that the actual network flows 3 are filtered and dynamically modelled. What is more, automatic flow control unit 13 can propagate filter rules 27 to other network devices, e.g. using the BCP protocol.

[0167] The invention according to one embodiment registers and stores the characteristics of monitored network flows, such as definitions 8 of network filters, threshold values 12, predicted flow characteristics 14, alarm signals 15, control signals 16, in the database of time series 26 supported by mechanisms invented for this purpose to manage the state of network flows of the flow observer unit 11.

[0168] Individual units responsible for utilizing the invention may be implemented as application specific integrated circuits (ASICs) or as modules of a single integrated unit or as field-programmable gate arrays (FPGAs) or as a computer, consisting at least of a processor, memory, mass memory and appropriate network interface connected to at least one networked device. In particular, individual units may be software modules performing a function on a suitably configured networked device or computer, consisting of at least a processor, memory, storage and an appropriate network interface connected to at least one networked device, as a computer program product with program coding means which are written on a computer readable storage medium, so as to perform the method when the computer program product is executed on a computer. In addition, software modules may be distinguished processes running in the operating system of a suitably configured network device or a computer with a network interface.

[0169] According to one embodiment, the flow management method facilitates effective monitoring of the protected network security and improves the response to security incidents. The sample unit analyses the information contained in the traffic samples received from at least one network device 4 or selected network devices. The analysis is carried out by the units of detectors, D0, D1, Di and flow observer unit 11 (observer). When any sensor detects a pattern of network traffic requiring monitoring, an object representing a set of packets matching the detected pattern is created in the unit. This object is also called an ‘isolated packet flow’ 9. The flow is strictly defined by the 8 definition (pattern) of network filter created from packet header attributes (terms) and generated by any of the system's detectors. An object created in this method can be in one of four states during its life in the unit. The four flow states are: isolated state of flow sw, monitored state of flow so, controlled state of flow st and expired state of flow se. The transition between the states can be described as a finite state machine, whose graph of states is showed in FIG. 3. The flow observer unit 11 (observer) is responsible for the change of the flow state.

[0170] The flow introduced into the database by any Di detector is isolated. This is the initial state of a flow from which it is autonomously moved by the flow observer unit 11 (observer) to the monitored state. The flow in this state is sampled (with a set sampling frequency) by the flow observer unit 11, which analyzes packet headers and calculates their signal representation, as well as time and frequency characteristics. The collected measurement results are recorded every sampling period in the flow database. At the same time, the observer process checks the conditions for changing the state of each registered flow. The transition condition is the logical product of the alarm signals that are generated for each flow in the unit. Alarm signals indicate that the supervised network flow signal has violated set limits, e.g. related to an autonomously identified upper limit of its value or a change in power in a selected part of its spectrum. If the flow conditions for mitigation are met, the flow observer unit 11 changes the flow state to controlled state. The flow in controlled state is sampled (at a set sampling rate) by the sampling unit 5. However, for the flow in this state, the automatic flow control unit (controller) also creates on the network device a network filter 19 for packets and configures the packet policing means 17 to shape the flow dynamics. Calculated control parameters and measurements of observed characteristics are recorded in the database every sampling period. A flow in a controlled state whose characteristics do not contravene the safety standards may be switched by the flow observer unit back again to the monitoring state. From this state, the flow can also return to the controlled state. A change of state occurs and is stored in the database if the safety standards are violated again. The flow rate can change several times between monitoring and control states.

[0171] If the flow in monitoring state meets the safety conditions, it will be moved by the flow observer unit 11 to the expiry state. In this state, monitoring of its parameters is finished and the data concerning this flow is deleted from the unit.

[0172] An embodiment of the unit implementation with the use of the database is showed in FIG. 4.

[0173] The control process according to the embodiment of the invention is implemented by managing the flow states s1 of the network. Each of the listed flow states s1 is responsible for a specific value of filtration rules 18 implemented on the network device 4 by means of network filters 19 and packet policing means 17. The flow observer unit 11 is responsible for the transition between states and current state control. Flow state s1 is strictly defined by values stored in the definition memory 8 of network filters and current control signals 16, values of alarm signals 15 or their absence, threshold values 12 and flow dynamics model 14. The embodiment of an invention describing the process of controlling with the use of flow states s1 does not limit the invention to the distinguished states only. The control process can be described in a completely different method while retaining key functionalities. The control process consists of the following actions: [0174] automatic and autonomous identification of isolated packet flows requiring monitoring and identification of disturbances in monitored packet flows, [0175] automatic isolation of sources of disturbances (attacks) by creating and sending to network devices a configuration describing the filtering rules 18 of filtered network flows 3, [0176] identification of a flow dynamics model and a packet policing mechanism dynamics model in order to calculate traffic volume predictions for each flow based on an identified dynamic model, creating predicted flow characteristics 14, [0177] adaptive attenuation of isolated disturbances (attack mitigation) dynamically tuning parameters of packet filtering rules 18, shaping the dynamics of selected network flows 3 in a way that prevents saturation of protected links.

[0178] The above process is carried out periodically at a set sampling rate in the of sampling periods, which can take values from 1 s to 15 s, for example. The sampling periods may vary from one unit to another, but the sampling unit 5 should then have the specified sampling period greater or equal to the longest sampling period or the smallest common multiple of the different sampling periods. The different sampling periods may be, for example, a period of 1 s for flow observer unit 15 and a period of 5 s for automatic flow control unit 13. The adaptive control system, according to the invention, creates a closed control loop with at least one network device 4 (e.g. switches or routers), using packet samples a1 from the output interface 2 as feedback signals to control network traffic load requiring supervision by means of synthesized filter rules 18. Based on the feedback signals, the dynamics of the actual network flows 3 potentially threatening the protected links are formed. Importantly, packets included in the formed network flows 3 are not rejected in their entirety, but are sent in a quantity that guarantees the security of the protected network. The safe share of controlled network flows in the total network traffic is calculated autonomously based on the solution of the fair link-sharing problem. An embodiment of its implementation is shown in FIG. 2.

[0179] The system communicates with the network devices connected to it (switches, routers), wherein it must consist of at least one network device 4, every specified sampling period in order to take packet samples of a1 network traffic from the indicated interfaces and configure packet filtering rules 18. These tasks are performed using standard protocols and remote configuration functions of the network device provided by the hardware manufacturer (e.g. netconf, flowspec).

[0180] According to another embodiment, the implementation of method in line with the invention on a network device 4 transforms said network device 4 (e.g. switch or router), equipped with network input 1 and output 2 interfaces, standard sampling units 5, network filters 19 and packet policing means 17 for conditional packet forwarding (policing/shaping), into an advanced device shaping the dynamics of packet flows (observed in layers L2-L4 of OSI). The extension of the functions of network device 4 can be achieved by modifying the architecture of the device with the necessary units according to the invention or by implementing the appropriate procedures according to the SDN architecture. The applied network engineering method distinguishes on the network device 4, two types of network interfaces: [0181] Input interface 1, dirty type: interfaces accepting packets to be analyzed by the system, [0182] Output 2 interface, clean type: interfaces that send packets analyzed by the system, providing to the unit a feedback signal that illustrates the effectiveness of the system's actions.

[0183] The system fetches samples (e.g. sFlow or netFlow) of a1 packets on both interface types (1, 2). Samples from dirty type 1 input interface are used to isolate packet flows 9, sets of packets matching the pattern autonomously built by the system, requiring observation (potential attacks) and creating in the system, a network traffic model associated with the observed actual network flow of 3 packets. Based on the identified packet flow model (dynamics) and predicted flow characteristics 14, the system makes further decisions about the rate of packet transfer forming the network flow.

[0184] Packet samples from the clean 2 output interface are used by the unit to determine the effectiveness of the flow rate shaping process. Shaping of flow dynamics is performed by means of filtering rules 18 (policing/shaping) established on dirty interfaces or on interfaces between dirty-clean interfaces. An example of a dirty-clean control system is shown in FIG. 5

[0185] Transparent network architecture based on OSI Layer L2-L4, using widely available network technologies, allows easy connection of client devices to input and output interfaces for packet distribution. The use of advanced network engineering methods allows performing system services even on a single network device 4, transforming a firewall into a smart network protection device.

[0186] In contrast to solutions based on signature (pattern matching, rule-based) techniques, which are known from the state of art, and compared to the observed network traffic to known patterns, the invented mechanism uses signal processing techniques to perform the tasks of automatic building of filtering rules 18 isolating the actual network flows 3 requiring monitoring.

[0187] According to the invention, detection of anomalous flows in network traffic or in a DDoS attack carrier is based on the analysis of a set of packet a1 samples, including data from layers L2-L4 of the OSI model, flowing through the observed input interface 1. The unit consists of interconnected detectors, at least one D0 detector, wherein at least one first order D1 detector and/or at least one higher order Di detector is preferable in the unit. The D0 detector extracts from the observed set of packet samples a1 of network traffic, the flows that stand out due to the given traffic statistics, for example, the number of bytes, the number of packets, the number of bits per second (bps), the number of packets per second (pps), the number of logical flows or the average number of bytes in the packet (bpp) recorded every set sampling period of o1.

[0188] Isolated packet flows 9 extracted in this way are described by the definition of a network filter 8 composed of a single base term 23 (containing a single attribute of a packet x1, e.g. a target IP address or protocol type). The detector performs a multi-criteria analysis of the observed data to isolate flows that have the characteristics of an anomaly in network traffic or attack. For this purpose, a multiple voting mechanism is used, in which candidates (in parallel elections) are flows defined by a single base term 23 (first list of candidates: dst ip A, dst ip B, second list: src port A, src port B, etc.), and voters of traffic statistics (bps, pps, flows, bpp, etc.).

[0189] The lists ordered by the voters from the candidate list are aggregated by the D0 detector to the summary lists 21, which combine flows that stand out in many ways at the same time. In this way, a set of base terms 23 is created, which correspond to the flows potentially requiring further observation. For each base term 23 stored in memory 10, the activity counter 24 is increased. The value of activity counter 24 in memory 10 is then checked. If at least one counter exceeds the set activity value 25, a definition of network filter 8 is created that corresponds to base term 23, whose counter exceeded the required activity value 25. Detector D0 is run cyclically every set sampling periods of o1.

[0190] To improve the detection of flows that require observation, it is preferable to use the representation of packet samples a1 in the form of time series 26, which describe the changes in the attributes x1 packets over time, while each time series 26 contains information about changes in any available attribute x1 of packet samples a1 stored in the packet buffer 6 over a given sampling period o1. The time series 26 are used to determine the predicted flow characteristics 14 of flows, to build dynamic models and to detect flows in detectors. The time series are created by either flow monitor 11 or automatic flow control 13 and correspond to the isolated packet flows 9 by the definitions 8 of network filters stored in memory 10.

[0191] Based on a set of flows extracted by the D0 detector, the higher order detector, D1, builds a new, more precisely defined set of flows. This set consists of flows that are a combination of flows extracted by the D0 detector 7. This combination 8 of network filters is the solution to a properly defined signal separation task. For each packet flow 9 isolated by D0 7, the corresponding time series 26 are taken. Then the time series 26 separated by the set characteristics are selected and the combination of previously isolated time series is determined. At least two 26 time series separated from each other are selected either because of maximum linear (orthogonal) independence or because of minimal correlation (stochastic independence) in the signal space 26 in the time domain or frequency domain. Determination of the combination of separated time series 26 includes calculating matrix mixing the time series 26 corresponding to the base terms 23 (created by the detector D0 7). To calculate mixing matrix, typical techniques of signal analysis in the time and frequency domain, known to persons skilled in art, are used. A mixing matrix is then used to synthesize composite terms 32 of base terms 23, and the corresponding sub-packet flows 29 are searched for in the packet samples a1 stored in the sample buffer 6. In the collected traffic samples, the D1 detector searches for flows being the logical product of the terms corresponding to the distinguished elements of mixing matrix. If the subpacket flow 29 described by composite term 32 exists in packet samples a1 stored in the sample buffer 6, a new definition of network filter 8 is created, which isolates the identified packet flow 9 and is stored in memory 10. Filters created in this method increase the resolution of flow observation, i.e. they divide the traffic distinguished by the D0 detector into components requiring monitoring. Detector D1 performs operations cyclically every set sampling periods of depending on whether detector D0 has isolated new packet flows or on the state of the flows analyzed by the flow monitor 11.

[0192] The higher order Di detector then builds a new subset of flows by analyzing observed flows directed to cluster time series 26, but this time based on a subset set by any other detector, preferably for at least one D1 detector. The higher-order detector shall select time series 26 correlated or similar according to another specified criterion, e.g. because of their similarity in the form of the time series 26, i.e. the proximity determined by the DTW method (dynamic time warping). The new definitions 8 of network filters are the logical product of the terms building the definitions of D1 detector network filters, which form an identified cluster. New composite term 32 is a product of terms building definitions 8 of network filters isolating packet flows 9 of packets, which correspond to the detected correlated time series or similar time series. If a subflow 29 described by composite term 32 exists in the collected packet samples a1, a definition 8 of network filter is created that isolates the identified packet flow of 9. As a result, the created network filter definitions 8 divide the traffic distinguished by any other detector, preferably by at least one D1 detector, into its components, increasing the resolution of the filtration. The Di detector is run cyclically at the set sampling intervals o1 of depending on whether another lower order detector has isolated new packet flows or depending on the state of the flows analyzed by the flow monitor unit 11.

[0193] The procedure of increasing filtration resolution, using the analysis in time and frequency domain of the extended signal base in the form of time series 26 (reshowing packet flows), can be repeated in an iterative process, building a hierarchy of D0, D1, Di detectors, which, in subsequent steps, isolates from the observed network traffic, the actual network flows 3 through more and more precisely defined filtering rules 18. This mechanism is showed in FIG. 6

[0194] The applied approach allows for autonomous detection of composite and time-varying attack vectors, as well as for adaptive construction of mechanisms mitigating composite attacks with variable dynamics. The diagram of iterative mechanism architecture for increasing detection accuracy is showed in FIG. 7.

[0195] According to one embodiment, the appropriate attenuation of the network packet flow isolated by a network filter 19 of network packets 3 flow by packet policing means 17 includes the rejection of the packet. Appropriate attenuation of network packet flow 19 isolated by a network filter 3 by packet policing means 17 includes packet buffering.

[0196] The individual steps in the process are carried out on different network devices or with at least one computer connected to the network via a suitable network interface. The sampling periods used for different network devices may vary from one network device to another.

[0197] An adaptive network traffic filtering unit, according to invention, for one or more networked devices that autonomously detects anomalies and volumetric attacks (DDoS), containing memory 10, a processor or controller and at least one network interface, further comprises a sampling unit 5 of input 1 and output 2 interfaces, a sample buffer 6, a detector unit D0 7, an observer unit 11, an automatic flow control unit 13, and packet policing means 17 and network filters 19. packet policing means 17 and network filters 19 may be located in the same network device or in another network device equipped with them.

[0198] Sampling unit 5 performs the function of reading and collecting packet samples a1 of actual network flow 3 and stores them in a dedicated sample buffer 6. This buffer is connected to the detector unit D0 7 isolating packet flows 9 by reading packet samples a1 written in the sample buffer 6 and generating definitions 8 of network filters and storing them in memory of 10. In addition, memory 10 is connected to an observer unit 11, which reads from memory 10 information about the definitions 8 of network filters and the corresponding packet samples a1 written in the sample buffer 6, and then sets the threshold values 12 and writes them to memory 10. Observer unit 11 also determines whether the determined threshold values 12 were exceeded by comparing the parameters of the isolated packet flow 9 with the previous threshold values 121 stored in memory 10, and if the previous threshold values 121 were exceeded, alarm signals 15 stored in memory 10 are generated. In addition, the observer unit 11 predicts flow characteristics 14 within the set prediction horizon and stores them in memory 10.

[0199] At the same time, memory 10 is combined with automatic flow control 13, which regulates and shapes the dynamics of actual network flows 3, which correspond to isolated packet flows 9, which do not meet the set threshold values 12 that control the packet policing means 17 and network filters 19. The automatic flow control unit 13 is connected to at least one other network device 27, which is configured so that it can be transmitted to them, based on network filters read from the definitions 8, memory 10 and packet policing means control signals 16, filtering rules 18. Network filters definitions 8 and packet policing control signals 16 are translated into packet filtering rules 18 and transmitted to packet policing means 17 and network filters 19, thereby isolating the packet flows of network traffic 3 and shaping their dynamics in the protected area of the network.

[0200] According to one embodiment, the automatic flow control unit 13 reads alarm signals 15 from memory 10, (predicted and observed) flow characteristics 14, network filters definitions 8, packet policing means current control signals 16 and fetches threshold values 12. The automatic flow control unit 13 is also connected to a sample buffer 6 from which it reads packet samples a1 from the input 1 and output 2 interfaces and adapts to the new control signals 16 of packet policing means for the respective isolated packet flows 9 and stores them in memory 10.

[0201] Between the sample buffer 6 and the flow monitor unit 11, there is a parallel first order D1 detector 28 connected to the D0 detector 7 and memory 10, or between the sample buffer 6 and the flow monitor 11, there may be a higher order Di detector 30 and memory connected to the D0 detector 7. In addition, the invention does not exclude the possibility that between the sample buffer 6 and the flow monitor unit 11, there is a first order D1 detector 28 serially connected to D0 detector 7, which is connected to memory 10, or that between the sample buffer 6 and the flow monitor unit 11, there is a higher order Di detector 30 serially connected to first order D1 detector 28, which is connected to memory 10.

[0202] The observer unit 11 can be configured so that, based on the definition 8 of network filters, alarm signals 15 and designated threshold values 12, it activates a first order D1 detector 28 or higher order Di detector 30 that isolates new flows 29 and stores the new definitions 8 of network filters in a memory of 10. In addition, the observer unit 11 based on the definition 8 of network filters, alarm signals 15 and threshold values 12 can be configured to activate the automatic flow control unit 13, which performs the function of control and the shaping of the dynamics of the isolated packet flow indicated by the observer unit 11. Additionally, the automatic flow control unit 13 is configured to communicate with other network devices 27.

INDUSTRIAL APPLICATION

[0203] The invention may be applied as an element of the unit of protection of ICT networks or network traffic exchange points. It can also be used, among other things, as a: [0204] a tool for monitoring the state of network security, [0205] a tool for active response to network security incidents, [0206] a tool to generate knowledge about network flows, [0207] a decision support system for ICT security operators, [0208] a tool for controlling selected parameters of the quality of network services.