Anomaly flow detection device and anomaly flow detection method

Abstract

An anomaly flow detection device and an anomaly flow detection method thereof are provided. The device can retrieve a plurality of training data transmitted between a monitored network and an external network, preprocess a plurality of packet headers of the pluralities of training data to obtain a plurality of training feature vectors, construct a flow recognition model with an unsupervised learning method, input the pluralities of training feature vectors to the flow recognition model to train the flow recognition model, retrieve a plurality of testing data transmitted between the monitored network and the external network, preprocess a plurality of packet headers of the pluralities of testing data to obtain a plurality of testing feature vectors, input the pluralities of testing feature vectors to the flow recognition model to identify whether the pluralities of packet headers of the pluralities of testing data are normal or abnormal, and determine the flow of the monitored network is abnormal according to the recognition result of the flow recognition model.

Claims

1. An anomaly flow detection device comprising: a network interface; a storage; and a processor electrically connected to the network interface and the storage; wherein the processor performs the following steps: retrieving a plurality of training data transmitted between a monitored network and an external network in a first time interval through the network interface; preprocessing a plurality of packet headers of the pluralities of training data to obtain a plurality of training feature vectors and storing them in the storage, comprising steps of: choosing the packet headers of the pluralities of training data with certain feature types, in which the feature types comprises numerical features and non-numerical features; filling up fields of missing feature values of the feature types; standardizing the numerical features in the feature types to obtain first sub-features; converting the non-numerical features in the feature types to binary value features to obtain second sub-features; and obtaining the training feature vectors according to the first sub-features and the second sub-features; constructing a flow recognition model with an unsupervised learning method and storing it in the storage; inputting the pluralities of training feature vectors to the flow recognition model to train the flow recognition model; retrieving a plurality of testing data transmitted between the monitored network and the external network through the network interface; preprocessing a plurality of packet headers of the pluralities of testing data to obtain a plurality of testing feature vectors and storing them in the storage; inputting the pluralities of testing feature vectors to the flow recognition model to identify whether the pluralities of packet headers of the pluralities of testing data are normal or abnormal; and determining the flow of the monitored network is abnormal by calculating an abnormal index according to a recognition result of the flow recognition model.

2. The according to claim 1, wherein the feature types of the packet headers of the pluralities of training data comprises the numerical features and the non-numerical features, wherein the numerical features comprise an IP (Internet Protocol) packet length, a TCP (Transmission Control Protocol) window size, a TCP header length, a TCP data length, a TCP urgent pointer, an UDP (User Datagram Protocol) length, an IP header length, and an IP time to live and the non-numerical features comprises TCP flags, an IP type of service (TOS), and IP flags.

3. The according to claim 1, wherein the step of determining the flow of the monitored network is abnormal by calculating an abnormal index according to the recognition result of the flow recognition model comprises the following step: determining the flow of the monitored network is abnormal when the abnormal index is greater than or equal to an abnormal threshold.

4. The according to claim 1, wherein the step of constructing a flow recognition model with an unsupervised learning method comprises the following step: constructing the flow recognition model with One-Class Support Vector Machine (OC-SVM) or Isolation Forest (IF) algorithm.

5. An anomaly flow detection method performed by an anomaly flow detection device, wherein the device comprises a network interface, a storage, and a processor electrically connected to the network interface and the storage, and the method comprises the following steps: retrieving a plurality of training data transmitted between a monitored network and an external network in a first time interval through the network interface; preprocessing a plurality of packet headers of the pluralities of training data to obtain a plurality of training feature vectors and storing them in the storage, comprising steps of: choosing the packet headers of the pluralities of training data with certain feature types, in which the feature types comprises numerical features and non-numerical features; filling up fields of missing feature values of the feature types; standardizing the numerical features in the feature types to obtain first sub-features; converting the non-numerical features in the feature types to binary value features to obtain second sub-features; and obtaining the training feature vectors according to the first sub-features and the second sub-features; constructing a flow recognition model with an unsupervised learning method and storing it in the storage; inputting the pluralities of training feature vectors to the flow recognition model to train the flow recognition model; retrieving a plurality of testing data transmitted between the monitored network and the external network through the network interface; preprocessing a plurality of packet headers of the pluralities of testing data to obtain a plurality of testing feature vectors and storing them in the storage; inputting the pluralities of testing feature vectors to the flow recognition model to identify whether the pluralities of packet headers of the pluralities of testing data are normal or abnormal; and determining the flow of the monitored network is abnormal by calculating an abnormal index according to the recognition result of the flow recognition model.

6. The anomaly flow detection method according to claim 5, wherein the feature types of the packet headers of the pluralities of training data comprises the numerical features and the non-numerical features, wherein the numerical features comprise an IP (Internet Protocol) packet length, a TCP (Transmission Control Protocol) window size, a TCP header length, a TCP data length, a TCP urgent pointer, an UDP (User Datagram Protocol) length, an IP header length, and an IP time to live and the non-numerical features comprises TCP flags, an IP type of service (TOS), and IP flags.

7. The anomaly flow detection method according to claim 5, wherein the step of determining the flow of the monitored network is abnormal by calculating an abnormal index according to the recognition result of the flow recognition model comprises the following step: determining the flow of the monitored network is abnormal when the abnormal index is greater than or equal to an abnormal threshold.

8. The anomaly flow detection method according to claim 5, wherein the step of constructing a flow recognition model with an unsupervised learning method comprises the following step: constructing the flow recognition model with One-Class Support Vector Machine (OC-SVM) or Isolation Forest (IF) algorithm.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Exemplary embodiments will be more readily understood from the following detailed description when read in conjunction with the appended drawings, in which:

(2) FIG. 1 depicts schematic view of an anomaly flow detection system according to one embodiment of the present disclosure.

(3) FIG. 2 depicts a schematic view of an anomaly flow detection device according to one embodiment of the present disclosure.

(4) FIG. 3 depicts a flow chart of an anomaly flow detection method according to one embodiment of the present disclosure.

(5) FIG. 4 depicts a flow chart of the steps of preprocessing the training data according to one embodiment of the present disclosure.

(6) FIG. 5 depicts a schematic view of a flow chart of the steps of preprocessing the testing data according to one embodiment of the present disclosure.

(7) FIG. 6 depicts a timing diagram of the testing data according to one embodiment of the present disclosure.

DETAILED DESCRIPTION

(8) For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numbers indicate like features. Persons having ordinary skill in the art will understand other varieties for implementing example embodiments, including those described herein. The drawings are not limited to specific scale and similar reference numbers are used for representing similar elements. As used in the disclosures and the appended claims, the terms “example embodiment,” “exemplary embodiment,” and “present embodiment” do not necessarily refer to a single embodiment, although it may, and various example embodiments may be readily combined and interchanged, without departing from the scope or spirit of the present disclosure. Furthermore, the terminology as used herein is for the purpose of describing example embodiments only and is not intended to be a limitation of the disclosure. In this respect, as used herein, the term “in” may include “in” and “on”, and the terms “a”, “an” and “the” may include singular and plural references. Furthermore, as used herein, the term “by” may also mean “from”, depending on the context. Furthermore, as used herein, the term “if” may also mean “when” or “upon”, depending on the context. Furthermore, as used herein, the words “and/or” may refer to and encompass any and all possible combinations of one or more of the associated listed items.

(9) Please refer to FIG. 1. FIG. 1 depicts schematic view of an anomaly flow detection system 1 according to one embodiment of the present disclosure. The anomaly flow detection system 1 may comprise an anomaly flow detection device 10, a router RT, a monitored network LAN (Local Area Network), and an external network WAN (Wide Area Network). The anomaly flow detection device 10 may be arranged in a computer apparatus, in which the computer apparatus may comprise a user equipment and a network apparatus. The user equipment includes, without limitation, computers, smart phones, PDAs (Personal Digital Assistants), etc. The network equipment includes, without limitation, a single network server and a server group composed of multiple network servers. The computer equipment can run alone to realize the present invention, or it can be implemented through the interactive operation of the router RT and other computer equipment in other networks, such as the monitored network LAN or the external network WAN. The network where the computer equipment is located includes, without limitation, the Internet, wide area network, metropolitan area network, local area network, VPN (Virtual Private Network), etc.

(10) According to one embodiment, the anomaly flow detection device 10 can receive packet data transmitted between the monitored network LAN and the external network WAN through the router RT, and construct a flow recognition model to analyze the packet data, so as to determine whether the network flow of the monitored network LAN is abnormal.

(11) According to another one embodiment, the anomaly flow detection device 10 may also be integrated into the router RT or arranged in the monitored network LAN. The anomaly flow detection method performed by the anomaly flow detection device 10 will be described below.

(12) In conjunction with FIG. 1, FIG. 2 depicts a schematic view of an anomaly flow detection device according to one embodiment of the present disclosure, FIG. 3 depicts a flow chart of an anomaly flow detection method according to one embodiment of the present disclosure, FIG. 4 depicts a flow chart of the steps of preprocessing the training data according to one embodiment of the present disclosure, and FIG. 5 depicts a schematic view of a flow chart of the steps of preprocessing the testing data according to one embodiment of the present disclosure.

(13) The anomaly flow detection device 10 may comprise a processor 101, a network interface 103, and a storage 105, in which the processor 101 may be electrically connected to the network interface 103 and the storage 105. The steps of detecting whether the network flow of the monitored network LAN by the anomaly flow detection device 10 is abnormal can be roughly divided into a training phase and a testing phase.

(14) The training phase may comprise the steps of retrieving a plurality of training data transmitted between a monitored network LAN and an external network WAN during a first time interval through the network interface 103 by the processor 101 (S301), in which the pluralities of training data are packet data; and preprocessing a plurality of packet headers of the pluralities of training data to obtain a plurality of training feature vectors of the training data and storing the pluralities of training feature vectors in the storage 105 by the processor 101 (S303).

(15) The step of S303 may comprise the step of choosing the packet headers of the pluralities of training data with certain feature types by the processor 101 (S401), in which the feature types comprise numerical features and non-numerical features. The packet headers may be represented in the form of a vector, such as the j-th feature type in the i-th data is represented by r.sub.i,j, in which i=1, 2, . . . , M (M is the total number of packets of data in the collected training data), and j=1, 2, . . . , N (N is the number of features used for training). The packet headers with numerical features chosen in the present embodiment may comprise, without limitation, a TCP window size, a TCP header length, a TCP data length, a TCP urgent pointer, an UDP length, an IP header length, and an IP time to live (r.sub.i,5, r.sub.i,6, r.sub.i,8, r.sub.i,9, r.sub.i,10, r.sub.i,11, r.sub.i,13, r.sub.i,15). The packet headers with non-numerical features chosen in the present embodiment may comprise, without limitation, TCP flags, an IP type of service (TOS), and IP flags (r.sub.i,7, r.sub.i,12, r.sub.i,14).

(16) When using TCP protocol to transmit packet data (training data, testing data), the value of feature r.sub.i,11 in the data is a missing feature value, such as the UDP length. If using UDP protocol to transmit packet data (training data, testing data), the values of the features r.sub.i,6, r.sub.i,7, r.sub.i,8, r.sub.i,9, r.sub.i,10 in the data are missing feature values, such as TCP window size, TCP flags, the TCP header length, TCP data length, and the TCP urgent pointer. In order to avoid too many missing feature values in the training data, the step of S303 may further comprise the steps of filling up fields of missing feature values of the feature types by the processor 101 (S403), for example, filling up the missing feature values with “−1”.

(17) For the packet headers with the numerical features in the training data, the processor 101 may scale the feature value according to the following formula. The step of S303 may further comprise the steps of standardizing the numerical features in the packet headers of the pluralities of training data to obtain first sub-features by the processor 101 (S405), for example, using the min-max normalization to process the features in training data. The processed training data is used as the first sub-features {circumflex over (r)}.sub.i,j, {circumflex over (r)}.sub.i,j may be presented a distribution with a standard deviation of 1 and a mean of 0, in which is the standardized result of r.sub.i,j. The first sub-features {circumflex over (r)}.sub.i,j may be approached to the standard normal distribution. The example of the first sub-features {circumflex over (r)}.sub.i,j may be {circumflex over (r)}.sub.i,5, {circumflex over (r)}.sub.i,6, {circumflex over (r)}.sub.i,8, {circumflex over (r)}.sub.i,9, {circumflex over (r)}.sub.i,10, {circumflex over (r)}.sub.i,11, {circumflex over (r)}.sub.i,13, {circumflex over (r)}.sub.i,15.

(18) For the packet headers with the non-numerical features in the training data, the step of S303 may further comprise the steps of converting the non-numerical features in the packet headers of the pluralities of training data to binary value features to obtain second sub-features by the processor 101 (S407). For example, the feature value of the packet headers of the non-numerical feature chosen by the processor 101 may be read by Wireshark (a packet analyzer) in hexadecimal. Therefore, the processor 101 may convert n-bit non-numerical features from hexadecimal into binary, and then convert them into n 1-bit sub-features, such that non-numerical features may be represented as sub-features. For example, the feature value of 6-bit TCP flags of one training data r.sub.i,7 is 0x18, which would be converted into a vector [0, 1, 1, 0, 0, 0] represented in a column. Therefore, the feature value of r.sub.i,7 would be converted into TCP flag sub-feature [r.sub.i,7,0, r.sub.i,7,1, r.sub.i,7,2, r.sub.i,7,3, r.sub.i,7,4, r.sub.i,7, 5]. Likewise, the feature value of a 8-bit IP DS (Differentiated Services) field of one training data r.sub.i,12 would be converted into IP DS field sub-feature [r.sub.i,12,0, r.sub.i,12,1, r.sub.i,12,2, r.sub.i,12,3, r.sub.i,12,4, r.sub.i,12,5, r.sub.i,12,6, r.sub.i,12,7], and the feature value of 3-bit IP flags of one training data r.sub.i,14 would be converted into IP flags sub-feature [r.sub.i,14,0, r.sub.i,14,1, r.sub.i,14,2]. The second sub-features ř are composed of the TCP flag sub-feature, the IP DS field sub-feature and the IP flags sub-feature.

(19) Next to the step S407, the step of S303 may further comprise the steps of constituting the training feature vectors f.sub.i=(f.sub.i,0, f.sub.i,1, f.sub.i,2, . . . , f.sub.i,24)=(r.sub.i,7,0, r.sub.i,7,1, r.sub.i,7,2, r.sub.i,7,3, r.sub.i,7,4, r.sub.i,7,5, r.sub.i,12,0, r.sub.i,12,1, r.sub.i,12,2, r.sub.i,12,3, r.sub.i,12,4, r.sub.i,12,5, r.sub.i,12,6, r.sub.i,12,7, r.sub.i,14,0, r.sub.i,14,1, r.sub.i,14,2, {circumflex over (r)}.sub.i,5, {circumflex over (r)}.sub.i,6, {circumflex over (r)}.sub.i,8, {circumflex over (r)}.sub.i,9, {circumflex over (r)}.sub.i,10, {circumflex over (r)}.sub.i,11, {circumflex over (r)}.sub.i,13, {circumflex over (r)}.sub.i,15) according to the first sub-features {circumflex over (r)}.sub.i,j according to step S405 and the second sub-features ř according to the step S407 (S409), in which f.sub.i,j is the j-th feature of the i-th training data.

(20) Next to the step S303, the training phase may further comprise the steps of constructing a flow recognition model with an unsupervised learning method by the processor 101 (S305). For example, the flow recognition model may be constructed with One-Class SVM (OC-SVM) algorithm or Isolation Forest (IF) algorithm.

(21) Next to the step S305, the training phase may further comprise the steps of inputting the pluralities of training feature vectors f.sub.i to the flow recognition model to train the flow recognition model by the processor 101 (S307). The above mentioned steps S301 to S307 complete the training phase.

(22) Next to the step S307, for detecting whether the network flow of the monitored network LAN is abnormal, the operation of the processor 101 enters the testing phase. The testing phase may comprise the steps of retrieving a plurality of testing data transmitted between the monitored network LAN and the external network WAN during a second time interval through the network interface 103 by the processor 101 (S309), in which the pluralities of testing data are packet data; and preprocessing a plurality of packet headers of the pluralities of testing data to obtain a plurality of testing feature vectors of the testing data and storing the pluralities of testing feature vectors in the storage 105 by the processor 101 (S311). The steps S307 to S311 are similar with the steps S301 to S305, and the steps S501 to S509 of the step S311 for preprocessing the pluralities of the packet headers of the pluralities of testing data are similar with the steps S401 to S409 Likewise, third sub-features are obtained by standardizing the numerical features in the packet headers of the pluralities of testing data (S505), fourth sub-features are obtained by converting the non-numerical features in the packet headers of the pluralities of testing data to binary value features (S507), and the testing feature vectors are obtained according to the third sub-features and the fourth sub-features (S509).

(23) Next to step S311, the testing phase may further comprise the steps of inputting the pluralities of testing feature vectors from the storage 105 to the flow recognition model to identify whether the pluralities of packet headers of the pluralities of testing data are normal or abnormal by the processor 101 (S313). More specifically, the processor 101 identifies whether the pluralities of packet headers of the pluralities of testing data are normal or abnormal according to an output value from the flow recognition model belongs to a set {0,1}, such as 0 means normal, and 1 means abnormal.

(24) Finally, next to step S313, the anomaly flow detection method may further comprise the step of determining whether the flow of the monitored network LAN is abnormal according to the recognition result of the flow recognition model (S315). More specifically, please refer to FIG. 6, if there are M packets judged to be abnormal in the K packets of the testing data during a detection period T by the flow recognition model, the abnormal index is M/K. When the abnormal index is greater than or equal to an abnormal threshold S, the flow of the monitored network is judged as abnormal, in which the abnormal threshold S may be between 0.1 and 0.9, without limitation.

(25) Above all, the anomaly flow detection device and the method of anomaly flow detection thereof of the present disclosure can inspect the first K packets in the time intervals to determine whether the flow of the monitored network is abnormal according to the recognition result of the flow recognition model. Therefore, the payload is small, the complexity is low, and there is no need to wait for the end of the entire stream, so the speed is faster, that is, the network manager can be notified via email or text message in the shortest time.

(26) While various embodiments in accordance with the disclosed principles are described above, it should be understood that they are presented by way of example only, and are not limiting. Thus, the breadth and scope of exemplary embodiment(s) should not be limited by any of the above-described embodiments, but should be defined only in accordance with the claims and their equivalents issuing from this disclosure. Furthermore, the above advantages and features are provided in described embodiments, but shall not limit the application of such issued claims to processes and structures accomplishing any or all of the above advantages.

(27) Additionally, the section headings herein are provided for consistency with the suggestions under 37 C.F.R. 1.77 or otherwise to provide organizational cues. These headings shall not limit or characterize the invention(s) set out in any claims that may issue from this disclosure. Specifically, a description of a technology in the “Background” is not to be construed as an admission that technology is prior art to any invention(s) in this disclosure. Furthermore, any reference in this disclosure to “invention” in the singular should not be used to argue that there is only a single point of novelty in this disclosure. Multiple inventions may be set forth according to the limitations of the multiple claims issuing from this disclosure, and such claims accordingly define the invention(s), and their equivalents, that are protected thereby. In all instances, the scope of such claims shall be considered on their own merits in light of this disclosure, but should not be constrained by the headings herein.