Search

The explosive growth in network traffic in recent times has resulted in increased processing pressure on network intrusion detection systems. In addition, there is a lack of reliable methods for preprocessing network traffic generated by benign applications that do not steal users’ data from their devices. To alleviate these problems, this study analyzed the differences between benign and malicious traffic produced by benign applications and malware, respectively. To fully express these differences, this study proposed a new set of statistical features for training a clustering model. Furthermore, to mine the communication channels generated by benign applications in batches, a semisupervised clustering method was adopted. Using a small number of labeled samples, our method aggregated historical network traffic into two types of clusters. The cluster that did not contain labeled malicious samples was regarded as a benign traffic cluster. The experimental results were compared using four types of clustering algorithms. The density-based spatial clustering of applications with noise (DBSCAN) clustering algorithm was selected to mine benign communication channels. We also compared our method with two other methods, and the results demonstrated that the benign channels mined through our method were more reliable. Finally, using our method, 1,811 benign transport layer security (TLS) channels were mined from 18,357 TLS communication channels. The number of flows carried by these benign channels comprised 65.37% of the entire network flows, and no malicious flow was included in our results, which proves the effectiveness of our method.

The popularity of encryption method brings a great challenge to malware traffic identification. Traditional classes defined by expert experience are usually classified based on the host behaviors of malware, such as banking malware or ransomware, which are often irrelevant to its communication traffic behaviors. It leads to the fact that the boundaries of traffic feature dataset of different malware classes are fuzzy and make these traditional classes unhelpful for classification based on traffic features. Meanwhile, traditional machine learning-based encrypted malware traffic identification methods, such as using the multi-classification supervised learning model, are inefficient both in model training and detection, and its detection accuracy cannot meet the demand. In this paper, we propose a distance-based method, which utilizes unsupervised learning algorithm Gaussian mixture model (GMM) and ordering points to identify the clustering structure (OPTICS) to calculate the Distance between malwares and make use of the Distance to define the new malware class called FClass. Then, a set of models are trained by XGBoost algorithm to build an identification framework based on the FClass. The performance of the proposed method has been evaluated by comparing it with the other four methods. The results show that the proposed distance-based method is more efficient and accurate.

For highly camouflaged Command and Control (C&C) communications, especially those using the Transport Security Layer (TLS) protocol, traditional classifiers which only based on statistical features or TLS handshake features gradually fail to detect such behavior. In this context, exploring features of other dimensions to build a more targeted recognition model is one of the ways to alleviate this problem. This paper proposed a new method of detecting malicious TLS traffic by using the communication channel as the detection unit, and a new set of modeling features for the communication channel was designed, including distribution features, the consistency features and statistical features of TLS communication channel. Experiments show that compared with other two types of features, the consistency features contribute most, and combining these three types of features together can train a better classifier which the precision reaches 92.57%. Comparative experiments show that the proposed method is more advantageous when faced with highly camouflaged TLS flows because the proposed method also achieved highest F1 score, and the accuracy is about 2% higher than the classifier based on the TLS handshake features, and 12% higher than the clustering model based on the statistical features of flow.

Domain Name System (DNS) tunnels, established between the controlled host and master server disguised as the authoritative domain name server, can be used as a secret data communication channel for malicious activities. Owing to the ready evasion of the DNS traffic to bypass the network security mechanism, DNS tunnelling can cause severe damage. Thus, an in-depth and comprehensive understanding of the various detection technologies is of considerable importance when facing this type of threat. However, most of the existing reviews focus on a single aspect of the DNS tunnel detection technologies, such as methods based on machine learning, traffic, and payload analysis. In addition, few studies have conducted comprehensive investigation that includes a sequentially integrated range of literature in this researchfield, or have analysed the latest literature on DNS tunnels. This paper reviews these detection technologies from a novel perspective of rule-based and model-based methods with descriptions and analyses of the DNS-based tools and their corresponding features. To the best of our knowledge, this is the first study to comprehensively discuss and analyse DNS tunnel detection in a novel and specific classification fashion from various aspects in detail, covering almost all the detection methods developed from 2006 to 2020. Furthermore, a comparative analysis of detection methods and several suggestions for future research directions are presented.

The transport layer security (TLS) protocol is widely adopted by apps as well as malware. With the geometric growth of TLS traffic, accurate and efficient detection of malicious TLS flows is becoming an imperative. However, current studies focus on either detection accuracy or detection efficiency, and few studies take into account both indicators. In this paper, we propose a two-layer detection framework composed of a filtering model (FM) and a malware family classification model (MFCM). In the first layer, a new set of TLS handshake features is presented to train the FM, which is devised to filter out a majority of benign TLS flows. For identifying malware families, both TLS handshake features and statistical features are applied to construct the MFCM in the second layer. Comprehensive experiments are conducted to substantiate the high accuracy and efficiency of the proposed two-layer framework. A total of 96.32% of benign TLS flows can be filtered out by the FM with few malicious TLS flows being discarded provided the threshold of the FM is set to 0.01. Moreover, a multiclassifier is selected to construct the MFCM to provide better performance than a set of binary classifiers under the same feature set. In addition, when the ratio of benign and malicious TLS flows is set to 10:1, the detection efficiency of the two-layer framework is 188% faster than that of the single-layer framework, while the average detection accuracy reaches 99.45%.