A communication-channel-based method for detecting deeply camouflaged malicious traffic

We present a novel method for detecting malicious TLS traffic based on communication channels that can detect deeply camouflaged malicious traffic. Moreover, we designed and extracted three types of channel features, namely, distribution features, consistency features of the Transport Layer Security (TLS) handshake field, and statistical features. Simultaneously, an efficacy feature selection method comprising a genetic algorithm is presented to obtain a global optimal feature subset, which reduces feature dimensions by 64% and increases accuracy by 1.5%. Comparison experiment results show that the proposed method possesses a more stable detection efficacy on different datasets with an accuracy of 97.65% and a much higher F1-score compared with other state-of-the-art classification methods.