Malware detection using bilayer behavior abstraction and improved one-class support vector machines.

Malware detection is one of the most challenging problems in computer security. Recently, methods based on machine learning are very popular in unknown and variant malware detection. In order to achieve a successful learning, extracting discriminant and stable features is the most important prerequisite. In this paper, we propose a bilayer behavior abstraction method based on semantic analysis of dynamic API sequences. Operations on sensitive system resources and complex behaviors are abstracted in an interpretable way at different semantic layers. At the lower layer, raw API calls are combined to abstract low-layer behaviors via data dependency analysis. At the higher layer, low-layer behaviors are further combined to construct more complex high-layer behaviors with good interpretability. The extracted low-layer and high-layer behaviors are finally embedded into a high-dimensional vector space. Hence, the abstracted behaviors can be directly used by many popular machine learning algorithms. Besides, to tackle the problem that benign programs are not adequately sampled or malware and benign programs are severely imbalanced, an improved one-class support vector machine (OC-SVM) named OC-SVM-Neg is proposed which makes use of the available negative samples. Experimental results show that the proposed feature extraction method with OC-SVM-Neg outperforms binary classifiers on the false alarm rate and the generalization ability. [ABSTRACT FROM AUTHOR]