Search

Detecting malware is crucial for ensuring the security of computer systems. Traditional machine learning models face challenges in effectively detecting malware, mainly due to the class imbalance problem, where the number of malware samples is significantly smaller than that of non-malware samples. Additionally, malware’s dynamic and evolving nature, continuously altering its structure and tactics, presents a substantial challenge for conventional artificial intelligence algorithms, further complicating the detection task. In pursuing an optimized malware detection technique, researchers initially explored traditional machine learning algorithms, focusing on the features of Portable Executable (PE) file headers. However, the inherent issues, such as imbalanced datasets and the deceptive nature of malware, have raised concerns about the credibility of the attained results. This can result in misclassifying malware as non-malware, leading to security vulnerabilities. One-Class Classification (OCC) methods have emerged as a promising approach to improve the detection of unknown malware. However, traditional OCC approaches face the challenge of the curse of dimensionality. This research proposes adapting subspace learning-based OCC methods to overcome the curse of dimensionality and effectively handle the class imbalance problem. We propose a pipeline for detecting malware using methods that jointly optimize a subspace and data description for OCC. We evaluate the performance of various one-class classifiers on three different datasets. We observed that the subspace-learning-based OCC is a promising approach. Evaluating various classifiers on three datasets reveals promising results, with a True Positive Rate (TPR) of 100 % for subspace-learning-based OCC. The proposed pipeline can serve as a valuable tool for improving the security of computer systems by accurately detecting malware and protecting against potential attacks.

The proliferation of cloud and public legitimate services (CLS) on a global scale has resulted in increasingly sophisticated malware attacks that abuse these services as command-and-control (C&C) communication channels. Conventional security solutions are inadequate for detecting malicious C&C traffic because it blends with legitimate traffic. This motivates the development of advanced detection techniques. We make the following contributions: First, we introduce a novel labeled dataset. This dataset serves as a valuable resource for training and evaluating detection techniques aimed at identifying malicious bots that abuse CLS as C&C channels. Second, we tailor our feature engineering to behaviors indicative of CLS abuse, such as connections to known CLS domains and potential C&C API calls. Third, to identify the most relevant features, we introduced a custom feature elimination (CFE) method designed to determine the exact number of features needed for filter selection approaches. Fourth, our approach focuses on both static and derivative features of Portable Executable (PE) files. After evaluating various machine learning (ML) classifiers, the random forest emerges as the most effective classifier, achieving a 98.26% detection rate. Fifth, we introduce the “Replace Misclassified Parameter (RMCP)” adversarial attack. This white-box strategy is designed to evaluate our system’s detection robustness. The RMCP attack modifies feature values in malicious samples to make them appear as benign samples, thereby bypassing the ML model’s classification while maintaining the malware’s malicious capabilities. The results of the robustness evaluation demonstrate that our proposed method successfully maintains a high accuracy level of 84%. In sum, our comprehensive approach offers a robust solution to the growing threat of malware abusing CLS as C&C infrastructure.

This study introduces an innovative all-in-one malware identification model that significantly enhances convenience and resource efficiency in classifying malware across diverse file types. Traditional malware identification methods involve the extraction of static and dynamic features, followed by comparisons with signature-based databases or machine learning-based classifiers. However, many malware detection applications that rely on transfer learning and image transformation suffer from excessive resource consumption. In recent years, transfer learning has emerged as a powerful tool for developing effective classifiers, leveraging pre-trained neural network models. In this research, we comprehensively explore various pre-trained network architectures, including compact and conventional networks, as well as series and directed acyclic graph configurations for malware classification. Our approach utilizes grayscale transform-based features as a standardized set of characteristics, streamlining malware classification across various file types. To ensure the robustness and generalization of our classification models, we integrate multiple datasets into the training process. Remarkably, we achieve an optimal model with 96% accuracy, while maintaining a modest 5 MB size using the SqueezeNet classifier. Overall, our model efficiently classifies malware across file types, reducing the computational load, which can be useful for cybersecurity professionals and organizations.

Over the last few years, the malware propagation on PC platforms, especially on Windows OS has been even severe. For the purpose of resisting a large scale of malware variants, machine learning (ML) classifiers for malicious Portable Executable (PE) files have been proposed to achieve automated classification. Recently, function call graph (FCG) vectorization (FCGV) representation was explored as the input feature to achieve higher ML classification accuracy, but FCGV representation loses some critical features of PE files due to the hash technique. This paper aims to further improve the classification accuracy of FCGV-based ML model by applying both graph and non-graph features. We propose an FCGV-SF based Random Forest classification model, which applies both FCGV features (graph features) and statistical features (SF, non-graph features) extracted from disassembled PE files. Six types of effective non-graph features are chosen for our integrated vector, namely, metadata, symbol, operation code, register, section and data definition. We evaluate our model on a dataset provided by Microsoft hosted at Kaggle, and the experimental results indicate that the classification accuracy increases from 0.9851 to 0.9957 compared with the existing model based on FCGV only.