A novel deep learning-based approach for malware detection.

Malware detection approaches can be classified into two classes, including static analysis and dynamic analysis. Conventional approaches of the two classes have their respective advantages and disadvantages. For example, static analysis is faster but cannot detect the malware variants generated through code obfuscation, whereas dynamic analysis can effectively detect variants generated through code obfuscation but is slower and requires intensive resources. This paper proposes a novel deep learning-based approach for malware detection. It delivers better performance than conventional approaches by combining static and dynamic analysis advantages. First, it visualises a portable executable (PE) file as a coloured image. Second, it extracts deep features from the colour image using fine-tuned deep learning model. Third, it detects malware based on the deep features using support vector machines (SVM). The proposed method combines deep learning with machine learning and eliminates the need for intensive feature engineering tasks and domain knowledge. The proposed approach is scalable, cost-effective, and efficient. The detection effectiveness of the proposed method is validated through 12 machine learning models and 15 deep learning models. The generalisability of the proposed framework is validated on various benchmark datasets. The proposed approach outperformed with an accuracy of 99.06% on the Malimg dataset. The Wilcoxon signed-rank test is used to show the statistical significance of the proposed framework. The detailed experimental results demonstrate the superiority of the proposed method over the other state-of-the-art approaches, with an average increase in accuracy of 16.56%. Finally, to tackle the problems of imbalanced data and the shortage of publicly available datasets for malware detection, various data augmentation techniques are proposed, which lead to improved performance. It is evident from the results that the proposed framework can be useful to the defence industry, which will be helpful in devising more efficient malware detection solutions. [Display omitted] • A novel hybrid framework combines deep transfer learning and machine learning for malware detection. • An image-based PE dataset is generated by transforming malicious and benign Windows executables into RGB-coloured images. • The second step involves a novel combination of deep learning and machine learning models for malware detection. • An in-depth analysis using 15 deep learning and 12 machine learning models is presented for malware detection. • The framework is scalable, cost-effective, and efficient. It eliminates the need for domain experts for reverse engineering tasks. [ABSTRACT FROM AUTHOR]