Visualized Malware Images using Hybrid Ensemble Deep Transfer Learning

Malware visualized images are increasingly being used to evade detection by traditional malware detection systems. In this study, we propose a hybrid deep transfer learning method for the detection and classification of Malware colored images. This paper presents a visualized RGB malware detection approach that employs a hybrid autoencoder for anomaly detection along with transfer deep learning for feature extraction. The proposed model first processes RGB images of malware samples, which are reconstructed using a hybrid autoencoder to highlight anomalies between normal and malicious code. Then, it uses a combination of deep learning techniques such as Convolutional Neural Networks (CNNs) architectures and the transfer learning approach to leverage the power of pre-trained models to extract features. The pre-trained models are then fine-tuned on the target dataset, which enables more efficient learning of image features than from scratch while reducing the risk of overfitting. Then, these architectures are ensembled to improve the system's accuracy and robustness in detecting malware threats. The proposed approach has been evaluated on a benchmark dataset for malware detection. The results show that the autoencoder algorithm is successful in identifying the outliers data. The Hybrid Ensemble Deep Transfer Learning (HEDTL) model outperforms individual architectures, with a higher detection rate and lower false positive rate, resulting in improved accuracy in detecting and preventing malware threats.