The growing number of malware threats is one of the most serious challenges facing modern cybersecurity, with obfuscated malware being especially difficult to detect. Obfuscated malware uses techniques such as polymorphism, metamorphism, and code encryption to change its structure and behavior at runtime, making signature-based and heuristic detection methods largely ineffective. This study proposes a hybrid deep learning framework that combines Deep Neural Networks (DNNs) for structural feature learning and Recurrent Neural Networks (RNNs) for capturing sequential behavioral patterns from memory data. Our proposed approach uses Particle Swarm Optimization (PSO) to automatically optimize key hyperparameters of the hybrid model, including learning rate, dropout, batch size, and layer dimensions, using validation performance as a fitness function. A preprocessing pipeline including normalization and feature selection is applied before training. The results show that the PSO-optimized hybrid model achieves 99.98% test accuracy on the MemMal-D2024 dataset, outperforming standalone DNN and RNN models. These findings demonstrate that the proposed framework is an effective and reliable solution for detecting obfuscated and memory-resident malware.
Alazba et al. (Mon,) studied this question.