Design of an Iterative Method for Malware Detection Using Autoencoders and Hybrid Machine Learning Models

In the evolving cyber threat landscape, one of the most visible and pernicious challenges is malware activity detection and analysis. Traditional detection and analysis methods face threats of data high-dimensionality, lack of strength against adversarial attacks, and non-efficient use of unlabeled data samples. In this context, we propose a comprehensive framework that applies machine learning methods to enhance evidence collection and malware activity analysis. The approach of our proposed model innovatively uses several advanced machine learning methods. First, in order to reduce the dimensionality of raw malware activity data by 50%, while at the same timestamp preserving critical information, as evidenced by minimal reconstruction error, we apply an autoencoder-based feature learning technique. This technique assists in the extraction of compact, informative, and feature representations covering both global and local discriminative patterns for accurate malware detection. With the addition of Gradient Boosted Decision Trees (GBDT) to features derived from Convolutional Neural Networks (CNN), we further improve the capability of the model. The hybrid model combines the outlier robustness and heterogeneous data handling capability of GBDTs with the hierarchical feature extraction capability of CNNs, resulting in a significant improvement in performance, with an F1-score of 0.95 on a validation set. In order to defend from evasion attacks, we incorporate adversarial training using Generative Adversarial Networks (GANs). It enables effective counteraction against adversarial strategies, reducing adversarial success rates by 60%. The model is trained using adversarial examples, and its parameters are optimized to minimize classification loss across both the normal and distorted inputs, thereby enhancing robustness. Expanding the applicability of the framework, we use semi-supervised self-training using Variational Autoencoders (VAEs) to use both labeled and unlabeled datasets & samples. This approach not only improves anomaly detection by 30% but also allows the model to learn probabilistic latent representations, thereby revealing underlying data structures. Finally, we address the challenge of temporal malware activity analysis through Long Short-Term Memory (LSTM) networks augmented with an attention mechanism. This configuration allows the model to be able to detect and adapt to evolving attack patterns, thus, by 25%, significantly improving the zero-day attack detection.