Enhancing malware detection and classification in network traffic using deep learning techniques

Abstract Malware detection and classification in network traffic is a critical challenge in cybersecurity, with evolving threats that traditional methods struggle to address. As network traffic becomes more complex, accurately identifying malicious activities while minimizing false positives is essential for real‐time monitoring systems. This study aims to enhance malware detection using deep learning (DL) techniques, focusing on improving accuracy, reducing false positives, and enabling real‐time detection in dynamic network environments. Several advanced DL techniques are introduced to address these challenges. Entropy‐Based Traffic Filtering (ETF) measures the randomness in network traffic to identify anomalies and malicious patterns, reducing noise and improving feature extraction. Self‐Supervised Learning for Anomaly Detection (SSLAD) detects malware without labeled data by learning normal traffic patterns and identifying anomalies, thus improving the detection of unknown threats. Graph Neural Networks for Malware Traffic Classification (GNN‐MTC) model network traffic as graphs, where devices are nodes, and communications are edges, capturing relational dependencies and anomalies to detect complex attack patterns like botnets and command‐and‐control (C2) communications. Context‐Aware Graph Attention Networks (CA‐GAT) further enhance detection by analyzing traffic as graphs while incorporating contextual factors like time and behavior, focusing on relevant interactions to improve attack detection. The proposed DL model achieves 98% accuracy, surpassing DeepMAL (95%) and an entropy‐based method by Huang et al. (97.3%). Its strong precision and recall demonstrate superior performance in detecting known and novel malware, making it well‐suited for real‐time network security applications. The model was implemented using Python. Future research could focus on integrating real‐time adaptive learning models, exploring hybrid DL architectures, and enhancing cross‐platform malware detection, ensuring scalability and robustness in evolving network security environments.