A quantum-inspired hybrid framework for ransomware detection using graph neural networks and support vector machines

Ransomware detection stands as a major cybersecurity problem because attackers use obfuscation methods together with encryption and polymorphic techniques to bypass static analysis systems. While classical machine learning and deep learning models achieve strong performance on known samples, their ability to model complex inter-feature dependencies under distribution shift remains limited. This paper proposes a Hybrid Quantum Graph Learning Framework (QGNN–QSVM) that integrates structured feature selection, relational modelling, and quantum-inspired kernel classification. A Quantum-Inspired Feature Selection (QIFS) mechanism first aligns and reduces structural Portable Executable (PE) attributes to a compact subset of twelve discriminative features using Laplacian-based locality preservation and quantum fidelity–based redundancy pruning. The selected features are organised as a correlation-driven graph and embedded using a Quantum Graph Neural Network, where entanglement-based interactions encode nonlinear dependencies. Final classification is performed using a Quantum Support Vector Machine with a Pauli feature map–based kernel. Experimental evaluation on the publicly available Ransomware Detection dataset by Mathur (GitHub) and the EMBER malware benchmark demonstrates within-dataset accuracies of 98.02% and 95.83%, respectively. Cross-dataset validation further indicates improved robustness compared to strong classical baselines, including gradient boosting and graph attention networks. Statistical testing confirms that the observed improvements are consistent across repeated runs. The results suggest that structured quantum-inspired relational modelling provides a stable enhancement for static ransomware detection under the evaluated configuration.