Enhancing Fraud Detection in Banking: A Hybrid Approach Combining Graph Algorithms and Machine Learning

Financial institutions face significant losses from increasingly sophisticated fraud attacks that evade traditional detection methods. This study proposes a hybrid approach combining machine learning (ML) (XGBoost, Random Forest, SVM, k-NN) with graph algorithms (PageRank, Community Detection, Degree Centrality) to enhance fraud detection accuracy in banking transactions. Using the BankSim dataset containing 587,443 legitimate and 7,200 fraudulent transactions we first address class imbalance through under sampling, then integrate graph-based features extracted via Neo4j to capture complex transactional relationships. Our methodology demonstrates that combining graph analytics with machine learning yields superior performance compared to standalone models, achieving precision scores up to 0.93 (k-NN) and recall rates of 0.87 (XGBoost). The hybrid approach also reduces training and prediction times by 2.9% and 6.8%, respectively, validated through 5-fold cross-validation. Key findings highlight that graph-augmented features improve F1-scores by 4–7% over conventional methods, with Random Forest and k-NN showing the most significant gains. This work contributes a practical framework for financial institutions to leverage interconnected transaction data, balancing detection accuracy (minimizing false negatives) and operational efficiency (reducing false positives). Future directions include testing this approach on real-time transaction streams and expanding to multi-modal fraud detection.