Hybrid deep learning approach with capsule networks for financial fraud detection

The rapid growth of digital financial services has significantly increased the complexity and volume of financial transactions, making the accurate detection of fraudulent activities a critical challenge for conventional machine learning approaches. This study proposes a hybrid intelligent fraud detection framework that integrates data preprocessing, feature selection, deep learning classification, and adaptive hyperparameter optimization to improve fraud detection accuracy while maintaining computational efficiency. Initially, transaction records are preprocessed through missing value treatment, feature normalization, outlier analysis, and Tomek Links undersampling to enhance data quality and address class imbalance. Subsequently, the Perfumer Optimization Algorithm (POA) is employed to identify the most informative feature subset, whereas a Capsule Network performs hierarchical feature representation learning to capture complex relationships among transaction attributes. The Fennec Fox Optimization Algorithm (FFOA) is further utilized to optimize the network hyperparameters, improving convergence and overall classification performance. The proposed framework was evaluated using a publicly available Kaggle financial fraud benchmark dataset and compared with several state-of-the-art deep learning models. Experimental results achieved an accuracy of 99.67%, precision of 99.52%, recall of 99.74%, F1-score of 99.65%, and an MCC of 0.993, while maintaining low computational complexity suitable for real-time applications. Furthermore, SHAP and permutation importance analyses demonstrated that behavioural attributes, particularly failed transaction attempts and transaction frequency, contributed most significantly to fraud prediction. The proposed framework advances financial fraud detection by combining optimization-driven feature selection, hierarchical representation learning, and adaptive hyperparameter tuning into a unified architecture, providing an accurate, interpretable, and computationally efficient solution suitable for next-generation intelligent financial security systems.