Enhancing credit card fraud detection with a hybrid approach using machine and deep learning

Abstract Credit card fraud is an important concern for banks, financial institutions and consumers, resulting in substantial financial losses annually. Traditional fraud detection systems are based on predefined rules, but as fraudsters develop more sophisticated techniques, these methods become less effective. Machine learning (ML) and deep learning (DL) offer powerful solutions to enhance the accuracy and efficiency of credit card fraud detection. However, a major challenge in credit card fraud detection is the highly imbalanced nature of transaction data, where fraudulent transactions are rare compared to legitimate ones. To address data imbalance, techniques such as Synthetic Minority Over-sampling Technique (SMOTE) and Mahalanobis distance Synthetic Minority Oversampling Technique–Edited Nearest Neighbors (SMOTE-ENN) hybrid sampling are applied to balance the dataset and improve model performance. The study evaluates various ML models and deep learning models for fraud detection and further evaluate learning dynamics, computational cost, and interpretability through SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations). Comprehensive error analysis confirms the robustness and transparency of the proposed approach. This study evaluated 37 models, and the two proposed stacking ensemble approaches showed significant advancements. The first proposed model effectively combines various algorithms: Extra Trees (ET), Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and eXtreme Gradient Boosting(XGBoost) as a meta-learner and the second proposed stacking ensemble approach integrates ET, Adaptive Boosting(AdaBoost) with Extra Trees as base, AdaBoost with Random Forest as base, XGBoost as meta-learner to maximize performance. This research highlights the importance of combining machine learning, deep learning, and data balancing techniques to improve credit card fraud detection. The proposed stacking ensemble approaches achieved exceptional results, with accuracy, precision, recall, F1-score, and Area Under the Curve (AUC) reaching 1.0, 0.9999, 1.0, 1.0 and 1.0, respectively. Experimental results indicate that ensemble learning techniques like Categorical Boosting (CatBoost) and XGBoost outperform traditional models, while deep learning methods, especially Feedforward Neural Network (FFNN), ANN (Artificial Neural Network) and Multilayer Perceptron(MLP) demonstrate strong performance in detecting fraud patterns.