Explainable Ensemble Machine Learning for Financial Transaction Fraud Detection: Insights from XGBoost and Deep Neural Networks

ABSTRACT The rapid digitalization of financial services has enhanced transaction speed and accessibility but also amplified exposure to fraud activities that undermine institutional integrity and consumer trust. Effective fraud detection requires analytical frameworks that are not only accurate and adaptive but also interpretable and compliant with financial regulations. This study develops a scalable, explainable machine learning framework for detecting fraud financial transactions using a hybrid ensemble of Extreme Gradient Boosting (XGBoost) and Deep Neural Networks (DNN). The proposed model integrates transactional, temporal, and identity-linked features to capture behavioral and contextual patterns in large-scale, high-frequency data. To address the extreme class imbalance inherent in fraud detection, we evaluate multiple strategies including focal loss, class weighting, and threshold optimization. Model transparency is enhanced through SHAP-based interpretability analysis, providing granular insights into the feature interactions driving fraud risk. Empirical evaluation on a real-world transaction dataset demonstrates that the hybrid ensemble achieves superior detection accuracy and recall relative to baseline models while maintaining explainability suitable for regulated financial environments. The results highlight the potential of combining interpretable machine learning with adaptive ensemble learning to enhance resilience and trustworthiness in modern financial risk management systems.