A Comparative Analysis of Original and Resampled Dataset Splits in Machine Learning Models for Credit Card Fraud Detection

The rise of digital payments has led to increased credit card fraud, causing significant financial losses. Due to the rarity of fraudulent transactions, datasets are highly imbalanced, making detection a major challenge for machine learning models, which often struggle to accurately detect these anomalies. Continuing research aims to improve the models. Therefore, selecting the right classifiers and sampling techniques is vitally important to build an effective model. Typically, the original dataset is split into training and test sets. Sampling techniques are applied only to the training set to prevent data leakage and preserve the integrity of the test set. This study presents a comparative analysis of model’s performance using both the original and resampled datasets split, evaluated across multiple classifiers and sampling strategies. The study utilizes a publicly available credit card fraud detection dataset. While models trained and tested on resampled dataset show higher accuracy, this improvement often diminishes when evaluated against the original test set. Resampling techniques, though useful, come with limitations. Findings emphasize the importance of maintaining an untouched test set from the original dataset to ensure reliable model evaluation and produce trustworthy performance metrics. This study highlights that using an unaltered test set is essential for achieving realistic and generalizable model performance.