A systematic evaluation of the reliability of feature selection methods and SHAP-based interpretability in machine learning models for arrhythmia analysis

Arrhythmia is one of the most prevalent cardiovascular diseases worldwide. The classification of arrhythmias plays a major role in the diagnosis of heart disease. Automated classification of electrocardiogram (ECG) signals using machine learning has emerged as a promising approach for the detection of arrhythmias. However, the comparative effectiveness of diverse classifiers and feature selection methods across different ECG datasets, along with statistical validation and interpretability assessment, remains limited. We systematically evaluated multiple machine learning models, including Logistic Regression, SVM, KNN, Decision Tree, Random Forest, XGBoost, and LightGBM across two datasets, Arrhythmia Dataset from UCI and for external validation of methodological pipeline we used Heterogenous Datset from Mendeley Data. Feature selection strategies (Boruta, RFECV, LASSO) were applied to reduce dimensionality, and stratified k-fold cross validation was employed to ensure robustness. Statistical significance was evaluated using the Friedman test on F1-scores across stratified cross-validation folds, followed by post-hoc analysis for pairwise model comparisons. Feature stability was examined across folds, and overlap between SHAP-derived important features and selected feature subsets was quantified to evaluate interpretability consistency. Computational efficiency was assessed using training and inference time analysis. Ablation studies assessed model performance at different feature proportions (0, 0.25, 0.5, 0.75, 1.0), and SHAP analysis was conducted to identify the most influential predictors. Ensemble methods (Random Forest, XGBoost, LightGBM) and SVM consistently achieved superior performance across both binary and multiclass tasks. Feature selection enhanced model efficiency, with LASSO and Boruta yielding stable results, while RFECV demonstrated both overfitting risks and unique feature insights. Stratified k -fold cross-validation confirmed the robustness of high-performing models, with accuracies remaining stable across folds. Ensemble-based models and SVM, combined with feature-efficient approaches and interpretable frameworks, offer reliable and clinically meaningful solutions for ECG classification. The combined evaluation of performance, statistical ranking, feature stability, interpretability overlap, and computational efficiency strengthens the methodological reliability of machine learning to support automated ECG analysis across diverse datasets.