HEFS-EXR: A Robust Hybrid Ensemble Feature Selection Algorithm

As data volumes grow, the performance of predictive models can be adversely affected by the presence of largely irrelevant and redundant features, especially in high-dimensional, small-sample data, where the risk of overfitting becomes more pronounced. Feature selection, therefore, emerges as a critical component of the machine learning process, enabling improved predictive performance, stability, feature quality, and greater explainability, particularly in "high-stakes" domains. Existing feature selection approaches are numerous; however, ensemble learning-based feature selection methods remain comparatively scarce, despite their potential to improve predictive performance, stability, and generalizability. No single method consistently balances predictive performance, stability, feature quality, and explainability across diverse problem settings. These gaps motivated the development of the HEFS-EXR method, a two-stage hybrid ensemble feature selection algorithm that leverages the strengths of multiple filter and wrapper feature selection methods, and automatically determines the feature subset size at each stage. We conducted experiments on 8 synthetic and 22 real-world datasets, and our findings indicate that HEFS-EXR consistently produces compact feature subsets, achieving dimensionality reductions ranging from 14% to 93% across both synthetic and real-world data from diverse domains, without compromising predictive performance and, in many cases, improving it. The results also demonstrated improved stability, highlighting the method's robustness.