As data volumes grow, the performance of predictive models can be adversely affected by the presence of largely irrelevant and redundant features, especially in high-dimensional, small-sample data, where the risk of overfitting becomes more pronounced. Feature selection, therefore, emerges as a critical component of the machine learning process, enabling improved predictive performance, stability, feature quality, and greater explainability, particularly in ``high-stakes'' domains. Existing feature selection approaches are numerous; however, ensemble learning-based feature selection methods remain comparatively scarce, despite their potential to improve predictive performance, stability, and generalizability. Because no single method consistently balances predictive performance, stability, and feature quality across diverse problem settings, we developed the HEFS-EXR method, a two-stage hybrid ensemble feature selection algorithm that leverages the strengths of multiple filter and wrapper feature selection methods, and automatically determines the feature subset size at each stage. We conducted experiments on eight synthetic and 22 real-world data sets, and our findings indicate that HEFS-EXR consistently produces compact feature subsets, achieving dimensionality reductions ranging from 14% to 93% across both synthetic and real-world data from diverse domains, without compromising predictive performance and, in many cases, improving it. The results also demonstrated improved stability, highlighting the method's robustness.
Bikaki et al. (Tue,) studied this question.