SHAP-Based Recursive Feature Elimination and Hyperparameter Optimization for Enhanced Financial Stock Forecasting

Financial time series forecasting faces two critical challenges: feature noise degrades model performance while the "black box" nature of deep learning models limits practical adoption in regulated environments. Existing approaches usually address feature selection and model optimisation separately, without considering temporal consistency in changing market conditions. This paper introduces SHAPRFE-LSTM: an integrated framework combining SHAP-based recursive feature elimination, LSTM networks and Optuna hyperparameter optimisation. Our approach employs three core innovations. The process of recursive feature elimination employs the use of SHAP values in order to systematically remove features that contribute noise. Optuna's Tree-structured Parzen Estimator facilitates the efficient search for hyperparameters. Temporal cross-validation employs weighted feature scoring to ensure robustness across different market regimes. The temporal cross-validation design addresses a key limitation in existing work by ensuring selected features maintain stable predictive importance across multiple market conditions. Rather than treating different time periods equally, our weighting mechanism gives greater emphasis to more recent market data while preserving historical patterns for robustness. Comprehensive experiments on EEM and DIA datasets validate the framework's effectiveness. SHAPRFE-LSTM achieves RMSE reductions of 45.9% and 61.5% respectively compared to baseline LSTM models. These improvements demonstrate that integrated feature selection and hyperparameter optimization can significantly enhance forecasting accuracy while providing the interpretability required for practical financial applications.