State-Dependent Value of News Sentiment in S&P 500 Direction Forecasting

Next-day S&P 500 direction forecasting matters for allocation, hedging, and risk management because broad-index movements transmit quickly across portfolios. Does structured news sentiment help predict next-day S&P 500 direction? We test four feature sets over 2008–2023 in an ablation sequence: technical indicators only (Set A), with FinBERT headline sentiment (Set B), with BERTopic topic-linked sentiment (Set C), and with realized-volatility weighting (Set D). This design makes two contributions: it separates the incremental value of increasingly structured sentiment features, and it tests whether sentiment value is state-dependent across volatility regimes. CatBoost, XGBoost, LightGBM, LSTM, and GRU are evaluated under walk-forward cross-validation, nested cross-validation, and formal statistical tests. On the full sample, sentiment does not deliver a measurable forecasting edge. Walk-forward AUCs sit near 0.50 for every feature set, and pairwise tests find no significant differences. However, this average masks a consistent pattern. Sentiment becomes more informative during high-volatility periods, suggesting that its value is state-dependent rather than uniform. Rolling AUC swings from 0.28 to 0.71 depending on the market period. When we split by VIX regime, Set D reaches 0.5684 AUC during high-volatility episodes (n=50, permutation p=0.213) while adding almost nothing in calm markets. Set D also has the lowest fold-to-fold variance and the shallowest drawdown in trading simulations. These results imply that the relevant question is not whether sentiment works in general, but when it does. Sentiment does not help on average; whether it helps during stress is suggestive but unconfirmed and needs more crisis-period data to settle.