A Comparative Analysis of Tree-Based Ensembles and Deep Learning for Equity Price Forecasting Using Feature-Engineered Advance Technical Indicators

Technical indicators are widely used by researchers and investors to predict the stock market. Finding the ideal quantity of technical indicators to use is essential for making accurate stock market predictions. In order to forecast short-term stock price fluctuations, this study methodically assesses the combination of sophisticated technical indicators and cutting-edge machine learning algorithms. The study developed a strong set of features, such as volatility-normalized Average True Range, RSI derivatives, VWAP-based measures, and Hull Moving Average, to be used as inputs for several models using 10 years of data (2014–2024) from ten randomly chosen NIFTY 50 stocks. To ensure economically significant signals, a binary classification target was established to forecast if the five-day ahead return will above a volatility-dependent threshold. Using a temporal train-validation-test split to ensure out-of-sample robustness and prevent look-ahead bias, the experiment contrasted a deep learning LSTM network, tree-based models (Random Forest and XGBoost), and a baseline dummy classifier. With the lowest maximum drawdown across stocks and the greatest average Sharpe ratio (1.72), the results showed that XGBoost performed noticeably better than any other model. It was especially successful because of its consistency and capacity to identify intricate, non-linear relationships in the feature-engineered tabular data. Despite being competitive in some situations, the LSTM performed worse overall and had larger variation, most likely as a result of the small dataset size and the natural benefit of using tree-based techniques with structured financial data. According to the study's findings, technically derived features have a great deal of predictive power, but their effectiveness is enhanced when they are processed using potent ensemble techniques like XGBoost, which provide better performance and robustness for real-world trading applications than more intricate deep learning techniques.