LSTM-RF Stock Prediction Algorithm via Short-Term Directional Probability-Based Model Selection

This paper proposes a hybrid machine learning algorithm that enhances stock price prediction accuracy by selecting the optimal model based on the predicted probabilities of short-term upward and downward trends. First, a long short term memory (LSTM) network and a random forest (RF) model are employed to forecast the next-day closing price. Then, based on each model’s statistical performance in predicting upward (HR+) and downward (HR−) trends over the preceding 60 trading days, the optimal model is selected, and the ultimate forecast is determined accordingly. Experimental results based on nine stocks from the Shanghai and Shenzhen Stock Exchanges, covering the period from January 1, 2018 to December 31, 2023, demonstrate that the proposed method outperforms RF, CNN, LSTM, GRU, CNN-LSTM, LSTM-RNN, LSTM-GRU, and AE + LSTM models. Specifically, it achieves superior performance in direction accuracy metrics (HR, HR+, and HR−), with overall HR improving by approximately 2–5% and MAPE decreasing by about 1–2%. Furthermore, the results indicate that the LSTM model performs better in upward trend prediction, while the RF model is more effective in downward trend prediction. In addition, the tanh activation function is found to outperform ReLU in deep learning models for stock prediction. These findings suggest that the proposed algorithm has practical value for the research on stock investment-related algorithms.