Development of Hybrid Deep Learning and Reinforcement Learning for Intelligent Trading in Forex Markets

Forex trading, a $6 trillion-a-day market, remains challenging to work with due to its non-linearity, volatility, and noise. Most traditional approaches, such as ARIMA and SARIMA, fail to capture these dynamics and therefore tend to have low forecasting accuracy. In this study, we propose a hybrid model combining a Convolutional Neural Network–Long Short-Term Memory (CNN-LSTM) network for Forex price forecasting and Deep Q-Learning (DQL) for trading strategy optimization. The CNN-LSTM model learns spatiotemporal features from historical data, evaluated using MSE, RMSE, and MAE. For trading decision-making, a DQL agent learns to maximize actions—buy, sell, or hold—through reward-based learning with epsilon-greedy exploration, experience replay, and target networks. Results showed that the CNN-LSTM model outperforms traditional models, achieving lower RMSE (0.0025) and MAE (0.0017). The DQL agent achieved a cumulative return of 49.2% and a Sharpe ratio of 2.87, surpassing rule-based methods. An ablation analysis confirmed the necessity of key components, such as experience replay and target networks, for stable learning. Statistical tests, such as the Diebold-Mariano test, further supported the predictive strength of the model. The hybrid model showed strong potential for real-time Forex trading, offering high precision and strong risk-adjusted returns. Future studies should incorporate macroeconomic variables, sentiment analysis, and multi-asset portfolios to enhance generalization and trading performance.