CIMTS: candlestick-integrated multi-modal trading system for stock indices

Stock price analysis based solely on technical indicators, such as in traditional stock price prediction studies, relies heavily on observations from individual sample markets and fails to consider insights from other modalities, making it imperative to study the implicit interdependence between different modalities. A Candlestick-Integrated Multi-Modal Trading System (CIMTS) is proposed that integrates 1D numerical time-series with 2D spatial candlestick chart-image data. The system analyzes multimodal data by employing Long Short-Term Memory and Vision Transformer models in parallel, identifying graphical signals of stock prices using a late fusion approach to merge the outputs from the two models. The system was tested over an out-of-sample period of approximately 3 years, partitioned from a 12-year dataset of key US and Korean stock indices, including the NASDAQ Composite, S&P500, KOSPI, and KOSDAQ, under different market conditions. The experimental results showed that the CIMTS outperformed single-modality models in terms of aggressiveness ( i.e ., higher cumulative returns and a greater number of trades) and stability ( i.e ., lower maximum drawdown and consistent win rates). In particular, the CIMTS outperformed the labeling method that considers next-day movements in the Korean market and the 5-day consecutive trend in the US market. The proposed system achieved cumulative returns of 42.52%, 51.37%, 39.36%, and 95.47% for NASDAQ, S&P500, KOSPI, and KOSDAQ, respectively (payoff ratios: 1.023, 0.877, 1.166, and 1.195; profit factors: 1.364, 1.638, 1.306, and 1.890, respectively), demonstrating consistently improved trading performance over the Buy & Hold benchmark across all tested market regimes.