A novel multimodal framework for ultra-short-term photovoltaic power forecasting via fusion of sky images and time-series data

This is the authors' abstract. We don't add key points for this paper.

Accurate forecasting of photovoltaic (PV) power generation is of great importance for reducing the uncertainty of distributed PV systems in power grid scheduling and enhancing operational safety and stability. Under variable weather conditions, PV output experiences significant fluctuations caused by changes in cloud patterns in the sky and the combined effects of multiple meteorological factors, thereby increasing the complexity of forecasting. To address these challenges, this study proposes a multimodal PV power forecasting framework, denoted as SFT, that integrates sky images and meteorological time-series. The framework consists of an improved Swin Transformer-based image encoder, a Temporal Fusion Transformer (TFT) branch for time-series modeling, and a gated-attention fusion module. The time-series branch encodes static, meteorological and PV power data, while the image branch converts sky images into numerical feature sequences using physics-based methods and introduces a one-dimensional Swin Transformer to learn deep representations. The fusion module adopts gated-attention to adaptively reweight and integrate the two modalities, ultimately outputting the PV power forecasts. Experimental results show that the proposed method achieves a mean absolute error (MAE) of 0.580 kW in the 10‑minute forecasting task, significantly outperforming baseline models and demonstrating superior accuracy and robustness.