SpanPatchTST: A self-supervised learning framework for short- and mid-term forecasting of power transformer load and oil temperature

Accurate short- and mid-term forecasting of transformer load and oil temperature enables improved operational reliability, preventive cooling, and efficient maintenance planning. This paper proposes SpanPatchTST, a novel self-supervised Patch Transformer that effectively captures the complex coupling relationships between different types of electric loads and oil temperature. First, SpanPatchTST divides multivariate time series data into multiple univariate time series, and segments each into patches of equal length. A subset of the patches is randomly masked using our proposed contiguous masking strategy. SpanPatchTST processes the unmasked patches using a modified Transformer encoder, incorporating advanced techniques such as talking-head attention, rotary position encoding, hybrid normalization, and sequential sampling, and is trained to reconstruct masked patches from the representations learned from the unmasked ones. After validating its reconstruction performance, SpanPatchTST is fine-tuned on limited downstream data to perform short- and mid-term forecasting of electric loads and oil temperature. SpanPatchTST outperformed several state-of-the-art Transformer-based and linear models across four real-world benchmark datasets and multiple forecasting scenarios. Extensive ablation studies and hyperparameter sensitivity analyses confirmed the robustness of SpanPatchTST and demonstrated the effectiveness of its architecture. Additionally, its patch-based attention mechanism enables efficient deployment in resource-constrained environments. For reproducibility, the code is publicly available at https://github.com/lear-ner97/SpanPatchTST