An autoregressive lag aware sparse transformer framework for demand forecasting

Abstract While traditional statistical methods have proven effective for demand forecasting in steady environments, they struggle with dynamic and non-stationary data. The rise of ML and DL models has provided more robust alternatives, with RNNs and LSTM networks showing significant promise in capturing temporal dependencies. However, these models often face challenges in efficiently capturing dominant temporal patterns in complex data. Recently, transformer-based models, leveraging self-attention mechanisms, have revolutionized time series forecasting by effectively handling long-range dependencies in data. This paper introduces Vaticinator, a statistically significant lag-aware sparse transformer-based framework. By combining the strengths of transformers and statistical methods, Vaticinator provides a principled approach to forecast demand fluctuations in complex, dynamic environments. On our chosen sales dataset with monthly aggregations, Vaticinator achieves a normalized RMSE of 0.23 and a MAPA of 97.59%, performing competitively against traditional statistical, ML and DL approaches evaluated on the same data. The proposed approach aims to improve forecasting reliability and interpretability, while reducing computational overhead associated with dense attention, thereby addressing key limitations of both traditional and modern forecasting techniques.