Transforming Image-Structured Financial Time Series for the Joint Model of Return and Covariance

Accurate forecasting of financial returns and their associated risks is fundamental to effective risk measurement, portfolio construction, and financial decision-making. However, expected returns and covariance structures are often modelled separately, potentially overlooking their interdependence and evolving cross-asset relationships. Motivated by the analogy between multi-asset financial time series and video data, this paper proposes an image-structured representation for jointly forecasting next-day mean returns and covariance matrices using the Time-Space Transformer (TimeSformer). Financial features are organised as temporal sequences of matrix-structured frames, enabling the model to learn temporal and cross-asset dependencies simultaneously. We evaluate shared two-head and separate two-tower architectures with multiple probabilistic and non-parametric loss functions for return and covariance prediction, while ensuring positive-definite covariance forecasts through Cholesky parameterisation. Empirical results demonstrate the proposed covariance-informed TimeSformer models substantially outperform the Vanilla Transformer and LSTM baselines across the principal mean-return and covariance forecasting measures, although the zero-return forecast and covariance persistence remain strong benchmarks for mean and covariance prediction, respectively. Incorporating historical covariance structure directly into the input further improves both forecasting tasks and yields the strongest covariance performance among the learned models. Diebold–Mariano tests confirm that several of these improvements are statistically significant, particularly for covariance forecasting and the covariance-informed representations. Economic evaluation of portfolio optimisation using global minimum variance portfolios further shows that covariance-informed TimeSformer models achieve strong risk-adjusted performance, with the best configuration attaining the highest Sharpe ratio among the evaluated models. Overall, the results demonstrate that spatiotemporal attention, task-specific objectives, and covariance-informed representations provide an effective framework for jointly modelling multi-asset return and risk dynamics.