Reasoning Across Modalities: A Chain-of-Modality Framework for Forecasting with Structured Data, Temporal Trends, and Strategic Text

The integration of heterogeneous modalitiesspecifically structured tabular data, temporal sequences, and domain-specific textpresents significant challenges for predictive modeling. Conventional large language models (LLMs) inherently lack mechanisms to reconcile the semantic and structural disparities among these data types. To bridge this gap, we proposeChain-of-Modality (CoM), a novel prompt engineering framework that enables pretrained LLMs to perform interpretable, sequential reasoning across modalities without requiring architectural modifications. CoM leverages lightweight modality-specific encoders to transform structured attributes and time-series patterns into natural language segments. These segments are semantically ordered (e.g.,Structured Temporal Textual) and composed with textual inputs to form a cohesive reasoning chain. This design emulates human analytical workflows, preserving modality-specific inductive biases while harnessing LLMs' generative capabilities. Fine-tuned viaparameter-efficient methods (LoRA), CoM achieves state-of-the-art performance: it reduces financial volatility prediction error by12.1% (RMSPE)compared to early fusion approaches and improves user behavior classificationAUC by 3.5%over text-only LLMs. Ablation studies confirm the essential contributions of all modalities and validate the significance of semantic ordering. Our work establishes a foundation for extensible, interpretable multimodal forecasting with minimal engineering overhead.