Abstract The collaboration between spatial encoders, such as CNNs and Vision Transformers (ViTs), and temporal modulators like LSTMs is fundamental to hybrid models, yet the dynamics of this interplay remain poorly understood. This study introduces a cross-domain framework, leveraging metrics from Information Theory and the Markov Information Bottleneck (MIB) principle, to quantify the internal information flows of these architectures. We analyze two distinct models—a CNN-LSTM for surveillance video and a ViT-LSTM for medical fMRI sequences—on both their native and foreign domains. Our analysis reveals two fundamentally different encoding strategies. The CNN employs a rigid, data-agnostic ‘Gradual Compression’ pipeline. The ViT, in contrast, demonstrates an ‘Adaptive Compression’ strategy, where the successful formation of an efficient information bottleneck is conditional on the data domain. We demonstrate that this adaptability is supported by a nuanced three-stage functional hierarchy within the ViT. Furthermore, we find the LSTM modulator’s role adapts to its partner: paired with the CNN, it acts as a conventional compressor, but when paired with the ViT, it switches to an ‘unpacker’ role to process the hyper-compressed representations it receives. We propose that this complex interplay exemplifies a principle of a ‘Dynamic Division of Labor,’ where functional roles are not fixed but emerge from the system’s interaction with the data. This insight challenges the static view of neural network components and opens a path toward more robust, context-aware AI.
Zvereva et al. (Mon,) studied this question.