Answers are grounded in this paper and cite their sources.
Empirical evaluation reveals capacity-gated computational friction from redundant and decorative inputs in vision-language models, indicating that non-converging modalities actively degrade...
The multimodal-learning benefit is standardly credited to the crossing of modalities; on a vision-language substrate the active ingredient is route convergence, and input that converges on nothing is a measurable cost rather than a neutral addition. A redundant congruent image adds nothing at matched text — an absent redundancy gain where the multisensory literature reports a robust redundant signals effect — across three independent designs and two model families (Qwen2-VL-2B and Idefics2-8B, a distinct SigLIP+Mistral encoder), while a decorative out-of-domain image attached to a sufficient text question raises the teacher-forced negative log-likelihood of the target (p = 0.001), graded by how far out of domain the decoration is. That cost decomposes, and what governs it is located: a generic token-contention component that scales with the number of irrelevant tokens (a matched-length span of null text costs too), plus a vision-specific increment — and its amplitude is capacity-gated, collapsing roughly tenfold from a 2B to a 7B model of the same family and near-absent on an 8B model with a distinct encoder. The larger model attends to the irrelevant image as much as the smaller one and pays almost no cost. Two further results: an installed image→name association does not compose into image→value retrieval at this scale, a clean dissociation; and more distinct encoding routes confer retention under subsequent interference, not depth alone. Two auxiliary probes extend the picture — an expertise-graded help-then-small-cost pattern consistent with expertise reversal, and, on a harder relational task, structural co-location rather than modality as what lets the model answer what serial prose cannot. Read through Friction Theory, the redundancy, split-attention and seductive-details effects of cognitive load theory, and load-theoretic distractor cost in attention, are recovered here as measurable friction in a non-biological substrate. The unifying mechanism — a non-converging input always starts a dismissal race, and whether that race measurably costs the target retrieval is capacity-gated — is adopted as an interpretive lens: the results are race-consistent, and the race model inequality that would separate a race from coactivation is a response-time test the substrate-native readout cannot run. Companion work: the per-token measurement standard this readout's theoretical companion comes from (10.5281/zenodo.20012654), the wider-track encoding result assumed as background (10.5281/zenodo.20059859), and the two cross-substrate siblings asking which friction signatures survive, in social-moral judgement (10.5281/zenodo.20678294) and in perceptual context effects (10.5281/zenodo.20678296). Target venue: Transactions on Machine Learning Research (TMLR).
No takes yet. Share an insight, caveat, or question.
Tomas Pødenphant Lund (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: