Asymmetric Multimodal Learning with Decomposed JEPAs

Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a sole objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom (Z; X₁, X₂), annihilating private modality-specific structure (Z; Xₘ) and cross-modal synergy (Z; X₁, X₂). We introduce Decomposed JEPA (D-JEPA), which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.