Collapse-Resistant Joint-Embedding Predictive Architectures: A Controlled Comparison of SIGReg, VICReg, and EMA

Joint-embedding predictive architectures (JEPA) learn representations by predicting the embedding of a target view from a context view, which makes them prone to representation collapse. Several stabilizers have been proposed, including exponential moving averages (EMA), variance-invariance-covariance regularization (VICReg), and isotropic Gaussian regularization (SIGReg). Comparisons in the literature often rely on single runs and differ in training budget. We propose a controlled protocol that compares these three stabilizers under identical architectures, budgets, and at least five random seeds, and that measures collapse with an intrinsic-dimension estimate on known synthetic manifolds and on sentence-embedding benchmarks. On synthetic manifolds, all three methods recover low intrinsic dimensions, but all underestimate at d=32, where VICReg retains the most embedding rank. Vision and language results are pending. Preprint. Results in this version come from small-scale pilot experiments; the full experimental protocol is described in the text and is pending.