Language JEPA for Reasoning: Predicting Answer Embeddings from Questions and Paraphrase Pairs

Autoregressive language models are trained to predict tokens, which couples the objective to surface form. We study a language joint-embedding predictive architecture that predicts the embedding of an answer from the embedding of a question, and that uses paraphrase pairs as an additional signal. We compare it with an autoregressive objective on arithmetic and multi-step reasoning tasks, and on a synthetic arithmetic setting where the correct answer is known by construction. A symbolic pilot shows that a small MLP readout fails beyond depth 2; the language JEPA results are pending. Preprint. Results in this version come from small-scale pilot experiments; the full experimental protocol is described in the text and is pending.