What does this research mean for the field?

In sub-3B math reasoning models under LoRA constraints, vanilla GRPO exhibits high seed variance and degrades performance, whereas adding an auxiliary language-consistency reward acts as an implicit regularizer that robustly improves reasoning performance. Novelty: ClaimNovelty.CONFIRMATORY. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

To investigate the effects of language-consistency rewards in improving math reasoning performance in models with limited capacity.

May 18, 2026Open Access

View Full Paper

Beyond English-Only GRPO: Training Language and Auxiliary Reward as Implicit Regularizers in Sub-3B Math Reasoning (v2.1)

DVDang Vu

Key Points

To investigate the effects of language-consistency rewards in improving math reasoning performance in models with limited capacity.
Compared four groups: English-only, Vietnamese-translated, English with auxiliary reward, and a constant-bias control across multiple seeds and tasks.
Used empirical data from AMC-23, MATH-500, AIME-2024, and a 10-language MGSM for assessment.
Only the group with the auxiliary language-consistency reward (A3) showed a significant improvement in AIME-2024 scores (+4.4 pp over base) across seeds.
English GRPO demonstrated notable variance and degraded performance in AIME-2024 for three seeds, signaling unreliability at the sub-3B + LoRA scale.
All conditions in the 10-language MGSM struggled to beat the untrained base performance, indicating no effective cross-lingual transfer.

Abstract

Multi-seed empirical study of GRPO post-training on DeepSeek-R1-Distill-Qwen-1.5B under a single-GPU LoRA (r=16) constraint. Four arms — English-only (A1), Vietnamese-translated (A2), English with auxiliary language-consistency reward R5 (A3), and a content-free constant-bias control (A4) — are compared across three seeds on AMC-23, MATH-500, AIME-2024, and 10-language MGSM. Three orthogonal findings: (i) only A3 produces a positive AIME-2024 maj@8 effect (37.8 ± 1.9%, +4.4 pp over base, robust across seeds); (ii) vanilla English GRPO shows σ = 11.3 pp seed variance on AMC-23 and degrades AIME-2024 pass@1 in 3/3 seeds (mean −12.2 pp), confirming that single-seed claims at sub-3B + LoRA scale are unreliable; (iii) on 10-language MGSM all four conditions converge to within ±0.5 pp of the untrained base — null cross-lingual transfer at this scale. The A4 constant-bias control does not reproduce the R5 effect (A3 − A4 = +5.58 pp on AIME-2024 maj@8, 95% bootstrap CI +1.13, +11.10), ruling out a pure reward-magnitude explanation.

Perguntar à IA

Bookmark

View Full Paper

Perguntar à IA

Bookmark

View Full Paper

Beyond English-Only GRPO: Training Language and Auxiliary Reward as Implicit Regularizers in Sub-3B Math Reasoning (v2.1)

Key Points

Abstract

Cite This Study

Also Consider

Also Consider