What question did this study set out to answer?

To assess the effects of language and auxiliary rewards on math reasoning performance post-GRPO training.

May 13, 2026Open Access

Beyond English-Only GRPO: Training Language and Auxiliary Reward as Implicit Regularizers in Sub-3B Math Reasoning

Key Points

To assess the effects of language and auxiliary rewards on math reasoning performance post-GRPO training.
Multi-seed empirical study with four experimental arms: English-only, Vietnamese-translated, English with auxiliary language-consistency reward R5, and constant-bias ablation.
Evaluation across three random seeds and four math benchmarks, under a single-GPU LoRA constraint.
Comparison of performance metrics including AIME-2024 maj@8 effects and seed variance on AMC-23.
R5 arm yielded the only positive AIME-2024 maj@8 effect (+4.4±1.9 pp over base).
English GRPO showed σ=11.3 pp seed variance on AMC-23, indicating unreliable single-seed claims.
10-language MGSM evaluation showed negligible cross-lingual transfer, with conditions within ±0.5 pp of baseline.

Abstract

A multi-seed empirical study of GRPO post-training of DeepSeek-R1-Distill-Qwen-1.5B under a single-GPU LoRA constraint. We compare four arms (English-only, Vietnamese-translated, English with auxiliary language-consistency reward R5, and a constant-bias ablation) across three random seeds and four math benchmarks. Three findings: (i) R5 arm achieves the only positive AIME-2024 maj@8 effect (+4.4±1.9 pp over base); (ii) vanilla English GRPO shows σ=11.3 pp seed variance on AMC-23 with consistent AIME degradation, confirming single-seed claims at this scale are unreliable; (iii) 10-language MGSM evaluation shows null cross-lingual transfer — all conditions within ±0.5 pp of base. Constant-bias ablation does not fully reproduce R5 effect

Perguntar à IA

Bookmark

View Full Paper