Training–Rollout Mismatch in Low-Precision LLM Reinforcement Learning: A Survey

Low-precision rollouts can reduce the cost of reinforcement learning for large language models, but numerical discrepancies between the sampler and learner can compromise policy optimization. Even with synchronized parameters and identical token histories, the two execution paths may assign different token probabilities, altering gradient estimation and update decisions. This survey examines how training–rollout mismatch arises, propagates through policy optimization, and evolves over training. We develop a component-level taxonomy that organizes existing approaches by intervention target into two complementary families: numerical design, which aligns execution across learner and sampler paths or preserves numerical fidelity and update visibility; and learning under residual mismatch, which corrects gradient estimation or controls policy updates. We compare the mechanisms, information requirements, and residual errors of these interventions to explain their limitations and how they can be combined. We further synthesize measurement practices and experimental evidence across precision regimes, architectures, and training configurations, examining how numerical discrepancies affect optimization stability, policy improvement, and computational cost. We clarify how differences in policy pairs, metric definitions, and training horizons shape the interpretation of reported results. Open challenges include predicting harmful mismatch, reducing mitigation overhead, sustaining policy improvement over longer training, and comparing methods at matched quality or end-to-end cost.