Uncertainty-Aware and Adaptive Constraint ReinforcementLearning Framework for Language Models (UAC-RL)

In this work, I propose UAC-RL, a reinforcement learning framework that addresses what I see as the fundamental contradiction in current pure RL approaches like DeepSeek-R1-Zero: the claim of "no human annotation" actually hides massive data cleaning costs. Having spent countless hours thinking about how to reduce reliance on human labor in AI training, I developed four key improvements to the GRPO algorithm. First, I designed an intra-group balancing function to prevent extreme reward values from dominating the learning process—a problem I've observed in practice. Second, I added a lightweight uncertainty head that outputs confidence scores based on Gaussian distribution, allowing the model to know what it doesn't know. Third, and most importantly, I created an endogenous safety constraint mechanism for MoE architectures, where each expert learns to evaluate and constrain its own outputs through shared safety representations. This eliminates the need for external safety classifiers. Fourth, I introduced an uncertainty calibration loss to ensure the model's uncertainty estimates actually reflect real errors. I also developed an adaptive λ scheduling mechanism that automatically balances these constraints during training, and an exploration reward that prevents the model from becoming overly conservative. Theoretical analysis shows game-theoretic guarantees, convergence properties, and regret bounds. The framework maintains or improves reasoning performance while reducing dependence on meticulously cleaned datasets—exactly the kind of efficiency I've been aiming for.