Focus on the Optimization of the RLHF Algorithm to Enhance the Training Effect After LLM

Post-training alignment of large language models is crucial for ensuring their safety, usefulness, and alignment with human preferences. Although reinforcement learning from human feedback (RLHF) is the mainstream approach, its training process is susceptible to issues such as reward model noise, unstable policy optimization, and catastrophic forgetting, leading to deviations between model outputs and true human preferences. This paper systematically reviews the optimization algorithms for enhancing the stability of RLHF in the past three years and categorizing them into three types: “Modifying signals”, aiming to improve the reliability of the reward signal; “Optimizing algorithms”, focusing on enhancing the performance of the reinforcement learning model to increase its noise resistance; “Developing Models”, reconstructing the training framework and multi-module collaboration from a system perspective to achieve fundamental optimization. This paper elaborates on the principles, representative algorithms, and advantages and disadvantages of each category, and conducts a horizontal comparison and analysis. This review aims to provide researchers with a clear algorithm classification framework and selection reference, while also pointing out the common challenges in current research, such as reward model bias, system implementation complexity, and generalization ability, in the hope of promoting subsequent research progress.