Reinforcement learning has become a central paradigm for post-training large language models , but prevailing RL-based post-training still relies heavily on sparse trajectory-level rewards and a passive reference policy that mainly regularizes against drift. A growing line of work expands this supervisory structure through on-policy self-distillation (OPSD), where a self-related or same-lineage teacher provides dense guidance on the student's own rollouts. The teacher's advantage may come from privileged runtime information, such as verified solutions, feedback, or hindsight hints, or from stronger checkpoints and specialist policy states produced during training. We present OPSD as a unified framework for supervisory mechanisms in RL-based post-training rather than only as knowledge compression. We organize this framework around two linked questions: where the teacher's advantage comes from, and what role the resulting signal plays in optimization. This framing separates informational source from optimization function, placing regularization, reward-anchored augmentation, and replacement-style supervision within one comparative picture. The formal connection is clearest for runtime-conditioned OPSD, where a stop-gradient teacher conditions on privileged training-time context while supervising the student's current on-policy rollouts. This formalization grounds OPSD as dense teacher-guided optimization coupled to RL, rather than as a separate post-hoc distillation step. Knowledge-and feedback-conditioned augmentation, selective teacher control, internalized teachers, and multi-teacher consolidation show that the practical design space is already broader, with performance depending on how privileged context is constructed, how teacher signal is represented and coupled to reward, and how the resulting system is stabilized. We conclude with open problems in adaptive teacher control, dynamic teacher construction, long-horizon and agentic supervision, and the benchmarks and scaling studies needed to make this design space cumulative.
Fu et al. (Fri,) studied this question.