O²PD: Overlap-Gated On-Policy Distillation on Reliable Prefix

On-policy distillation (OPD) supplies dense token-level supervision by aligning a student to the teacher's next-token distribution on its own generated rollouts. Existing works mostly implement top-k OPD, applying reverse Kullback-Leibler () divergence between the teacher's and the student's renormalized distributions over teacher top-k support. However, the teacher scores student-generated rollouts that are inherently off-policy for itself, leaving the reliability of its next-token guidance unclear. Restricting the divergence to the teacher top-k support also discards the remaining mass outside that support. These gaps leave two questions implicit: 1) which tokens to align, and 2) how to align those tokens. To answer these questions, we propose Overlap-gated On-Policy Distillation (O²PD). As for 1), we quantitatively analyze the reliability of teacher supervision and find that alignment on the student's trajectory prefix is more reliable than alignment on the full context. Therefore, our O²PD is applied only to the 20% prefix tokens of each student rollout. As for 2), we theoretically characterize two problems of top-k OPD: the Teacher-support Uncovering and the Dominant-Mode Mismatch. Based on these analyses, we propose two support strategies: the Reset Support strategy attaches a reset atom to every token, pooling mass outside the teacher top-k, and the Support Convergence strategy converges the support to the already aligned intersection. Our O²PD employs the designed overlap-gated $α$-divergence to interpolate between mode-covering and mode-seeking for per-token alignment, where the overlap is used as a gate to switch the support and adaptively modulate $α$.