Exploring and enhancing the transfer of distribution in knowledge distillation for autoregressive language models

Key Points

• Improved transfer of distribution leads to higher training efficiency in autoregressive models, enhancing their performance.
• Key evidence shows that fine-tuning sampling methods results in a notable efficiency increase during the training process.
• Analysis involved exploring various techniques to enhance knowledge distillation within autoregressive language models.
• These findings support the importance of optimized training methods, indicating further development may significantly impact model applications.