Adaptive temperature distillation with curriculum learning for lightweight multilingual language teaching models

Knowledge distillation can compress large multilingual models into lightweight alternatives; however, existing approaches employ a uniform temperature setting that does not account for the varying characteristics of different languages. This limitation results in suboptimal knowledge transfer, particularly for low-resource languages. We present an adaptive temperature distillation framework with curriculum learning for multilingual knowledge transfer. We introduce (1) a learnable temperature predictor that dynamically adjusts the softmax temperature according to language resource availability and sample difficulty, by assigning higher temperatures to low-resource languages to encourage generalization and lower temperatures to high-resource languages to preserve precise decision boundaries; (2) a curriculum learning strategy that organizes training progressively from high-to-low-resource languages and easy-to-hard samples. We evaluate our framework on 5 benchmarks covering more than 40 languages. Our method achieves an average accuracy of 84.1%, corresponding to 97.8% of the teacher model's performance (XLM-R-Large), while using only 11.8% of its parameters (66 M vs. 560 M) and running at 8.5× faster inference speed. Compared to standard distillation, we achieve + 4.6% improvement overall and up to + 7.8% for low-resource languages. Ablation studies confirmcomplementary contributions: adaptive temperature alone yields + 3.2%, curriculum learning alone yields + 2.9%, and their combination achieves + 4.8%, exceeding either individual component. The framework also reduces the number of training epochs required for convergence by approximately 40% and improves zero-shot cross-lingual transfer by + 6.0%, demonstrating its effectiveness for multilingual model compression. It should be noted that the current framework is evaluated exclusively on discriminative tasks; its applicability to generative tasks such as machine translation and abstractive summarization remains an avenue for future work.