What question did this study set out to answer?

The aim is to enhance knowledge transfer from a teacher model to a student model using meta learning techniques.

January 1, 2022Open Access

BERT Learns to Teach: Knowledge Distillation with Meta Learning

Key Points

The aim is to enhance knowledge transfer from a teacher model to a student model using meta learning techniques.
Introduced MetaDistil, where the teacher model adapts based on student performance feedback.
Implemented a pilot update mechanism to improve alignment within meta learning algorithms.
Conducted experiments on various benchmarks to validate the effectiveness of MetaDistil.
MetaDistil showed significant performance improvements over traditional knowledge distillation algorithms.
The method was less sensitive to variations in student model capacity and hyperparameters.
Facilitated broader application of knowledge distillation across different tasks and models.

Abstract

We present Knowledge Distillation with Meta Learning (MetaDistil), a simple yet effective alternative to traditional knowledge distillation (KD) methods where the teacher model is fixed during training. We show the teacher network can learn to better transfer knowledge to the student network (i.e., learning to teach) with the feedback from the performance of the distilled student network in a meta learning framework. Moreover, we introduce a pilot update mechanism to improve the alignment between the inner-learner and meta-learner in meta learning algorithms that focus on an improved inner-learner. Experiments on various benchmarks show that MetaDistil can yield significant improvements compared with traditional KD algorithms and is less sensitive to the choice of different student capacity and hyperparameters, facilitating the use of KD on different tasks and models.

KI fragen

Bookmark

View Full Paper