Continuous Adaptive Knowledge Distillation for Few-Shot Relation Extraction

The goal of continuous few-shot relation extraction is to enable the model to continuously learn new relation types under conditions with limited labeled training data while avoiding the forgetting of previously learned relations. The primary challenges include catastrophic forgetting of old relations and overfitting due to data sparsity. To address these challenges, this paper proposes a hierarchical distillation model that innovatively combines contrastive distillation with orthogonal adversarial distillation techniques. Specifically, we introduce a contrastive distillation approach in the feature distillation layer, integrating adaptive cosine techniques and negative sampling strategies to ensure that the model effectively retains and utilizes knowledge from previous tasks when learning new ones. Additionally, we employ orthogonal adversarial distillation in the hidden distillation layer to alleviate overfitting in low-resource scenarios. Experimental results demonstrate that our proposed method significantly outperforms the current state-of-the-art models for continuous few-shot relation extraction on two benchmark datasets, validating its effectiveness in handling few-shot data and knowledge transfer.