Towards Unified Evaluation of Continual Learning in Spoken Language Understanding

Continual learning has emerged as an important challenge across various tasks, including Spoken Language Understanding (SLU). The evaluation of continual learning algorithms typically involves assessing the model's stability, plasticity, and generalizability. However, existing continual learning metrics primarily focus on only one or two of the properties. They neglect the overall performance across all tasks, and do not adequately disentangle the plasticity versus stability/generalizability trade-offs in the model. In this work, we propose an evaluation metric that provides a unified evaluation on stability, plasticity, and generalizability. By employing the proposed metric, we demonstrate how introducing various knowledge distillations can improve these three properties, and we apply it to the SLU model evaluation. We further show that our proposed metric is more sensitive in capturing the impact of task ordering in continual learning, making it better suited for practical scenarios.