The Impact of Knowledge Distillation on the Energy Consumption and Runtime Efficiency of NLP Models

Context. While models like BERT and GPT are powerful, they require substantial resources. Knowledge distillation can be employed as a technique to enhance their efficiency. Yet, we lack a clear understanding on their performance and energy consumption. This uncertainty is a major concern, especially in practical applications, where these models could strain resources and limit accessibility for developers with limited means. Our drive also comes from the pressing need for environmentally-friendly and sustainable applications in light of growing environmental worries. To address this, it is crucial to accurately measure their energy consumption. Goal. This study aims to determine how Knowledge Distillation affects the energy consumption and performance of NLP models. Method. We benchmark BERT, Distilled-BERT, GPT-2, and Distilled-GPT-2 using three different tasks from 3 different categories selected from a third-party dataset. The energy consumption, CPU utilization, memory utilization, and inference time of the considered NLP models are measured and statistically analyzed.