Layer-Wise Attention with Pivot Layers for Effective Fine-Tuning of Encoder-Based Language Models

Fine-tuning pre-trained encoder-based language models for down-stream tasks is typically performed by exploiting the output of the last encoder layer. However, an alternative line of research suggests that leveraging representations from multiple encoder layers may yield richer linguistic information. Previous studies found that different layers convey different linguistic knowledge, suggesting that the last layer might not be optimal for all down-stream tasks. In this paper, we propose a layer-wise attention mechanism using a pivot layer as a new fine-tuning method. The pivot layer is used to compute attention scores of encoder layers, and we define three types of pivot layers. We also examine four attention functions and demonstrate through experiments that the attention function plays an important role in layer-wise attention for fine-tuning. The best-performing combination of our proposed mechanism outperformed the standard fine-tuning method and other recent methods in the General Language Understanding Evaluation (GLUE) benchmark. By visualizing the attention distributions, we found that the last layer is not always preferable for every GLUE benchmark task, and that differences in attention distribution are associated with task performance.