What question did this study set out to answer?

This work investigates the necessity and redundancy of the Feed Forward Network in Transformer architectures.

January 1, 2023Open Access

One Wide Feedforward Is All You Need

Key Points

This work investigates the necessity and redundancy of the Feed Forward Network in Transformer architectures.
Explored the role of the Feed Forward Network (FFN) in Transformer models.
Reduced FFN parameters by removing it from decoder layers and sharing a single FFN across the encoder.
Scaled architecture by increasing the hidden dimension of the shared FFN.
Achieved substantial reduction in model parameters with only a modest drop in accuracy.
Gains in accuracy and latency observed after optimizing the FFN structure.
The modified architecture led to improved performance compared to the original Transformer Big.

Abstract

The Transformer architecture has two main non-embedding components: Attention and the Feed Forward Network (FFN). Attention captures interdependencies between words regardless of their position, while the FFN non-linearly transforms each input token independently. In this work we explore the role of the FFN, and find that despite taking up a significant fraction of the model’s parameters, it is highly redundant. Concretely, we are able to substantially reduce the number of parameters with only a modest drop in accuracy by removing the FFN on the decoder layers and sharing a single FFN across the encoder. Finally we scale this architecture back to its original size by increasing the hidden dimension of the shared FFN, achieving substantial gains in both accuracy and latency with respect to the original Transformer Big.

Perguntar à IA

Bookmark

View Full Paper