This is the authors' abstract. We don't add key points for this paper.
Recent studies show that the attention heads in Transformer are not equal. We relate this phenomenon to the imbalance training of multi-head attention and the model dependence on specific heads. To tackle this problem, we propose a simple masking method: HeadMask, in two specific ways. Experiments show that translation improvements are achieved on multiple language pairs. Subsequent empirical analyses also support our assumption and confirm the effectiveness of the method.
Answers are grounded in this paper and cite their sources.
No takes yet. Share an insight, caveat, or question.
Sun et al. (2020) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: