Answers are grounded in this paper and cite their sources.
Demonstrates improved computational efficiency in transformer networks, suggesting significant advancements in processing ultra-long sequences.
Transformer networks have been increasingly successful in various fields. The input sequence lengths have become much larger as the algorithm and task complexity develops, which is challenging due to high computational and storage cost. Softmax and LayerNorm are bottleneck nonlinear operators in ultra-long sequence Transformer networks. To improve the efficiency of Softmax, assumption-based and quantization-based Softmax approaches are introduced. However, the sparsity potential to accelerate Softmax itself is not fully discovered. To improve the efficiency of LayerNorm, some works reduce the input size, and some works explore the pipeline. However, the sparsity potential is also not yet explored. To address these challenges, this article presents the ULSeq-TA software–hardware co-design framework. The software includes 1) the grouped sparse Softmax method to leverage the data magnifying characteristic to explore the middle and post-Softmax sparse processing and 2) the dual-path sparse LayerNorm method which explores the dimensional significance for sparse calculation. The hardware includes 1) an attention fusion architecture which reduces the on-chip memory with fused operators; 2) the grouped sparse Softmax core; and 3) the dual-path sparse LayerNorm core. Experiments show that the software achieves <tex-math notation="LaTeX">4.45× </tex-math> and <tex-math notation="LaTeX">7.59× </tex-math> computation reduction with little output difference for Softmax and LayerNorm, respectively. The hardware architecture supports at most 32768 sequence length with only 186-kB on-chip memory and achieves <tex-math notation="LaTeX">1.75× -1.98× </tex-math> and <tex-math notation="LaTeX">3.22× -4.32× </tex-math> speedups for sparse Softmax core and sparse LayerNorm core with little accuracy loss, respectively.
No takes yet. Share an insight, caveat, or question.
Wang et al. (2023) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: