HyAtt: A Hybrid Attention Accelerator for Transformers With Analog In-Memory Computing

Transformer-based models have achieved state-of-the-art performance in various deep learning domains, yet their heavy computation and memory demands, particularly in the self-attention mechanism, present significant challenges for efficient hardware implementation. In this work, we proposeHyAtt, a hybrid attention accelerator that integrates both analog and digital computing paradigms to optimize Transformer-based workloads. Our architecture offloads token pruning to a computing-in-memory (CIM)-based analog core while delegating attention computations for unpruned tokens to a dedicated digital processor. An efficient task scheduling procedure parallelizes the token pruning, data fetching, and computations for enhanced performance. In the analog core, we introduce a transposable memory array with 9T SRAM cells for charge sharing-based CIM operations with an analog-to-digital converter (ADC)-less scheme to enhance energy efficiency.HyAttprunes 77.9% of tokens with only a 0.65% accuracy drop, requiring the digital core to process the attention only for the remaining 22.1% of tokens. The silicon prototype ofHyAttis fabricated in a 65-nm process and achieves an operational frequency of 1.1 GHz. It delivers a peak energy efficiency of 14.8 and 1.65 TOPS/W at 0.7 V and a peak area efficiency of 976.6 and 79.4 GOPS/mm2at 1.2 V at CIM and chip levels, respectively.