Towards Floating Point-Based AI Acceleration: Hybrid PIM with Non-Uniform Data Format and Reduced Multiplications

Neural networks (NNs) have exhibited excellent performance in various fields of artificial intelligence. However, the primary operations in these mainstream models, including matrix-vector multiplication (MVM), element-wise multiplication (EWM), and depth-wise convolution (DWConv), require massive data movements during computation, which greatly impacts NNs’ inference performance. The emerging Processing-In-Memory (PIM) architectures have shown great potential to overcome the memory wall problem. However, constrained by the supported data format and operator type, directly adopting PIM architectures for neural network acceleration faces three challenges: (1) Floating-point (FP) format has been widely adopted for ensuring high algorithm accuracy. However, Resistive Random-Access Memory (RRAM)-based analog PIM architectures perform integer (INT) MVMs in the analog domain, limiting their application to the more accurate FP format; (2) Static Random-Access Memory (SRAM)-based digital PIM architectures require additional circuits to support the FP format, and the SRAM capacity cannot satisfy the storage requirement of latest large language models (LLMs); (3) When performing the operators with few accumulation steps, such as EWMs and DWConvs, only few memory units in PIM architecture are activated, resulting in severe device under-utilization. To tackle the above challenges, this paper proposes an RRAM and 3D-SRAM-based hybrid PIM architecture, achieving FP-based algorithm accuracy, high device utilization, and high energy efficiency. At the software level , we first analyze the impact of quantization errors on NN’s inference accuracy. For the quantization error-insensitive MVM operations, we propose the PIM-oriented exponent-free non-uniform (PN) data format. The proposed PN format can be flexibly adjusted to fit the non-uniform distribution and approach FP-based algorithm accuracy using bit-slicing-based full INT operations. For the quantization error-sensitive EWM/DWConv operations, we introduce the multiplication-free approximated FP multiplications to reduce the additional hardware overhead. At the hardware level , we propose a hybrid PIM architecture, including an RRAM analog PIM using shift-and-add for PN-based MVMs, and a 3D-SRAM digital PIM with high utilization for DWConv/EWM operations. Extensive experiments on CNNs and attention-free LLMs validate that the proposed PIM architecture achieves up to 99.4 × and 33.9 × speedup with 5697.7 × and 8.2 × energy efficiency improvement compared to GPU and PIM-baseline, respectively. With the proposed PN format and approximated FP multiplications, the algorithm accuracy of CNNs and attention-free LLMs can be improved by up to 3.01% and 10.18%, respectively.