ViTFCIM: A Sparsity-Aware Time--Frequency Computing-in-Memory Accelerator for Vision Transformers

Vision Transformers (ViTs) offer strong global modeling capability for vision applications. However, deploying ViTs on resource-constrained edge devices remains challenging due to the high computational and memory demands and the irregular sparsity. This work presents ViTFCIM, a hybrid Time-Frequency Computing-in-Memory (TF-CIM) architecture designed for sparsity-aware mixed-precision ViT inference. The architecture uses a Digital-to-Time Converter (DTC) to encode inputs as time-domain pulses and a Digitally Controlled Oscillator (DCO) to encode weights as frequencies. TF-CIM then performs time-frequency multiplication and in-situ phase accumulation in a multiplier-free manner. A data-driven time-domain sparsity-aware scheme is combined with an outer-product-based dataflow to adaptively skip ineffectual computations. Furthermore, a spatio-temporal bit-splitting strategy partitions input bits temporally and weight bits spatially across dual-SIMD processing elements (PEs), enabling bit-scalable operation. Post-layout simulations show that the 65-nm TF-CIM macro achieves a peak energy efficiency of 71.98 TOPS/W at 1.2 V in INT4 mode. Evaluations of DeiT-Tiny MHSA GEMM kernels show that the sparsity-aware encoding achieves average reductions of4.22×in energy and1.89×in latency. The mixed-precision bit-splitting scheme further achieves average reductions of2.89×in DCO-array power and2.70×in MHSA GEMM latency.