HAVIT: An Efficient Hardware-Accelerator for Vision Transformer with Informative Patch Selection Techniques

Vision Transformers (ViTs) are widely utilized in the domain of computer vision. However, they encounter challenges related to high computational expenses, particularly for real-time inference on devices with limited resources. To tackle this challenge, we design HAVIT, an efficient ViT accelerator that integrates lightweight informative patch selection techniques with the core ViT acceleration module. The informative patch selection techniques introduce minimal computation, comprising only about 0.03% on average compared to the savings achieved, as they rely on edge detection methods like Sobel and Canny, in contrast to the computationally intensive AI models. After the edge detection, we use three proposed algorithms—Density-based Patch selector (DSP), Row Column Intersection patch selector (RCI), and Bounded Envelop Patch selector (BEP)—each offering unique paths for selecting informative patches with distinct accuracy-performance trade-offs. Upon receiving the informative patches, the ViT acceleration portion of the proposed HAVIT employs parallelism by identifying independent data to enhance the performance further while being energy efficient. Experimental results show that our policies can significantly reduce computational workload by around 27% − 57% compared to the baseline (no_patch_selection). The proposed HAVIT also outperforms state-of-the-art accelerators in terms of overall system performance.