SA-ADFQ: structure-aware activation distribution friendly quantization for vision transformers in medical image analysis

Vision Transformers (ViTs) have exhibited impressive performance in medical image analysis tasks like classification, detection, and segmentation. However, due to their computational and memory costs, it is difficult to efficiently deploy ViTs in resource-constrained clinical environments. Quantization is an efficient approach for mitigating this problem. However, existing Post-Training Quantization (PTQ) approaches face performance degradation at low-bit precision due to the complicated activation distributions in ViTs. To address these issues, Structure-Aware Activation Distribution Friendly Quantization (SA-ADFQ), a new extension to the existing quantization method, specifically for medical image analysis tasks, is introduced. By inheriting the Activation Distribution Friendly Quantization (ADFQ) method, which includes the per-patch outlier quantizer and shift-log2 quantizer to address the irregular and non-uniform distributions. Moreover, to further enhance quantization performance, a new module-wise optimization approach that maintains spatial dependencies and key feature representations in medical image data is also introduced. Furthermore, the method is extended to include medical image classification, detection, and segmentation tasks, demonstrating its flexibility for various image and video tasks. Through extensive experiments on medical image datasets, it is shown that the proposed method significantly improves performance over existing quantization methods, especially at low-bit precision, and is computationally efficient.