FedSD: A Communication-Efficient Federated Learning Method With Scaled Stochastic Quantization and Dynamic Caching Mechanism

Communication cost is a significant bottleneck in Federated Learning (FL). Integrating gradient quantization in FL is a favored approach to mitigate communication costs. However, these methods often tend to lose important information by converting many elements to zero. Moreover, they typically overlook the need to compress data sent back to devices and ignore the efficiency gains from reducing communication rounds and limiting device participation. To overcome these challenges, we introduce the Scaled Stochastic Quantization (SSQ) method, designed to prevent many elements from being set to zero after quantization. SSQ improves Stochastic Quantization (SQ) by adding a scaling step, which can maximize the mapping range of compression elements in the compressed space and reduce quantization error. Addressing local model staleness from downstream data compression, we also present a Dynamic Caching Mechanism (DCM) tailored for SSQ. DCM ensures model synchronization through periodic transmission of compressed updates and efficient management of backups. By combining SSQ and DCM into FL, alongside methods for increased local iterations and random device selection, we establish a new framework named FedSD. Additionally, we provide a theoretical analysis of the convergence bounds of FedSD. Our extensive testing shows that FedSD requires significantly lower communication costs compared to other methods to achieve the same level of accuracy.