FReNG: Federated Optimization by Using Regularized Natural Gradient Descent

Federated learning (FL) is an important area in machine learning that addresses the challenges of distributed data by collaboratively training a global model. One major issue with FL is the increased communication rounds or FL iterations between the server and the participating data sources or clients, which is caused by the presence of heterogeneous data across all the clients. To address this issue, this paper introduces FReNG, which utilizes regularized natural gradient descent with exponential moving average of the global gradient on the server. The server-based utilization of regularized natural gradient descent and exponential moving average of the global gradient leads to faster convergence of the global model in heterogeneous settings without additional local space and computational complexities, resulting in comparatively fewer communication rounds, while achieving the targeted accuracy from the global model. As FReNG uses regularized natural gradient descent, it can directly compute the update step without calculating and storing the full Fisher information matrix (FIM), which is one of the key advantage of FReNG. Extensive experimental results on image classification tasks using various heterogeneously partitioned datasets show that FReNG outperforms existing state-of-the-art FL algorithms such as FedProx, SCAFFOLD & FedInit and can compete with other state-of-the-art FL algorithms, FedGA, & DONE (with less per FL iteration communication cost & less local time complexity) in terms of reduced communication rounds, while achieving a certain precision of convergence and a targeted test accuracy from the global model.