Gradient Free Personalized Federated Learning

Federated Learning, as an emerging edge artificial intelligence paradigm, enables a group of clients to collaboratively train a global model without revealing their local data. The conventional FL algorithms usually depend on the access to exact gradient or Hessian matrix, which may be inaccessible due to resource limitation or application restriction. Meanwhile, Federated Learning intrinsically suffers from data heterogeneity, which restricts the global model from performing well on each clients' task. To simultaneously tackle these two challenges, we propose a gradient free personalized federated learning framework, namely pFedZO. We utilize infimal convolution to bridging the gap between personal and global knowledge, and exploit zeroth-order gradient estimator to solve the problem. We theoretically show the local approximation can converge sublinearly and the global problem converge to the neighbourhood of the optimal with a same speed. We further propose pFedZO-Heur to accelerate training procedure. Experimentally, we verify that pFedZO excels at test accuracy with the vanilla Zeroth-Order Optimization (ZOO) based FL by Math 1. We also show pFedZO-Heur can achieve the same performance level with lower time consumption.