FNBench: Benchmarking Robust Federated Learning against Noisy Labels

Robustness to label noise within data is a significant challenge in federated learning (FL).From the data-centric perspective, the data quality of distributed datasets can not be guaranteed since annotations of different clients contain complicated label noise of varying degrees, which causes the performance degradation.There have been some early attempts to tackle noisy labels in FL.However, there exists a lack of benchmark studies on comprehensively evaluating their practical performance under unified experimental settings.To this end, we propose the first benchmark study FNBench to provide an experimental investigation.Meanwhile, we provide observations to understand why noisy labels impair FL.Our evaluation incorporates sixteen related state-of-the-art methods under diverse settings on five datasets.These datasets include three label noise types: (i) Manual-injected synthetic noisy datasets, (ii) Noisy datasets with human annotation error patterns, and (iii) One systematic large-scale noisy dataset crawled from well-noted online shopping websites.Through our empirical investigation, we draw diverse conclusions to guide future studies.We additionally exploit a representation-aware regularization method to enhance the robustness of existing methods against noisy labels based on our observations.Finally, we discuss the limitations of this work and propose three-fold future directions.Our source code will be opensourced to facilitate related communities.