Evaluating federated learning under data heterogeneity with analysis of failure modes and practical mitigations

Federated Learning (FL) allows for collaborative model training across decentralized clients while maintaining data privacy; yet real-world deployments always include heterogeneous and non-identically distributed (non-IID) client data. Existing FL research primarily assesses system performance using aggregate metrics like global or average accuracy, which can hide differences in client behavior. This study provides a thorough empirical evaluation of federated learning under controlled data heterogeneity, with a major emphasis on worst-client performance as a measure of client-level robustness. Experiments are carried out on MNIST, CIFAR-10, and CIFAR-100 utilizing normal FedAvg training and commonly accepted baselines, such as FedProx and q-FedAvg, at various levels of Dirichlet-based label skew. The results show that, as heterogeneity increases, worst-client accuracy falls quickly, revealing silent errors that were previously disguised by aggregate evaluation. A comparison of mitigation options reveals that server-side client re-weighting improves worst-case performance under modest heterogeneity but becomes unstable and damaging under strong non-IID situations. In contrast, simple client-side data balancing reliably improves worst-client accuracy across all heterogeneity regimes without requiring additional infrastructure or algorithmic complexity. These findings emphasize the limitations of aggregate evaluation in federated learning, as well as the need of client-aware metrics and mitigation measures that increase local update quality while dealing with client-level inequalities in the face of true data heterogeneity.