On the Dynamics of Non-IID Data in Federated Learning and High-Performance Computing

This paper investigates the symbiosis of Federated Learning (FL) and High-Performance Computing (HPC) architectures, unraveling challenges introduced by the intricate interplay of heterogeneity and non-Independently and Identically Distributed (non-lID) data. By leveraging the Flower framework, our research delves into the nuanced implications of FL in diverse HPC environments. We provide a comprehensive exploration of the heterogeneity within contemporary HPC architectures, spanning node organizations, memory hierarchies, and special-ized accelerators, emphasizing adaptability to this complexity. Methodologically, we simulate a FL scenario within our research laboratory, leveraging Flower to orchestrate collaborative model training across heterogeneous nodes. The experiments involve variations in the Dirichlet beta parameter, offering insights into the effects of non-lID data. Results encompass communication efficiency, energy efficiency, and global model accuracy, providing a holistic understanding of the performances across diverse HPC infrastructures. This research contributes to the ongoing discourse on efficient and scalable algorithms, providing insights for collaborative learning in the era of diverse HPC architectures.