Federated Learning (FL) is often evaluated in simulation, which overlooks network variability, system heterogeneity, and energy costs in geo-distributed settings. We propose a deployment-aware methodology that triangulates analytical modeling, simulation, and real-world deployments within a unified FL evaluation framework. For a given series of experimental scenarios, the methodology allows to assess the consistency of performance trends across the three evaluation approaches, quantifying deviations in key metrics such as run time, communication overhead, and energy consumption. This further enables cross-validation of the reliability of multiple measurement tools, highlighting discrepancies in commonly reported metrics such as the energy usage. The methodology is validated on FL workloads by comparing analytical predictions and simulations against large-scale deployments on the Grid'5000 testbed, spanning 51 nodes across four geographically distant sites. By varying key FL components such as aggregation algorithms, client sampling rates, and datasets, we characterize how different FL design choices affect the reliability of the three evaluation approaches. Our findings reveal significant divergences: analytical models accurately capture communication patterns and preserve the relative performance of the scenarios, simulations reflect broad trends but often lead to performance rankings of different configurations inconsistent with those found through actual deployment, while only the latter uncovers hidden costs, such as increased energy consumption due to data imbalances. We conclude with practical guidance on what simulations can and cannot capture, the complementary role of analytical models, and which aspects require deployment-level validation.
Valli et al. (Thu,) studied this question.