The rapid growth of data volumes from large scientific collaborations, such as the Large Hadron Collider (LHC), presents significant challenges for the High Energy Physics (HEP) community. With annual data volumes projected to increase by a factor of thirty by 2028, efficient data management has become a critical concern. The HEP community’s reliance on wide-area networks for global data distribution often results in redundant long-distance transfers, leading to network congestion and degraded application performance. This study investigates the effectiveness of regional data caches in mitigating network congestion and enhancing application performance, using a large-scale dataset of millions of access records from regional caches in Southern California, Chicago, and Boston, which serve the LHC’s CMS experiment. Our analysis reveals the substantial potential of in-network caching to transform large-scale scientific data dissemination, enabling faster and more efficient data access for researchers and scientists. Additionally, neural networks trained on data from multiple regional caches demonstrate enhanced predictive accuracy, particularly benefiting caches with limited historical data through transfer learning, thereby validating their robust generalization capability.
Sim et al. (Tue,) studied this question.