Answers are grounded in this paper and cite their sources.
Radio astronomy observations generate massive heterogeneous datasets with file sizes ranging from kilobytes to gigabytes and total volumes reaching terabytes per session. Telescope sites in remote areas face limited network connectivity, making efficient data transfer a significant challenge. The conventional rsync tool, while widely used for incremental transfer, suffers from serial limitations that prevent full bandwidth utilization, particularly for extreme file-size distributions characteristic of radio astronomy data. This paper presents a heterogeneity-aware parallel data transfer method built upon the rsync framework, designed for datasets from facilities such as the Qitai radio telescope (QTT). Our approach introduces three key contributions: (i) a heterogeneity-aware task partitioning strategy with an adaptive correction factor κ ( C v ) that isolates files into singleton chunks; (ii) a mathematical performance model quantifying the impact of data heterogeneity on parallel speedup through a load imbalance factor γ ( N , C v , ρ ) ; and (iii) a checkpoint-based fault tolerance mechanism that eliminates costly directory re-traversal overhead upon interruption. Experimental evaluation on both simulated and authentic ASKAP observation data demonstrates the effectiveness of our method. On the representative data-mixed workload, our method outperforms Optimized Rsync by 16.8% on simulated data and by 28.8% on real data, while maintaining CPU utilization below 1.14% and a memory footprint approximately three times smaller than Standard Rsync. The efficiency ratio shows at least two orders of magnitude higher resource efficiency than all baseline tools. The proposed method offers an alternative transmission solution for the QTT data transfer system, seamlessly integrating with the existing rsync ecosystem while consuming minimal computing resources.
No takes yet. Share an insight, caveat, or question.
Wang et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: