Optimal distributed subsampling for joint mean and variance models in heterogeneous multi-source big data

For the first time, the theories and methods of subsampling techniques for joint mean and variance models are studied in the multi-source heterogeneous big data. Additionally, simulations are conducted to evaluate the significant effectiveness of the proposed method. With the rapid advancement of information technology, the generation and accumulation of multi-source big data have created rich resources for data analysis and decision-making. However, processing and analyzing large-scale data from diverse sources presents significant challenges for traditional statistical methods, including insufficient computational resources, transmission delays, and privacy concerns. As joint modeling of mean and variance is fundamental to statistical analysis and significantly impacts the accuracy and effectiveness of data interpretation, efficiently applying optimal subsampling techniques to massive joint mean and variance models in a distributed environment has become a critical issue in modern data analysis. In this paper, we propose a distributed subsampling algorithm based on joint mean and variance models to effectively address the subsample selection problem in multi-source big data environments. First, the asymptotic properties of the subsample estimator are analyzed under the assumption of full data. Next, the optimal subsampling probabilities and allocated subsample sizes for the k-th sub-dataset are derived. Finally, a two-step, approximately optimal subsampling algorithm based on joint mean and variance models is introduced. The performance of the proposed method is evaluated through simulation studies and a real-world energy consumption data analysis.