Leveraging Fst and Genetic Distance to Optimize Reference Sets for Enhanced Cross-Population Genomic Prediction

Genomic selection often faces challenges of insufficient prediction accuracy in cross-population applications, primarily due to differences in linkage disequilibrium patterns between populations. This study proposes an Fst-based strategy to enhance prediction performance by constructing a cross-population reference set with high genetic similarity to the target population (PopA). By integrating Fst-mediated SNP screening and Euclidean genetic distance analysis, the top 10%, 15% and 20% of individuals genetically most similar to PopA were screened from PopB and PopC, respectively, leading to the generation of six reference sets characterized by different mixing proportions. The results demonstrate that incorporating the top 10–20% of the most similar individuals significantly improves the accuracy and robustness of genomic estimated breeding value predictions. Among the methods evaluated, ssGBLUP and wGBLUP performed best, with prediction accuracy increasing as the mixing proportion rose up to 20%. This approach effectively mitigates structural bias caused by inter-population genetic differences and significantly enhances prediction efficiency. The multi-level mixing experiment not only validates the practical value of Fst and Euclidean distance but also provides theoretical support and a feasible solution for the efficient integration of cross-population germplasm resources.