Integrating machine learning with single-step GBLUP for enhanced genomic prediction in white spruce

We evaluated the predictive performance of machine learning-augmented single- and multi-trait single-step genomic best linear prediction (ssGBLUP) models for 30 complex traits spanning productivity, chemical defense, and climate adaptability in white spruce ( Picea glauca ) from Alberta, Canada, genotyped at 211,061 SNPs. We extend a novel integration of the ssGBLUP with non-linear kernels for forest tree improvement. Using the conventional ssGBLUP model with VanRaden’s linear genomic relationship matrix as baseline, we compared two non-linear extensions: an averaged Gaussian kernel (GK) with trait-specific bandwidths and an arc-cosine kernel (AK) with 1–20 layers (depth). To relate prediction gains to genetic architecture, we also fitted an extended GBLUP model explicitly including additive, dominance, and epistatic (additive × additive and additive × dominance) effects. Additive effects predominated in 21 traits, dominance contributed > 50% of variance in several chemical-defense traits, and epistasis explained ~ 25% in key physiological traits. Predictive performance varied by kernel type and trait architecture. AK achieved the highest accuracy in 29 traits, improving prediction by up to 27% (predictive differences up to ~ 0.10), particularly for traits with more complex genetic architectures. GK yielded modest gains (< 7%; predictive differences < 0.03), while linear G kernel remained competitive for core productivity traits. In multi-trait models, non-linear kernels performed similar to the linear alternatives. Notably, low-heritability traits such as drought resistance improved from 0.446-0.517 to 0.461-0.547 accuracy with multi-trait integration. These findings suggest that matching kernel complexity to genetic architecture may enhance genomic prediction and highlight the potential of non-linear kernels, particularly AK, in forest tree breeding.