Spectral similarity shapes species discrimination of herbarium specimens in a taxonomically challenging group

Premise Supervised machine learning models trained on reflectance spectra are emerging as powerful tools for assisting taxonomic identification of herbarium specimens. However, it is unclear how species composition in model training affects performance. We used the genus Vitis to test how classification errors are driven by spectral similarities among species and whether prediction probabilities decline when classifying species not included in training. Methods We trained a global partial least‐squares discriminant analysis (PLS‐DA) on foliar herbarium hyperspectral data from 18 Vitis species. We then ran 18 reduced versions of the global model, where each reduced model omitted one species from training but was tested on all 18 species. We calculated pairwise changes in per‐species sensitivity () between the global and reduced models, tested whether correlated with pairwise spectral similarity, and evaluated whether prediction probabilities were lowest when classifying species omitted from training. Results The global PLS‐DA had 74.1% accuracy. Across reduced models, sensitivity significantly increased for 31 unique species pairs and significantly decreased for three. Sensitivity improvements were greatest when the focal and omitted species had high spectral similarity and declined nonlinearly with increasing spectral differentiation. Prediction probabilities were highest for correct predictions, intermediate for misclassifications, and lowest for species omitted from training. Conclusions Our results demonstrate a nonlinear relationship between classification accuracy and spectral similarity, suggesting limits of spectral separability among intrageneric species. Prediction probability may be a useful metric for evaluating the likelihood that a specimen represents a species not included in training.