Answers are grounded in this paper and cite their sources.
Machine learning (ML) is revolutionizing field and laboratory studies of animals. However, a challenge when deploying ML for classification tasks is ensuring the models are reliable. Currently, we evaluate models using performance metrics alone (e.g. precision, recall and F1 score), but these can overlook the ultimate aim, which is not the outputs themselves (e.g. detected species or individual identities or behaviour) but their incorporation for hypothesis testing. As improving performance metrics has diminishing returns, particularly when data are inherently noisy (as human-labelled, animal-based data often are), researchers are faced with the conundrum of investing more time in maximizing metrics versus answering biological questions. This raises the question: how much noise can we accept in ML models? Here, we start by describing an underreported factor that can cause metrics to underestimate model performance. Specifically, ambiguity between categories or mistakes in labelling test data generate hard ceilings that limit performance metrics. This likely widespread issue means that many models could be performing better than their metrics suggest. Next, we argue and show that imperfect models (e.g. low F1 scores) can still be useable. Using a case study on ML-identified behaviour from vulturine guineafowl accelerometer data, we first propose a simulation framework to evaluate the robustness of hypothesis testing using models that make classification errors. Second, we show how to determine the utility of a model by supplementing existing performance metrics with ‘biological validations’. This involves applying ML models to unlabelled data and using the models' outputs to test hypotheses for which we can anticipate the outcome. Together, we show that effect sizes and expected biological patterns can be detected even when performance metrics are relatively low (e.g. F1 scores, 60%–70%). In doing so, we provide a roadmap for validation approaches of ML classification models tailored to research in animal behaviour, and other fields with noisy, biological data. • Evaluating machine learning (ML) models must go beyond performance metrics. • Mislabels in test data lead to underestimation of the model's performance. • Underestimated metrics can cause research delays, despite models being useful. • Simulations and biological validations provide additional evaluation for ML models. • Models with ‘low’ standard metrics can still be powerful for hypothesis testing.
No takes yet. Share an insight, caveat, or question.
Christensen et al. (2025) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: