Estimating how Site-Level Differences in Acoustic Environments Affect Species Detection by Machine Learning Models

Machine learning models are vital in species detection from acoustic data, due to the large amount of recordings produced, especially in passive acoustic monitoring studies. However, the acoustic environment of a site can significantly impact the detectability of species using such models, potentially leading to biased estimates of species abundance and distribution. This impact is poorly understood, and current manual validation methods for assessing acoustic detectability are intractable when working with many species, across multiple locations and datasets, due to the intensive time requirements to manually identify false negatives. We introduce a new method, Noise-Augmented Detectability Estimation (NADE), to estimate the impact of site-specific environmental noise on the acoustic detectability of different species. We use vocalisations augmented with noise from the sites themselves to alleviate the need for manual validation of detectability. Our approach produces site- and species-specific estimates of the complex relationship between background noise, vocalisation amplitude, and acoustic detectability when using a machine learning detection model. We evaluate this method using BirdNET on six bird species, in the UK, demonstrating its potential for broader application in acoustic ecology. Our results show that there are large differences in the ability of a machine learning model to detect and classify vocalisations across different sites and species. For example: the detectability of loud vocalisations can be 90\% at one site and 10\% at another; the detectability at a site can be halved during the course of a survey; and that the amplitude of the background noise alone is insufficient to correctly estimate the detectability of a vocalisation. Detectability at a site can also vary considerably by species: a site can have the relatively high detectability for one species but poorer detectability for another. By providing a quantitative measure of how site-specific background noise affects acoustic detectability, NADE could be used to improve the accuracy of species abundance estimates derived from passive acoustic monitoring, helping to identify sites and times where the detectability is likely to be lower. This approach provides a significant step towards making acoustic detectability estimation more feasible, allowing more robust estimates of species abundance and distribution from acoustic data.