Integrating human judgement and few-shot machine learning to annotate baleen whale songs

Among baleen whales, humpback whales' hirearchically structured songs represent a peak in complexity. The smallest acoustic components of these songs, termed units, are classified into over 100 distinct types based on frequency content and duration. Annotating songs to the unit level is key to studying their structure and temporal evolution. Manual annotation poses challenges, being time-consuming and often requiring a degree of subjectivity. Nonetheless manual annotation is still undertaken, as variations in song structure, variability between singers, contextual factors, and the emergence of novel or hybrid unit types challenge the development of automatic methods. We address this challenge using a workflow that combines regular expert input and a prototypical network, which is a few-shot learning framework that can achieve high classification accuracies with limited training examples. Our model was initially trained on humpback whale recordings from New Caledonia across 2 years. Then, with minimal human input to adapt to subsequent years' recordings, the model achieves accuracies above 75% on previously observed unit types and 70% on never-before-seen unit types. This human-in-the-loop approach creates a scalable framework that allows for the quick annotation of humpback whale songs while maintaining the flexibility and rigour provided by human input.