BioLP-bench Measuring understanding of AI models of biological lab protocols

Abstract Language models rapidly become more capable in many domains, including biology. Both AI developers and policy makers 1 2 3 are in need of benchmarks that evaluate their proficiency in conducting biological research. However, there are only a handful of such benchmarks4, 5, and all of them have their limitations. This paper introduces the Biological Lab Protocol benchmark (BioLP-bench) that evaluates the ability of language models to find and correct mistakes in a diverse set of laboratory protocols commonly used in biological research. To evaluate understanding of the protocols by AI models, we introduced in these protocols numerous mistakes that would still allow them to function correctly. After that we introduced in each protocol a single mistake that would cause it to fail. We then presented these modified protocols to an LLM, prompting it to identify the mistake that would cause it to fail, and measured the accuracy of a model in identifying such mistakes across many test cases. Only OpenAI o1-preview scored similarly to the performance of human experts, while other language models demonstrated substantially worse performance, and in most cases couldn’t correctly identify the mistake. Code and dataset are published at https://github.com/baceolus/BioLP-bench