Natural language querying of biological databases with large language models

It is attractive to be able to query biological knowledge bases using queries written in a natural language. Natural language queries are typically translated into structured queries using Large Language Models (LLM). However, unassisted naive LLMs often fail to interpret the scientific queries correctly and also may produce linguistically correct but factually false output colloquially known as hallucinations. Here we report the outcomes of a systematic evaluation of several techniques aimed at querying biological databases in natural language with LLMs. We tested multiple LLMs, both proprietary and open-source, on a knowledge graph representing a subset of the Open Targets database frequently used in drug discovery. We find that the best balance between accuracy and flexibility in this context is achieved by multiple LLM agents that can challenge the outputs of each other and interact with a human user. This strategy is very flexible and does not require prior knowledge of the data structure, query templates, secondary databases, or adaptor language models, and it exceeds the query accuracy of the other techniques. We also highlight the need for appropriate benchmarks for testing the ability to query large biological knowledge bases with LLMs, and propose future directions for this work.