Abstract: Bioinformatics is one of the main areas in the health sciences with great potential for the application of Large Language Models (LLMs), as it mainly involves computational analysis of results derived from experimental and high-throughput analysis of human, animal, and cellular models. Testing available general-purpose LLMs for the presence of inaccurate answers, such as confabulations, is of great interest in the fields of bioinformatics and computational genomics. In the current study, we carried out an analysis of the performance of six freely available LLMs (Gemini, ChatGPT, Grok, Claude, Llama, and DeepSeek) in a number of tasks commonly used in bioinformatics and computational genomics, with varying levels of difficulty. The selected tasks were: converting different identifiers (for two organisms), simulating a bisulfite conversion of DNA sequences, identifying the effects on amino acids of DNA polymorphisms, retrieving orthologues in mouse, identifying GO ontologies and KEGG pathways for lists of genes, interpreting a Volcano plot for gene expression, and automatically generating R code to visualize data. In general, our analysis identified a multiplicity of confabulations for different types of results generated by the tested LLMs. Our results highlighted a high number of errors in the output of the LLMs and identified automatic generation of code as a promising area. Future studies will be needed for a better understanding of the causes of confabulations of LLMs in research-related tasks.
Diego A. Forero (Thu,) studied this question.