Benchmarking Large Language Models Against Web of Science

Key Points

• This research aims to assess the accuracy of large language models in generating bibliometric data compared to the Web of Science.
• Evaluated several large language models for their bibliometric data accuracy.
• Conducted comparative analyses between LLM outputs and Web of Science data.
• Identified limitations and strengths of each model in producing bibliometric information.
• ChatGPT-4o performed the best among the large language models evaluated.
• Significant limitations were noted in all models when compared to Web of Science outputs.
• The findings highlight the need for caution when using LLMs in bibliometric analyses.