How Effectively Do LLMs Automate Data Analysis? A Comparative Study with ChatGPT's Data Analyst, Grok, and Qwen

Artificial Intelligence (AI) tools are increasingly becoming integral to analytical processes. This paper evaluates the potential of Large Language Models (LLMs), specifically OpenAI’s ChatGPT’s Data Analyst, Grok 3, and Qwen2.5-Max in data analysis. We conducted a structured experiment employing this tool in 108 questions spanning descriptive, diagnostic, predictive, and prescriptive analyses to assess its effectiveness. The study revealed an overall efficiency rate of 72.22% for ChatGPT’s Data Analyst, outperforming Grok 3 at 45.37% and Qwen-Max 2.5 at 8.33%. By discussing the strengths and limitations of a state-of-the-art LLM-based tool in aiding data scientists, this study aims to mark a critical milestone for future developments in the field, particularly as a reference for the open-source community.