Challenges in Using Conversational AI for Data Science

Large Language Models (LLMs) are transforming data science, offering assistance in coding, preprocessing, analysis, and decisionmaking. However, data scientists face significant challenges when interacting with LLM-powered agents and implementing their suggestions effectively. To explore these challenges, we conducted a mixed-methods study comprising contextual observations, semi-structured interviews (n=14), and a survey (n=114). Our findings reveal key obstacles, including difficulties in retrieving contextual data, crafting prompts for complex tasks, adapting generated code to local environments, and refining prompts iteratively. Based on these insights, we propose actionable design recommendations, such as data brushing for improved context selection and inquisitive feedback loops to enhance communication with conversational AI assistants in data science workflows.