The rapid rise of artificial intelligence (AI) has inspired scholarship that seeks to make sense of the impacts of AI technology on a wide range of human activity, including social scientific research. The power of AI to synthesise literature, compile information, and generate content is well recognised, yet its performance on some core dimensions of inductive/abductive research, including qualitative data analysis, is less documented and understood. This paper explores capabilities and implications of AI technology by focusing on the central position of the human experience in constructivist-interpretive research and its representability by dis-experienced AI models. We report on a novel qualitative experimental approach developed by the authors to examine inter-coder reliability between ‘human only’ and ‘human-AI’ approaches. Key findings highlight fundamental shortfalls, including AI generated output that is disconnected from the transcript data, and the inability of AI to generate robust descriptive and theoretical categories from raw data despite training. We conclude use of AI in qualitative analysis will require the development of entirely new skillsets, such as ‘analytical coaxing’, to navigate the vagaries of AI models. Our Three-Step Framework proposes a way to establish and communicate rigour in AI assisted analysis, foregrounding epistemological reflexivity and justifications that acknowledge the centrality of the human experience.
Messner et al. (Fri,) studied this question.