Reading Between the Lines: A Scoping Review of Large Language Models in Conversational Data Analysis

Q: What does this research mean for the field?

Large language models, predominantly GPT-4, are most commonly used for thematic analysis of conversational data, but current literature suffers from inconsistent reporting and validation practices that require standardized guidance. Novelty: ClaimNovelty.SYNTHESIS. Consensus alignment: ConsensusAlignment.NEUTRAL.

Sebastian Halicki; Yechan Choi; Riley Wicklander; Elliana Dunlap; Jieun Kim

doi:10.1177/10711813261484479

September 6, 2026Proceedings of the Human Factors and Ergonomics Society Annual Meeting

Reading Between the Lines: A Scoping Review of Large Language Models in Conversational Data Analysis

View at Publisher

Ask AI

Bookmark

Share

Key Points

To systematically examine how transformer-based large language models are applied and validated in the qualitative analysis of interview and conversational data.
Conducted a scoping review adhering to PRISMA-ScR guidelines across ProQuest, Web of Science, and IEEE Xplore for publications between 2020 and 2025.
Screened 3,102 retrieved records and assessed 207 full-text articles, ultimately synthesizing data from 84 peer-reviewed studies.
Thematic analysis was the primary qualitative task (43/84, 51%), most frequently performed on full transcripts via zero-shot prompting.
Decoder-only architectures represented 82% (69/84) of applications, with GPT-4 (51%) and GPT-3.5 (21%) used most often, and Llama leading open-source models (10%).
Model validation lacked consistency, relying primarily on human-to-model comparisons (n = 33) or repeated model runs (n = 12), alongside irregular reporting of prompts and model versions.

Implication

Scoping review reveals dominant use of large language models for thematic analysis in conversational data, highlighting the need for standardized reporting.

Abstract

Transformer-based large language models are becoming increasingly popular in supporting qualitative analysis, but their applications and validation are often poorly characterized. This scoping review followed PRISMA-ScR rules to understand how such models are used in the analysis of interview and conversational data. Searches of ProQuest, Web of Science and IEEE Xplore (2020–2025) returned 3,102 records; 207 were assessed at full text, and 84 peer-reviewed studies were included. Support for thematic analysis was the most common application (43/84, 51%), typically with full transcripts analyzed using zero-shot prompting. Decoder-only models dominated (69/84, 82%), with GPT-4 (51%) and GPT-3.5 (21%) most frequent and Llama the leading open-source alternative (10%). Validation, when present, relied mainly on human-LLM comparison ( n = 33) or repeated model runs ( n = 12). Trends regarding reporting model versions and prompts were inconsistent. Standardized reporting and ethical guidance are needed to support reproducible use.

Discussion

No takes yet. Share an insight, caveat, or question.

Cite This Study

Halicki et al. (2026) studied this question.

synapsesocial.com/papers/6a9fd74958e84d0ff5b45cff https://doi.org/10.1177/10711813261484479

View at Publisher

Ask AI

Bookmark

Share

Reading Between the Lines: A Scoping Review of Large Language Models in Conversational Data Analysis

Key Points

Implication

Abstract

Discussion

Cite This Study

Reading Between the Lines: A Scoping Review of Large Language Models in Conversational Data Analysis

Key Points

Implication

Abstract

Discussion

Cite This Study

Keep reading

Keep reading