Prompt-Driven LLM Pipeline for Topic Modeling of Multiple Sclerosis Social Media Discussions

Despite advances in topic modeling, extracting coherent themes from short, noisy, health-related social media texts remains a methodological challenge. This paper presents a preliminary and exploratory empirical investigation of a prompt-driven Large Language Model (LLM) pipeline for topic modeling of 504 multiple sclerosis (MS)-related posts from Platform X. Two in-context learning strategies—zero-shot and few-shot prompting—were compared using GPT-4o-mini, with topic quality assessed through a multi-layered evaluation framework that leveraged GPT-4 for automated coherence and diversity scoring, complemented by human validation for coherence and document-level faithfulness. Within this experimental setting, few-shot prompting achieved the highest human agreement score (87.5%) alongside high coherence (4.9/5.0), while zero-shot prompting yielded the highest coherence score (5.0/5.0), with 79.2% human agreement. Both configurations demonstrated high thematic diversity (4.6/5.0) and produced themes largely judged as faithfully grounded in the source posts across the full corpus, with few-shot prompting demonstrating consistently stronger faithfulness and greater thematic stability. Nonetheless, these findings suggest that prompt-driven LLM pipelines — particularly few-shot approaches—show promise as a human-aligned and interpretable method for topic modeling of short, noisy, health-related social media texts.