What question did this study set out to answer?

This study aims to assess the performance of reasoning-enabled LLMs in classifying stances towards COVID-19 vaccinations and identify chain-of-thought reasoning failures.

August 10, 2026Journal of Medical SystemsOpen Access

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection

View Full Paper

Ask AI

Bookmark

Authors

APAndreas PraschkMedical University of Graz VFValentin Fischill-NeudeckParacelsus Medical University TCThomas CaspariParacelsus Medical University

Discussion

Member takes

Implication

Mixed-methods study evaluates reasoning failures in LLMs classifying COVID-19 vaccination stances on Twitter, suggesting areas for improvement.

Key Points

This study aims to assess the performance of reasoning-enabled LLMs in classifying stances towards COVID-19 vaccinations and identify chain-of-thought reasoning failures.
Evaluated two LLMs (o4-mini and Gemini) on 3,060 COVID-19 vaccination tweets against human-annotated labels.
Measured accuracy, macro-F1 score, and CoT availability while qualitatively analyzing dual-error cases using Mayring’s content analysis.
Quantitatively compared the CoT outputs for reasoning coherence among misclassified tweets.
o4-mini achieved an accuracy of 0.819 compared to Gemini's 0.799; p = 0.0015 indicating significant difference.
Of the tweets with CoTs, 14.9% were dual-errors, with both models misclassifying the same tweet in 88.8% of cases.
Identified reasoning failures included target confusion, literal interpretation of sarcasm, and inconsistencies between labels and rationales.

Cite This Study

Praschk et al. (2026) studied this question.

synapsesocial.com/papers/6a797d479c20a9bbd31848c8 https://doi.org/10.1007/s10916-026-02449-3

View Full Paper

Ask AI

Bookmark

Implication

Mixed-methods study evaluates reasoning failures in LLMs classifying COVID-19 vaccination stances on Twitter, suggesting areas for improvement.

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection

Discussion

Member takes

Implication

Key Points

Cite This Study

Beyond Accuracy: A Mixed-Methods Audit of Chain-of-Thought Failures in LLM-Based COVID-19 Vaccine Stance Detection

Discussion

Member takes

Implication

Key Points

Cite This Study

Also Consider

Also Consider