WeHMiT-Bench: Evaluating Large Language Models for Explainable Health Misinformation Detection in WeChat Article Titles

Driven by competition for attention, WeChat public accounts often use sensational or weakly supported health claims in article titles. Such titles may mislead users or be accepted, repeated, or amplified by large language models (LLMs). However, dedicated benchmarks that test whether LLMs can recognize and explain potentially misleading health claims in Chinese WeChat article titles remain limited. This study introduces WeHMiT-Bench, a benchmark containing expert annotations of Chinese WeChat article titles concerning traditional Chinese medicine and related health information, including traditional wellness, folk remedies, health supplements, and disease claims. Multiple LLMs are evaluated using Chinese titles alone and without external retrieval. The evaluation combines a main LLM comparison with focused analyses of the reliability, interpretability, and practical limitations of model judgments. Generated explanations are further evaluated using a framework based on relevance, details specific to each claim, title grounding, avoidance of unsupported additions, and usefulness for readers of health information. The main LLM comparison shows that under common evaluation criteria, LLMs achieve a peak classification accuracy of 0.822 on the Core Health Set constructed using purposive sampling, and explanations accompanying correct predictions consistently receive higher quality scores. Using WeHMiT-Bench, the study clarifies the capabilities and limitations of the evaluated LLMs in identifying and explaining cues of potentially misleading health claims in social media titles, informing efforts to uncover such information.