Evaluating Locally Grounded Generative AI for Health-Policy Support: A Multilevel Mixed-Methods Pilot Study

Generative artificial intelligence (GenAI) may assist evidence-informed health-policy work, but favorable user perceptions and well-written outputs do not establish factual reliability or implementation readiness. A multilevel mixed-methods pilot evaluation examined MUSYA AI, a locally grounded policy-support platform in Thailand. Thirty-five health-sector personnel completed questionnaires, seven participated in Think-Aloud observation, sixteen Policy Briefs were assessed using an expert rubric, and seventy factual claims underwent evidence auditing. The document comparison included eight MUSYA AI and eight ChatGPT (GPT-5.6, OpenAI) outputs under common task instructions and system-specific information access. Mean System Usability Scale, perceived usefulness, and trust scores were 97.29/100, 4.70/5, and 4.57/5, respectively. Ceiling effects and limited internal consistency constrained interpretation of these descriptive indices; user experience, ease of use, and behavioral intention were interpreted at the item level. Observations documented reading and editing, but successful error detection and correction were not evaluated. MUSYA AI documents received higher overall quality ratings than comparator documents (95.38 ± 2.39 versus 72.25 ± 11.90; difference, 23.13 points; bootstrap 95% confidence interval, 15.63–31.38; Hedges’ g = 2.55). However, classification–rationale inconsistencies affected forty-five claims (64.3%), leaving comparative factual-accuracy findings provisional. The small, nonrandomized document sample and absence of verified task pairing prevented causal attribution of quality differences. MUSYA AI elicited favorable pilot-user perceptions and produced higher-rated documents within the evaluated sample. These findings support continued supervised evaluation; institutional readiness requires traceable claim adjudication, demonstrated oversight effectiveness, and broader controlled assessment.