Artificial Intelligence Benchmarking of Community Health Worker Diagnostic Chatbots Under Unmeasured Code-Mixed Vernacular Intent Drift: A Healthcare Safety Evaluation

Diagnostic chatbots deployed to support community health workers in multilingual settings are benchmarked almost exclusively on curated, monolingual test sets that do not represent the code-mixed vernacular in which frontline consultations actually occur. the present analysis treats that mismatch as a design problem rather than a data problem: when intent drift is unmeasured, benchmark scores cannot bound the safety of the deployed system, because the quantity that matters clinically is the probability that a drifted utterance is silently mapped to a confident but wrong triage recommendation. A replication-based evaluation of the before-after-control-impact estimator is then reported, using a declared variance structure (between-cluster SD = 6.0, residual SD = 4.5, intraclass correlation = 0.64) and 2000 replications per design cell across cluster counts of 8, 16, 24 and 32 and effect sizes of 3, 5 and 8 percentage points. The analysis indicates that 54 clusters are required to detect a 5 percentage point effect at 80% power under this variance structure, and that the intraclass correlation of 0.64 dominates the design's operating characteristics. The contribution is a benchmark protocol that reports drift-conditional safety margins alongside aggregate accuracy, and a demonstration that cluster-randomised evaluation of chatbot safety is feasible only when the design accounts for the clustering that code-mixed deployment induces.