From benchmark to user: an end-to-end evaluation audit of an AI-based oral health screening prototype

Background Artificial intelligence (AI) for oral screening is usually judged by aggregate detection metrics on public benchmarks whose partitions, labels, and relevance to users are rarely checked. Methods We audited a prototype combining a YOLOv8m detector for six conditions (1,500 test images), a DenseNet169 histopathology network, and a ResNet18 router. Under a pre-specified plan, we measured duplication across partitions, tested removal of duplicated test images against 10,000 matched controls, compared duplicated copies' labels and predictions, compared the library versions, and ran every test image through the application pipeline. Results Near-duplicates of training images accounted for 17.3% of test and 17.1% of validation images (lower bounds). Removing duplicated test images changed mean average precision (mAP)@0.5 by +0.001 (95% confidence interval [CI] −0.012 to +0.014). Only 22% of duplicated photographs had identical labels on both copies; exploratory analysis suggested memorization of training labels in orientations seen in training. A library update moved mAP@0.5:0.95 by −0.012 with identical predictions. The router rejected 23% of in-domain photographs and 51% of photographs with ulcers, so ulcer sensitivity per image was 0.36 despite the highest benchmark recall (0.91). A class-mapping defect, still live in a public demonstration in September 2026, gave the wrong name to 73% of findings. The histopathology classifier classified every image as positive; its label was not a diagnosis. Conclusion Duplication compromised the benchmark's independence without detectably changing aggregate mAP; the defects that changed what was reported or displayed lay in the labels, evaluation software, and pipeline. Evaluations should verify partitions and labels, pin the library, and test deployed systems end to end.