What question did this study set out to answer?

To evaluate the performance of vision-language models for quality assessment in object detection across multiple domains.

May 28, 2026Open Access

Vision-Language Models for automated quality control: a benchmarking framework and comprehensive study

MAMohamed AbdelkaderPrince Sultan University

Key Points

To evaluate the performance of vision-language models for quality assessment in object detection across multiple domains.
Comprehensive benchmarking of nine state-of-the-art vision-language models across five domains.
Employs multi-class classification to assess semantic correctness of detection outputs with accuracy metrics and statistical testing.
Analysis includes evaluating performance using ground-truth-annotated samples and statistical metrics such as coefficient of variation and Kruskal-Wallis testing.
Overall accuracy ranges from 8.5% to 82.8%, with a mean of 45.7% and standard deviation of 18.5%.
LLaVA-13B achieves the highest overall performance at 48.6% accuracy, highlighting significant challenges in medical domains with HAM10000 averaging 7.3%.
Statistical analysis indicates substantial inter-model differences (p<0.001, effect sizes $$\eta ^2$$=0.82-0.96), confirming meaningful performance variations.

Abstract

Automated object detection systems require robust quality assessment mechanisms to maintain performance when deployed in environments that deviate from training distributions. While traditional monitoring relies on statistical drift detection, these approaches lack semantic understanding necessary for triggering appropriate model adaptations. This paper presents the first comprehensive benchmarking of Vision-Language Models (VLMs) for semantic-level quality assessment of multi-domain object detection outputs. We systematically evaluate nine state-of-the-art VLM models across five diverse domains spanning medical imaging (Brain Tumor, HAM10000), aerial surveillance (VisDrone), industrial inspection (Carparts), and general detection (COCO) using ground-truth-annotated samples. Our rigorous statistical evaluation employs multi-class classification where VLMs assess the semantic correctness of detection outputs, with comprehensive analysis including accuracy metrics, coefficient of variation, and Kruskal-Wallis testing. Results reveal substantial performance heterogeneity across models and domains, with overall accuracy ranging from 8. 5% to 82. 8% (mean: 45. 7%, SD: 18. 5%). LLaVA-13B achieves the highest overall performance (48. 6% accuracy, CV: 23. 8%), while medical domains prove most challenging (HAM10000: 7. 3% mean accuracy vs. VisDrone: 55. 9%). Statistical analysis reveals significant inter-model differences within all domains (p<0. 001, effect sizes ²=0. 82-0. 96), confirming meaningful performance distinctions despite substantial cross-domain variation. Based on deployment criticality requirements, we establish three operational tiers: production-assistants (medical 80%, industrial 70%, surveillance 60%), supervised deployment, and research-stage systems. Our findings demonstrate that current VLMs are suitable for supervised rather than fully autonomous deployment, providing essential benchmarks and evidence-based guidelines for implementing VLM-based quality control in production computer vision systems. The benchmark code is open-source and available at https: //github. com/mzahana/vlm-bench.

Ask AI

Helpful

Bookmark

View Full Paper