Auto Dataset Builder: An Automated Annotation Framework with Vision-Language Verification and Neural Dataset Quality Scoring

The construction of high-quality training datasets remains a primary bottleneck in computer vision model development, with annotation reportedly consuming 60-80% of project time. We present Auto Dataset Builder (ADB), an automated framework that converts unlabeled images into YOLO-compatible annotated datasets through a three-stage pipeline: (1) YOLOv11 proposal generation, (2) SAM2 mask-guided bounding-box refinement, and (3) zero-shot semantic verification, for which the framework supports either a Vision-LLM backend (Qwen-VL-Chat / LLaVA) or a CPU-friendly CLIP ViT-B/32 zero-shot classifier. We additionally introduce the Neural Dataset Quality Score (Neural DQS), a lightweight regressor that estimates the expected mAP@0.5 a YOLOv11 model will achieve after training on a given dataset, computed from six interpretable dataset-level features without requiring any model training. On 96 controlled degradation variants of COCO128, Neural DQS achieves a 5-fold cross-validated Pearson correlation of r=0.929 (R^2=0.854, p<0.001) with actual mAP@0.5; ablation and SHAP analyses identify CLIP-space semantic diversity as the dominant predictor, with its removal dropping r to 0.679. On a 597-image COCO2017 motorcycle subset, ADB's fully automated annotations reach 94.2% of the mAP@0.5 obtained from human (Manual) annotations (0.441 vs. 0.468) with zero manual labeling effort. A single active-learning iteration (ADB+AL), which re-annotates the 100 most uncertain images from a held-out unlabeled pool, closes this gap entirely and matches the strongest baseline at mAP@0.5=0.476. Code, trained models, and benchmark dataset:- https://github.com/ericchen931209/auto-dataset-builder- https://huggingface.co/EricChenWei/neural-dqs- https://huggingface.co/datasets/EricChenWei/neural-dqs-benchmark