Vision-enhanced multi-modal learning framework for non-destructive pavement damage detection

Road infrastructure damage detection is crucial for maintaining transportation safety and efficiency. While current methods like machine learning and computer vision are effective, they have difficulty analyzing and describing complex pavement conditions. This paper presents DamageQwen, a vision-language model framework for non-destructive testing of pavement damage. The framework integrates YOLO v11x for initial damage detection with vision-language models (VLMs) to enhance detection accuracy and reduce reliance on large labeled datasets. By combining object detection capabilities with natural language understanding, DamageQwen enables more comprehensive damage assessment through visual question answering. The framework incorporates CLIP-based repeat defects elimination and supports few-shot learning with just 1–4 shots. Compared to generic VLMs, our approach significantly improves the accuracy of descriptions while greatly reducing the labelling effort. The framework's ability to process both visual and semantic information allows non-experts to effectively detect and assess pavement damage through simple prompts. • Proposed DamageQwen framework based on Vision-Language Models for pavement damage detection. • Innovative combination of YOLO and CLIP to generate prompts, enhancing VLMs' recognition accuracy. • Supports simple user prompts so that non-experts can accurately describe pavement damage. • Achieves significant accuracy improvements with minimal training, reducing data labeling workload.