Clip-based road-marking detection with LLM-guided driving prompts

The advancement of artificial intelligence (AI) has significantly improved the perception and decision-making abilities of autonomous vehicles (AVs), yet real-time and accurate road marking detection remains difficult under faded markings, nighttime scenes and adverse road–weather conditions. This paper presents RoadGPT, a vision–language pipeline that couples a CLIP-based detector (RoadCLIP) with a planner-facing, LLM-generated advisory layer. The model is trained and evaluated on 3,696 road marking images covering 14 UK Highway Code classes, comprising 2,576 real images (69.7 %) from Google Street View and 1,120 text-to-image synthetic images (30.3 %) that broaden rare appearances and degraded conditions. We use 2,968 images for training (2,072 real, 896 virtual) and 728 images for testing (504 real, 224 virtual), keeping the same 70:30 real–virtual ratio in both splits. On this test set, RoadCLIP attains 98.5 % precision, 97.9 % recall and 98.2 % F1 for non-lane markings, while lane-marking subclasses reach up to 89.8 % F1. The advisory layer transforms recognised markings into structured driving prompts and is assessed via semantic similarity using Sentence-BERT (all-mpnet-base-v2) cosine scores against Highway Code-based references, achieving 89.3 % similarity, alongside an external LLM-as-judge rating of 4.68/5 for accuracy, completeness, and concise effectiveness. The full camera-to-advisory path runs in real time at 135 FPS (batch size 1, 224 × 224) on an RTX 4070 under a unified timing protocol. A remaining limitation is that visually similar lane-marking classes and extreme low-light scenes still reduce discriminability compared with symbol-like, non-lane markings.