Enhanced YOLOv8n-based steel surface defect detection using adaptive threshold focal loss, swin transformer, and a high-resolution detection head

Steel surface defect detection in industrial environments faces three persistent challenges. First, strong reflections and complex textures often confuse defect features with background patterns. Second, due to limited spatial resolution in deep feature pyramids, micro defects are frequently missed. Third, detection-level imbalance and hard-sample imbalance cause rare defect types to be underrepresented during training, leading to poor recall. To address these issues, this paper proposes ATFL-Swin-YOLO. It is an enhanced YOLOv8n-based detection framework for steel surface defect detection. Several advanced components are carefully integrated and adapted. The standard classification loss is replaced with Adaptive Threshold Focal Loss (ATFL). This loss function was originally developed for infrared small target detection. It dynamically adjusts the learning emphasis between easy and hard samples. To better model long-range spatial dependencies and suppress large-scale background interference, Swin Transformer blocks are embedded into the backbone network. A high-resolution P2 detection head is also introduced. It fuses shallow 160× 160 feature maps with upsampled semantic representations. This helps recover fine-grained details of micro defects. Extensive experiments are conducted on the NEU-DET and GC10-DET benchmarks. The proposed method reduces the number of parameters by 16.0%, from 3.157M to 2.650M. It improves mAP@50 from 76.7% to 77.6% and mAP@50-95 from 41.0% to 42.5%. Notably, detection performance for critical micro-defect categories (e.g., inclusions and pitted surfaces) is substantially enhanced. Further comparisons with additional SOTA models confirm that ATFL-Swin-YOLO attains a more favorable trade-off between precision and parameter efficiency. Ablation studies and comparisons with state-of-the-art methods confirm that the synergistic design of these components consistently improves accuracy without increasing the model parameter count.