Attention‐Guided Lightweight CNN‐Transformer Fusion for Real‐Time Traffic Sign Recognition in Adverse Environments: HACTNet

ABSTRACT Autonomous driving is also impossible without traffic sign recognition (TSR; also known as traffic sign‐on‐road), which limits its reliability to domain changes, unfavourable weather, obstruction and hardware capacity. This paper proposes HACTNet, a low‐complexity CNN‐Transformer hybrid model that pushes the state‐of‐art in TSR by making a noteworthy set of contributions including (i) efficient convaps to model parts of the image, (ii) transformer encoder to capture the global context and (iii) an attention‐based fusion block to dynamically combine the two complementary sets of features. This synergy facilitates strong recognition in presence of blur and occlusion and in varying illumination. In addition to accuracy, HACTNet achieves high robustness (52.8%) against strong PGD adversarial attacks (8/255), but is still efficient (7.9 M parameters and 22.1 FPS) on the NVIDIA Jetson Nano. Moreover, the comparative analysis between the hybrid models (EATFormer, local‐ViT) and HACTNet proves that HACTNet has a better accuracy‐efficiency ratio. The extraordinary capability to counteract adverse weather conditions, fog, night, rain, snow etc., which is proven by the extensive testing of the real‐world ACDC adverse conditions data set, supports the viability of the proposed solutions in the real world. It is plug and play modularity with on‐going learning via elastic weight consolidation (3.3% less forgetting) and unsupervised domain adaptation via MMD loss (5.3% better on TT100K with no labels). Moreover, INT8 quantization with quantization‐aware training (QAT) incurs little accuracy loss (less than 0.5 percent) and much lower energy (0.27 J/sample) usage, which forms an edge deployment preparedness. Additionally, when adjusting to new traffic signs over time, the model shows compatibility with continuous learning, achieving a low forgetting rate (3.3%), highlighting its practical viability for long‐term autonomous deployment. Overall, HACTNet produces a versatile and expandable solution for next‐generation intelligent transportation systems by striking a balance between accuracy, robustness and efficiency.