FEE-RTDETR: dynamic frequency convolution guided motion-blur-aware detection of dense rockfalls and track intrusions on railway slopes

Intelligent detection of railway-slope rockfalls and track-intrusion regions is essential for improving operational safety. However, dense rockfall events and motion blur during rapid rolling often reduce detection accuracy. To address these challenges, this study proposes a Frequency-Enhanced and Edge-Aware Real-Time Detection Transformer (FEE-RTDETR) based on RT-DETR. The proposed model integrates Frequency Dynamic Convolution (FDConv), an Edge-aware Dynamic Frequency Feed-Forward Network (EDFFN) and an Efficient Up-Convolution Block (EUCB) to enhance dense-object detection, motion-blur-aware feature representation and computational efficiency. Experiments on the in-house dataset show that FEE-RTDETR achieves a precision of 91.3%, recall of 91.1%, mAP50 of 91.4%, mAP50–95 of 86.0% and an F1-score of 0.912, with 50.7 GFLO Ps. Compared with representative YOLO variants, the proposed model demonstrates competitive detection performance. Qualitative results indicate strong robustness under dense scenes, motion blur and sleeper-like distractors. Additional experiments on external datasets and real-site engineering data further verify its generalization capability. The proposed method provides an effective solution for intelligent monitoring of railway-slope rockfall and track intrusion.