Binary Transformer Detectors for Automatic Modulation Detection Under Realistic Radio Frequency Impairment Conditions

Automatic modulation classification (AMC) is a core capability for spectrum monitoring, adaptive receivers, and electronic support. Most radio-frequency machine learning (RFML) studies train multi-class classifiers on benchmark datasets that contain a single modulation per recording at baseband. In operational settings, however, the objective is often to detect only a small set of signals of interest, making large multi-class models unnecessarily expensive to train and deploy. In addition, multi-class formulations can increase false-alarm risk due to confusion among non-essential classes and may allocate model capacity inefficiently to distinctions that are irrelevant for the operational objective. This paper investigates an alternative workflow based on targeted binary transformer detectors and evaluates their robustness under practical RF complications. Using the RadioML 2018.01A dataset, we construct binary detection tasks with BPSK as the signal of interest and introduce three increasingly realistic conditions: (i) center-frequency shifts away from baseband, (ii) sampling-rate mismatches via decimation and interpolation, and (iii) multi-signal mixtures where modulations co-occur either in frequency (simultaneous transmissions) or in time (temporal concatenation). The results show that baseband-trained detectors do not generalize to center-frequency-shifted signals, and multi-signal interference can cause complete detection failure unless explicitly modeled during training. We investigate early-exit transformer inference to reduce computation on high-confidence examples, showing it maintains (and occasionally improves) detection performance. We also evaluate inter-modulation transfer learning and intra-modulation adaptation from baseband to mixed- and multi-signal scenarios.