Robust Transformer Neural Network based Signal Detector for Optical-OTFS in Doubly Dispersive Channels with BER and PSD Evaluation

This paper presents a comprehensive investigation of deep learning (DL)-assisted joint channel estimation and signal detection using a Transformer Neural Network (T-NN) for optical orthogonal time frequency space (OTFS) systems operating over doubly dispersive optical turbulence fading channels. Extensive simulations employing 256-ary quadrature amplitude modulation (256-QAM) demonstrate that the proposed T-NN significantly outperforms conventional and recent state-of-the-art OTFS detection approaches, including minimum mean square error (MMSE), QR-decomposition-based maximum likelihood detection (QRM-MLD), message passing (MP), approximate message passing (AMP), expectation propagation (EP), Deep Unfolding, and Bi-LSTM-based detectors. The considered optical channel model incorporates atmospheric attenuation, log-normal turbulence fading, delay-Doppler interference, and pointing error impairments to accurately represent practical free-space optical (FSO) communication environments. Bit error rate (BER) evaluations indicate that at a BER of , the proposed T-NN achieves substantial signal-to-noise ratio (SNR) gains of approximately 8.4–10 dB over baseline Optical-OTFS systems and nearly 1.4–8 dB improvements over advanced OTFS detectors, even under severe 10–20% channel estimation error conditions. Power spectral density (PSD) analysis further confirms that the proposed T-NN achieves the lowest PSD floor among all compared methods, thereby significantly reducing out-of-band spectral leakage under optical turbulence fading conditions. This enhanced spectral confinement improves spectral efficiency and minimizes adjacent-channel interference in dense optical communication environments. Complexity analysis reveals that while conventional and iterative OTFS detectors exhibit cubic to exponential computational complexity, the proposed T-NN maintains efficient quadratic complexity due to its self-attention mechanism. These results validate the robustness, adaptability, and generalization capability of the proposed T-NN for reliable, high-throughput, and energy-efficient Optical-OTFS communication systems under realistic imperfect channel conditions.