Nonlinearity-Robust IM/DD THz Communication System via Two-Stage Deep Learning Equalizer

Linear and nonlinear impairments restrict the transmission performance of high-speed terahertz (THz) communication systems. To improve transmission performance, we propose a two-stage nonlinear equalizer (NLE). In the first stage, a memory-controlled long short-term memory (LSTM) neural network learns channel nonlinearity and compensates for it through nonlinear waveform regression. In the second stage, a low-complexity deep random forest (RF) network identifies nonlinear boundaries among individual QAM symbols and adjusts the standard hard decision thresholds of the QAM demodulator to align with the distribution of received symbols. This study experimentally validates the proposed two-stage NLE on a dual-channel THz-over-fiber transmission system using an intensity modulation and direct detection (IM/DD) scheme, achieving a successful 20 Gbps line rate up to a 4.5-meter wireless link at both 125/300 GHz frequency bands. The proposed scheme outperforms a Volterra nonlinear equalizer in all tested scenarios, surpassing a linear equalizer (LE) by reducing the bit error rate (BER) from2.47 × 10⁻³to2.61 × 10⁻⁴in the 300 GHz link and from3.42 × 10⁻³to5.64 × 10⁻⁴in the 125 GHz link.