Joint Multi-Channel Dual-Polarization Autoencoder for Scalable End-to-End Long-Haul Optical Transmission

This paper introduces a Joint 4-channel wavelength-division multiplexed (WDM) Dual-Polarization Autoencoder (J-4WDM-DPAE) framework to address the scaling limitations of existing end-to-end learning architectures for long-haul coherent optical transmission. The proposed approach employs a unified one-dimensional residual convolutional neural network (CNN) decoder that processes all eight complex symbol streams (4 WDM channels × 2 polarizations) at the symbol rate, enabling simultaneous exploitation of inter-channel and inter-polarization correlations with low inference latency. The transceiver is trained through a fully differentiable dual-polarization Manakov split-step Fourier method (SSFM) model including span-wise amplified spontaneous emission (ASE) noise and an effective combined transmitter–local-oscillator phase-noise process, enabling co-optimization of a shared geometric constellation shaping (GCS) encoder under realistic nonlinear and linewidth constraints. Robustness is further enhanced by randomized launch powers and signal-to-noise ratio (SNR) conditions during training. Additional robustness is assessed by cross-SPS evaluation (SSFM-resolution mismatch) and SSFM convergence checks, indicating that the reported achievable-rate trends are not an artifact of the baseline SSFM discretization. Evaluations over standard single-mode fiber (SSMF) for 16-, 32-, and 64-quadrature amplitude modulation (QAM) show that at 1000 km, the learned constellations achieve generalized mutual information close to the dual-polarization limits, with pre-FEC bit-error rates remaining below an adopted threshold of 2×10−2 (used as a representative soft-decision FEC operating target) up to 4000 km when the model is re-trained for each distance. Complexity analysis further indicates that the unified WDM-aware decoder provides a quantitative performance–computational trade-off compared to existing counterparts under matched link conditions.