Trellis-coded OAM-SK multiplexing system based on spatial indexing and a multi-task neural network for turbulence-suppressed transmission

To meet demands for high-capacity optical transmission, multi-state orbital angular momentum shift keying (OAM-SK) systems offer an effective solution. However, atmospheric turbulence (AT) degrades vortex beams (VBs), reducing transmission quality and limiting long-distance transmission. In this paper, we propose a trellis-coded OAM-SK multiplexing system based on spatial indexing and a multi-task neural network (MTNN). Virtual channels and spatial indexes within the constellation space of trellis-coded modulation create a source-interaction framework, and the MTNN suppresses AT effects. The system can directly extract information from virtual channels via spatial indexing without increasing channel overhead. Additionally, it employs a designed Spatial-OAM encoder to reduce the dimensionality of encoded multi-state OAM during two-source multiplexing, which can lower detection complexity, improve transmission capacity, and distance. Experimental results show that the system can effectively utilize spatial indexes for inter-source switching while suppressing turbulent effects, without increasing channel redundancy. For C n 2 ≤ 2×10 −13 m −2/3 , the MTNN maintains spatial index classification accuracy above 99.28%. We then encode 16-bit multi-state OAM using a 16,000-length pseudo-random binary sequence. Through the Spatial-OAM encoder, the proposed scheme can convert 16-bit multi-state OAM modes into 4-bit single OAM modes, followed by multiplexing without altering transmission characteristics. Furthermore, when C n 2 =5×10 −14 m −2/3 , the bit error rate (BER) is 1.5 × 10 −3 at a transmission distance of 1000m, demonstrating the system’s potential for long-distance transmission.