Abstract This paper focuses on improving the efficiency of multiple-input multiple-output hybrid radio frequency/free-space optical (RF/FSO) communication systems. This is achieved by employing a combination of hybrid on-off keying (OOK) modulation, M-ary digital pulse position modulation (M-ary DPPM), and M-pulse amplitude and position modulation (M-PAPM). The study aims to analyze and enhance bit-error-rate performance using techniques such as the moment generating function, the modified Chernoff bound, and the Gaussian approximation, while accounting for challenges like amplified spontaneous emission noise, atmospheric turbulence (AT), pointing errors (PEs), and interchannel crosstalk. The proposed system model is based on a passive optical network (PON) that utilizes wavelength division multiplexing (WDM) for dense WDM (DWDM). By implementing eight DWDM channels in the C-band, each transmitting 2.5 Gbps data streams across eight spatial paths, the system achieves an aggregate throughput of 160 Gbps while maintaining compatibility with standard RF/FSO PON fiber networks. The integration of adaptive optics is also suggested to mitigate the effects of AT and PE, thereby improving modulation efficiency. The study reveals that the proposed M-ary hybrid DPPM-M-PAPM solution increases receiver sensitivity compared to OOK, ensuring greater reliability. It achieves a lower power penalty of 0.2–3.0 dB at a low coding level (M) of 2 under weak turbulence conditions.
Elsayed et al. (Wed,) studied this question.