On the Effect of Keyhole Channel in RSMA Networks: A Theoretical Outage Analysis

This paper examines the theoretical performance of rate-splitting multiple access (RSMA) in keyhole fading channels. This rank-deficient environment is notorious for degrading the performance of traditional single-input single-output (SISO) systems. For a two-user downlink channel, exact and asymptotic outage probability results of RSMA with perfect and imperfect successive interference cancellation (pSIC and ipSIC) are derived. Closed-form solutions are derived by comparing the product of two independent Nakagami-m fading channels that model the keyhole effect. To further demonstrate RSMA’s robustness in such an environment, we also examine the diversity order and unveil the effect of keyhole-induced rank deficiency on system reliability. Our results demonstrate that RSMA retains a performance edge over non-orthogonal multiple access (NOMA), especially with imperfect SIC or low SNR. Numerical results andMonte Carlo simulations confirm the theoretical formulas and show that RSMA can combat the harmful effects of the keyhole channel better than other conventional schemes. The results confirm the promise of RSMA for future wireless systems operating in severe fading environments.