RNN based MMSE detection for massive MIMO systems with OTA and imperfect channel state information

This study investigated the bit-error-rate (BER) performance of various detection algorithms for massive multiple-input multiple-output (MIMO) systems under realistic channel impairments, including channel estimation errors, Rayleigh fading, and over-the-air (OTA) conditions. A novel recurrent neural network–minimum mean-square error (RNN-MMSE) detection scheme is proposed that synergistically integrates RNN learning with MMSE estimation to enhance detection robustness. Under a severe 20% channel estimation error, the proposed approach achieves a BER of 10 − 3 at approximately 12.2 dB, offering a substantial signal-to-noise ratio (SNR) gain of 7.3 dB over the traditional zero-forcing equalizer (ZFE) and approximately 3–4 dB over a machine learning-based convolutional neural network (CNN) and RNN detectors. With reduced channel error (10%), the RNN-MMSE attains the same BER at 9.8 dB, maintaining impressive gains of 8.7 dB over ZFE and 3.4 dB over RNN. In Rayleigh fading, the method requires only 8.5 dB to reach 10 − 3 BER, yielding a nearly 9.7 dB improvement over ZFE. In practical OTA tests, the scheme consistently outperforms, requiring just 9.8 dB—about 9.2 dB better than ZFE and 3 dB better than RNN. Complexity analysis shows that while RNN-MMSE introduces additional computational overhead, it remains tractable \(\:O(T{N}²+{N}³),\) relative to the exponential complexity of maximum likelihood detection (MLD). Overall, the proposed method significantly lowers the required SNR, enhances the energy efficiency, and ensures robust communication across diverse practical scenarios, making it a strong candidate for next-generation fifth-generation (5G) and sixth-generation (6G) massive MIMO deployments, where maintaining link reliability under imperfect channel conditions is critical.