Optimized deep learning‐based channel estimation for pilot contamination in a massive multiple‐input‐multiple‐output‐non‐orthogonal multiple access system

Summary One of the advanced field in 5G cellular networks is the Massive Multiple‐Input‐Multiple‐Output (MIMO), which creates a massive antenna array by offering numerous antennas at the destination. This grows as a hot research topic in the wireless sectors as it enhances the volume and spectrum usage of the channel. The spectral efficiency (SE) is maximized using the abundant antennas employed by MIMO using spatial multiplexing of consumers, which needs precise channel state information (CSI). The SE is affected by both pilot overhead and pilot contamination. To mitigate the contamination and to estimate the suitable channel for communication, an efficient strategy is introduced using the proposed Namib Beetle Aquila optimization (NBAO) Deep Q network (DQN). Here, the optimal pilot location is identified by employing NBAO, which is an integration of Namib beetle optimization (NBO) and Aquila optimizer (AO). Moreover, DQN is introduced to determine the suitable channel and metrics, such as bit error rate (BER) and normalized mean square error (MSE) is used for evaluation. The normalized MSE channel estimation is utilized to mitigate the effects of pilot contamination. Additionally, designed NBAO + DQN have attained a value of 0. 0006 and 0. 0005 for BER and normalized MSE.