Q-Learning-Based Adaptive Power Control in Wireless RF Energy Harvesting Heterogeneous Networks

This article investigates adaptive power control in wireless radio frequency energy harvesting (EH) femtocell heterogeneous networks (HetNets), where some EH devices desire to harvest energy from the signals transmitted from both macro base stations (BSs), and femtocell BSs. An optimization problem is formulated to maximize the sum capacity of femtocells while satisfying the EH requirements of energy users, and information quality of service (QoS) requirements of macrocell users by adaptively controlling the transmit power of femtocell HetNets, where nonlinear EH model is adopted. Due to the discrete variables of transmit power, the formulated optimization problem is with combinatorial computational complexity, and cannot be solved with known solution methods. Thus, a Q-learning-based algorithm is presented, and a segmented reward function based on the distance factor, and the penalty parameter is designed. Simulation results show the effectiveness of the proposed Q-learning-based algorithm, and the presented segmented reward function. It is also demonstrated that by using the nonlinear EH model can effectively avoid the deviation brought by the traditional ideal linear EH model. Additionally, it shows that the convergence time of our proposed scheme grows linearly w.r.t. the number of femtocell BSs, and the number of selectable transmit power levels of femtocell BSs roughly.