User Access Control in Open Radio Access Networks: A Federated Deep Reinforcement Learning Approach

Targeting at implementing the next generation radio access networks (RANs) with virtualized network components, the open RAN (O-RAN) has been regarded as a novel paradigm towards fully open, virtualized and interoperable RANs. Through particularly introducing RAN intelligent controllers (RICs), machine learning (ML) can be unprecedentedly installed, adapting to various vertical applications and deployment environments without sophisticated planning efforts. However, the O-RAN also suffers two critical challenges of load balancing and frequent handovers in the massive base station (BS) deployment. In this paper, an intelligent user access control scheme with deep reinforcement learning (DRL) is proposed. To optimize the performance of distributed deep Q-networks (DQNs) trained by user equipments (UEs), a federated DRL-based scheme is proposed with a global model server installed in the RIC to update the DQN parameters. To further predictively train a global DQN with acceptable signaling overheads, the upper confidence bound (UCB) algorithm to select the optimal UE set and a dueling structure to decompose the DQN parameters are developed. With the proposed scheme, each UE effectively maximizes the long-term throughput and avoids frequent handovers. The simulation results well justify the outstanding performance of the proposed scheme over the-state-of-the-arts, to serve as references for the O-RAN standardization.