Reinforcement Learning-based Bandwidth Decision in Optical Access Networks: A Study of Exploration Strategy and Time with Confidence Guarantee

Reinforcement learning (RL) has recently emerged as a promising solution for intelligence bandwidth decisions that reduce latency in optical access networks. Even though RL drives model-free self-adaptive bandwidth decisions, the learning time cost and the widely-known exploration-exploitation dilemma of when to apply the best decision learnt are challenging to address in the bandwidth decision context. This paper for the first time explores how to rapidly learn an optimal bandwidth decision with a known confidence level of the decision for minimizing optical access network latency. We investigate critical aspects, including reward acquisition and strategies to explore decisions, in an RL-based bandwidth allocation scheme. Applying renewal theory, we address the timing for the central office (CO) to acquire rewards from optical network units for accurate decision value evaluation. Further, we derive the relationship between the decision practice times and the confidence of the optimal decision in closed-form. A reward variance-oriented (RVO) exploration strategy is proposed, in which the CO selects bandwidth decisions with probabilities proportional to the reward variances. We prove that the RVO is the most time efficient in learning an optimal decision with a confidence guarantee. With numerical and extensive simulations, we validate the theory and compare several common strategies with the RVO.