Digital Twin Enhanced Federated Reinforcement Learning With Lightweight Knowledge Distillation in Mobile Networks

The high-speed mobile networks offer great potentials to many future intelligent applications, such as autonomous vehicles in smart transportation systems. Such networks provide the possibility to interconnect mobile devices to achieve fast knowledge sharing for efficient collaborative learning and operations, especially with the help of distributed machine learning, e.g., Federated Learning (FL), and modern digital technologies, e.g., Digital Twin (DT) systems. Typically, FL requires a fixed group of participants that have Independent and Identically Distributed (IID) data for accurate and stable model training, which is highly unlikely in real-world mobile network scenarios. In this paper, in order to facilitate the lightweight model training and real-time processing in high-speed mobile networks, we design and introduce an end-edge-cloud structured three-layer Federated Reinforcement Learning (FRL) framework, incorporated with an edge-cloud structured DT system. A dual-Reinforcement Learning (dual-RL) scheme is devised to support optimizations of client node selection and global aggregation frequency during FL via a cooperative decision-making strategy, which is assisted by a two-layer DT system deployed in the edge-cloud for real-time monitoring of mobile devices and environment changes. A model pruning and federated bidirectional distillation (Bi-distillation) mechanism is then developed locally for the lightweight model training, while a model splitting scheme with a lightweight data augmentation mechanism is developed globally to separately optimize the aggregation weights based on a splitted neural network structure (i.e., the encoder and classifier) in a more targeted manner, which can work together to effectively reduce the overall communication cost and improve the non-IID problem. Experiment and evaluation results compared with three baseline methods using two different real-world datasets demonstrate the usefulness and outstanding performance of our proposed FRL model in communication-efficient model training and non-IID issue alleviation for high-speed mobile network scenarios.