Energy-Efficient Content Fetching Strategies in Cache-Enabled D2D Networks via an Actor-Critic Reinforcement Learning Structure

As one of the important complementary technologies of the fifth-generation (5G) wireless communication and beyond, mobile device-to-device (D2D) edge caching and computing can effectively reduce the pressure on backbone networks and improve the user experience. Specific content can be pre-cached on the user devices based on personalized content placement strategies, and the cached content can be fetched by neighboring devices in the same D2D network. However, when multiple devices simultaneously fetch content from the same device, collisions will occur and reduce communication efficiency. In this paper, we design the content fetching strategies based on an actor-critic deep reinforcement learning (DRL) architecture, which can adjust the content fetching collision rate to adapt to different application scenarios. First, the optimization problem is formulated with the goal of minimizing the collision rate to improve the throughput, and a general actor-critic DRL algorithm is used to improve the content fetching strategy. Second, by optimizing the network architecture and reward function, the two-level actor-critic algorithm is improved to effectively manage the collision rate and transmission power. Furthermore, to balance the conflict between the collision rate and device energy consumption, the related reward values are weighted in the reward function to optimize the energy efficiency. The simulation results show that the content fetching collision rate based on the improved two-level actor-critic algorithm decreases significantly compared with that of the baseline algorithms, and the network energy consumption can be optimized by adjusting the weight factors.