Dynamic Computation Offloading With Energy Harvesting Devices: A Hybrid-Decision-Based Deep Reinforcement Learning Approach

Mobile-edge computing (MEC) with energy harvesting (EH) is becoming an emerging paradigm to improve the computation experience for the Internet-of-Things (IoT) devices. For a multidevice multiserver MEC system, the frequently varied harvested energy, along with changeable computation task loads and time-varying computation capacities of servers, increase the system's dynamic. Therefore, each device should learn to make coordinated actions, such as the offloading ratio, local computation capacity, and server selection, to achieve a satisfactory computation quality. Thus, the MEC system with EH devices is highly dynamic and face two challenges: 1) continuous- discrete hybrid action spaces and 2) coordination among devices. To deal with such problem, we propose two deep reinforcement learning (DRL)-based algorithms: 1) hybrid-decision-based actor-critic learning (Hybrid-AC) and 2) multidevice hybrid-AC (MD-Hybrid-AC) for dynamic computation offloading. HybridAC solves the hybrid action space with an improvement of actor-critic architecture, where the actor outputs continuous actions (offloading ratio and local computation capacity) corresponding to every server, and the critic evaluates the continuous actions and outputs the discrete action of server selection. MDHybrid-AC adopts the framework of centralized training with decentralized execution. It learns coordinated decisions by constructing a centralized critic to output server selections, which considers the continuous action policies of all devices. Simulation results show that the proposed algorithms achieve a good balance between consumed time and energy, and have a significant performance improvement compared with baseline offloading policies.