Hierarchical Game-Theoretic and Reinforcement Learning Framework for Computational Offloading in UAV-Enabled Mobile Edge Computing Networks With Multiple Service Providers

We present a novel game-theoretic (GT) and reinforcement learning (RL) framework for computational offloading in the mobile edge computing (MEC) network operated by multiple service providers (SPs). The network is formed by MEC servers installed at stationary base stations (BSs) and unmanned aerial vehicles (UAVs) deployed as quasi-stationary BSs. Since computing powers of MEC servers are limited, the BSs in proximity can form coalitions with shared data processing resources to serve their users more efficiently. However, as BSs can be privately owned or controlled by different SPs, in any coalition, the BSs: 1) take only the actions that maximize their long-term payoffs and 2) do not coordinate their actions with other BSs in the coalition. That is, inside each coalition, BSs act in an independent and self-interested manner. Therefore, the interactions among BSs cannot be described by conventional coalitional games. Instead, the network operation is modeled by a two-level hierarchical model. The upper level is a cooperative game that defines the process of coalition formation. The lower level comprises the set of noncooperative subgames to represent a self-interested and independent behavior of BSs in coalitions. To enable each BS to select a coalition and decide on its action maximizing its long-term payoff, we propose two algorithms that combine coalition formation with RL and prove that these algorithms converge to the states where the coalitional structure is strongly stable and the strategies of BSs are in the mixed-strategy Nash equilibrium (NE).