HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Discrete-continuous hybrid action space is a natural setting in many problems, such as robot control and game AI. However, most previous Learning (RL) works only demonstrate the success in controlling either discrete or continuous action space, while seldom take into account hybrid action space. One naive way to address hybrid action RL is to the hybrid action space into a unified homogeneous action space by or continualization, so that conventional RL algorithms can be. However, this ignores the underlying structure of hybrid action space also induces the scalability issue and additional approximation, thus leading to degenerated results. In this paper, we propose Action Representation (HyAR) to learn a compact and decodable latent space for the original hybrid action space. HyAR constructs the space and embeds the dependence between discrete action and continuous via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is to be semantically smooth through unsupervised environmental dynamics. Finally, the agent then learns its policy with conventional DRL in the learned representation space and interacts with the by decoding the hybrid action embeddings to the original action. We evaluate HyAR in a variety of environments with discrete-continuous space. The results demonstrate the superiority of HyAR when compared previous baselines, especially for high-dimensional action spaces.