Deep Dive Into Model-Free Reinforcement Learning for Underwater Locomotion: Theory and Practice

Aquatic animals and underwater robots operate in a complex physical world and must coordinate their bodies to achieve behavioral objectives such as navigation and predation. With recent developments in deep reinforcement learning (RL), it is now possible for scientists and engineers to synthesize sensorimotor strategies (policies) for specific tasks using physically simulated bodies and environments. However, beyond solving individual control problems, these methods offer an exciting framework for understanding the organization of an animal sensorimotor system in connection with its morphology and physical interaction with the environment, as well as for deriving general design rules for bioinspired underwater robots. Although algorithms and code implementing both learning agents and environments are increasingly available, the basic assumptions and modeling choices that go into the formulation of an embodied feedback control problem using deep reinforcement learning may not be immediately apparent. In this tutorial, we provide a self-contained introduction to model-free reinforcement learning for embodied agents in underwater environments, with a focus on actor-critic methods. We first present the mathematical formulation of RL, highlighting where physical modeling choices enter. We then discuss the practical aspects of implementing actor-critic algorithms. Drawing on recent examples of RL-controlled swimmers, we provide guidelines for choosing observations, actions, and rewards consistent with biological behavior, and we outline how RL can be used as a tool to explore hypotheses about the feedback control underlying animal and robotic behavior.