Evolution of Reinforcement Learning in Uncertain Environments: A Simple Explanation for Complex Foraging Behaviors

Abstract. Reinforcement learning (RL) is a fundamental process by which organisms learn to achieve a goal from interactions with the envi-ronment. Using Artificial Life techniques we derive (near-)optimal neu-ronal learning rules in a simple neural network model of decision-making in simulated bumblebees foraging for nectar. The resulting networks ex-hibit efficient RL, allowing the bees to respond rapidly to changes in reward contingencies. The evolved synaptic plasticity dynamics give rise to varying exploration/exploitation levels from which emerge the well-documented foraging strategies of risk aversion and probability matching. These are shown to be a direct result of optimal RL, providing a biolog-ically founded, parsimonious and novel explanation for these behaviors. Our results are corroborated by a rigorous mathematical analysis and by experiments in mobile robots. 1