Reinforcement learning agents often struggle in sparse-reward environments where feedback is limited and appears only after a sequence of correct actions. In partialobservable navigation tasks, simple exploration strategies are often insufficient. This study investigates intrinsic motivation mechanisms, specifically focusing on the “Don’t Do What Doesn’t Matter” (DoWhaM) method, which rewards rare but effective actions. To address its limitations in spatial tasks, we propose Area-aware DoWhaM Adaptation (ADA). This method extends action-usefulness with spatial novelty bonuses to encourage expanding the visible area. We evaluate ADA against DoWhaM and a Count-Based baselines in various MiniGrid environments. Results indicate that ADA improves sample efficiency in the early stages of training. In dynamic environments where the layout changes in every episode, ADA significantly outperforms the Count-Based baseline and learns faster than DoWhaM. These findings suggest that combining action-usefulness with spatial novelty provides a robust heuristic for exploration in procedurally generated tasks.
Eren et al. (Tue,) studied this question.