Adaptive Prioritized Experience Replay for Deep Q-Networks via TD-Error Dispersion

Prioritized Experience Replay (PER) samples transitions in proportion to their Temporal Difference (TD) error, which improves the sample efficiency of Deep Q-Networks (DQN). Conventional PER, however, keeps the prioritization strength α fixed for the entire training run. This paper proposes a TD-error adaptive PER strategy: after each optimization step, α is reset from the standard deviation of the batch's TD errors and bounded within [0.2, 0.8], so that how strongly the buffer is prioritized changes with what the network is currently learning. The method is evaluated in CartPole-v1 against Uniform DQN and fixed-α PER with several α values. Across five seeds, the adaptive strategy showed the smallest cross-seed variance in training performance (26.92) and the highest mean reward under greedy evaluation (239.10), even though its final training score of 115.97 stayed below the best-tuned fixed-α setting. Adaptive prioritization therefore looks promising, but it has not yet beaten a carefully chosen fixed-α in a consistent way.