Static Oracles Fail under Distribution Shift: The Primacy of Endogenous Horizon Generation in Reinforcement Learning

Traditional reinforcement learning (RL) typically frames time as an exogenous budget: a fixed horizon or discount factor that mechanically truncates agent trajectories. This paper explores an alternative, complementary perspective: time may be treated as an endogenous resource that agents can extend through targeted exploration. We introduce the Endogenous Horizon Markov Decision Process (EH-MDP), where the remaining temporal budget τ_t is included in the augmented state space, and its depletion rate is tied to the agent’s information gain. Within this framework, agents learn to adaptively allocate temporal resources: visiting novel states slows time consumption, effectively granting additional steps to reach sparse, delayed rewards. We first validate the core mechanism in the MiniGrid-Empty-8x8 environment using a strict causal identification protocol with six orthogonal ablations across 15 random seeds. Our full model achieves a mean success rate of 10.4% (90% CI [3.7%, 17.8%]), while all baseline models (Reward Only, Random Time, Time Only, Permuted Time Placebo) yield near-zero success rates (<2.0%). We then evaluate the framework under extreme distribution shift: agents trained exclusively on the 8x8 maze are directly transferred to a 16x16 maze (shortest path ≈30 steps) while the initial physical time budget remains fixed at 12 steps. The Placebo model—which shares the same intrinsic reward and empirical time-consumption distribution as our full model, but lacks the state-dependent causal structure—collapses to a 2.0% success rate, exhibiting what we term cognitive rigidity. In contrast, our full model adaptively generates a new temporal distribution, achieving a 12.4% success rate (p = 0.0305 vs. placebo) and, in some seeds, reaching the goal in 40 physical steps—3.33 times the initial budget. Wasserstein distance analysis confirms a measurable shift between time-consumption distributions in the 8x8 and 16x16 environments (unconditional W-dist = 0.0825, p < 0.01). These results suggest that the ability to dynamically generate adaptive temporal distributions, rather than relying on a static distribution learned in training, may support more robust generalization under distribution shift, and that fixed oracles can be brittle in out-of-distribution settings.