Geometry-Aware Conformal Temperature Calibration for Entropic Action Selection in Reinforcement Learning

Entropy-based soft-min operators and Gibbs policies are widely used in reinforcement learning to smooth greedy decisions and regulate stochastic action selection. Their behavior depends critically on an inverse temperature parameter, which is often chosen through fixed values, annealing schedules, entropy targets, or worst-case bounds. These rules do not directly control the local error induced by soft action selection and do not account for the geometry of the action-value vector. This paper develops a geometry-aware conformal calibration framework for selecting inverse temperatures in discounted finite-action reinforcement learning. The analysis distinguishes the operator-level soft-min approximation error from decision-level Gibbs excess and shows how both quantities depend on local value gaps and a near-optimal action structure. A conformal order statistic rule is then used to obtain finite-sample marginal control of the selected local score under exchangeability. A geometry-conditional extension assigns different temperatures to different action-value geometries. The Bellman analysis clarifies that fixed temperature maps preserve contraction, while data-dependent conformal selectors are best interpreted as post-training action selection rules. A finite-MDP experiment empirically confirms the conformal coverage behavior and shows that calibration must be applied to a decision-relevant cost representation when value estimates contain optimistic value estimation traps. An Atari SpaceInvaders experiment with a frozen deep Q-network shows that the learned action-value geometry is heterogeneous, that decision-level calibration is less conservative than operator-level calibration, and that gated geometry-conditional selection preserves most of the greedy return while providing explicit marginal score control of randomized decision excess with low online overhead.