Entropy-Driven Action Randomization: A Deployment-Time Defense for Environment Privacy in Deep Reinforcement Learning

Deep reinforcement learning (DRL) agents implicitly memorize the structure of their training environment, allowing an attacker to reconstruct it by querying the action outputs of a deployed model and causing environment-privacy leakage. To address this, this study proposes Entropy-Driven Action Randomization (EDAR), a deployment-time action randomization defense. Inspired by the exponential mechanism from differential privacy, EDAR replaces deterministic greedy action selection with probabilistic action sampling based on a normalized Q-value utility function, without altering the trained policy. A state-adaptive dynamic privacy budget is further designed, guided by the Shannon entropy of the action distribution. A simulated-annealing-based policy inversion attack is used to quantify privacy leakage, measured by the normalized recovery rate (NRR). Experiments on GridWorld environments ranging from 7 × 7 to 13 × 13 show that, at a privacy level comparable to a strong fixed budget, the proposed dynamic budget raises the reward retention from 6.2% to 78.9% while keeping a comparable NRR reduction. These results indicate that adapting the privacy budget to per-state policy determinism yields a markedly better privacy–utility trade-off than existing training-perturbation-based methods. We clarify that the differential privacy property invoked here is a per-query, mechanism-level guarantee of indistinguishability between neighboring observation states under a fixed trained model. The protection of the environment structure itself is established empirically through the policy inversion attack. Experiments are conducted on discrete GridWorld navigation tasks; the conclusions are scoped to small value-based navigation settings.