Evaluative Expressivity: Why Optimal Multi-Objective Policies Require State-Dependent Valuation

We prove a strict expressivity gap in multi-objective reinforcement learning: some Pareto-optimal policies cannot be made Bellman-optimal by any constant scalarization weight, even when their value vectors lie on the convex hull, because the Bellman optimality conditions at different states impose contradictory requirements on the weight. This limitation is distinct from the known convex hull limitation (Das and Dennis, 1997) and operates at the policy level rather than the value level. State-contingent weights recover these policies, and feasibility is testable by a standard linear program. Across random instances, 25–42% of Pareto-optimal policies require state-contingent weights. Agents with constant weights recover zero such policies across all tested seeds; agents with per-state weights recover 50–100%. The gap persists under neural function approximation. A bilevel algorithm, evolutionary multi-objective optimization of per-state weights with Bellman-consistent inner-loop training, recovers all per-state-only policies in both tabular and continuous-state neural settings.