Offline reinforcement learning for care management: addressing sparse rewards, temporal dependencies, and fairness in Medicaid populations

Abstract Reinforcement learning offers a principled framework for optimizing sequential clinical decisions, yet deployment in healthcare remains limited by extreme variance inflation in off-policy evaluation when action spaces are high-dimensional. We present an offline reinforcement learning framework centered on doubly robust off-policy evaluation, integrated with factored action space decomposition, multi-component reward shaping, bidirectional LSTM networks with attention, and fairness-constrained training. Applied to 9,998,139 weekly observations from 160,264 Medicaid beneficiaries enrolled in population health management programs (2023–2025), standard weighted importance sampling achieved an effective sample size of 0.0% due to variance inflation, while doubly robust evaluation achieved 44.2% (147-fold improvement), enabling stable policy assessment. The learned policy was statistically non-inferior to observed clinician behavior. Temporal modeling with attention significantly outperformed feedforward baselines, and fairness constraints reduced racial/ethnic intervention disparities from 3.5 to 0.8 percentage points. This framework provides methodological infrastructure for prospective clinical trials of reinforcement learning-based decision support systems in Medicaid populations.