HM-CDSS: A Hierarchical Multi-Agent Reinforcement Learning Framework with Conservative Q-Learning and Retrieval-Augmented Clinical Reasoning for Sepsis Decision Support

Decision-making in healthcare in the Intensive Care Unit (ICU) is a very complex and multi-dimensional process. Sepsis management requires well-coordinated, simultaneous decisions across vasopressor titration, fluid restoration, antibiotic supervision, and haemodynamic monitoring, each operating on different physiological timescales. Despite advances in reinforcement-learning-based clinical decision support, existing systems treat these as a single flat action space and cannot capture the multi-level structure of real clinical reasoning. Moreover, most systems lack explainability mechanisms, making them unsuitable for accountable clinical practice. This study proposes the Hierarchical Multi-Agent Clinical Decision Support System (HM-CDSS), a four-phase hierarchical multi-agent reinforcement learning framework designed to address these limitations. Using a cohort of 5,282 adult ICU sepsis patients from the Medical Information Mart for Intensive Care (MIMIC-IV) database, approximately 78,200 offline trajectory transitions are constructed. A single-agent Dueling DQN baseline achieves a mean test reward of 4.385 but exhibits conservative policy collapse, reaching only 10.3% clinician agreement. The proposed Hierarchical Multi-Agent Reinforcement Learning (HMARL) system introduces a meta-agent that assigns high-level clinical goals to four specialist sub-agents trained with Conservative Q-Learning (CQL), using agent-specific regularization weights (α = 1.0 for vasopressor and fluid agents; α = 0.5 otherwise) to prevent out-of-distribution Q-value overestimation. A three-condition ablation study confirms that CQL adds 21.8% points of clinician agreement above a hierarchical DQN baseline, and that hierarchy and distributional constraint are complementary mechanisms. Overall clinician agreement reaches 49.9%, a gain of 39.6% points over the single-agent baseline. Additionally, a retrieval-augmented generative reasoning layer, based on Sepsis-3 and Surviving Sepsis Campaign guidelines, produces chain-of-thought clinical justifications evaluated through a proposed five-metric framework comprising Guideline Grounding Score, Hallucination Rate, Clinical Consistency Score, Recommendation Completeness, and Uncertainty Calibration (UC; note that UC could not be computed empirically in the current evaluation due to uniformly high agent confidence). Across 98 evaluated timesteps, the system achieves 0% hallucination and 100% recommendation completeness across the 98-timestep evaluation sample; these results should be interpreted as preliminary, given the limited sample size. HM-CDSS proposes an integrated framework combining offline hierarchical MARL, guideline-based explainability, and quantified reasoning-quality assessment for ICU clinical decision support.