To build AI systems capable of decision-support assistance, such as AI-assisted healthcare, it is essential to develop user-centric decision-making processes that are robust, interpretable, and capable of effectively processing and acting on natural language interactions. Instruction-based prompting of large language models has demonstrated considerable success in supporting humans with information assistance tasks, including creative writing and content generation. However, recent studies reveal that language models exhibit limitations in performing complex reasoning and planning tasks, such as constructing compositional or hierarchical plans involving multiple reasoning steps. To address these challenges, we propose a neurosymbolic framework that integrates large language models with symbolic knowledge graphs, graph-based reasoners, and constraint-aware planning modules. This hybrid approach leverages the open-ended generative capabilities of language models—for instance, generating initial plans—while incorporating domain-specific structured representations through knowledge graphs to manage and refine these plans. The integration facilitates scalable and robust planning and reasoning in complex domains while adhering to domain-specific constraints. Through an example in the healthcare context, we explain how this framework offers the potential to enforce safety standards, ensure compliance with domain-specific regulations, and maintain logical consistency throughout extended decision-making processes. We further show the utility of this approach with examples drawn from complex decision-making environments in other domains (e.g., manufacturing), emphasizing its relevance and broader applicability in scenarios where trustworthy, safety-constrained planning and reasoning are critical.
Sheth et al. (Sat,) studied this question.