What question did this study set out to answer?

The objective is to evaluate how effectively large language models and large reasoning models can generate triaging rules for clinical decision support systems.

January 24, 2026Open Access

Large language models and conditional rules in clinical decision support systems

Key Points

The objective is to evaluate how effectively large language models and large reasoning models can generate triaging rules for clinical decision support systems.
Prompted various large language models (LLMs) and large reasoning models (LRMs) using different techniques.
Compared LLM-generated rule sets to clinical rule sets from the Pandemic Intervention Monitoring System (PiMS).
Evaluated effectiveness based on accuracy, interpretability, and rule complexity.
LLMs produced less complex and lower interpretability rule sets compared to the PiMS rule set.
Accuracy of LLM-generated rules ranged from 31.62% to 70.71%.
LRMs exhibited variable interpretability and improved accuracy ranging from 31.62% to 81.70%.

Abstract

Abstract Background Clinical Decision Support Systems (CDSS) improve patient outcomes and support sustainable health services by enhancing medical decisions. Developing rules for a CDSS is expensive due to delays in capturing and defining the rules through multiple iterations between clinicians and developers as the role of a clinician is patient care. Objective We investigate the effectiveness of large language models (LLMs) and large reasoning models (LRMs) in generating a triaging rule set for a CDSS. Methods We prompt various LLMs (GPT-3.5, GPT-4, GPT-4o, Gemini, Claude 3.5 Sonnet) and various LRMs (GPT-o1-mini, Grok-4, Claude 4 Sonnet) using alternative prompting techniques. We compare the LLM generated rule sets against the clinical rule set from our Pandemic Intervention Monitoring System (PiMS); a triaging CDSS built in collaboration with clinicians to monitor COVID-19 positive patients. Effectiveness is evaluated based on the accuracy, interpretability, and rule complexity. Results We identified that LLMs generated COVID-19 screening rule sets compared to triaging rule sets when not specifying the variables from our PiMS rule set. By including PiMS variables in our prompts, we discovered LLMs 1) had lower interpretability and rule complexity compared to the PiMS rule set, and 2) resulted in an average accuracy between 31.62% ± 0.19% and 70.71% ± 0.02%. While for LRMs, we identified that 1) interpretability varied between 3 and 94 compared to 41 identified in our PiMS rule set and 2) resulted in an average accuracy between 31.62% ± 0.19% and 81.70 ± 0.05%. Conclusions LLMs are limited in emulating clinical rule sets due to their simplicity and lack of complex reasoning. Despite LRMs improving effectiveness, they are still limited. LLMs and LRMs can generate a feasible initial rule set for CDSS. This can reduce time invested by clinicians and developers by minimising the number of iterations for refinement. Future work can explore integrating LLMs and LRMs with decision trees to improve effectiveness.

Read Full Paperexternally

KI fragen

Bookmark

View Full Paper

Cite This Study

Sivasothy et al. (Wed,) studied this question.

synapsesocial.com/papers/697460acbb9d90c67120a8e6 https://doi.org/https://doi.org/10.1007/s13755-026-00428-z

KI fragen

Bookmark

View Full Paper