LLM-in-the-Loop execution of clinical quality language

Objective: We present a method and open-source prototype to augment Clinical Quality Language (CQL) using a large language model (LLM) in the execution loop for querying unstructured text. Materials and Methods: We modified a popular CQL engine to trigger an LLM-in-the-Loop pipeline (LitL) whenever CQL references unstructured FHIR resources, without altering the existing structured resource handling. LitL generates a binary response to natural language queries from CQL using an LLM. Results: We present an open-source prototype that includes: an implementation of the LitL pattern, and a modified CQL execution engine that triggers LitL for unstructured data. The feasibility testing with two locally hosted LLMs achieved an accuracy of 72% and 93% respectively. An end-to-end prototype is provided to facilitate implementation with configurable models, prompts and hyperparameters. Conclusion: The LitL pattern extends CQL to unstructured data, preserving compatibility with existing CQL and reducing hallucination. All components of the prototype are free and open source.