Prompt-driven large language model for evidence extraction in implementation science: exploring semantic similarity between manual and automated coding across studies

Despite rapid scientific advancements that generate clinical interventions to optimize disease prevention, diagnostics, and treatments, significant gaps remain between evidence and clinical adoption. Implementation science evaluates targeted strategies systematically designed to promote the adoption, integration, and sustainment of evidence-based interventions, programs, and policies in real-world settings. However, identifying and coding factors associated with implementation success reported in the literature is time-consuming and labor-intensive. Large language models (LLMs) offer potential to automate and scale this process. We aimed to evaluate the potential of LLMs to accurately extract determinants associated with implementation in oncology aligned with a well-recognized implementation framework. This exploratory pilot study assessed the feasibility of using a Llama-based locally deployed LLM to extract CFIR-aligned barriers and facilitators from a purposively selected sample of twelve published studies, including oncology-focused implementation studies (containing structured implementation data) and published clinical intervention studies (containing unstructured implementation data). An ontology-based schema guided data structuring and three prompting strategies – zero-shot (task instructions only), few-shot (task instructions plus illustrative examples), and chain-of-thought (CoT) (instructs the model to generate step-by-step reasoning) – were evaluated to assess extraction performance. Semantic similarity between LLM-generated and human-coded outputs was assessed using both human rating and automated rating with a sentence transformer model. A total of 144 extraction outputs were generated. For implementation science studies, zero-shot prompting showed the highest average semantic similarity for both barriers and facilitators, compared with few-shot and CoT prompting. In clinical intervention studies, few-shot prompting showed higher average similarity scores of 0.78 (barriers) and 0.76 (facilitators) via sentence transformer, and 0.68/0.67 via human rating, than zero-shot and CoT. Across all prompting approaches, human ratings were slightly lower than automated ratings, although the reasons for this difference were not directly examined. Overall, the LLM reduced annotation time by approximately 75–95%, decreasing per article coding from several hours to under 20 min. In this exploration, a locally deployed LLM showed preliminary potential to support extraction of implementation science determinants, with the prompting strategies varying by study type. The small purposively selected sample limits external validity, and the findings should be interpreted as feasibility evidence rather than definitive validation of LLM extraction quality. Larger, systematically sampled, independently annotated, expert-adjudicated datasets are needed before broader claims can be made about accuracy and generalisability.