Clinical insight from mesh implant narratives using zero-shot Retrieval-Augmented Generation approach

Aim: Mesh implant surgeries for hernia repair are frequently associated with adverse events that can compromise patient outcomes. Extracting structured clinical insights from large-scale, unstructured data sources - such as the U.S. Food and Drug Administration’s Manufacturer and User Facility Device Experience (FDA MAUDE) database - remains a significant challenge due to the variability and subjectivity in patient-generated narratives. Methods: This study evaluates the use of zero-shot prompting with a generative artificial intelligence (AI) model, both independently and in combination with a Retrieval-Augmented Generation (RAG) approach, to automate the extraction of structured summaries and adverse events from mesh-related FDA reports. The LLaMA 2 (13B) model was used to generate structured summaries (e.g., report date, hernia type, adverse event flag) and identify adverse events. Results: The model outputs were compared to a manually annotated Benchmark Baseline. With zero-shot prompting alone, the model achieved accuracies of 67% for report date, 60% for hernia type, and 83% for adverse event flag. After integrating the RAG approach, these accuracies improved to 81%, 82%, and 99%, respectively. The accuracy for adverse event extraction increased from 60% to 86%, and the Jaccard similarity improved from 75% to 88.9%. Chi-square tests confirmed statistical significance (P < 0.05) for most of the observed improvements. Conclusion: This study demonstrates that combining zero-shot generative AI with retrieval augmentation can effectively convert unstructured patient reports into structured data. This approach offers a scalable and interpretable method for adverse event monitoring in mesh implant surgeries and supports data-driven evaluation of patient-reported outcomes.