Leveraging Data Science for Conducting Observational Studies: Highlighting Advantages and Limitations Throughout the Evaluation of a Use Case

This is the authors' abstract. We don't add key points for this paper.

Objective To develop a data science pipeline for data extraction and collection to conduct an observational study using institutional medical informatics and AI tools. Patients and Methods: skin excision cases exposed to intra-incisional clindamycin (from February 5 to October 14, 2025) and non-exposed (from January 1 to December 31, 2022) were defined as our cohorts of interest. The outcome of our study was surgical site infection (SSI) 30-days following surgery. An LLM-supported data science pipeline was used for cohort identification, case screening, and extraction of procedural information. LLM screening and data extraction were validated in 300 random cases. A GEE model was used to analyze the effect of intra-incisional clindamycin on SSI. Results The LLM achieved high accuracy for screening cases for procedures done in the head (accuracy: 99.0%, 95%CI:[97.1-99.8%]) and skin excisions (97.0%, 95%CI:[94.4-98.6%]). Additionally, it achieved remarkable performance for extraction of clinical information, with the highest accuracy observed for extraction of anatomical location associated with the procedure (accuracy: 100.0%, 95%CI:[98.8-100.0%]). The final dataset included 2247 skin excision cases (594 exposed; 1653 non-exposed) from 1923 patients. Cases that received intra-incisional clindamycin had lower odds of SSI, however, this association was not statistically significant (aOR: 0.57, 95%CI:[0.31-1.07]). Conclusion The implementation of medical informatics tools and robust AI algorithms will enable the conduct of end-to-end research pipelines. However, it is important to always consider humans in the loop, as oversight ensures reliability and accuracy, and robustness.