Benchmark to bedside: Building modular multimodal machine learning infrastructure for healthcare at scale

Healthcare systems generate vast amounts of comprehensive data daily; yet, translating this information into robust machine learning (ML) models that improve patient outcomes remains a challenge. The demand for scalable, practical, and problem-driven systems that use this heterogeneous data to its full capacity is growing as healthcare complexity and costs rise alongside the aging population. The first contribution establishes reproducibility as a cornerstone of clinical ML research. Despite the increasing availability of open-access intensive care unit (ICU) datasets and publications, the field lacks standardized approaches to data preprocessing, task definition, and evaluation. Researchers often employ custom datasets, ambiguous definitions, and dataset-specific pipelines, making it impossible to compare models meaningfully or build upon previous work. We address this need with Yet Another ICU Benchmark (YAIB) --- a modular, extensible framework designed to support reproducible clinical ML experiments across multiple datasets --- and ReciPies --- a concise, human‑readable, and reproducible way to declare, execute, and share preprocessing pipelines. YAIB standardizes the entire modeling workflow from raw data harmonization through clinical concept definition to model fitting and evaluation, thereby reducing implementation overhead and enabling transparent, comparable research. Critically, the framework demonstrates that dataset selection, cohort definitions, and preprocessing choices often have a more substantial impact on prediction performance than the choice of model architecture itself. The second contribution demonstrates the value of prospective clinical trials for specific healthcare Artificial Intelligence (AI) applications. As an interdisciplinary team of computer scientists and surgeons, we created a dataset that combines non-intrusive wearable vital signals with 1,285 patients, 270,603 hours of telemetric vital signs, and more than 8 million clinical measurements. We aimed to detect subtle endpoints in the general ward, which is often undermonitored and underrepresented in ML research. We developed a comprehensive machine-learning-driven early warning system that unifies preoperative parameters, intraoperative measurements, ICU data, clinical text, and continuous high-resolution vital signs from wearable devices deployed on surgical wards. The approach detects surgical site infections, pancreatic fistulas, hematomas, and bile leaks in the general ward. High-resolution sensor data provide improvements of up to 8% in the area under the receiver operating curve (AUROC) and 109% in the area under the precision-recall curve (AUPRC), underscoring the importance of capturing dynamic physiological complexity as an early marker of clinical deterioration. Our findings suggest the need for hybrid wards with custom AI early warning systems that lower diagnostic costs and care burden. The third contribution addresses the need for a standardized healthcare data infrastructure. To truly scale clinical AI across multiple institutions and datasets, the field requires tools that enable collaboration, reproducibility, and interoperability. This work contributes to the Medical Event Data Standard (MEDS) --- a lightweight, ML-first data schema designed to standardize the representation of electronic health record (EHR) data for AI use. MEDS provides a flexible, low-level abstraction layer that integrates with existing EHR processing tools and modeling pipelines, reducing the friction in clinical machine learning research. In this framework, we have developed multiple interoperable tools for data transformation to MEDS, exploration, and harmonization. The open-source ecosystem dramatically improves reproducible research output. Moreover, this thesis includes an open-source benchmark of 11 clinical tasks across eight open-access datasets and a multicenter health system comprising 12 million patients, demonstrating the value of reproducible, multi-institutional research efforts. Moreover, we benchmark state-of-the-art EHR foundation models trained on 6.8 million patients in various data regimes. Low data regimes demonstrate promising results for these models. MEDS enables scalable, privacy-preserving model evaluation across datasets and health systems, which should drive compounding iterative improvements to health AI across institutional boundaries. Throughout this work, a pattern emerges: successful clinical machine learning requires not only algorithmic innovation across the domain but also systematic attention to data quality, experimental methodology, and software infrastructure. The envisioned goal is to improve and prolong patient lives through trustworthy and equitable healthcare AI for the many.