Robust AI-Driven Spectral Imaging for Perioperative Care

Physicians face major challenges in perioperative decision-making, as they need to rely on clinical intuition and limited information for critical real-time judgments. Spectral imaging (SI) could support this process by rapidly and non-invasively revealing changes in tissue composition that alter spectral signatures. While such changes often remain invisible to the human eye or conventional RGB imaging, SI captures subtle variations in tissue reflectance spectra at each pixel. Combined with machine learning (ML), this high-dimensional data could efficiently yield clinically relevant insights. Numerous proof-of-concept studies have demonstrated the potential of SI, particularly for estimating functional tissue parameters such as oxygenation, thereby enabling non-invasive distinction between perfused and ischemic tissue during surgery. However, several important clinical applications of SI remain underexplored: Clinical Gap: Automated Surgical Scene Segmentation Visual discrimination of tissue types remains an important challenge for surgeons, and automated surgical scene segmentation is a key component of surgical data science applications such as surgical phase recognition and robot-assisted surgery. However, SI-based segmentation, particularly in open surgeries, has received little attention. Consequently, it remains unclear whether SI offers advantages over other imaging modalities (e.g., RGB imaging) for surgical scene segmentation and how to optimally represent the input data in terms of spatial granularity (e.g., pixels, entire images). Leveraging the largest semantically annotated SI database to date, we close this gap and demonstrate that SI consistently outperforms RGB across all spatial granularities. Our image-based SI segmentation reaches performance comparable to a second human expert. Clinical Gap: Sepsis Diagnosis and Mortality Prediction in Critically Ill Patients Sepsis remains a leading cause of mortality and critical illness. Early detection is vital to reduce mortality risk, yet reliable biomarkers for timely diagnosis and outcome prediction are still lacking. Sepsis diagnosis in intensive care unit (ICU) patients is particularly challenging due to high baseline illness severity. SI could potentially close this gap by capturing early signs such as edema formation and microcirculatory dysfunction. However, prior studies compare sepsis patients to healthy volunteers or narrowly selected cohorts, introducing a substantial risk of shortcut learning from confounding factors such as age and treatment regimens. We address this critical gap through a prospective study in ICU patients, comprising the largest SI patient cohort to date, in which we diagnose sepsis and predict mortality on the day of admission. Our SI-based ML models achieve high accuracy, particularly when combined with minimal clinical data, and outperform widely used biomarkers and scores, while enabling rapid, non-invasive, cost-effective and mobile assessments. Technical Gap: Investigation of Domain Shifts A key challenge for SI analysis is its clinical translation. Numerous studies outside medical SI have shown that domain shifts between training and real-world application data can severely degrade algorithm performance, yet this issue has received little attention in medical SI. We are the first to investigate the impact of important real-world domain shifts: Illuminant and hardware-related shifts in functional tissue parameter estimation, geometric shifts (e.g., situs occlusions) in surgical scene segmentation, and population shifts in sepsis diagnosis and mortality prediction. Our results show that such shifts can substantially degrade downstream task performance. Technical Gap: Mitigating performance degradation under domain shifts We propose methods to mitigate the performance degradation under domain shifts and improve algorithm robustness. To address drops in functional tissue parameter estimation due to illuminant changes, we introduce the first intraoperative, live illuminant estimation approach. Our method outperforms state-of-the-art illuminant estimation techniques from nonmedical domains, achieving accuracy close to the ideal scenario of a perfectly known illuminant. Additionally, we provide recommendations to mitigate hardware-related bias in SI study design. To enable robust surgical scene segmentation under geometric domain shifts, we introduce a surgery-inspired data augmentation strategy which restores in-distribution performance across diverse out-of-distribution scenarios. In conclusion, this thesis contributes substantial advancements towards the robust and reliable application of ML-based SI analysis in real-world clinical settings. Specifically, it enables, for the first time, (1) intraoperative functional tissue parameter estimation under illuminant and hardware-related shifts, (2) automated surgical scene segmentation under geometric domain shifts, and (3) automated sepsis diagnosis and mortality prediction among ICU patients. Our findings are supported by extensive validation studies which are among the largest in the field of medical SI to date. To support the research community and facilitate the clinical translation of SI, we have publicly released datasets, as well as our code and pretrained models.