Multimodal PPG-Based Arrhythmia Detection Using a CLIP-Initialized Multi-Task U-Net and LLM-Assisted Reporting

Figures

• 

  Figure 2. Example of a normal PPG segment used in this study. The waveform shows stable amplitude and nearly periodic beat-to-beat intervals.

• 

  Figure 3. Example of an arrhythmia PPG segment with the corresponding abnormal mask (shaded region). The highlighted interval indicates the time span labeled as arrhythmic.

• 

  Figure 4. Representative arrhythmia PPG segments for (a) Atrial Fibrillation (AF), (b) Bradycardia, and (c) Tachycardia. Each shaded region corresponds to the abnormal interval used to generate the segmentation mask.

• 

  Figure 6. Overview of the CLIP-style contrastive pretraining framework. PPG waveforms are encoded via ResNet1D, clinical variables via an MLP, and rhythm descriptions via BioBERT. All are projected into a shared 128-dimensional space and optimized with InfoNCE losses for wave–clinical and wave–state alignment.

• 

  Figure 12. Example AF case correctly classified by the proposed model. x-axis: sample index (0–286) corresponding to resampled time; y-axis: normalized amplitude.

• 

  Figure 13. Example Bradycardia case correctly classified and segmented. x-axis: sample index (0–286) corresponding to resampled time; y-axis: normalized amplitude.

• 

  Figure 1. Overview of the proposed multimodal PPG-based arrhythmia detection framework integrating contrastive pretraining, multi-task learning, and LLM-based explainability. Physiological inputs consisting of PPG segments and HRV features, structured clinical metadata, and rhythm-specific textual descriptions are first aligned through CLIP-style contrastive learning, where waveform, clinical, and text representations are projected into a shared embedding space. The pretrained encoder is then reused to initialize a multi-task 1D U-Net that simultaneously performs four-class rhythm classification (Normal, Atrial Fibrillation, Bradycardia, and Tachycardia) and temporal segmentation of abnormal intervals. The model outputs, together with similar-patient retrieval results and guideline excerpts, are provided to a Retrieval-Augmented large language model (LLM), which generates structured and clinically grounded diagnostic reports. This architecture enables unified representation learning, temporal localization, and interpretable clinical reasoning within a single end-to-end system.

• 

  Figure 5. Baseline multi-task 1D U-Net. The encoder–decoder path predicts a segmentation mask for abnormal intervals, while a squeeze-and-excitation (SE) block refines the bottleneck feature, which is then fused with HRV features and passed to a classification head for four-class rhythm prediction.

• 

  Figure 7. Proposed CLIP-initialized multi-task U-Net. The ResNet1D encoder pretrained with the CLIP-BioBERT framework acts as the U-Net encoder (blue blocks), and its bottleneck output is projected to a 128-dimensional CLIP embedding. HRV features and 13 structured clinical variables (green blocks) are concatenated with this embedding and passed to a classification head for four-class rhythm prediction, while the decoder path produces a 1D segmentation mask that localizes abnormal intervals.

• 

  Figure 8. User interface for patient data management and aggregate analysis. (a) The registration form for inputting 13 clinical variables (e.g., demographics, electrolytes), which are essential for the clinical encoder. (b) The arrhythmia summary dashboard providing the distribution of predicted rhythms and retrieving relevant medical guidelines via the RAG framework.

• 

  Figure 9. The on-demand “Deep Diagnosis” reporting interface. (a) RAG-based clinical report: the system retrieves similar patient cases from the vector database to provide statistical evidence (e.g., “AF was a significant outcome in 45.6% of similar instances”) and clinical impressions. (b) Pure waveform analysis: the vision-capable LLM analyzes the morphological features of the signal image, identifying rhythm irregularities and signal quality issues (e.g., baseline drift, compensatory pauses) to support the final diagnosis.

• 

  Figure 10. Confusion matrix of the baseline multi-task U-Net on the test set. Axes: true label (y) vs. predicted label (x).