Accelerating Machine Learning in Health Care by Addressing the Labeling Bottleneck: Development and Evaluation Study

Background: Timely detection of postoperative arrhythmias after pediatric cardiac surgery is essential for preventing hemodynamic compromise, yet machine learning models are constrained by the scarcity of labelled datasets reflecting real-world monitoring. Existing approaches rely on adult 12-lead electrocardiograms, which are rarely used continuously in pediatric intensive care units (ICUs) and fail to capture age-dependent waveform variability. Rare arrhythmias such as junctional ectopic tachycardia are especially difficult to find and label within large continuous monitoring streams. Objective: We evaluated how efficiently a staged, self-bootstrapping labeling pipeline surfaces a rare arrhythmia class relative to naive random review of an unlabelled dataset, and how that efficiency evolves across the pipeline's dependent stages. Methods: Leveraging a repository of over 1.6 million hours of unlabelled continuous lead II electrocardiogram (ECG) from more than 9,000 pediatric patients, we implemented four sequential sourcing stages, each enabling the next: retrospective MUSE (GE Healthcare Inc.) studies and real-time ICU observations produced an expert-labelled seed set, which trained a preliminary classifier that then guided two active-learning stages, uncertainty sampling and embedding-based retrieval. All labels were assigned by clinical experts; the classifier was used only to prioritize which segments they reviewed. To benchmark the pipeline, we labelled 200 randomly drawn segments and compared junctional yield using the Fisher exact test. Results: Random sampling yielded junctional arrhythmia in only 2 of 200 segments (1.0%). The full pipeline produced a dataset that was 15.3% junctional by labelled hours and 24.7% by patient, a roughly 15-fold and 25-fold enrichment over random sampling, respectively (Fisher exact test, P<.001). Yield rose at each successive stage, from 7.6% of labelled hours (MUSE) to 35.7% (embedding retrieval), reflecting efficiency compounding as the pipeline matured. Of segments retrieved from the junctional embedding region, 60.2% were confirmed junctional once labelled. The final dataset comprised 181.2 hours of expert-labelled ECG from 1,447 unique patients. Conclusions: A staged pipeline that bootstraps model-guided sampling from a small expert-labelled seed set enriches a rare arrhythmia class roughly 15-fold over random review, directing scarce clinician effort toward informative segments. We make no claim about the guiding model's diagnostic performance, which will be reported separately. This methodology is broadly applicable to clinical domains where expert annotation is limited and rare events dominate.