A time-aware reproducible benchmark of full and lightweight AutoML, machine learning, and deep learning frameworks for heart disease detection

Reproducible evaluation frameworks are essential for translating machine learning advances into medical decision-support systems, yet comparisons of classical machine learning (ML), deep learning (DL), and automated machine learning (AutoML) often lack standardized preprocessing, consistent validation, and transparent runtime reporting. We present a reproducible, time-aware benchmark for heart disease detection using a unified tabular dataset merging four University of California Irvine (UCI) Heart Disease sources: Cleveland, Hungary, Switzerland, and Veterans Administration (VA) Long Beach. The framework applies shared preprocessing and evaluates ML, DL, and AutoML under nominal framework-level time limits of 180 and 60 s, while reporting predictive metrics, observed wall-clock runtime, memory usage, and serialized model size. In the original single-split benchmark, Extreme gradient boosting (XGBoost) achieved an accuracy of 84.78%, an area under the receiver operating characteristic curve (ROC-AUC) of 91.79%, and an F1-score of 86.67% in the full regime. Under the lightweight regime, AutoGluon matched the accuracy and F1-score, increased area under the receiver operating characteristic curve (ROC-AUC) to 92.80%, and completed in 9.87 s, although with a larger serialized artifact. Robustness was assessed using repeated stratified 80/20 holdout evaluation across five pre-specified seeds. AutoGluon_Light achieved the highest descriptive mean F1-score and ROC-AUC (83.97 ± 1.95% and 89.16 ± 2.29%, respectively), while ClassicML_Light remained competitive (82.86 ± 2.68% F1-score; 88.09 ± 2.29% ROC-AUC) and achieved the shortest mean runtime (3.41 ± 3.52 s). Pairwise Wilcoxon signed-rank tests with Holm correction showed no statistically significant differences among lightweight frameworks. Budget compliance analysis showed that only AutoGluon_Light and ClassicML_Light satisfied the nominal 60-s budget in all repeated runs; FLAML_Light and H2O_Light exceeded it in all runs, whereas MLJAR_Light complied in one of five. Target sensitivity analysis showed lower performance when preserving disease severity: the best binary macro-F1-score was 81.57 ± 3.69%, compared with 59.33 ± 1.45% for the ordinal three-class formulation and 39.63 ± 2.19% for the original five-class formulation. Overall, the study provides an engineering-oriented comparison of predictive performance, runtime behavior, budget compliance, and deployment trade-offs rather than evidence of universal model superiority or clinical effectiveness. Findings are specific to the pooled and randomly partitioned merged UCI Heart Disease benchmark and should not be interpreted as evidence of cross-institutional transferability or external clinical validity.