Designing Reliable Harnesses for Autonomous Machine Learning Experiments: Failure Modes, Safeguards, and Lessons Learned

Autonomous AI agents offer a promising venue to accelerating machine learning research through iterative experimentation with limited human intervention. However, reliable autonomous experimentation requires carefully designed experimental environments that prevent unintended modifications and preserve evaluation integrity. Drawing on more than 15,000 autonomous experimental runs across three biomedical segmentation benchmarks, we present practical recommendations and lessons learned for designing robust experimental harnesses. Our findings highlight common agent behaviors and provide guidelines for improving the reliability, reproducibility, and efficiency of structured autonomous machine learning experiments.