When to Use ML, LLM, or Hybrid Approaches for Software Test Failure Classification:A Framework-Aware Comparative Study

Automated classification of software test failures into failed testcase (defective test code)or bug (defective application) is a critical but understudied problem in QA automation.We present the first framework-aware comparative study of three classification strategies:Pure LLM (Llama 3.2), Pure ML (XGBoost with 52 engineered features), and a HybridML+LLM pipeline. Evaluated on 30 labeled test failures spanning four testing frameworks(Selenium, pytest, pytest-requests, SQLAlchemy), our results reveal that no single methoddominates across all frameworks: XGBoost achieves 90% accuracy on Selenium testsbut only 33% on API tests, while LLMs achieve 100% on database tests but only 60% onSelenium tests. We introduce framework-adaptive classification, a strategy that selectsthe optimal method per detected framework, achieving an estimated 85% accuracy acrossall domains — 8.3 points above any single method. Our findings provide practical guidelinesfor QA teams building automated test failure analysis systems.