Industrial case study uncovers seven recurring failure defect classes in an automated SaaS test suite, highlighting how displaced and suppressed errors mask true software bugs.
End-to-end (E2E) user-interface test suites at industrial scale are widely reported to make flakiness their dominant maintenance cost. Less widely examined is a second problem that compounds it: the site at which a failure is reported often diverges from the site of its true defect, and a further, under-examined subset of defects suppresses the failure signal entirely rather than merely displacing it. This paper reports an industrial case study of a 48-feature, approximately 852-scenario Playwright and Cucumber behaviour-driven test suite exercising a commercial multi-tenant software-as-a-service platform across three environments. Through practitioner-embedded root-cause analysis, latency instrumentation of the application under test, and codebase-wide pattern search, a taxonomy of seven recurring defect classes is derived and organized along a displacement/suppression axis. Five classes cause a correct expectation about test behaviour to fail at a location distant from the true cause; two classes cause a broken test to report a pass while verifying nothing. Measured latencies that motivate specific fixes are reported, the propagation of one defect class across eighteen page-object classes is quantified, and a dependency-partitioned parallel execution architecture adopted as a mitigating intervention for suite run time is described. The taxonomy is positioned against the flaky-test and test-smell literature, noting where the suppression classes are structurally analogous to previously reported “rotten green test” phenomena in unit testing, and the threats to validity that follow from a single-suite, single-team study are set out.
No takes yet. Share an insight, caveat, or question.
V Soniya (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: