Comparing change detectors for adaptive execution tracing: Large-scale validation and design insights

Context: Performance debugging in large-scale production systems is constrained by high tracing overhead and the difficulty of activating tracing at the right time. Objective: This paper investigates whether simple statistical detectors are sufficient for adaptive execution tracing compared with more complex models, while maintaining efficiency. Methods: We study the Dynamic Adaptive Execution Tracing (DAET) framework, which consists of three phases: tracing-function selection using call-stack profiling and coefficient-of-variation ranking, change detection via anomaly scoring with a dynamic threshold, and trace configuration adjustment based on a tolerance-based gating mechanism. We conduct a large-scale evaluation of five change-detection methods (ARIMA, LAST, SMA, EWMA, and a compact Transformer) on two Mozilla CI/CD performance-telemetry datasets (1819 signatures: 342 Firefox-Android and 1477 mozilla-beta; over 1.39 million telemetry observations; 9095 detector–signature evaluation runs) using a unified chronological protocol, including replay-based analysis and multi-seed validation. Results: Simple statistical methods consistently match or outperform more complex models while requiring significantly lower computational cost. Across both datasets, SMA and EWMA provide the best overall trade-off between detection performance and efficiency; validation-based detector-specific gate tuning places them first and second on both datasets. All methods substantially reduce trace storage. A small heuristic audit illustrates that unmatched detections cannot automatically be treated as verified non-anomalies, but it does not estimate true anomaly precision. Conclusion: Model complexity is not the primary driver of performance in adaptive tracing systems. Instead, careful parameterisation and gating strategies have a greater impact, providing practical guidance for designing efficient and effective tracing mechanisms.