Intra-service tracing for shared and stressed datacenters: efficient management and intelligent analysis

With the rapid expansion of data centers, the escalating complexity of software and cluster scales demands more efficient and precise application observability. To identify performance bottlenecks and diagnose anomalies, it is essential to leverage execution tracing to accurately determine the causal relationships within application behaviors. However, we observe that in complex cloud data center environments-characterized by workload co-location and high hardware pressure, existing execution tracing tools typically incur a 5%∼10% performance overhead, hindering their seamless deployment across large-scale clusters. Although hardware tracing capabilities in modern processors are designed to reduce overhead, the four-dimensional design trade-offs among time efficiency, space overhead, data coverage, and management complexity remain insufficiently explored. Consequently, current solutions fail to meet the stringent overhead requirements of complex data centers. To address these challenges, this paper proposes ETrace, which optimizes time efficiency at both the node and cluster levels. At the node level, we leverage the four-dimensional design trade-offs of hardware tracing to minimize unnecessary control operations, thereby ensuring application performance, and dynamically allocate limited trace storage based on the application's real-time state. At the cluster level, we implement flexible and efficient multi-node trace management through a versatile sampling strategy and a cloud-native triggering framework. Evaluations across various benchmarks demonstrate that ETrace incurs an average performance degradation of only 0.97%, representing up to 85% overhead reduction compared to state-of-the-art solutions. Furthermore, using typical intelligent applications in Alibaba data centers as case studies, Also, ETrace's instruction-level tracing capability greatly enhances datacenter manageability with newer insights on anomaly diagnosis and bottleneck localization.