STAR: A Long short-term workload-aware optimizer for online layout reorganization

In large-scale data analytics systems such as cloud data warehouses, workloads are inherently complex, involving a vast number of rows and columns, and evolve over time, leading to excessive data scanning. Optimized data layout can skip a large number of irrelevant partitions, thereby improving scan performance. Nowadays, numerous static optimized approaches have been proposed for offline layout optimization based on historical workloads, but they struggle to maintain performance under unpredictable workload shifts. Online layout reorganization can address such issues by dynamically adapting data layouts to workload evolution. However, effectively and efficiently adopting dynamic adaptation strategies for online layout reorganization remains challenging in the face of evolving workloads, which arises from the limitations of existing dynamic adaptation approaches in precisely identifying switching moments and accurately selecting target layouts. These limitations lead to delayed or inappropriate layout changes, resulting in limited query performance gains and excessive reorganization cost. In this paper, we propose the framework STAR, a Long \({{S}}\) hort- \({{T}}\) erm Workload- \({{A}}\) ware Optimizer for Online Layout \({{R}}\) eorganization. STAR contains two methods: the \({{D}}\) ual- \({{A}}\) daptive \({{R}}\) eward-Threshold \({{A}}\) wareness method (DARA) and the \({{Du}}\) al-Source \({{E}}\) nsemble Hybrid-Probabili \({{t}}\) y Switching method (DUET). DARA overcomes the limitation in precise switching moment awareness and DUET addresses the limitation in accurate switching selection. In addition, STAR introduces an optimized initial layout to reduce reorganization cost. Experiments on real-world datasets show that STAR achieves up to 27.7% improvement in combined query and reorganization time over SOTA, and up to a 63.4% decrease in reorganization cost.