Ultron: History-Based Query Optimization at Databricks

Databricks, a modern lakehouse data platform, operates at massive scale (over 50 million VMs, processing billions of queries per day) with limited data statistics, making traditional cost-based query optimization challenging. Databricks uses adaptive query execution (AQE) to mitigate many runtime inefficiencies, but certain decisions, such as join operator selection, could benefit significantly from accurate prior knowledge. We present Ultron, a production-grade history-based query optimization (HBO) framework recently deployed at Databricks. Since analytical workloads are often repetitive, Ultron can learn from prior executions of the same query and apply precise, subplan-level historical insights during optimization. To achieve this, Ultron introduces a three-tier architecture consisting of (1) a distributed cache for query history, (2) a low-latency local prediction layer with data-drift awareness, and (3) modular optimization applications that modify query plans without risky exploration. We demonstrate three applications of Ultron using data from the Databricks fleet and synthetic benchmarks, showing that Ultron improves the median latency of eligible joins by 25% in production, optimizes approximately 23% of all joins across the fleet, and achieves 96% post-hoc decision accuracy. Ultron operates with modest overhead (< 5% median additional optimization time) and scales elastically with load, demonstrating that precise history-based optimization can safely complement adaptive execution in large-scale cloud data platforms.