Late Commits and Zombie Runs: A Study of Watermark Designs for Incremental Data Loads

This is the authors' abstract. We don't add key points for this paper.

Incremental loads driven by a high watermark are commonly used to keep an analytical copy of an operational database current. I study three hazards that make such loads erroneous: a crash in the middle of a run, a source transaction that commits after the watermark has passed its timestamp, and a zombie run, in which a stalled run resumes after the orchestrator has already replaced it. I built loadcrash, an open-source harness which injects each hazard into thirteen load patterns and checks every result against an exact oracle, and ran 8,850 trials. Commit-horizon extraction, which reads only rows older than the oldest open source transaction, never lost or duplicated a row and read each row exactly once. Combined with an atomic commit, it also made an unkeyed append safe under crashes. Its cost is freshness: under heavy-tailed commit lag, its 99th-percentile delay reached about ten scheduling intervals, whereas designs that re-read stayed under one. Lookback windows, the usual alternative, could not be sized for heavy-tailed lag. A window four times the bounded-lag requirement lost rows, and a window that grows from the late rows it catches lost 0.07 to 0.46% of rows, because the rows it misses never inform it. For zombie runs, the common defense of moving the watermark only forward turned a self-healing anomaly in partition-overwrite pipelines into permanent loss. Compare-and-set fencing of the watermark protected only the patterns that commit data and watermark together. Code, raw results for all 8,850 trials and analysis scripts: https://github.com/IanO-136/loadcrash (archived at https://doi.org/10.5281/zenodo.23225718).