Answers are grounded in this paper and cite their sources.
"Just use DuckDB" and "you need Spark" are both common advice for batch transformation work, and both are usually given without numbers. This study pre-registered six hypotheses (PREREG.md, committed before any data was generated) and measured where a single-node embedded engine stops dominating a local-mode distributed engine on one fixed workload: six transformations (filter, join, aggregate, window, type conversion, sort and limit) over TPC-H-derived data at scale factors SF0.1, SF1, and SF10, run identically on DuckDB 2.0.0, on Apache Spark 4.0.1 in local mode, and as a dbt project (dbt Core 1.12.6) on the same DuckDB engine, on one 4-core machine with 7.7 GiB of RAM. DuckDB was faster on the total workload at every scale tested: the total time ratio (DuckDB divided by Spark) is 0.125 at SF0.1, 0.162 at SF1, and 0.248 at SF10, and no single transformation at SF10 reaches the pre-registered 1.5 crossover ratio (the best is 0.467), so no crossover is observed inside the ladder tested (H1 supported, H2 refuted). Correctness held everywhere it was tested: output checksums agree across all three arms in 18 of 18 (scale, transformation) cells, with run-to-run determinism on re-runs (H3 supported). Incremental dbt models with unique_key reproduced the full-refresh checksums under four late-data scenarios at every scale (H4 supported), and dbt data-quality gates configured as promotion gates caught 10 of 10 injected faults with zero missed critical faults (H5 supported). The incremental saving against full refresh is 87.1 percent at SF10, but SF1 also exceeds 50 percent (70.5 percent), where the pre-registration predicted small scales would stay below it, so the economics hypothesis is only partially supported (H6). The data is derived from TPC-H; these are not official TPC-H results and no comparability to official TPC-H results is claimed.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: