Answers are grounded in this paper and cite their sources.
What does a portable pipeline model cost, and what does it buy? This study pre-registered the question against an existing record. The six batch transformations of an already published Spark and DuckDB study (Track B2 of the data-engineering-scale-lab, closed 2026-11-07) were re-implemented as Apache Beam pipelines in the Python SDK (Apache Beam 2.77.0), run on the DirectRunner on the same machine that produced the comparison record, and scored under rules frozen in a pre-registration before any Beam run existed. Three hypotheses were registered. At the smaller scale (data derived from TPC-H at scale factor 0.1), all six transformations completed: the Beam total was 169.92 s against the Spark local total of 11.86 s, a portability tax of 14.327 times, unevenly spread (per-transformation ratios from 5.831 to 22.137, a spread of 3.796, secondary evidence only). On correctness, Beam outputs matched the comparison record's run-1 SHA-256 checksum on 7 of the 12 registered cells with zero mismatches, and run 5 reproduced run 1 on every matched cell. At scale factor 1, the central join (T2, 6,001,215 output rows) exhausted the machine in nine consecutive passes across the two runner configurations the frozen protocol allowed, at peak process-tree memory between 4,497,064 KB and 5,429,284 KB on a 7.7 GiB shared virtual machine; only T1 ever completed (mean 152.97 s over five timed runs). Under the frozen rules all three verdicts are therefore INCONCLUSIVE: the tax is measured only at the smaller scale (H1, H3), and exact reproduction is proven on 7 of 12 cells, not contradicted on the rest (H2). The claims are deliberately narrow: one machine, one workload, one runner. Within those bounds, the portability is real (identical bytes under a canonical checksum), the premium at small scale is about fourteen times the Spark runtime, and the reference runner has a resource floor that the larger scale crosses.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: