Answers are grounded in this paper and cite their sources.
"Just use a script on one machine" and "you need a cluster" are both standard advice for batch document processing, usually given without numbers. This study pre-registered the numbers. On one named machine (a Dell PowerEdge R610), Hadoop MapReduce on HDFS (3.3.6, Docker pseudo-distributed cluster capped at 8 CPUs and 24 GiB) is compared with a tuned single-node baseline (same tokenizer, aggregation in DuckDB) on two document jobs over the English Wikisource corpus (648,182 documents, 1.85 GiB of extracted text), at 0.1 GiB, 1 GiB, and the full corpus. Every run's output had to hash identically to the baseline's before its time counted. Four hypotheses were frozen in advance. H1, scale crossover, is CONFIRMED: the MapReduce-to-baseline wall-time ratio for the inverted-index job falls from 2.288 at 0.1 GiB to 0.963 at the full corpus, where MapReduce is (narrowly) faster. H2, the small-file effect, is CONFIRMED at a magnitude far beyond its 15 percent pass line: at 1 GiB, storing the corpus as 1,055 files of at most 1 MiB instead of 9 files costs 1,251.7 percent of job wall time (median 7,737.209 seconds against 572.420), a factor of about 13.5. The cell had been stopped unmeasured in an earlier phase for practicality and was measured in a later phase under the owner's explicit authorization, with the deviation record public. H3, replication cost, is INCONCLUSIVE: replication factor 3 costs 29.5 percent on ingest write time but minus 0.1 percent on job time, failing the registered joint rule. H4 is REFUTED: Spark on YARN, the optional arm, is slower than MapReduce at this scale (median 1,087.0 against 954.5 seconds), not 30 percent faster. The study's claims are deliberately narrow: one machine, one corpus, one capped cluster. Within those bounds, every number was registered before it was run and every deviation is on the record.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: