Answers are grounded in this paper and cite their sources.
Adding a second GPU node is the standard first step when one machine feels too small for training, and on a slow link it is often the wrong step. This study pre-registered the measurement before running it. Two NVIDIA DGX Spark nodes (one GB10 GPU each), joined by commodity 1 GbE through an office switch, ran data-parallel training (PyTorch DDP, DDP with a bf16 gradient-compression hook, and FSDP2) on GPT-style models of 5.3M, 29.5M, and 91.5M parameters, across gradient-accumulation factors of 1, 4, 16, and 64, against a single-node arm at the same global batch. Seven hypotheses with frozen pass lines were committed before any main-grid run. Five pass. Two nodes are slower than one in 9 of 12 DDP cells, down to a speedup of 0.069 (95 percent bootstrap interval 0.067 to 0.075) for the 91.5M model at accumulation 1, and faster only at accumulation 64 (1.31x to 1.47x). A two-parameter link model, t(S) = alpha + S / beta, fitted on 126 all-reduce trials before any two-node training run (alpha 0.40 ms, beta 117.4 MB/s, R squared 0.9999998), predicted every two-node DDP step time within 2.7 percent, inside its registered 25 percent line, and called the faster-or-slower side correctly in all 12 cells. Correctness is exact, not approximate: after 30 steps, two-node DDP and FSDP2 parameters differ from a single-process replay by 0.0, and crash recovery at steps 13, 25, and 37 of a 40-step run ends bit-identical to the uninterrupted run in all three cases. Two hypotheses missed, narrowly, and are reported as missed: the bf16 hook cut step time by 29.4 percent in one qualifying cell against a 30 percent line (range across qualifying cells 29.4 to 48.7 percent, with the loss condition met everywhere), and FSDP2 peak memory was 65.3 percent of DDP's against a 65 percent line. The claims are deliberately narrow: two nodes, one slow link, short throughput runs. Within those bounds, the crossover is measured, the prediction was locked before the runs it predicts, and no pass line moved.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: