A physicist completes a computation. The computation derives a physical quantity — say, the primordial helium abundance — from a chain of input values and intermediate calculations. The physicist writes a paper. The paper is typeset in LaTeX. The equations are formatted. The derivation is described in prose. The prose reads something like this: "Using the Planck 2018 value of the baryon density parameter Ωb = 0. 0490 ± 0. 0006, we derive the baryon-to-photon ratio η via the standard relation involving the CMB photon number density (Fixsen 2009) and the critical density. Substituting into the Pitrou et al. (2018) BBN fitting formulae, we obtain Yₚ = 0. 2486, in reasonable agreement with the Aver et al. (2015) measurement of 0. 2449 ± 0. 0040. " The reader who wants to verify this result must perform the following steps. Find the Planck 2018 paper. Locate the Ωb value in its tables. Find the Fixsen 2009 paper. Locate the CMB temperature value. Calculate the photon number density from the temperature using the standard formula, which requires knowing that formula and looking up ζ (3). Find the critical density, which requires the Hubble constant and the gravitational constant, each of which requires its own lookup. Compute η from these inputs. Find the Pitrou 2018 paper. Locate the fitting coefficients. Apply the fitting formula. Check whether the result matches the paper's stated value. This process has specific failure modes. The reader may copy a value incorrectly from one of the cited papers — transcription error. The paper may use a symbol that means different things in different conventions and not state which convention is intended — notational ambiguity. The paper may skip steps the author considered obvious — derivation gaps. The paper may have used a specific numerical method with specific parameters and not reported them — unstated numerical choices. The computation may have been performed in double-precision floating point, where rounding errors accumulated across the chain and the final digits depend on the implementation rather than on the physics — floating-point artifact. These are not descriptions of bad practice. This is standard practice. The best papers in the best journals follow this format. The format was designed for an era when computation meant pen-and-paper derivation and publication meant printed pages. In that era, prose description of a computation was the best available technology for communicating the computation to others. The format served that era. The era is over. Computation is performed by machines. Storage is effectively unlimited. Distribution is instantaneous and global. The infrastructure that made prose description the only viable format no longer constrains us. What constrains us is convention — the inherited practice of describing computations rather than publishing them. The structural problem is precise. A prose description of a computation is not a computation. It is a set of instructions for performing a computation, written in natural language, subject to ambiguity, and requiring manual re-execution by every reader who wishes to verify the result. The distance between the description and the computation is the space where irreproducibility lives. Every ambiguity, every unstated choice, every transcription opportunity is a point where two independent readers may arrive at different results from the same paper. The prose description does not prevent this divergence. It enables it.
Geoffrey Howland (Fri,) studied this question.