Answers are grounded in this paper and cite their sources.
Headline writers use question framing on the theory that an open question opens a curiosity gap. This study asks three things of one public archive of headline A/B tests, under a pre-registered plan with a locked confirmatory split. First, do question headlines change click-through within randomized tests? Pooling 465 eligible tests from the locked split of the Upworthy Research Archive by DerSimonian and Laird random-effects meta-analysis gives a pooled odds ratio of 0.8645 (95 percent CI 0.8335 to 0.8966): question headlines reduce clicks, and the pre-registered verdict rule returns CONFIRMED. Between-test heterogeneity is large (tau 0.3325, I squared 75.8 percent), so the pooled number is an average over contexts, not a universal constant. Second, how badly does the common practice of repeatedly peeking at a fixed-horizon test inflate false positives, on streams calibrated to this archive's arm sizes? In 200,000 simulated A/A streams, seven looks at a nominal 5 percent test reject 16.32 percent of null streams, against 5.03 percent for a single final look; a mixture sequential probability ratio test (mSPRT) rejects 0.0 percent on the same streams. Third, can a cross-test observational estimate recover the randomized benchmark on the same platform? On the development split, propensity-weighted and matched estimates both land within bootstrap measurement error of the randomized benchmark, so the pre-registered hypothesis that they would miss is Refuted; the study reports the measured gaps rather than a success story for either side. The contribution is not the headline answer, which prior work has already published on this archive's confirmatory set, but the pairing: one pre-registered, one-command-reproducible pipeline in which a randomized estimate, an observational-miss measurement, and a peeking calibration are all run under the same frozen rules, with a public audit trail a third party can re-run end to end.
No takes yet. Share an insight, caveat, or question.
Jesun Ahmad Ushno (2026) studied this question.
Synapse has enriched 8 closely related papers on similar clinical questions. Consider them for comparative context: