Does economic policy matter? A note on the narrative approach and exact inference

This note examines uncertainty in time-series inference from rare episodes, focusing on the narrative approach. A small number of randomly drawn episodes may falsely suggest policy effects because they are associated with macroeconomic shocks that do not cancel out in inference. We illustrate this using Fisher-style exact inference. Applying our test to Romer and Romer’s (2023) analysis, we find substantial uncertainty. Although the unemployment rate’s peak response to an identified monetary contraction exceeds the 95-percent confidence bands of the counterfactual distribution based on randomly drawn months—suggesting systematic policy effects—this finding is reversed once additional controls are included. • Small numbers of narrative shocks risk capturing unrelated macroeconomic noise. • Fisher-style inference quantifies uncertainty in identifying policy effects. • We find that this type of uncertainty is substantial in US macroeconomic data. • Results highlight fragility of inference from rare policy shock episodes.