Conceptualizing Surprise with the Jensen–Shannon distance: A Bayesian Information-Theoretic approach for the Social Sciences

Generally in the social sciences, results are informally deemed surprising if their associated 𝑝-value is sufficiently small. This implicit interpretation, however, is both conceptually and mathematically inappropriate, and such misuse of the 𝑝-value can lead to erroneous conclusions about the novelty of results. To solve that issue, this paper builds on Bayesian inference, Information theory, and recommendations made to related fields, to argue for the adoption of a more appropriate conceptualization of surprise as the relative entropy between prior and posterior knowledge. This formal conceptualization enables researchers to appropriately measure surprise as the Jensen-Shannon distance, for which the paper provides easily implementable software and a demonstration of its use in relation to empirical data.© Keywords: Surprise; Novelty; Jensen-Shannon distance; JS distance; Jensen-Shannon divergence; JS divergence; Relative entropy; Distance; Divergence; Dissimilarity; 𝑝-value; 𝑆-value; Information theory; Bayesian inference