Interpreting statistical evidence with S values: an illustration with trials of goal‐directed haemodynamic therapy

Asking clinical researchers regarding their relationship with p values, the answer would likely be “it's difficult”. Despite known challenges with p values with respect to clinical utility and misinterpretations, they remain ubiquitous in the medical literature [1, 2]. In search of new ways to interpret the evidence resulting from data collection, a new statistical quantity, the S value, was proposed recently [3, 4]. The S value (also termed Shannon-information or surprisal) is a statistic quantifying the amount of evidence against a test hypothesis, for example the null hypothesis underlying a randomised trial. It is defined as the negative logarithm of an observed p value p: s = −log2(p). The unit of S values are called bits when using the binary logarithm. In contrast to p values, S values have a simple and intuitive interpretation related to physical experimentation, akin to tossing a coin [3-5]. Imagine tossing a fair coin four times and getting four heads. Assuming an equal chance of heads and tails, the associated p value is 1 2 × 1 2 × 1 2 × 1 2 = 1 2 4 1/2× 1/2× 1/2× 1/2=(1/2)⁴ . The corresponding S value is 4 bits and represents the evidence against the test hypothesis of a fair coin. Note that the conventional p value of 0.05 corresponds to an S value of 4.3 bits. To aid interpretation of S values with physical experimentation, the S values are rounded to the nearest integer. Thus, the evidence against a test hypothesis when getting a p value of 0.05 can be compared with the situation of getting four heads (or tails) when tossing an unbiased coin four times. Here, we illustrate S values with trials investigating the possible benefits of goal-directed haemodynamic treatment and discuss their clinical utility in interpreting conventional p values. We used published data from a previous meta-analysis comparing goal-directed haemodynamic treatment with control treatment [6]. Outcome data for: mortality; sepsis; surgical site infection; pneumonia; urinary tract infection (UTI); acute kidney injury (AKI); paralytic ileus; the need for re-operation; and the incidence of ≥ 1 complication were available. Using published outcome data, we recalculated p values using a random-effects model to pool the individual studies. Associated S values are computed with these recalculated p values for each outcome. All analyses were performed with R (R Foundation for Statistical Computing, Vienna, Austria). Figure 1 displays the p and S values for the nine outcomes. The black dotted line displays the relationship from the S value's definition: s = −log2(p). The S values for the re-operation, sepsis, mortality and paralytic ileus outcomes are small with zero to two bits. The evidence against the null hypothesis of no treatment benefit for the outcomes AKI and UTI are 4 bits and 5 bits, respectively. In contrast, the S values for pneumonia, ≥ 1 complication and surgical site infection are 10 bits, 19 bits and 25 bits, respectively. Table 1 provides an overview of the numeric values. The non-linear relationship between S and p values has important implications with respect to interpretation when comparing two p values. For example, the p values for sepsis (p = 0.61) and mortality (p = 0.21) differ by 0.4 but differ in their amount of evidence against the null hypothesis only by 1 bit. In contrast, the p values for UTI (p = 0.038) and pneumonia (p = 0.0008) differ by only 0.037, which is a magnitude smaller. However, these two outcomes differ in their S values by 5 bits – a rather large difference. This difference corresponds to the probability of getting five heads when tossing a fair coin five times vs. the scenario of getting 10 heads when tossing an unbiased coin 10 times. Overall, there are an abundant number of cases in the medical literature where treatment benefits are evaluated only with respect to the conventional significance level of 0.05. The proposed S value does not aim to replace the traditional p value. However, its aim is to provide a more intuitive interpretation of the collected evidence against a test hypothesis. Additionally, S values help interpreting different p values. Imagine two outcomes with corresponding p values of p = 0.049 and p = 0.051, respectively. The evidence against the test hypothesis is 4.4 bits and 4.3 bits, respectively. In the context of the coin-tossing experiment, the evidence against the test hypothesis would be the same in these cases, which questions the practice of dichotomising the evidence into being either significant or non-significant. I have shown that S values provide an intuitive interpretation of the pooled evidence from multiple randomised controlled trials with respect to a treatment effect of goal-directed haemodynamic treatment and can be compared with the situation of getting all heads (or tails) when tossing a fair coin four times (AKI); five times (UTI); 10 times (pneumonia); 19 times (≥ 1 complication); and 25 times (surgical site infection). Although some studies have been analysed by means of S values [5], a broad assessment of their clinical utility and their adoption in clinical research is still missing. Yet in terms of the potential of S values to change the interpretation of evidence against a test hypothesis, Kenneth J Rothman put it succinctly: “If the S value moves us away from focusing on significance testing, it is worth a try” [7]. No competing interests declared. Open access publishing facilitated by Inselspital Universitatsspital Bern, as part of the Wiley - Inselspital Universitatsspital Bern agreement via the Consortium Of Swiss Academic Libraries.