Significant significance?

Modern medicine is a game of reason. The times of harmonizing bodily fluids and quackery seem long gone. We no longer believe in phlebotomy or burning women to free our communities from bewitched spirits, even though bloodletting had occasionally worked based on chance for some medical practitioners during the Middle Ages. Individual experience is still an invaluable skill for any clinician, albeit more in the realm of visual diagnosis than sensing imbalanced spirits. Ultimately, it was the 18th century's period of Enlightenment and the subsequent ~250 years of scientific, rational, objective, and most importantly evidence-based research that has paved the way for our current state of rapidly evolving and continuously improving medical comprehension. So much for the theory. As with most modern scientific theories, this is not a bad one. Hence, life expectancies have been growing rapidly since the advent of modern medicine and are now projected to outgrow barriers by 2030 which had commonly been deemed unconquerable at the beginning of the century.1, 2 These are splendid results, facilitated by a plethora of advances in individual medical fields: HIV treatment is in a state which not only allows for longer patient survival but may enable disease eradication plans.3 New kinds of cancer treatments have substantially changed lives for millions of patients around the world.4, 5 These and countless more phenomenal developments allowed evidence-based medicine to make headway in ridding our world from the scourges of humanity. What happens though when the desire for scientific advancement outruns healthy scepticism? It turns out, in modern medicine all that glitters is not gold. The infamous “Meta-Analysis of Multiple Primary Prevention Trials of Cardiovascular Events Using Aspirin” by Bartolucci et al impressively highlighted the pitfalls associated with the uncritical and unchallenged quest for medical evidence.6 The indisputably well-written study examined acetylsalicylic acid (ASS) capability of preventing myocardial infarction (MI) by comprising data of over 22.000 patients for a timespan of on average five years. The authors reported a P-value of P < .00001, which led to the recommendation of ASS for the general population. The reported P-value indicated a positive effect of ASS for MI risk reduction beyond any measure of reasonable doubt. Another victory of modern medicine, based on a highly significant statistical analysis: Sounds great, doesn't it? Not so much on second sight: There was a simple but very important insight missing while recommending ASS for basically everybody. While it was impressively shown that taking ASS had an effect on the rate of MIs, nobody cared for what this effect actually was. This is where statistics get tricky: It turned out to be barely anything. Even though a P-value smaller than a tenth of a percent of a percent was reported there was only a risk reduction of 0.77% and a correlation coefficient between ASS intake and reduced MI of r2 = .001. The actual effect of ASS on a single patient seemed not only clinically negligible but furthermore did most certainly not make up for ASS’s adverse side effects. Subsequently, the correct question to ask is: How could this happen? It is worth the time to take a step back and think about what happened in this study. The researchers reported a P-value, indicating an effect, which in this case was the reduction of MI events in patients under ASS treatment. The critical misinterpretation is to link a low P-value with a relevant effect. P-values indicate the statistical certainty with which a hypothesis can be claimed to be true and are tested against an alpha-level. Less technical, a P-value tells us how likely it is that an observed effect is not truly there and is much rather observed by chance, just like there is a chance of rolling a six for five times straight on a fair die. A P-value would tell us how likely this event is to be observed by chance and thought backwards, how sure we can be that the observed effect is not a product of chance. The alpha-level is our decision boundary – P-values below the chosen alpha are considered significant, while P-values greater than alpha are generally interpreted as no shown difference. Notably, the value of alpha is arbitrary and commonly set to 0.05 due to nothing else but convention. Quite intuitively, after around 10 straight rolled sixes we all would heavily suspect the die to be loaded and most statistical tests would suggest a P-value well below commonly chosen alpha-levels. Measures of certainty play a critical role in medical science: Nobody would want to take a drug "proven" to work if there was a 40% chance of this "prove" being fallacious. Then again, there is something else you would want to know before taking a drug: How much does it help? Frustratingly, P-values do not tell us anything about effect size. Coming back to our die, rolling a six three of ten times is not suspicious at all but rolling 3.000 sixes out of 10.000 rolls would leave us convinced that the die is loaded. Markedly, in both cases, it could have been the same loaded die. When playing games most people do not care about being cheated a little – after all, there are no completely equal games (such as no die is manufactured with infinite precision). What people do rightfully care a lot about is whether they are cheated badly. Translated into scientific terms: Most people do not care about P-values. They care about effect sizes. What is to be learned from the ASS example is that there are more layers to statistical interpretation than just the calculation of P-values and that the sizes of discovered effects often add more to our medical understanding than P-values. Luckily, effect sizes are in most cases easily and intuitively calculated. The most straight-up options are 95% confidence intervals and simple regression coefficients. Gail Sullivan and Richard Feinn have authored a highly impactful editorial about the correct interpretation of P-values and effect sizes which can be recommended as a short handbook for scientific reporting.7 Equally important to reporting effect sizes is to interpret them correctly and comprehensibly. At this point researchers are asked to look at the effects they have shown and tell their readers what they think about them from their scientific perspective. This is not about intellectual paternalism but much rather about enhancing papers readability and correct interpretability for readers. This is increasingly important, given the fact that overall research output has skyrocketed while scientific advances have created more and more specialized fields of research.8 Of course, only reading papers from your very own field of research might limit the problem of readability but it is mostly reading and understanding things very different from one's everyday workflow that spark the creation of truly original ideas. Unsurprisingly, all of this is not limited to clinical sciences, but might be even more pronounced in basic research contexts. Fundamental medical research is based on a myriad of elaborate methods and spans across a vast field of medical domains. Scanning through recent physiological research presents the reader with a multitude of methods and statistical tests, all to be interpreted differently. Not to forget, the most fundamental limitation of biomedical sciences: Every experiment, every result, and every interpretation is manmade and therefore a flawed model of reality. P-values allow us to estimate the statistical probability that our model's results are due to chance. The choice of an appropriate model is therefore the most important feat of any scientific pursuit. Staggeringly low P-values promise nothing more than false-positive conclusions if the chosen model does not generalize adequately to real-world scenarios while, vice versa, unsound model selection can lead to large P-values even if there is a very significant and clinically important dependency to be found. No clear-cut way exists to determine which model is best to use as the field is too diverse, but a good rule of thumb would be to focus on a model's predictive scope. Biochemical models are best suited to describe biological dependencies on a cellular level.9-11 Deeper down, they even can be capable of characterizing specific intracellular pathways, broadening our understanding of the most basic biological levels.12-14 The examination of more complex systems depends on models focussing more on tissue and organ-related, functional models, both on a microscopical15, 16 and macroscopical17, 18 levels. Research about organ system interactions and multifocal dependencies is often too complex for a deep-down functional analysis and must rely on models based on systemic physiological and mechanistic understanding.19-21 Then, there are research questions which require investigation of whole body morphological or behavioural traits which can only be observed in living animals22 and lastly, there are the research questions which depend on human subjects of research and the heaps of models related to research in humans.23, 24 It is a hard task to choose the correct model and in many cases, only a combination of multiple models can lead to insightful results. Interpreting those models based on P-values only will lead to described false positive and negative interpretations. Physiologically comprehensible, well-described models and ideally a reported dose-response relationship can be crucial first measures of model integrity and therefore offer a more complete synopsis of research results than crude P-values. Even if all of this sounds very difficult, the situation is not too dire. Researchers should try to address the concern of declining interprofessional readability, but the tools to do so are right there and easy to administer. Explaining and interpreting effect sizes can go a long way and emphasizing those ideas in research papers helps to guide the reader's attention to the most pivotal results. Explaining to what degree muscles fibres were expected to grow and how much protein levels would have to rise to have a significant effect are key elements of excellent research papers, much more so than hunting P-values down to another decimal place. Our capability of understanding, interpreting and being able to trust evidence-based research works are the foundation of modern medicine's wonders. We as the research community have the tools available to make our results more accessible to readers and prevent them from flawed interpretations. Emphasizing effect sizes over plain P-values is another important contribution to distance our medical understanding from foregone maladministration. So, the next time writing a paper we are to make sure to take that step and not only present significant P-values but a truly significant significance. There is no conflict of interest to declare.