Reviewing research reporting in randomised controlled trials: Confidence and P-values

INTRODUCTION Despite the significant strides made by journals and funding agencies in enhancing research reporting standards, a notable gap persists in the field of anaesthesia research about statistical tools. To bridge this gap, we undertook a cross-sectional study of all the randomised controlled trials published in 2023 in the Indian Journal of Anaesthesia (IJA). Our study focused on reporting confidence intervals (CIs) and P values, a topic that has not been extensively explored in this context.[1] METHODS We conducted a cross-sectional analysis of research papers published in the Indian Journal of Anaesthesia from January 2023 to December 2023. We (NS and VG) independently searched each monthly issue on the IJA official website, screened the titles and abstracts, and populated the randomised controlled trials that met the inclusion criteria. Our exploration of statistical reporting elements was based on a comprehensive questionnaire, partially derived from the Consolidated Standards of Reporting Trials (CONSORT) 2010 standards. This questionnaire [Table 1] was rigorously pilot-tested on 20% of the RCTs, with each trial examined independently over 2 weeks. Any disagreements were resolved through discussion, ensuring the reliability of our findings. Our assessment focused on key statistical reporting areas, including baseline data, imbalance assessment, and outcome reporting as CIs. These areas are crucial for the interpretation and replication of research findings.Table 1: Summary of the results of the questionnaire assessing reporting characteristicsAs we aimed to include and describe all the randomised studies in the past year, no formal sample size calculation was done. All analyses were performed using Google Sheets and R Core Team (2023): R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria (https://www.R-project.org/), and Rversion 4.3.2 ('eye holes'). RESULTS The last search date was 21st January 2024, and candidate studies were retrieved from the IJA website. All 50 included studies were parallel-group trials [Figure 1].Figure 1: Study flow. n=number of patientsBaseline demographic data were compared using P-values in 58% of the trials. However, the frequency of trials reporting these decreased over the months [Figure 2b]. Regarding outcome reporting, CIs were reported for the primary and secondary outcomes in 24% of the studies. The proportion of studies reporting these increased over months [Figure 2c]. The trends of reporting characteristics are shown in Figure 2a-d.Figure 2: Trends of reporting characteristics. (a) Appropriate measures of central tendency, (b) P-value based comparison of baseline variables, (c) Confidence interval of primary outcome, (d) Statistical test for primary outcomeDISCUSSION Applying null hypothesis significance testing in comparing baseline variables using P values does not seem valid. For instance, if the mean age were 46 in the treatment group and 52 in the control group with P < 0.05, we would reject the null hypothesis of no difference in age, and one may read the alternate hypothesis as 'the participants were younger because they were allocated into the treatment group'. Similarly, one should not give too much importance to only the outcomes with P < 0.05. To consider the results/effects significant only if P < 0.05 is too simplistic and the term 'significant' loses its semantic meaning when clinically significant treatment effects are ignored as soon as P = 0.05. For example, trialists exploring early airway pressure release ventilation against low tidal volume ventilation in 138 patients with ARDS had concluded that the ICU mortality was similar between the groups even when there was an absolute risk difference of − 14.6% (P = 0.053).[2] One cannot find a logical reason to explain why a P value of 0.053 would be less significant than a P-value of 0.049, especially in the face of hard outcomes such as mortality with a clinically important difference. A P > 0.05 does not mean that the null hypothesis is true; it only means that we could not find sufficient evidence to reject the null hypothesis.[3,4] The absence of evidence is not the same as evidence of absence. Reporting confidence intervals (CIs) for the difference between treatment arms can help distinguish statistical significance from clinical or practical importance.[3,5,6] The CI denotes the range of values within which an estimate/population parameter is anticipated to fall a specified percentage (the confidence level or coverage level) of the time when the experiment is replicated, or the population is re-sampled using the same methodology. In a 95% CI, there is a 95% probability that the range of the interval derived from the sample contains the true value of the population parameter.[3] The authors of RCTs in IJA in 2023 have reported CIs for primary and/or secondary outcomes and CIs for the effect size estimates.[7-18] The awareness of the relevance of CIs and the willingness to adopt CI reporting are indeed encouraging. However, it is important to understand that merely reporting the CIs is not enough; interpreting results based on CIs, especially the CI of between-group differences (treatment effects), is essential. This should include explicitly interpreting the width of the CI and not restricting oneself to merely commenting if it consists of the null. This will improve the quality of research reporting and facilitate decision-making. The American Statistical Association released a 'Statement on Statistical Significance and P-values' with six principles.[19] A few editorials have explained the same, which has also made its way into current major textbooks.[20,21] Current reporting checklists, such as the CONSORT checklist, also require confidence intervals to be reported.[22] CONCLUSION There is a massive commitment in terms of time and resources when we perform clinical research and peer review each other's work. As physician-scientists, we need to strengthen our understanding of statistical concepts involved in conducting our research and ensure robust reporting of the same. There appears to be an improving trend in reporting the statistical elements over the months in the RCTs published in the IJA, especially in providing confidence intervals for effect estimates. In addition, placing excessive reliance on P values alone should be discouraged. Financial support and sponsorship Nil. Conflicts of interest There are no conflicts of interest.