Retrospective use of the PRECIS-2 trial design tool to assess design choices in randomised controlled trials; An empirical review

The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS-2) tool has been widely used to help investigators design randomised trials, facilitating the task of aligning design choices with an explanatory or pragmatic primary trial intention. PRECIS-2 is increasingly being used to retrospectively assess the degree of pragmatism or explanatoriness among published trials within reviews. There is little information on the interrater reliability of the tool and no consensus on the preferred method of achieving an accurate and reliable judgement of trial 'pragmatism' when using PRECIS-2 retrospectively. The aims of this study were to assess the level of pragmatism or explanatoriness of trials that cite PRECIS-2, and to assess inter-rater reliability of PRECIS-2 using different scoring approaches. We compared agreement between two independent ratings within a single pair with agreement between consensus scores reached by two independent pairs of reviewers; and whether widening the agreement criteria increased inter-rater reliability. Thirty RCTs were randomly selected from trials citing the PRECIS-2 tool. Two pairs of reviewers: a clinician paired with a methodologist in each case, were trained and independently scored each trial and reached a consensus score within pairs. Agreement between reviewers within pairs, and between consensus scores across pairs, was assessed using kappa statistics for each of the nine PRECIS-2 domains. RCTsciting PRECIS-2 had predominantly pragmatic design features. Inter-rater reliability within pairs was low across all domains, with the highest levels found in the two domains: analysis (0.32) and follow up (0.33) domains. Agreement across pairs on the consensus scores was similarly low. Agreement between reviewers and reviewer pairs was above 70% when agreement was reclassified as 'within one point difference on the scoring scale' for eight domains, but no improvement was obtained for the remaining domain. Trials citing PRECIS-2 tend to have predominantly pragmatic design features. When using PRECIS-2 to retrospectively score trial publications, agreement between consensus scores across pairs of reviewers was no better than agreement within pairs. Reconfiguring the PRECIS scoring scale and improving scoring guidance may provide a more meaningful, easily interpreted measure of 'pragmatism' for trialists wishing to use PRECIS-2 as a review tool.