Multivariable modelling for meta‐epidemiological assessment of the association between trial quality and treatment effects estimated in randomized clinical trials

Methodological deficiencies are known to affect the results of randomized trials. There are several components of trial quality, which, when inadequately attended to, may bias the treatment effect under study. The extent of this bias, so far only vaguely known, is currently being investigated by 'meta-epidemiological' re-analysis of data collected as part of systematic reviews. As inadequate quality components often co-occur, we maintain that the suspected biases must be evaluated simultaneously. Furthermore, the biases cannot safely be assumed to be homogeneous across systematic reviews. Therefore, a stable multivariable method that allows for heterogeneity is needed for assessing the 'bias coefficients'. We present two general statistical models for analysis of a study of 523 randomized trials from 48 meta-analyses in a random sample of Cochrane reviews: a logistic regression model uses the design of the trials as such to give estimates; a weighted regression model incorporates between-trial variation and thus gives wider confidence intervals, but is computationally lighter and can be used with trials of more general design. In both models, heterogeneity in the bias coefficients can be incorporated in two ways. A stratification approach pools the estimates from models estimated on subgroups of the data. We explore stratification by reviews and by broad trial types, the latter of which gives larger subgroups of the data, circumventing instabilities. A multilevel approach also avoids instabilities and addresses the more fundamental problem of interpretation of the pooled multivariable effect in the presence of heterogeneity.