Do all roads lead to Rome? Critical reassessment of the individual patient meta-analysis on bypass grafts by Gaudino et al.

The recent individual patient data meta-analysis by Gaudino et al. [1], comparing outcomes with different grafts for coronary artery surgery from not for this question-randomized trials, was met with questions on the methodology given that some of the results proved controversial. The EJCTS reacted to various comments on the methodology and suggested an independent reanalysis of the data. The author group of the paper, led by Mario Gaudino, should be applauded for their scientifically sound cooperation in this exercise. In the spirit of open data and open science, the authors shared their entire dataset and the codes with the journal. The authors of this Editorial wish to express their gratitude and esteem for this behaviour, which exemplifies excellence in science. The journal was particularly keen to have this process initiated because the topic is important, and the main players have considerable expertise in CABG research and statistics. Importantly, the authors used a legitimate way to analyse the data and the outline statistical analysis plan was predefined and registered together with another paper [2]. When performing the same analysis, we came to similar results, confirming that the data handling was numerically and statistically correct. During the reanalysis process, we identified imbalances in the matching, which we preferred to analyse conventionally in separate propensity score matching for the 2 comparisons. Utilizing 2 separate comparisons (BITA versus LITA + RA; BITA versus LITA + SVG), we achieved excellent balanced characteristics on observation of the matched baseline data in both comparisons (Tables 1 and 2). BITA versus LITA–RA: baseline characteristics for matched and unmatched groups BITA: Bilateral internal thoracic artery; CVA: cerebrovascular accident; IQR: interquartile range; LVEF: left ventricular ejection fraction; LITA: left internal thoracic artery; MI: myocardial infarction; NYHA: New York Heart Association; PCI: percutaneous coronary intervention; PVD: peripheral vascular disease; RA: radial artery; SMD: standardized mean difference. BITA versus LITA–RA: baseline characteristics for matched and unmatched groups BITA: Bilateral internal thoracic artery; CVA: cerebrovascular accident; IQR: interquartile range; LVEF: left ventricular ejection fraction; LITA: left internal thoracic artery; MI: myocardial infarction; NYHA: New York Heart Association; PCI: percutaneous coronary intervention; PVD: peripheral vascular disease; RA: radial artery; SMD: standardized mean difference. BITA versus LITA–SVG: baseline characteristics for matched and unmatched groups BITA: Bilateral internal thoracic artery; CVA: cerebrovascular accident; IQR: interquartile range; LITA: left internal thoracic artery; LVEF: left ventricular ejection fraction; MI: myocardial infarction; NYHA: New York Heart Association; PCI: percutaneous coronary intervention; PVD: peripheral vascular disease; SVG: saphenous vein graft, SMD: standardized mean difference. BITA versus LITA–SVG: baseline characteristics for matched and unmatched groups BITA: Bilateral internal thoracic artery; CVA: cerebrovascular accident; IQR: interquartile range; LITA: left internal thoracic artery; LVEF: left ventricular ejection fraction; MI: myocardial infarction; NYHA: New York Heart Association; PCI: percutaneous coronary intervention; PVD: peripheral vascular disease; SVG: saphenous vein graft, SMD: standardized mean difference. Unlike the findings of Gaudino et al., we found no systematic difference between the groups on the important outcome of mortality (Fig. 1). With the hazard ratio on mortality for BITA versus LITA + RA of HR 0.93 (95% confidence interval 0.69–1.26; P = 0.66); and for BITA versus LITA + SVG of HR 1.02 (0.86–1.22; P = 0.81). The point of making 2 separate comparisons in the context of 3 separate treatment strategies of interest is to utilize conventionally the statistical principle of transitivity; that is if x > y and y > z, then we can infer that x > z. This approach also matches one of the scenarios provided in the motivating example by Gaudino et al. [1]. Data comparison of 2 different methods running the meta-analysis with the same data set. Different approaches to analysis can lead to qualitatively different results. In a fascinating and relevant study, Silberzahn et al. [3] found that analyses conducted independently by 29 analytical teams found important differences, with point estimates varying qualitatively (e.g. moving from increased risk to decreased risk in the vexed question of whether soccer referees were more likely to ‘red card’ dark-skin-toned players). Some 69% found a significant positive relationship, while the remainder did not. Propensity score-matched analyses use a statistical model (a logistic regression) to calculate a likelihood-based score (described on the logit, or log(e)odds, scale) for each subject to identify their ‘risk’ of being treated with the strategy of interest, accounting for a range of potential patient-level characteristics. Rosenbaum and Rubin [4] who first developed the approach described how the method was unbiased when, having accounted for the propensity score, the actual exposure to the treatment of interest carried no extra information on the risk of the subject. Therefore, to give the right answer, the propensity score has to capture all of the risks faced by a subject regardless of whether they received the treatment of interest or not. Achieving the perfect adjustment described by Rosenbaum and Rubin is not practically possible. We have imperfect knowledge on the risks faced by individuals and we measure the available risk factors imperfectly. For example, a complex clinical process such as diabetes may be described as simply ‘present’ or ‘absent’. A strength, but also a weakness, of the propensity score approach is that we can include many explanatory variables in the creation of the score without being concerned with technical issues such as overfitting because it is only the point estimate that is used from the model, not its uncertainty. While this allows us to include all relevant risk factors, it also facilitates the dilution of an important risk factor in the score because there are many less important risks included. An important risk factor may be hard to identify because statistical significance alone will not guide us; instead, we should assess the appropriateness of the achieved match between cases and controls and only proceed to the analysis of outcomes when we are content that the groups are closely alike. While different analysis strategies are possible, propensity score-matched analyses are attractive because they limit comparisons to subjects who are all potential candidates for the treatment of interest, at least as described by the observed characteristics and thus the propensity score, rather than including subjects who could never be included. This is nicely described for example by work comparing TAVI and SAVR, where the matched subjects tend to be in the middle range of risk [5], with lower-risk subjects in the data set receiving SAVR and high-risk subjects receiving TAVI; the latter particularly are very unlikely to be candidates for SAVR and thus really have no place in a statistical model comparing the 2 strategies. Because of concerns about the effectiveness of the matching process, we use approaches such as the standardized mean difference (SMD) to assess how similar the resulting groups are over each important subject characteristic. The SMD is helpful because, unlike a statistical test, it does not rely on having sufficient numbers and thus power. Threshold values of the SMD that are considered adequate are arbitrary, but we often look for values <0.1 to suggest that a good match has been achieved. The authors mentioned that the statistical plan was prespecified. However, the protection from the bias of prespecification is rather less for observational-based approaches compared with randomized comparisons (even when the data for the non-randomized comparisons come from randomized trials), as the risk of ‘getting it wrong’ is substantially greater for observational-based approaches [6] than for randomized comparisons. The solution where possible is to contrast observational analysis with randomized trials using a target trial emulation approach [7] and use these as a cautious bridge into new analytic ground. That said, prespecification is very helpful as it may avoid the lure of an interesting answer. However, the prespecified analysis should be subject to considerable supportive or sensitivity analyses to establish its robustness to other reasonable strategies. The analyses we have conducted should be considered part of that validation strategy. The analytic approach undertaken by the authors (contrasting 3 strategies in a single match, and using matching with replacement) does bring some extra challenges to pitch against the clear benefits of having a level playing field for comparison across all 3 strategies. While the standardized mean difference is quite helpful in identifying imbalance, there is a potential challenge applying standard criteria for the SMD over 3 groups since an imbalance in 1 group may be somewhat obfuscated by similarity in the other 2. Also, standard methods SMD for categorical variables (e.g. those with >2 levels) can give quite surprising results, which may not reflect the importance of observed baseline differences. Replicating the matching achieved by the authors led to the identification of several systematic imbalances in the baseline characteristics of subjects and, thus, we question how well they are matched. While the authors argue for the merits of a randomized trial to address this question, it is not clear from reanalysis of these data that this might come to firm conclusions if it is to consider mortality as the primary end point. However, if substantial clinical uncertainty remains, there is no substitute for a properly randomized and conducted trial to address this question. Such a trial will however need to be large. The analyses presented by the authors are not under criticism as the approach is scientifically sound and the results of the implementation of the statistical methodology appear correct. We have conducted additional sensitivity analyses, which approach the question in slightly different although more orthodox ways, and we have not found the substantial differences in clinical outcome identified by the authors in their analyses. Firm conclusions await the results of properly conducted randomized trials. We thank the authors for participating in this fascinating exercise in open science. We strongly believe that the overall rigour of scientific publishing profits from open data exchange and interpretation. Conflict of interest: Nick Freemantle declares that UCL receives a grant from EACTS to provide methodological and educational input and that this work was conducted as part of that arrangement. Patrick O. Myers declares that he is Secretary General of EACTS. Matthias Siepe declares that he is Editor-in-Chief of this journal and Trustee of EACTS.