Strengthening Real‐World Evidence on Question Not Answered by Randomized Trials: A Trial Calibration Approach

Real world evidence studies the effectiveness and safety of medical products in clinical practice by analyzing real-world data, often from electronic health records, insurance claims, or registry data 1. Real world evidence has been used for decades to inform regulatory decisions about medication safety, but more recently to inform regulatory decision about treatment effectiveness. There is interest for real world evidence to complement randomized controlled trial evidence to inform treatment guidelines and regulatory decisions when randomized controlled trials are no available. Such situations occur in rare diseases where real world evidence is frequently included in regulatory decision making. However, in many prevalent chronic diseases important clinical questions on medical products are unlikely to be studied with human experiments because of a lack of financial incentive, for example, combination of two drugs manufactured by competitors, and the difficulty to recruit subjects, for example, infrequent cancer subtypes, or the questionable ethicality, for example, pregnant women. Real world evidence derived from studies conducted using data recorded by clinical practice can provide valid estimates of treatment effects, as evidenced by many database studies that were subsequently confirmed by randomized controlled trials. Canagliflozin, a sodium-glucose cotransporter-2 inhibitor first approved for the treatment of Type 2 diabetes lowering HbA1c values, and its effect on reducing the risk of cardiovascular events in patients with Type 2 diabetes was first reported in a large-scale real world evidence study conducted in insurance claims data against alternative treatments 2, and only subsequently confirmed in a cardiovascular outcomes trial (CANVAS) 3. More recently, the effect of gonadotropin-releasing hormone (GnRH) antagonists compared with GnRH agonists in patients with prostate cancer and the occurrence of cardiovascular events was first reported in a real world evidence study 4, while a subsequent randomized controlled trial (PRONOUNCE) provided similar findings but included fewer participants 5. Of course it is recognized that there are plenty of high-profile examples where non-randomized studies, sometimes using secondary data, have misled treatment recommendations, for example postmenopausal hormone therapy and vitamin E to reduce cardiovascular events 6 or statins to reduce the risk of cancers 7. There are many more such examples, raising the question whether there are specific design and data issues that, if addressed adequately, would predictably lead to real world evidence studies with robust causal conclusions on treatment effects. Following the mandate of 21st Century Cures Act of 2016, the FDA proposed a framework to perform real world evidence studies with clinical practice databases to support effectiveness claims 1, acknowledging challenges due to the absence of randomization in observational studies and the identification of clinically relevant measures of outcomes that are not usually captured in secondary data such as electronic health records and insurance claims data. Since 2016, several guidance documents on the use of real-world evidence to approve medical devices, drugs, and biologics have been published. In 2020, 90% of FDA drug approvals incorporated real world evidence, but real world evidence was considered as essential evidence in only 10% 8. The main issues raised by FDA reviewers included insufficient data accuracy or completeness, concerns about residual confounding without baseline randomization, and the lack of pre-specified protocols. A frequently-used approach to avoid self-inflicted bias is to explicitly emulate a hypothetical trial by using non-randomized data from clinical practice 9. All design and measurement aspects of a real world evidence study then mimic those of the emulated hypothetical "target trial," which provides clarity in key design aspects, including eligibility criteria, outcome measurement, treatment strategy, start and end of follow-up, causal contrast, and the analysis plan 10. The next logical step is to calibrate database studies against actual trials if measurement quality allows. If the emulation against an actual trial produces similar findings it empirically demonstrates the ability of the database study design to support causal conclusions. The randomized controlled trial-DUPLICATE initiative emulated more than 30 completed and 2 ongoing trials using clinical practice data in a consistent, prospectively registered, transparent, and reproducible process that increases acceptability by decision makers. For trials that could be emulated well in design and measurement, the effect estimates were highly concordant (r2 = 0.94) 11. A new era of real-world evidence that is anchored in trial evidence is emerging (Figure 1). Evidence on drug effects does not come out of thin air and builds on increasingly larger and refined (randomized) studies in humans. Expanding on the lessons learned from systematic trial emulations, a two-stage approach will instill further confidence in real world evidence when randomized trial evidence are not available: In a first stage, an existing high-quality randomized controlled trial that studies the medication of interest will be identified. This trial will then be emulated as closely as possible in terms of design and measurement using data from clinical practice. This limits the breath of this approach to those clinical questions where key variables are observable in a fit-for-purpose real-world data source, for example, exposure, outcome, population identifiers to determine eligibility criteria, and key confounders. If an emulation seems feasible, Institutional Review Board (IRB) approval should be obtained, a protocol is developed and deposited before the analysis begins. If the first stage real world evidence emulation reveals similar findings as the randomized controlled trial, one moves to a second stage: Using the same real world evidence data infrastructure and analytic approach as in the successful first stage trial emulation, the investigator may now vary a single study parameter in the real world evidence study to address a relevant knowledge gap, for example s/he may replace a surrogate endpoint with a clinical endpoint, may expand the study population from predominantly younger males to include older adults and more women, or change the comparator from a single drug to a combination of two drugs. This two-stage approach will bolster confidence that the second stage real world evidence study will come to the same findings as a hypothetical trial addressing the same question, given the success in the first stage. The first stage provides an important diagnostic besides limitations inherent to any real-world evidence studies (Table 1). If we cannot emulate the existing trial design and measurement well, it implies a need to consider alternative measurement, design, or analytic choices in the real world evidence study. However, successful emulation in the first stage does not guarantee success in the second stage, particularly if the measurement characteristics from Stage 1 to Stage 2 substantially worsen, if the treatment channeling mechanisms are substantially different, or if key confounders related to a different outcome definition from Stage 1 to Stage 2 are not captured. Published RCTs addressing a research question close to the RWE study of interest Research question addressed by varying a single study parameter compared to stage 1, for example, As an example, we demonstrated this two-stage approach with a series of studies on the treatment of immune-mediated inflammatory disease, specifically inflammatory bowel disease (IBD) 12-14. The therapeutic armamentarium in IBD dramatically increased in the last decade, but head-to-head and add-on controlled trials are lacking and the optimal position of each drug remains largely unknown. A major clinical question is whether the combination of a biologic and a co-immunosuppressant is more effective than biologic monotherapy. While two randomized controlled trials have shown that the combination of infliximab and thiopurines is more effective than infliximab monotherapy in patients with Crohn's disease and ulcerative colitis 15, 16, the impact of adding thiopurines to vedolizumab, the first targeted biologic agent approved for the treatment of both inflammatory bowel diseases after the era of tumor necrosis factor antagonists, remained unknown. Ideally, this question would have been addressed by a randomized controlled trial, but no such randomized controlled trials were planned. In our setting, we first successfully replicated the findings of the two randomized controlled trials (SUCCESS, SONIC) assessing the effectiveness of infliximab and thiopurines compared with infliximab monotherapy using two US and one French claim database. Results were highly consistent with the randomized controlled trials and across databases 12, 13. We subsequently emulated a hypothetical target trial assessing the effectiveness of vedolizumab plus thiopurines compared with vedolizumab monotherapy using the same databases and methodological framework, including outcome definitions 14. As SUCCESS and SONIC were restricted to patients not previously exposed to biologics while the majority of patients exposed to vedolizumab have been previously treated with biologics in real-life setting, the only variable added as confounder from Stage 1 to Stage 2 was the number of prior biologics. All other confounders considered were similar between Stage 1 and Stage 2. Bolstering our confidence in the results of real-world evidence studies is critical to inform healthcare decision making. A two-stage approach will help build confidence in real world evidence studies in some settings by incorporating learnings from calibration against actual randomized controlled trials with related clinical questions. Such an approach would reproducibly lead to robust non-experimental methodology applied to fit-for-purpose data leading to randomized controlled trial-calibrated real world evidence findings. The authors disclose the following: Julien Kirchgesner received research support from the French National Society of Gastroenterology (SNFGE), consulting fees from Janssen, Lilly, Celltrion, Abbvie, Takeda, Roche, and Pfizer for unrelated studies. Shirley V. Wang has consulted for Exponent Inc. and MITRE, a federally funded research and development center for the Centers for Medicare and Medicaid Services. Sebastian Schneeweiss is participating in investigator-initiated grants to the Brigham and Women's Hospital from UCB and Boehringer Ingelheim unrelated to the topic of this study. He is a consultant to Aetion Inc., a software manufacturer of which he owns equity.