Benchmarking evidence from nonrandomized healthcare database studies against randomized clinical trials could strengthen confidence that database studies can reproduce known trial results and extend analyses to questions not directly addressed in trials. However, the process can be challenging if differences in effect modifier distributions persist despite the same eligibility criteria between trials and their emulations. Post hoc population standardization can address this by aligning observable population distributions. Across four cardiovascular outcome trials previously emulated in claims data by the RCT-DUPLICATE initiative, we implemented post hoc population standardization by potential effect modifiers to study whether effect estimates meaningfully move closer to the trial findings. Exposures and comparators were 1:1 propensity score-matched on > 100 baseline characteristics. In all trials, we standardized age and sex distributions to match those of the emulated RCTs and, additionally, some cardiovascular risk factors, data permitting. Standardization resulted in close alignment of the select baseline characteristics, yet produced minimal changes in hazard ratios (HRs) and 1-year risk differences. Variance increased in some analyses, reflecting large population differences in certain trial-database comparisons. The benefits of population standardization must be weighed against challenges such as weight variability, bias-variance tradeoff, and limited overlap in covariate distributions. Importantly, differences in effect modifier distributions may not alter effect estimates if modifiers do not change the exposure-outcome relationship on the multiplicative scale, underscoring the distinction between statistical and biological interactions. In our four example trials, post hoc population standardization equalized populations but did not result in closer alignment of treatment effect estimates and expanded 95% confidence intervals.
Htoo et al. (Tue,) studied this question.