A simulation-based bias analysis to assess the impact of unmeasured confounding when designing non-randomized database studies

Unmeasured confounding is often raised as a source of potential bias during the design of nonrandomized studies, but quantifying such concerns is challenging. We developed a simulation-based approach to assess the potential impact of unmeasured confounding during the study design stage. The approach involved generation of hypothetical individual-level cohorts using realistic parameters, including a binary treatment (prevalence 25%), a time-to-event outcome (incidence 5%), 13 measured covariates, a binary unmeasured confounder (u1; 10%), and a binary measured "proxy" variable (p1) correlated with u1. Strengths of unmeasured confounding and correlations between u1 and p1 were varied in simulation scenarios. Treatment effects were estimated with (1) no adjustment, (2) adjustment for measured confounders (level 1), and (3) adjustment for measured confounders and their proxy (level 2). We computed absolute standardized mean differences in u1 and p1 and relative bias with each level of adjustment. Across all scenarios, level 2 adjustment led to improvement in the balance of u1, but this improvement was highly dependent on the correlation between u1 and p1. Level 2 adjustments also had lower relative bias than level 1 adjustments (in strong u1 scenarios: relative bias of 9.2%, 12.2%, and 13.5% at correlations of 0.7, 0.5, and 0.3, respectively, vs 16.4%, 15.8%, and 15.0% for level 1). An approach using simulated individual-level data is useful to explicitly convey the potential for bias due to unmeasured confounding while designing nonrandomized studies, and can be helpful in informing design choices. This article is part of a Special Collection on Pharmacoepidemiology.