Propensity score estimation: assessing machine learning methods for an observational prison cohort

Background Propensity scores are an important tool for using observational data to answer causal questions. Machine learning methods for estimating propensity scores have out-performed more commonly used logistic regression methods in studies considering large, synthetic datasets. To inform propensity estimation methods for smaller datasets from real-world observational studies, we describe the implementation of machine learning algorithms using data from a prison-recruited cohort. Methods This study describes a procedure to use logistic regression, gradient-boosting, random forest and single-hidden-layer neural networks to balance confounders between a treatment and control group. We assess balance using the average standardised absolute mean difference (ASAM), where a lower ASAM indicates better balance. We provide an application to an observational cohort of Australian incarcerated men for a causal question for the effect of emotional support in prison on emergency department presentations within 100 days post prison release. Results There were 328 participants, of whom 231 (70%) reported accessing emotional support during incarceration and 74 (23%) presented to the emergency department within 100 days post release. ASAM in the unweighted sample was 17.7. Single-hidden-layer neural networks achieved the lowest ASAM (10.6) followed by random forest (13.7), logistic regression (14.3) and gradient-boosting (17.1). Conclusions All weighting techniques decreased the ASAM, with single-hidden-layer neural networks performing the best. This finding strengthens the evidence for the role of machine learning methods for achieving balanced datasets using real-world observational data.