Guide for Comparing Estimators of Policy Change Effects on Health

To the Editor: The effects of policy changes on human health are of great interest to epidemiologists. Policies are often introduced in a nonrandomized fashion, and researchers must use observational data to estimate the effects of the policy change. Around 2020, new estimators were proposed mostly in the econometric literature (with one article in the epidemiologic literature) to estimate the average treatment effect on the treated (ATT) for difference-in-differences (DID) settings where polices are introduced at different times across treated units. The two-way fixed-effect estimator, which had previously been the standard approach for these settings, was shown to be biased and/or to estimate an unintuitive target parameter when treatment effects were dynamic or heterogeneous (when effects differed over time or across adoption cohorts, respectively).1 Newly introduced estimators that addressed the limitations of the two-way fixed-effects estimator include the Group-Time ATT estimator introduced by Callaway and Sant’Anna,2 the Cohort ATT estimator introduced by Sun and Abraham,3 and the Target Trial estimator developed for this context by Ben-Michael et al.4 Another tool, the Goodman Bacon decomposition,1 allows researchers to see how the traditional two-way fixed-effects estimator can be biased as a result of dynamic effects, or can estimate a nonintuitive parameter by aggregating effects across heterogeneous effects using weights that epidemiologists would be unlikely to choose. To illustrate the need for new estimators and facilitate their use by epidemiologists, we created an online guide (https://rpubs.com/corinne-riddell/guide-to-did-estimators) that uses R code to compute the ATT estimates across nine scenarios of increasing complexity and describes the findings. A policy can be rolled out in a multitude of ways, and how the policy is introduced and the shape of its effect on the health outcome matter in determining which estimator is appropriate for the analysis. For any policy analysis, the following characteristics can be observed by the researcher and can affect the performance of the two-way fixed-effects estimator and the parameter being estimated: Timing of policy adoption: When do the policy changes occur in reference to the total observation window? Staggered treatment adoption: A policy that is introduced at the same time by all treated units does not have staggered adoption. A policy change that is introduced at different time points by different units has staggered adoption. In addition, the following characteristics cannot be known a priori but also contribute to whether a traditional two-way fixed-effects analysis will produce an unbiased estimate of the policy effect or if a new estimator is more appropriate: Heterogenous treatment effects: If there are multiple adoption cohorts (groups of states that introduce the policy change at the same time), the effect of the policy change is heterogeneous if the magnitude of the effect varies across adoption cohorts. Dynamic treatment effects: A policy has a dynamic treatment effect if its effect on the health outcomes varies as a function of time since the policy’s introduction. These four characteristics, in varying combination, can affect the two-way fixed-effects estimator’s ability to produce an unbiased effect as well as the underlying parameter being estimated.1 For example, heterogeneity of the treatment’s effect across adoption cohorts (characteristic no. 3) changes the parameter being estimated because the two-way fixed-effects estimate is a weighted average of all ATTs, where the weights are a function of both the number of units treated at each time and the treatment variance, which is influenced by the time of policy adoption during the study period (characteristic no. 1).1 This matters because units treated closer to the middle of the study period have larger weights. This is an important aspect of the new literature, as it shows how the two-way fixed-effects estimate can be affected by the timing of policy introductions in terms of the total length of the panel analyzed by the researcher.4 A researcher can obtain a different two-way fixed-effects estimate of the policy’s effect on the outcome solely by adding more prepolicy time (or postpolicy time) to the analysis, because of the contribution of the treatment variance to the weights. Taking a weighted average is common in epidemiology, but the weights chosen by the two-way fixed-effects estimator are hidden by the model, and do not correspond to weights that any “reasonable” epidemiologist would choose. In this setting, the two-way fixed-effects estimate is providing a valid estimate but to the wrong question. Fortunately, the new estimators overcome this issue. This is illustrated in the guide, where we demonstrate how decreasing the amount of time observed before the policy change shifts the two-way fixed-effects estimate but not the other estimates. A separate issue arises with dynamic effects (characteristic no. 4) under staggered introductions (characteristic no. 2). Here, dynamic effects can introduce bias because the two-way fixed-effects model includes already-treated units as controls for some comparisons. Suppose that an earlier-treated unit has a larger dynamic effect than the later-treated unit (e.g., the effect of the policy change has a slope of 3 for earlier-treated units versus a slope of 2 for later-treated units). At the time of the later-unit’s policy change, the pre-post difference for the earlier-treated state will be larger than the pre-post difference for the later-treated state, making the effect of treatment look preventive (e.g., 2 − 3 = −1, where a negative number is denoting prevention) for this particular contrast, although for both treated units the treatment increased harm. This bias is a violation of the exchangeability assumption because using already-treated units violates the parallel trends assumption when there are dynamic effects. The guide demonstrates this bias; it uses the Goodman Bacon decomposition to show that the two-way fixed-effects estimate is influenced by comparisons between earlier- and later-treated units that no epidemiologist would actually want to make. It then shows how the new estimators can be used to estimate the treatment effect without this bias. In summary, if treatment effects are heterogeneous across adoption cohorts, then the parameter estimated by the two-way fixed-effects estimator uses weights to aggregate the effects that are unintuitive to the investigator. On the other hand, the presence of dynamic effects across adoption cohorts introduces a violation in the exchangeability assumption which biases the two-way fixed-effects estimate. Newly introduced estimators overcome both issues. The field of policy analysis is undergoing a shift in real-time. Estimating the impact of policies on health outcomes is vital as policies are some of the key mechanisms for intervening on health outcomes and improving health equity. We hope that this guide will serve as a valuable tool in educating epidemiologists on when traditional two-way fixed-effects estimates are appropriate, and when to use and how to estimate effects using three newer estimators. Each of the illustrative scenarios has an underlying spreadsheet containing the policy and outcome data that can be easily edited by the reader to investigate how the estimators perform for scenarios they envision. These data are imported into R, visualized, and analyzed using the various approaches and tools.