Answers are grounded in this paper and cite their sources.
The conventional approaches to mediation analysis such as path analysis and structural equation modeling typically involve specifying two structural models, one for the mediator and the other for the outcome. We employ an alternative approach that avoids some strong identification assumptions invoked by the conventional approaches. By applying a new weighting procedure to the observed data, we estimate the average potential outcome if the entire population were treated, the average potential outcome if the entire population were untreated, and the average potential outcome if the entire population were treated and if every individual unit's mediator value would counterfactually remain at the same level as it would be when untreated. The estimated differences among these average potential outcomes provide estimates of the total effect, the natural direct effect, and the natural indirect effect. Applying this approach to multilevel educational data, we evaluate the total effect of the algebra-for-all policy in the Chicago Public Schools by comparing the math achievement of two ninth-grade cohorts. We further investigate whether the policy effect was mediated by the policy-induced change in class peer ability. Combining weighting with prognostic score-based difference-in-differences adjustment enables us to reduce both measured and unmeasured confounding. Keywords: Causal inferencecausal mechanismdirect effectindirect effectmarginal mean weighting through stratificationpotential outcomespropensity scoreprognostic scoreratio-of-mediator-probability weighting ACKNOWLEDGMENTS This research was supported by a major research grant entitled "Improving Research on Instruction: Models, Designs, and Analytic Methods" funded by the Spencer Foundation, a Scholars Award from the William T. Grant Foundation and the start-up funds from the University of Chicago for the first author. Additional support came from the Institute of Education Sciences, U.S. Department of Education (Grant No. R305R060059). Opinions reflect those of the authors and do not necessarily reflect those of the granting agencies. Notes To empirically identify the lower ability students among those attending policy-affected schools, we specify a model predicting class peer ability in the prepolicy year C(0) as a function of student and school characteristics and apply it to both cohorts. Similarly, a model predicting class peer ability in the postpolicy year C(1) is applied to both cohorts. In addition, we specify a model predicting a student's conditional probability of taking algebra in the prepolicy year M(0) and apply it to both cohorts. A student is identified to be in the lower ability subpopulation if the student would likely experience a change in course-taking (M(0) < .9) and a rise in class peer ability (C(1) – C(0) > 0.3 SD). Let a i denote the policy effect on student i's class peer ability; and let b i denote the class peer ability effect on math outcome conditioning on policy exposure. Furthermore, let a=E(ai ) be the coefficient for policy exposure in the first path model; and let b=E(bi ) be the coefficient for class peer ability in the second path model. As Bullock et al. (Citation2010) showed, if individual-specific a i and b i are not independent of each other, the indirect effect E(aibi ) will not be equal to the product of coefficients ab. Instead, we have that . Petersen et al. (Citation2006) proposed a mean independence assumption implied by Assumption 8: . In other words, the direct effect of the policy no longer depends on class peer ability among those with the same pretreatment characteristics. For example, in a sequential randomized experiment, suppose that the probability of having high-ability peers is .6 under the experimental condition and .2 under the control condition. To estimate E{Y[1, C(0)]}, the ratio-of-mediator-probability weight for an experimental student who was assigned to have high-ability peers would be .2/.6=1/3, whereas the weight for an experimental student who was assigned to have low-ability peers would be .8/.4=2. In general, adjusting for a posttreatment covariate W would introduce bias because W = W(0) is observed only among students in the prepolicy cohort, whereas W = W(1) is observed only among those in the postpolicy cohort, in which case W is a function of policy exposure Z. In our application, however, W(0) and W(1) are presumably observed for every student. When z takes a fixed value, W(z) is a function of observed and unobserved pre-treatment covariates only and is no longer a function of Z. Hence adjusting for W(z) would not introduce bias in estimating the policy effect. According to Hansen (Citation2008), when the prognostic score is given, the potential outcome associated with the control condition becomes independent of the observed pretreatment covariates. Hence the prognostic score greatly reduces the dimension of covariates to be controlled for. Even though a propensity score estimated as a student's conditional probability of attending an affected school would similarly reduce the dimension of covariates, the prognostic score-based adjustment has important advantages in the current application. Most important, the propensity score for attending an affected school makes no use of the outcome information. In contrast, because the prognostic scores are predicted prepolicy math outcomes and are strongly correlated with the postpolicy math outcomes, conditioning on the prognostic scores, the local confounding effect defined in Equation 3 can be more precisely estimated. Furthermore, the propensity score for attending an affected school would be a function of student characteristics only. The prognostic scores are functions of both student and school characteristics and thus enable us to identify comparable students in comparable schools. Student-level covariates include demographics, incoming skills, social status, poverty status, residence in the attendance zone, mobility, and ELL status prior to high school. School-level covariates include average incoming skills, demographic composition, SES composition (average social status and poverty), mobility rate prior to high school, proportion from the attendance zone, and cohort size in school. A student's incoming math ability is a stronger predictor of the prepolicy math outcome if attending a school affected by the policy than if attending a school unaffected by the policy. This is because the student would probably take remedial math in a school affected by the policy but would take algebra instead in a school unaffected by the policy. We implement this procedure as follows. Analyzing Equation 2, we estimate the coefficients α(g=0) 1 and α(g=0) 2 using the prepolicy students attending unaffected schools. Similarly, α(g=1) 1 and α(g=1) 2 are estimated through analyzing the prepolicy students attending affected schools. Next, we apply the model (Equation 2) to predict a pair of prognostic scores for the entire sample. However, the prognostic score model for the pre-policy cohort would not apply to the postpolicy cohort if there were important between-cohort differences in covariate relationships with Y(0). Hence, we apply the marginal mean weight ω z in all the analysis. Because the between-cohort demographic differences might not be the same across the two types of schools, ω z is obtained separately for affected schools and unaffected schools. To predict prepolicy class peer ability, we used all student covariates and the following school-level covariates: demographic composition, SES composition, percentage of students from attendance zone, percentage of students from non-CPS schools, vocational school, algebra for all students in the prepolicy year, prepolicy algebra enrollment rate, higher math course enrollment rate before and after the policy, algebra enrollment rate among special education students before and after the policy, mean and standard deviation of math incoming skills before and after the policy, and percentage of students in special education. To predict postpolicy class peer ability, we used a similar set of covariates and additionally included the percentage of students receiving ELL services in eighth grade and the percentage of students moved prior to high school. Because is computed directly as a ratio of the two probabilities estimated from the sample, the product of ω z and may become unduly large. We opt for trimming the values larger than 10 by replacing them with a constant 1.0. We then normalize the final weight to restore the sample size. Covariance adjustment for student incoming skills is seemingly inappropriate because it assumes, for example, that a prepolicy student at the 50th percentile is comparable to a postpolicy student at the 20th percentile displaying the same level of incoming skills. Propensity score-based adjustment shows limitations due to the lack of overlap in the distribution of incoming skills between the two cohorts. As shown in our results, covariance adjustment for incoming skills led to a conclusion that the policy generated a positive total effect on math achievement. In contrast, the result from prognostic score-based adjustment with comparable precision indicates that the average total effect of the policy was indistinguishable from zero.
No takes yet. Share an insight, caveat, or question.
Hong et al. (2012) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: