Temporal and Spatial Variation in Environmental Impact Assessment

We compare two approaches to designing and analyzing monitoring studies to assess chronic, local environmental impacts. Intervention Analysis (IA) compares Before and After time series at an Impact site; a special case is Before–After, Control–Impact (BACI), using comparison sites as covariates to reduce extraneous variance and serial correlation. IVRS (impact vs. reference sites) compares Impact and Control sites with respect to Before–After change, treating the sites as experimental units. The IVRS estimate of an “effect” is the same as that of the simplest BACI (though not of others), but IVRS estimates error variance by variation among sites, while IA and BACI estimate it by variation over time. These approaches differ in goals, design, and models of the role of chance in determining the data. In IA and BACI, the goal is to determine change at the specific Impact site, so no Controls are needed. IA does not have controls and BACI's are not experimental controls, but covariates, deliberately chosen to be correlated with the Impact site. The goal given for IVRS is to compare hypothetical Impact and Control “populations,” so the Controls are essential and are randomly chosen, perhaps with restrictions to make them independent of each other and (presumably) of Impact. IA and BACI inferences are model based: uncertainty arises from sampling error and natural temporal processes causing variation in the variable of concern (e.g., a species' abundance); these processes are modeled as the results of repeatable chance setups. IVRS inferences are design based: uncertainty arises from variation among sites, as well as the other two sources, and is modeled by the assumed random selection of Impact and Control sites, like the drawing of equiprobable numbers from a hat. We outline the formal analyses, showing that IVRS is simpler, and BACI more complex, than usually supposed. We then describe the principles and assumptions of IA and BACI, defining an “effect” as the difference between what happened after the impact and what would have happened without it, and stressing the need to justify chance models as reasonable representations of human uncertainty. We respond to comments on BACI, some of which arise from misunderstanding of these principles. IVRS's design-based justification is almost always invalid in real assessments: the Impact site is not chosen randomly. We show that “as if random” selection by “Nature” is untenable and that an approximation to this, while a possibly useful guide, cannot be used for inference. We argue that, without literal random assignment of treatments to sites, IVRS can only be model based. Its design and analyses will then be different, using and allowing for correlation between sites. It is likely to have low power and requires strong assumptions that are difficult to check, so should be used only when IA or BACI cannot be used, e.g., when there are no Before data.