The Role of Known Effects in Observational Studies

SUMMARY When treatments are not randomly assigned, treated and control subjects may be quite different prior to treatment, so that straightforward comparisons of responses in treated and control groups may give a distorted impression of the effect of the treatment. While adjustments for observed pretreatment differences can help, there is often reason for concern that important differences were not measured and not controlled by statistical adjustments. This paper concerns methods for detecting and indicating the probable impact of such unobserved pretreatment differences. The methods discussed here use known effects of the treatment on certain supplementary responses included in the study to provide information about unobserved pretreatment differences. Previous work has noted that known effects provide the basis for a statistical test of the assumption that adjustments for observed covariates suffice in removing bias. Here, the matter is taken several steps further by addressing the following questions. What are the formal properties of such tests? Under what circumstances are the tests effective at detecting unobserved pretreatment differences? Or to put it another way, what sorts of supplementary response variables provide the most effective checks? If unobserved pretreatment differences are detected, what can be said about the direction of the biases they produce? These statistical tests are, of course, as influenced by the available sample size as they are by the magnitude of the unobserved pretreatment differences they are intended to detect. Given that the pretreatment differences in question are not observed directly, what can be concluded about their magnitude? To motivate the discussion, two examples are discussed throughout the paper. 1. Examples of the Use of Known Effects in Observational Studies In observational studies, subjects are not randomly assigned to treatments, making interpretation difficult, as differences in outcomes in treated and control groups may not reflect effects of the treatment, but rather pretreatment differences between the groups. Effective adjustments for observed covariates certainly help; however, there is usually reason for concern that some important pretreatment differences may not have been measured, so that adjustments for observed covariates may not suffice to render the groups comparable. This paper considers methods for detecting such unobserved pretreatment differences. These methods use knowledge about the effects of the treatment on certain supplementary response or posttreatment variables included in the study specifically to provide information about unobserved pretreatment differences. A brief outline follows. The current section contains motivating examples of the use of known effects in observational studies. A brief review of concepts and notation for observational studies is contained in Section 2; there is little new material in this section. Section 3 establishes a result concerning underadjustment, that is, concerning the direction of the bias remaining after adjustment for observed covariates. Section 4 contains material that is specific to the use of known effects in observational studies. Throughout, it is assumed that comparisons of treated and control groups are made after an effective