Generalizing across stimuli as well as subjects: A neglected aspect of external validity.

In order to generalize the results of an experiment beyond the specific stimuli employed, it is necessary to consider variance due to stimulus sampling. This can be accomplished by treating stimuli as a random effect rather than the traditional procedure of treating stimuli as a fixed effect. The serious consequences of the traditional approach are illustrated using examples from applied psychology. Statistical and design considerations for generalizing the results of experiments are discussed. In the design of psychological research, it is of utmost importance to be able to separate the effects of experimental manipulations from the effects of extraneous variables. It is inevitably the case that subjects possess attributes that are unmeasured, uncontrolled, and have unpredictable effects. However, it is well known that randomization allows the researcher to estimate the magnitude of the effects of these extraneous variables and, through the use of standard techniques of statistical inference, to determine the probability that differences between conditions as large as (or larger than) those obtained would occur if these extraneous variables were operating alone. Just as in the case of subjects, it is likely that unmeasured and uncontrolled attributes of stimuli affect the experimental outcome in unpredictable ways. The potential for this problem exists in many areas of applied psychology where extraneous aspects of stimuli are difficult to assess. For example, consider a hypothetical study of sex discrimination in which four males and four females play the role of an applicant interviewing for a managerial position. Although applicants read the same script, there are (unavoidably) differences in poise, physical attractiveness, and presentation style. If the male applicants were rated as being significantly more qualified than the female applicants, would the Requests for reprints should be addressed to David