Evaluating the performance of paper people.

Several recent studies of performance appraisal have used the approach, in which raters read performance vignettes and then rate the performance of several hypothetical ratees. This approach may lead to systematically different outcomes from studies in which ratings are based on the direct or indirect (e.g., via videotape) observation of ratees' behavior. One hundred and eleven studies published between 1975 and 1984 were grouped into five major substantive categories, and a comparative meta-analysis was used to contrast the outcomes of paper people studies to those of similar studies in which ratings were based on the observation of ratee behavior. Effect sizes were found to be significantly larger in paper people studies, although this difference was not uniform across all research areas. Results are discussed in terms of differences in signal-to-noise ratios across the two methods. Many methods have been used in research on performance appraisal. One dimension for classifying these methods is based on whether they use a approach, in which subjects read a series of vignettes containing scaled behavior descriptions and then rate the performance of one or more hypothetical ratees, or an approach that is based on the direct or indirect observation (i.e., via film or videotape) of the behavior of one or more ratees. The latter method has been used in studies ranging from those examining annual perfomance appraisals conducted for administrative purposes in organizations (e.g., Cleveland & Landy, 1981) to laboratory studies, in which the ratees are actors carrying out performance scripts (e.g., Murphy, Balzer, Kellam, & Armstrong, 1984). The critical difference between the two approaches is that in a paper people study, subjects receive written descriptions of the performance they are to evaluate, whereas in studies involving behavior observation, subjects must observe, interpret, and recall the behavior of ratees in the process of evaluating their performance. The use of paper people in research on the interview has been strongly criticized (Gorman, Clover, & Doherty, 1978), although Dipboye, Stramler, and Fontenelle (1984) noted that this approach in fact simulates several features of the interview process. There are also grounds for criticizing the use of paper people in performance appraisal research. First, the question of whether raters observe or read summaries of the performance they are to evaluate has a substantial impact on the nature of the rating task. For example, raters' ability to accurately observe, encode, and recall performance is a critical concern in much of the recent research on performance appraisal (DeNisi, Meg