Beyond the Big Test: Noncognitive Assessment in Higher Education (review)

Reviewed by: Beyond the Big Test: Noncognitive Assessment in Higher Education Nathan R. Kuncel (bio), Lisa L. Thomas (bio), and Marcus Credé (bio) William E. Sedlacek. Beyond the Big Test: Noncognitive Assessment in Higher Education. 2004. Jossey-Bass Higher and Adult Education Series. San Francisco: Jossey-Bass, 273 pp. Cloth: 41. 85. ISBN: 0-7879-6020-9. Standardized tests, though valid predictors of subsequent success, are designed to measure one specific set of individual differences. Other characteristics of students and the environments in which they study also affect their success. Unfortunately, current efforts to quantify other relevant characteristics are generally relegated to the unstructured and unstandardized personal statement, essay, interview, and letter of recommendation. We can do better with current measurement technology. Sedlacek's Beyond the Big Test discusses other potentially useful measures for admissions as well as opportunities for counseling and program development. The author focuses most of his attention on the characteristics that are central to his own research and offers in the appendices measures he has developed. A book of this type can make two major contributions. The first is as a general review and summary of the literature, thereby serving as a reference for other researchers. The second is as a resource for practice either by providing new measures and instruments or by discussing how other existing instruments may be used. As a review of the state of the literature, the book is mixed. The discussion of general psychometric issues was quite accessible and might serve to inform people new to this domain; but this book also presents positions that are not supported by current research. A major guiding thesis of the book is that no test can yield equal validity for different groups. Yet considerable research on predictor bias and differential item functioning (DIF) indicates that test scores do yield comparable information across racial groups and gender groups when other variables are considered (e. g. , Cleary et al. , 1975). Moreover, it is current professional practice to discard items that display DIF before they are used in a test. Sedlacek does not consider this research and professional practice in his review. In another instance, the author argues that standardized tests at the college and graduate level predict little beyond first-year grades. Numerous large-scale studies and meta-analysis suggest that they do, in fact, predict other outcomes, including grades at all years, faculty ratings, research productivity, evaluations of creativity, profession-specific licensing exam scores, evaluations of career potential, degree attainment, and even job performance (e. g. , Kuncel, Hezlett, & Ones, 2001, 2004). Finally, it would have been desirable to have had a deeper review of the literature related to predictors not developed by the author. Although Sedlacek's thesis that additional measures are needed to obtain a more complete picture of students is unquestionably true, his analysis neglects a thorough and accurate discussion of existing measures and research. As a resource for practice, the author should be commended for providing a large number of measures for assessing campus climate, attitudes toward other groups, and predictor measures for student admissions. Often, however, the discussion of the efficacy of these measures was not sufficiently extensive. In some cases, the information was even misleading. For example, Sedlacek cites what appear to be a reanalysis of virtually the same data to suggest that his admissions measure, the Non-Cognitive Questionnaire (NCQ), has consistently been found effective (e. g. , Tracey & Sedlacek, 1984, 1987). This creates the appearance of a substantial body of evidence when a smaller one actually exists. Similarly, some of the research cited gives the impression of strong supporting empirical evidence when little is actually available. In one case the author claims that research demonstrates that the NCQ has good test-retest reliabilities, a critical feature for admissions and counseling purposes, but fails to mention that the cited evidence is based on a sample of only 18 students. Our own meta-analysis of the predictive validity of the NCQ has found an average validity of nearly zero across 47 studies and more than 9, 000 students (Thomas, Kuncel, & Credé, 2004). While our research was not a comprehensive analysis of all of the instruments presented in this book, our findings. . .