The Causal Analysis of Cognitive Outcomes in the Coleman, Hoffer and Kilgore Report

In their report to the National Center for Education Statistics, James Coleman, Thomas Hoffer and Sally Kilgore (henceforth referred to as CHK) (1981) address three questions with respect to cognitive outcomes (= test scores) in the American high school system (= 3 sectors: public, Catholic and other-private): (1) Do mnean test scores differ across the three sectors? (2) Do the sectors actually produce the differences in outcome? (3) How do the sectors produce those differences-that is, by what policies do they accomplish their effects? CHK provide these answers: (1) Yes: Mean test scores are higher in the private sectors than in the public sector. (2) Yes: Test scores are higher in the private sector even after controlling for differences in student characteristics. Furthermore, the private sectors produce larger gains in test scores during the last two years of high schoolthan does the public sector. Furthermore, the Catholic come closer to the common schoorl ideal, by educating students of varying background more nearly alike than do the other-private and public schools. (3) The private produce their higher test scores by placing higher academic demands and imposing stricter discipline on their students than do the public schools. Our task is to assess the validity of these answers-that is, to evaluate the evidence and reasoning that generated them. We have been handicapped by the style of the Report, a document of 233 pages + appendices. Elaborate calculations from various regression equations are given, but the equations themselves are rarely presented. Sample sizes are seldom in-, dicated. The definitions of variables are often cryptic. In such circumstances one might want to rely on the objectivity and scientific judgment of the authors as a substitute for the documentation one expects to find in a scientific report. But there is so much advocacy in the CHK Report that this option was unavailable to us. We have, however, been assisted by access to some of the authors' computer output provided to us on request and by access to unpublished studies by others who have been reanalyzing the original data set. Finally, we have tapped CHK's article, Cognitive outcomes in public and private schools (Coleman et al., 1982), and Coleman's own restatement, Private schools, public and the public interest (Coleman, 1981). Our summary assessments are that the methods and interpretations used by CHK fall below the minimum standards for socialscientific research, and that the CHK answers to the questions posed above are not warranted by their evidence.