Considerations and Techniques for the Analysis of NAEP Data

The special characteristics of the National Assessment data that affect the validity of conventional techniques of statistical analysis are considered. In contrast to the assumptions underlying standard methods of statistical analysis, the NAEP samples are obtained by means of a stratified multistage probability sampling design in which clusters of students are selected and in which certain subpopulations are sampled at a higher rate. The resulting samples have different statistical characteristics from those of a simple random sample. Analytic techniques that take the structure of the data into account are discussed and their properties explored. A second feature of the National Assessment data arises from the fact that multiple matrix sampling techniques are used for the assignment of subsets of the pool of exercises to subsamples of students. National Assessment accounts for the effect of the missing data on its scaling models through the use of multiple imputation procedures, creating vectors of “plausible values” for the scale scores. The use of plausible values and their effect on analysis are illustrated. The analyses are illustrated with results from the NAEP assessment of mathematics.