Measurement Issues in the Alignment of Standards and Assessments

This study examined the impact of rater agreement on decisions concerning the alignment between the Golden State Examination in High School Mathematics (California Department of Education, 2001a Academic Senate of the California State University. 1997. Statement on competencies in mathematics expected of entering college students Sacramento, CA: Author. [Google Scholar]) and the University of California (UC) Statement on Competencies in Mathematics Expected of Entering College Students (Academic Senate of the California State University, 1997 Academic Senate of the California State University. 1997. Statement on competencies in mathematics expected of entering college students Sacramento, CA: Author. [Google Scholar]). The UC faculty and high school mathematics teachers (N = 20) rated the mathematics items of the Golden State Examination relative to the expectations identified in the UC competency statement, identifying item features related to content and dimensionality. Raters assigned values for a primary topic, a secondary topic, item/topic centrality, and depth of knowledge. Agreement within these criteria was the basis of the assessment of alignment. Results showed considerable variability in judgments across raters and different pictures of alignment depending on the particular subset of raters providing the ratings. A few differences emerged between rater types.