Still weighting for the right criteria to validate student evaluations of teaching in the IDEA system.

This study clarifies issues raised in a series of studies about the Instructional Development and effectiveness Assessment (IDEA) system of students' evaluations of teaching effectiveness (SETEs) and pursues research stimulated by these studies. On the basis of a paragigm proposed by D. P. Hyot (1969), W. E. Cashin, R. G. Downey, and G. R. Sixbury (1994) used agreement between student ratings of progress on 10 course objectives and teacher ratings of the importance of each objective to validate the ratings. The major new contribution of the present investigation is to evaluate the construct validaty of progress ratings (dimensionality and relation to importance), the central feature of the IDEA system. Much of the modest agreement between progess and importance ratigns was explained by discipline, and the extent of agreement was nearly unrelated to teaching effectiveness. Two factors underlying the 10 progress ratings (Subject/Professional Mastery and personal Development) were found, but the correlation between the factors was large (.757). This apparent lack of discrimination among the progress ratings may explain in part why various weighted and unweighted averages of the progress ratings performed so similarly. Studies validating IDEA ratings in relation to traditional criteria in SETE research (e.g., student learning, teacher self-evaluations, evaluations by former students, interventions designed to improve teaching) are needed.