An Index of Studentsʼ Satisfaction with Instruction

The purpose of this study was to determine whether a students' satisfaction index (SSI), derived from responses to a single rating of a faculty member's overall instructional ability, is a reliably valid tool for identifying those medical school faculty members whose instruction is in need of improvement. Background Debates have been held since the 1950s on the validity of students' evaluations of faculty (SEF).1,2,3,4,5,6 While opinion is split over the application of SEF to the management of the professoriate,7,8,9 the majority of the researchers addressing this issue support the use of SEF in the areas of faculty development and instructional improvement.10,11,12 The primary premises underlying use of students' ratings of the instructional abilities of faculty has been repeatedly addressed by educational researchers. The reliability and validity of such measures have been the subject of numerous studies.13,14,15 While their results have not been consistent, they have predominantly supported the stability and validity of students' evaluations of faculty. The use of a single global measure to assess instructional ability has also received the attention of the researchers, and it has been established that such a measure can be valid.18,19,20 Students evaluate every instructor at the Texas College of Osteopathic Medicine. The items on the required evaluation form can be changed to correspond to the needs of academic departments and the unique characters of instructional programs. However, the final item on each and every evaluation form is constant across all courses. That item states “Overall, this is an effective instructor.” Responses to this question are made on a five-point scale and reported on a 80-point scale (strongly agree equates to an SSI of 100, agree = 80, neutral = 60, disagree = 40, and strongly disagree = 20). In practice, the SSI works like grades assigned to students in graduate and professional training programs. Medical students seldom use the lower half of the rating scale. Their responses result in SSI scores from the 60s through the 90s, with a mean score of 77. The SSI score is transmitted, along with responses to the other concepts assessed, to the faculty member and the department chair. Predictably, when an SSI drops below 70 the department's chair begins to comb through other data on teaching performance to find out why the faculty member is not doing well. During the past 15 years faculty members have expressed a variety of concerns about the SSI. Faculty members think the SSI reflects the students' moods at the moment and, thus provides unreliable results. They feel that students are not capable of judging their instruction and that their peers would provide different (presumably more favorable) results. They suggest that the students' ratings correlate highly with the grades the students receive. Many feel that an assessment made at the end of the course disadvantages those who teach early. Each of these beliefs casts doubt on the reliability and validity of the SSI. These are the questions that are addressed in this study: (1) Is the SSI reliable? (2) Is the SSI valid? (3) Is the SSI biased by grades? (4) Is the SSI biased by the time lag between performance and measurement? Method Eighty-five second-year medical students in a five-month course in internal medicine agreed to evaluate quality of instruction for each of the 124 lectures given during the course. The 24 faculty teaching the course agreed to end their classes ten minutes early each day to allow the students time for evaluation. The departments of Medicine and Medical Education agreed to have faculty members present to evaluate each lecture. As an inducement for participation, the Department of Medical Education generated individual formative assessments and suggestions to improve the quality of instruction for each faculty member who taught for three or more hours in the course. These reports were delivered after all the data in the study had been collected. Data were collected following each of 124 lectures from the entire portion of the 85 students who attended the lecture. Faculty also evaluated each of the lectures in the course. Complete data were collected for 24 instructors. One instructor was removed from the study due to the onset of acute illness during instruction. The 23 remaining instructors were each assessed by ten to 310 students, and by two to nine faculty members. One of the faculty evaluators in each session was from the Department of Medical Education; the rest were from teaching faculty in the Department of Medicine. At the end of the course the normal instructor-evaluation process was conducted and a post-course SSI was derived at that time. Results Is the SSI Reliable? To answer this question a test-retest was conducted. Students' evaluations at the end of the course were compared with those obtained from the same students at individual lectures. In Figure 1, faculty have been ranked by their end-of-course evaluations. Figure 1 presents a mean end-of-lecture SSI and a mean end-of-course SSI for each of the 23 faculty members. The end-of-lecture SSI is the average response of the students who attended the lectures presented by the instructor. The end-of-course SSI is the average response of the students who evaluated that instructor at the end of the course. The correlation between these two sets of satisfaction indexes is r =.847.Figure 1: Students' ratings of 23 faculty immediately following each faculty member's lecture (broken line) and again at the end of the course (solid line). The faculty all delivered lectures at different points during the five-month-long internal medicine course.There is some difference between the means (end-of-lecture mean = 83.21, end-of-course mean = 82.72). A paired t-test of the difference between the means of the measures was not significant (p =.647). The practical significance of observed differences depends on their interpretation. One pair of observed differences could cause an instructor to be viewed as a member of a different group. Students' satisfaction with Instructor 2 shifted from moderate agreement to neutral. In the other 22 cases the instructors' ratings remained in the same relative positions on the criterion scale. Is the SSI Valid? A concurrent validation was performed. Students' SSI scores were compared with those derived from the observations of the faculty members. Figure 2 presents end-of-lecture ratings from students and faculty for each of the 23 instructors. The students' end-of-lecture satisfaction index is the mean response of students who attended the lectures presented by the instructor. The faculty's end-of-lecture satisfaction index is the mean response from faculty members who attended the same lectures. The correlation between these two sets of satisfaction indexes is r =.846.Figure 2: Students' (solid line) and faculty's (broken line) ratings of 23 instructors immediately following the lectures they delivered during a five-month-long internal medicine course.The measures had different mean values (students = 83.21, faculty = 79.23). A paired t-test of the difference between the means of the measures was statistically significant (p =.012). These instructors were consistently rated almost four points lower by faculty than they were rated by students. On a practical basis, several pairs of these observed differences could cause an instructor to be viewed as a member of a different group. Instructors 3, 6, 10, 15, and 19 could be viewed as less competent on the basis of their peers' ratings. Is the SSI Biased by Grades? Four non-cumulative examinations were given in the course. The grades received on the four exams were correlated with the average satisfaction index given by each student during the time block covered by each exam (n = 84). The resultant correlations were: r = −.01,.025,.089, and.090. There was no systematic relationship between grades and SSI scores. Is the SSI Biased by Lag Time to Evaluation? End-of-course evaluations of the 23 instructors in this study followed their lectures by as much as five months. To assess the effect of lag time on the evaluations, each instructor was assigned to one of four groups based on the number of months that passed between his or her last lecture and the end-of-course evaluation. Table 1 presents descriptive data for the resultant groups. By subtracting the end-of-course satisfaction index from that obtained following their lectures, difference scores were generated. A one-way fixed-effects analysis of variance of the difference scores compared across groups was not significant (p =.821).TABLE 1: Students' End-of-course Satisfaction Indexes by Months of Lag from Lecture to Evaluation*Incidental Findings. There is a ceiling effect in medical students' responses to the question that forms the basis for the satisfaction index. As a result, for both the end-of-lecture SSI and the end-of-course SSI there was less than one standard deviation available above the mean response in 52% of the cases. This reflects the fact that distributions of SSI responses were negatively skewed (−1.289). The behaviors of the medical students changed during the study. They came to the lectures in far greater numbers than they had before. In fact, a secondary issue the authors wanted to investigate regarding the effect of attendance on performance could not be addressed. There were too few students in the “non-attending” pool to do an analysis. Conclusions The SSI demonstrated sufficient reliability and validity in this population of students and faculty to support its use by our institution as a tool to identify faculty members whose instruction is questionable. The measure did not appear to be biased by either earned course grades or lag time to evaluation. The marked negative skew and ceiling effect observed in the data limit the application of the SSI to its intended purpose, flagging poorly-performing instructors. Differentiation among the instructors at the upper end of the SSI is practically impossible. Discussion The findings of this study support the assumption that a single item can be used to assess the global effectiveness of a faculty member's instruction.17 However, caution should be used in generalizing the findings. The strength that the student satisfaction index has demonstrated may have been due, in part, to its use with second-year medical students. Medical students are highly focused intelligent respondents whose backgrounds are academically homogeneous. The strength of the index is also partially attributable to the fact that the respondents commonly used only half of the scale's values. This keeps the satisfaction index values for most instructors in a relatively narrow band and gives rise to a pronounced negative skew in their distribution. The concurrent validity of the SSI was assessed by comparing students' responses with the responses of teaching faculty from the Department of Medicine and PhD-level medical educators. The study was conducted in a medical school that routinely evaluates every faculty member in every course. Data were collected and analyzed by a department that the students had come to trust for its grading of their examinations and forwarding of their assessments of faculty and courses to the administration. Both the students and the faculty knew they were engaged in a study of the SSI. The extent to which these environmental and research variables affected the results of this study will be known only when researchers subject the SSI to further analysis. Finally, an ethical issue may be raised by the SSI. The SSI was developed as a flag to inform the faculty member and department chair that there could be a problem in the area of instruction. As such, it does not tell the user why the respondents feel that the instruction they have received is suspect; neither does it give guidance to assist in remedying the problem. It is incumbent upon an institution to develop the means for both accurately identifying the nature of the problem18,19 and also addressing it productively20,21,22,23,24 before it elects to discover which of its faculty has the problem.