Gauging the Outcomes of Change in a New Medical Curriculum

After decades of concern about the lack of momentum in reforming medical curricula, a number of schools have introduced significant revisions and innovations in recent years. In most cases, the goals of these changes have followed the general principles promulgated by the Association of American Medical Colleges' (AAMC's) General Professional Education of the Physician (GPEP) and College Preparation for Medicine Report and other similar documents.1,2 Objectives consistent with these goals have been codified and disseminated through the AAMC's Medical School Objectives Project (MSOP).3 Several new educational strategies (e.g., problem-based learning) and course domains (e.g., courses in professional skills and perspectives) have become common elements of the resulting curricular initiatives at many medical schools.4 Given the need to track the effects and effectiveness of change in medical education programs, 5,6,7,8 Makoul developed the Student Perception Survey, 9 which focuses on how students view both the learning environment and their own learning experiences. It was first administered at Northwestern University Medical School in 1993, and has since been used by medical schools at the University of Chicago, Washington University, University of Utah, Medical University of South Carolina and, most recently, the University of Minnesota at Duluth. This study limits analysis to data collected at Northwestern between 1993 and 1999. Context In 1993, Northwestern University Medical School implemented a totally new first- and second-year (M1-M2) curriculum. Other, less sweeping, changes in the clinically oriented third- and fourth-year curriculum have been made more incrementally over the past decade, and are not a focus of this report. While some improvements have been made in our nearly seven years of experience with the M1-M2 curriculum, the basic concept and format are still firmly in place. The curriculum is composed of four courses, each presented in a series of discrete, topically focused units.10 Each course and nearly every unit are interdisciplinary in nature and draw faculty from a number of departments; all are managed and funded centrally by the dean's administration. Two areas of emphasis differentiate the current M1–M2 curriculum from its predecessor. The first is a change in the way we expect students to learn medicine. Our students are now explicitly regarded as adult learners, with a wide variety of backgrounds, aptitudes, and learning styles. Adult education models embrace this diversity and provide a framework for continuous self-directed education beyond the formal curriculum. Moreover, the very nature of the profession demands that students learn to “think on their feet,” relating different areas of knowledge one to another and serving as critics of their own and others' reasoning processes. Accordingly, the curriculum provides a variety of learning formats, with an emphasis on interactive, discussion-based small-group activities. In addition, the clinical skills units include peer observation and feedback on a regular basis.10,11 The second emphasis is a dramatic increase in the attention paid to issues of professional perspectives and professional skills. As detailed by Curry and Makoul,4 attention to students' interpersonal skills and attitudes and to the interface of the medical profession with society at large had grown steadily for some years. Not until the early 1990s, however, did schools begin to address these issues comprehensively. Since then, professionalism has become much more visible on the medical education agenda.3 The conceptual framework of patient-centered medicine (also referred to as relationship-centered medicine), which highly values the physician's capacity for empathy, attentive listening, and concern for the patient's perspective,12 has been instrumental in bringing about these changes. The very breadth and comprehensiveness of significant educational reform make it difficult to reliably evaluate the specific impact of any component. Further, consistent with the focus on adult learning and professional development (i.e., we want our students to mature as self-aware professionals), we consider students' perceptions to be an important element of curriculum evaluation. We used the Student Perception Survey as our program evaluation tool because it offers a broad view of students' attitudes and experiences. For instance, we were interested in assessing, over a period of years, whether the new M1-M2 curriculum affected students' perceptions about the importance of key educational goals, and whether it had an effect on their perceived progress toward those goals. Educational Goals: Importance. There is some concern that medical students become less idealistic and more cynical as they progress through the curriculum.13,14 On the other hand, students are likely to place more emphasis on areas relevant to clinical practice as they approach the clinical clerkship phase of their education. To assess whether students place more or less value on key educational goals after their first two years of medical school, we can compare responses to the Student Perception Surveys administered to incoming students with those to surveys administered to the same students at the end of their second year (just before clinical clerkships begin). Since we expect that incoming students will highly value all of the goals, thus generating a ceiling effect, we do not expect the importance ratings to rise. Neither do we expect them to fall, since the new curriculum attempts to reinforce the value of these goals. Thus, our expectations regarding importance ratings are phrased as our first (null) hypothesis: There will be no statistically significant difference in the importance ratings when Student Perceptions Surveys administered to incoming students are compared with those administered at the end of the second year. Educational Goals: Progress. Attending physicians' comments regarding the readiness and performances of students in their clerkships provide one good indication of whether a new M1-M2 curriculum is effective. However, it is difficult to systematically evaluate progress toward a variety of goals with such a method. Since we have a pass—fail grading system, the only grade-like metric available is the U.S. Medical Licensing Examination (USMLE) Step 1 score, also poorly suited to address a diverse set of goals. The Student Perception Survey allows us to assess students' views about the extent to which the curriculum has helped them progress toward each of the goals listed in Table 1. A brief “In Progress” article published in Academic Medicine reported immediate positive changes in ten of the 16 educational goals when data collected from the class of 1996, which progressed through the first two years before the curriculum was implemented, were compared with data from the classes of 1997 and 1998, the first cohorts to complete the new M1-M2 curriculum.9 Since we expect the revised curriculum to prove effective in maintaining those changes, we offer the second hypothesis: Students who have progressed through the new curriculum will report more progress toward educational goals than will students who completed the survey before the new curriculum was in place.TABLE 1: Responses to Importance of Educational Goals Section of the Student Perception Survey by Incoming and Experienced Students at Northwestern University Medical School, Classes of 1997–2001*Method Student Perception Survey. The survey gathers information about medical students' perceptions regarding faculty contact, educational goals, educational activities, and patient-centered tasks of care. It also gauges learning orientation, social orientation, career plan, conceptions of health, and demographic information. It is administered longitudinally via scan-form or computer: once at the beginning of medical school (i.e., during orientation week) and again at the end of the second year (i.e., just before clerkships). (We ran a study in 1997 to compare pencil-and-paper, scan-form, and computer versions of the survey; no difference in response patterns was detected.) This report includes data collected at both time points from students in the classes of 1996–2001. The survey is usually completed by all students in each cohort; it was distributed to fewer second-year students in 1995 and 1996, and fewer incoming students in 1998, due to administrative errors. Social security numbers serve as identification tags, allowing us to match surveys from incoming and experienced students without accessing their names or creating another set of identification numbers. Educational Goals. In 1990, the dean, with the approval of all department chairs and senior deans, established eight goals for medical school education.10 The 16 goals assessed in the Educational Goals section of the Student Perception Survey (see Table 1) were developed by explicating these original eight (e.g., operationalizing “communication”) and then expanding the list to include four additional goals expressed by faculty who had developed the new curriculum for the first two years of medical school. Table 1 indicates which of the goals were added. Nunnally emphasized that the plan and procedure of an item's generation is a primary determinant of its content validity.15 Drawing the items directly from goals outlined by the medical school certainly enhanced content validity. Further support comes from the observation that these goals are not unique to Northwestern; they are reflected in blueprints for medical education,1,2,3 deemed relevant by the other schools using the Student Perception Survey, and in the expressed values of practicing physicians.16 The items also have representational validity, as pilot tests conducted during the survey-development process indicated that medical students understood these items as intended.17 The Educational Goals section of the survey asks both incoming and experienced students to rate the importance of these 16 goals on a scale ranging from 0 = “not at all important” to 4 = “absolutely essential.” The intervening scale points are labeled 1 = “slightly important,” 2 = “moderately important,” 3 = “very important.” The survey administered at the end of the second year also asks students to indicate the extent to which their medical school experience has helped them progress toward each goal. The scale for measuring progress ranges from 0 = “not at all” to 4 = “completely.” Importance. To test our first (null) hypothesis which posits little change in how students value the various educational goals, we performed paired t-tests on data from surveys administered to incoming and experienced students in the classes of 1997 through 2001, all of whom had been exposed to the new curriculum. Since we assert the null hypothesis, statistical power is an important consideration. Simply stated, the power of a test is the probability of rejecting the null hypothesis when it is indeed false. Given the large sample of matched pairs (n = 511), we chose a fairly conservative α level to avoid highlighting differences of trivial magnitude. At α =.01 (two-tailed), we have statistical power greater than.98 for detecting small to medium effect sizes.18 Progress. To test our second hypothesis, which states that the new curriculum should be associated with greater perceptions of progress toward the educational goals, we performed independent-sample t-tests on data from surveys administered to experienced students (those at the end of their second year). (One-way ANOVAs indicated that data from the classes of 1997 through 2001 could be combined because they were statistically similar. Thus, we ran t-tests to facilitate presentation and interpretation of results.) We compared the perceptions of students in the class of 1996 (n = 165), who had experienced the old curriculum, with those of students in the classes of 1997 through 2001 (n = 603). Again, the large sample size affords good statistical power. At α =.01 (two-tailed), we have statistical power greater than.80 for detecting small to medium effect sizes via these independent-sample t-tests.18 Results Importance. On average, the students rated all of the educational goals from “very important” to “absolutely essential” (see Table 1). When surveys administered at the two time points were matched and importance ratings were compared via paired t-tests, we found statistically significant, though relatively small, differences (Δ) in how the students valued four educational goals. Importance ratings increased for “become more proficient at learning on your own” (Δ =.11, p <.01) and “improve your problem solving skills” (Δ =.14, p <.001); they decreased for “become proficient in clinical decision making” (Δ = −.10, p <.001) and “become more aware of ethical issues in medicine” (Δ = −.13, p <.001). Progress. The students' mean ratings of the extent to which their experiences had helped them accomplish each goal were closer to the scale's mid-point that were the importance ratings (see Table 2). Students completing the new M1-M2 curriculum reported significantly more progress toward ten of the educational goals than did the cohort that progressed through the first two years before the new curriculum was implemented. The biggest changes were associated with “master skills for providing information to patients” (Δ =.50, p <.001), “gain a full appreciation for political, economic, and social influences on health care” (Δ =.42, p <.001), “become more comfortable when being assessed by your peers” (Δ =.35, p <.001), “become more proficient at learning on your own” (Δ =.33, p <.001), and “improve your problem solving skills” (Δ =.32, p <.001). The only decrease was associated with “master physical examination skills (Δ = −.10, ns).TABLE 2: Experienced Students' Perceived Progress toward Educational Goals* While in the Old Curriculum (Class of 1996) Versus the New Curriculum (Classes of 1997–2001), Northwestern University Medical SchoolSince distributions for some of the importance and progress items were not normal, we also ran nonparametric tests (Wilcoxon signed-ranks test for importance, Wilcoxon—Mann—Whitney test for progress). The power efficiencies of these tests are about 95% when compared with their parametric counterparts.19 We obtained exactly the same patterns of statistical significance, reinforcing the notion that parametric tests are robust when it comes to the assumptions of normality.20 Discussion A number of measures and methods (e.g., written tests, clinical skills exams, faculty reports) can provide data for assessment of students and curriculum evaluation. However, such data are relatively particular in nature. Just as clinical outcomes researchers obtain patients' perceptions to complement more objective measures of health,21 medical educators interested in the outcomes of curricular reform have gained important information by measuring students' perceptions in the areas of well-being,22 learning activity,23 learning environment,24 and long-term effects.25 This study's findings indicate the value of gauging students' perceptions regarding a variety of education goals as well. While there were statistically significant differences in importance ratings for 25% of the educational goals, there was no trend in terms of directionality. Thus, our first (null) hypothesis received general support; the value students placed on the educational goals remained relatively stable between orientation week and the end of the second year of the curriculum. As shown in Table 2, our second hypothesis, which focused on progress estimates, received general support as well. Students who had progressed through the new curriculum reported more progress toward ten of the educational goals than did students who completed the survey before the new curriculum was in place. All of the statistically significant differences in progress estimates were larger than any of the differences in importance ratings. This pattern of results was immediate9 and has been sustained over the years. It appears that the Patient, Physician & Society (PPS) course, which extends throughout the first two years, 10 contributes to increases in the students' perceived progress toward their educational goals. More specifically, the PPS course emphasizes providing information to patients, incorporates peer assessment and feedback, and explores the political, economic, and social influences on health care. We were pleased to find that, when compared with the students in the old curriculum, the students who had experienced the current M1-M2 curriculum reported more perceived progress toward the goals of becoming more proficient at learning on their own and developing skills to enhance lifelong learning. We attribute this change to the adult-learner and active-learning approach taken by all four of the M1-M2 courses. However, we did not see a similar gain in the area of identifying strengths and weaknesses in academic and clinical abilities, an important component of lifelong learning and mindful practice.26 The results suggest that we also need to focus our attention on helping students learn to manage time more effectively and understand how the stresses of life as a physician will affect their personal lives, two goals voiced by faculty who developed the new M1-M2 curriculum. Regarding the goal of developing skills for practicing health promotion and disease prevention, we are planning to move to a more clinically oriented PPS unit on health risks, in part because the students reported little increased progress in this area at the end of their second year. Finally, despite a well-received first-year unit on physical examination skills in PPS, we observed a decrease in perceived progress toward this skill set, a consistent and rather troubling finding over the years. We will continue to work toward improving students' confidence and competence in physical exam skills within the PPS course, as the first and second years of medical school offer an opportunity to ensure a consistent approach to teaching and learning basic skills. Our aim is to provide a solid foundation that can be built upon during the clerkships. While it would have been preferable to collect the Student Perception Survey's data for more than one cohort in the old curriculum, the survey could not be implemented until it was designed and tested. Still, the pattern of results is clear and consistent, and changes in progress estimates can be logically linked to changes in the curriculum. Further, results from other schools using the Student Perception Survey reinforce the findings regarding progress. For instance, progress estimates also increased at the University of Utah after a curricular revision. Interestingly, significant progress toward a similar number of goals was evident at both Northwestern and Utah, but the pattern of results (i.e., mix and magnitude of changes) differed. (We will be working with Dr. Neal Whitman and colleagues at Utah to determine the extent to which observed changes reflect the emphases of M1-M2 curricular reform at that institution.) Students' perceived progress toward their educational goals did not increase at schools that did not make substantial changes in their M1-M2 curricula during the period they have used the Student Perception Survey. Taken together, these observations highlight the generalizability and sensitivity of this approach to curriculum evaluation. The Student Perception Survey has proved a very useful tool for gauging the effects of curricular reform and identifying areas in need of more attention. We consider students' perceptions one important component of curriculum evaluation,27 and we will continue to monitor them carefully. At present, we are working to develop a questionnaire for residency program directors and another one for medical school alumni, each of which will draw on aspects of the Student Perception Survey. As noted by Gerrity and Mahaffy,5 this type of outcome data serves the important function of documenting where we have been and helping us better understand where we are going.