The Deanʼs Letter Summary Statement

The dean's letter is a statement of evaluation constituting a critical component of medical students' applications for residency positions. In 1989, the Association of American Medical Colleges (AAMC) proposed guidelines for the dean's letter in an effort to standardize its content. Among the guidelines presented, use of “comparative performance data” was recommended to allow residency admission committees to estimate a candidate's performance in comparison with those of his or her peers. 1 Within the summary section of the dean's letter, many institutions have adopted the use of a summary statement, otherwise known as a final descriptor, key word, or key phrase, as a means of providing this comparative information. 2,3 The use of summary statements varies considerably. Studies have demonstrated that between 53% and 85% of U.S. medical schools report the use of summary statements. 2–5 The different scales vary as well, with “good, very good, excellent, outstanding” being among the more common. Although readers of dean's letters have valued this quantifiable method of comparing large numbers of applicants, the meanings of numerous summary statements are often unclear. 2,4,5 One study showed, nevertheless, that students whose schools provided summary statements or other “comparative” performance data achieved higher rankings from the residency selection committee than did students from “non-comparative” schools. 6 Despite their value, there is no literature examining the methods by which medical schools create such ranking scales. Prior to 1995, Mount Sinai School of Medicine (MSSM) assigned students' summary statements based solely on third-year academic performances. The algorithm was modified in order to reflect more accurately those attributes and skills valued by the school: involvement in community service, demonstration of leadership, and publication of abstracts and papers. This study evaluated the impact of the new algorithm on the summary statement in the dean's letter. Methods The current algorithm. The sample for this study was all graduates from MSSM in New York City over a recent three-year period (1999–2001). The current algorithm for the dean's letter summary statement, in effect since 1995, was based on a point system incorporating five categories of performance. Possible scores ranged from 0 to 10 points, divided as follows: clinical clerkship performance (0–5 points), U.S. Medical Lincensing Examination (USMLE) Step 1 score (0–1 point), authorship of published research (0–2 points), and leadership and community service (0–2 points). Corresponding summary statements were good (total score = 0–1/2 point), very good (1–2 points), excellent (21/2–41/2 points), and outstanding (≥5 points). For each of the five major third-year clinical clerkships, students achieved 1 point for a grade of honors and 1/2 point for “just short of honors.” Students earned 1 point for a USMLE Step 1 score ≥230. In response to rising scores, this value was increased to 235 in 2001. For authorship, points were distributed as follows: first author on a published paper was 1 point, other than first author on a paper was 1/2 point, or first author on an abstract was 1/2 point, with a maximum total of 2 points. For school leadership and community service, a committee of deans and faculty discussed each student individually and credited him or her with up to 2 points. The student was required to demonstrate more than mere involvement in extracurricular activities in order to receive leadership points. However, a few students who played major participatory roles in numerous activities, as confirmed by telephone correspondence with activity advisors, received such credit. The old algorithm. Based on grades in the five clinical clerkships and membership in the Alpha Omega Alpha (AOA) honor society, a student was assigned a summary statement of good with 0 honors, very good with 1–2 honors, excellent with ≥3 honors, and outstanding with ≥3 honors plus membership in AOA. Calculation of summary statements. Summary statements were determined by the deans, using the current algorithm, at the time of preparation of the letters. These data were abstracted and given in a blinded fashion to one of the authors (SK), who applied the old algorithm to the data. Theoretical summary statements for all students were determined. The paired data were analyzed by chi-square to determine significance of differences between the current and old algorithms. Demographic analysis. Students' genders and modes of medical school entry were abstracted for analysis of an effect of the adjusted algorithm on certain subsets of the sample population. Students enrolled at MSSM via five routes: (1) regular entry; (2) Humanities and Medicine Program (HAM), an early-assurance-of-admission program in which students are accepted to MSSM in the college sophomore year and pursue humanities or social science majors; 7 (3) MD—PhD graduate degree program; (4) City College of New York (CCNY) Biomedical Program, a five-year combined undergraduate and medical program from which students enter directly into the third-year class; and (5) underrepresented minorities (URMs). The regular entry and URM entry programs are mutually exclusive. There were three students, however, who overlapped categories: one MD—PhD student who was also a URM and two HAM students who were also URMs. Such minimal numbers do not have an impact on the data. The genders and entry types of students whose summary statements differed with use of the two algorithms were compared with those of students whose summary statements remained unchanged. The proportions of students in each entry type whose summary statements “changed” were compared with the regular-entry MSSM group. Statistical significance was determined by chi-square analysis. Results The total sample consisted of 345 graduating seniors from MSSM between 1999 and 2001. The resulting distribution of summary statements based on the old algorithm revealed that the majority of students were good or very good (31% and 36% respectively), as shown in Figure 1. Only one third of the sample constituted both the excellent and outstanding categories, 16% in each. The current algorithm, in comparison, produced a more bell-shaped distribution, with an overall shift toward the higher summary statements: 17% good, 26% very good, 34% excellent, and 24% outstanding. The difference between the two algorithms was found to be statistically significant, p < .0001.Figure 1: Distributions of summary statements of 345 medical students at Mount Sinai School of Medicine based on old and current algorithms, 1999–2001. The authors compared the institution's current and old algorithms for assigning summary statements in the dean's letter. Summary statements via the current algorithm were determined for all graduating students at the time of preparation of the dean's letters. Students records were abstracted in a blinded fashion, and the old algorithm was applied to the same subset of students.The distribution shift is the result of “change” in summary statements for 149 students, or 43% of the sample. We refer to “change” in summary statements in a theoretical sense, defining those students whose current summary statements differed from the hypothetical summary statements assigned by the old algorithm. In all cases, the change was from a lower-value summary statement to a higher one. Most were one-step improvements, including 41 students whose summary statements changed from good to very good, 73 students who went from very good to excellent, and 23 who changed from excellent to outstanding. A few students achieved two-step improvements: eight changed from good to excellent, while four changed from very good to outstanding. The institution's algorithm revision did not affect students differently by gender. Of the 175 male and 170 female students in the sample, 75 (43%) and 74 (44%), respectively, received summary statements that differed between the two algorithms. Table 1 shows the proportions of students in the entry programs whose summary statements changed. Compared with the regular-entry MSSM students (42% changed), the proportions of URM and CCNY biomedical students were significantly higher and lower, respectively (57%, p < .05; 25%, p < .05). The proportions of HAM students and MD—PhD students whose summary statements changed were not significantly different from that of MSSM students.TABLE 1: Proportions of Medical Students at Mount Sinai School of Medicine Whose Summary Statements “Changed” Based on Entry Program, 1999–2001*Of the students whose descriptors changed, 39% were accounted for by the inclusion of their research activities, 23% for community service, and 46% for leadership activity. Some individuals overlapped in these categories. Discussion This study compared two methods of assigning summary statements for dean's letters for MSSM students. The old algorithm, based solely on academic performance, was modified by the institution to include an evaluation of student leadership, community service, and research activity. The new algorithm assigned summary statements in a manner more consistent with the values of the institution and the expectations of its students. Minority students, in particular, benefited from this algorithm modification, while students in the CCNY Biomedical Program did not. The current algorithm produced a distribution shift to the right, revealing the extent to which the old algorithm undervalued our students. Two thirds of the student body were described as good or very good, which, in an era of medical education when “good” often means “the worst student,” was highly discrediting. The current distribution more accurately reflects the valued attributes of leadership, dedication to one's community, and motivation toward scientific research. Nevertheless, we must continue to evaluate the implications of this distribution shift. Although adjustments of cut-off values may be considered in the future, the innovation of the current algorithm is an improvement in the method by which we rank our students. As Dr. Cohen insisted in his President's Address at the 108th Annual Meeting of the Association of American Medical Colleges, medical educators must demand “more than academic achievement and scientific aptitude.” He challenged the medical profession to, once certain scholarship requirements are demonstrated, “[make] selections on the basis of character traits and aptitude for serving others… by documented evidence of prior public service, of leadership, of experiences with volunteerism, of a penchant for self-sacrifice in the interest of others.” 8 Two subsets of the sample population, URM and CCNY biomedical students, were significantly affected by the algorithm modification. The current algorithm significantly improved the summary statements of URMs, perhaps reflecting a refinement of a system that traditionally placed minority students at a disadvantage. Studies show that URMs tend to score lower on standardized examinations. In a cohort study of 14,725 medical students, initial mean USMLE Step 1 scores and pass rates were lower for African Americans and Hispanics than for other ethnic groups. 9 CCNY students, on the other hand, undergo an accelerated training process that may compress the time available to them to become involved in community service or research. Entering MSSM during the third year of medical school may also restrict their likelihood of assuming leadership roles, as many students initiate involvement in extracurricular activities during the first and second years. These limitations may account for the smaller number of CCNY students who benefited from the new algorithm. A limitation of the study is that the data may appear to have inflated students' summary statements. There should be caution, however, in forming this conclusion, as no nationally accepted algorithm or distribution yet exists. The old algorithm, in fact, undervalued our students, and perhaps the bell-shaped curve of the new algorithm is more representative of a normal population distribution. In an institution-based survey of residency directors in 1999 and 2000, those directors rated 84% of “outstanding” students, 71% of “excellent” students, 60% of “very good” students, and 32% of “good” students as superior, compared with the summary statements of their MSSM dean. This trend supports the predictive value of the summary statements, although further research is needed. One might also consider the limitation of assigning numeric values to qualitative measures. Upon instituting our current algorithm, we attempted to minimize potential bias by involving a committee of faculty and deans; particularly individuals familiar with the students, and telephoning relevant sources to clarify students' involvement. Without efforts to provide comparative data on such qualitative attributes, we risk reverting to numbers and test scores as sole selection criteria. In line with other national efforts to improve the medical school dean's letter, the summary statement is an important item for further investigation because it is a valuable source of comparative data with unmet potential. Medical educators have already recommended widespread use of clustering systems that categorize students into performance groups and include the percentage of students in each group as a method superior to traditional class ranking. 4,5 Such a system, as our current algorithm demonstrates, provides the comparative data that residency admission committees demand, while valuing our students' scholarship, leadership, and dedication to community, as well as academic achievement. Future study might investigate alternative algorithms used by other institutions and the implications of a nationwide standard algorithm for assigning summary statements.