Assessing Students for Medical School Admissions: Is It Time for a New Approach?*

Our methods for admitting students, whether to medical school or even to undergraduate and graduate school, are rather archaic. We use tests that we know are only modestly to moderately predictive of success in medical school and that measure content that covers only a small fraction of the skills necessary for success as a medical professional. Might there be more effective ways of devising devices to help us make decisions in admissions situations? List 1 illustrates some potential bases for essays that might be used in medical school admissions. These items have not yet been tried out, but readers who wish to experiment with them are welcome to do so. Why do we believe that items such as these will provide incremental validity over traditional measures in predicting success in medical school and later, on the job?List 1 Sample Questions for Medical School AdmissionsIn this article, I first describe the theory of successful intelligence, which has served as a basis for a number of our admissions assessments, and then I present some studies that have tested the theory in practice. The Theory of Successful Intelligence The theory of successful intelligence suggests that students’ failures to achieve at a level that matches their potential often results from teaching and assessments that are narrow in conceptualization and rigid in implementation.1,2 The ways in which teachers teach do not always match the ways in which students learn. The traditional methods, in essence, typically shine a spotlight on a small number of students with certain ability-based styles, and they almost never focus on a large number of students who have the ability to succeed but whose ability-based styles do not correspond to the patterns of learning and thinking valued by the schools. To rectify this situation, one must value other ability-based styles and then change teaching and assessment so that these other ability patterns can lead to success in school. According to the proposed theory, successful intelligence is (1) the use of an integrated set of abilities needed to attain success in life, however an individual defines it, within his or her sociocultural context. People are successfully intelligent by virtue of (2) recognizing their strengths and making the most of them while at the same time recognizing their weaknesses and finding ways to correct or compensate for them. Successfully intelligent people (3) adapt to, shape, and select environments through (4) a balanced use of their analytical, creative, and practical abilities.1,3,4 Underlying all three abilities is the role of knowledge, because one cannot think analytically, creatively, or practically if one does not have knowledge stored in long-term memory to which one can apply one’s thinking. People typically balance the three kinds of abilities. They need creative abilities to generate ideas, analytical abilities to determine whether they are good ideas, and practical abilities to implement the ideas and to convince others of the value of those ideas. Most people who are successfully intelligent are not equal in these three abilities, but they find ways of making the three abilities work together harmoniously and advantageously. There are other theories that might serve as a basis for constructing new instruments. For example, Gardner5 has proposed a well-known theory of multiple intelligences, which posits that people can learn in different ways. He has applied this theory to instruction and assessment.6 Studies Testing the Theory of Successful Intelligence for Use in Admissions Assessments My collaborators and I have done several studies suggesting the efficacy of the theory of successful intelligence as a basis for admissions decisions. The Rainbow Project The Rainbow Project and related collaborations are fully described elsewhere by Sternberg and the Rainbow Project Collaborators.7–9 The Rainbow measures supplement the SAT. The SAT is a three-hour examination currently measuring verbal comprehension and mathematical thinking skills, with a writing component to be added in the near future. A wide variety of studies have shown the utility of the SAT as a predictor of college success, especially as measured by GPA (grade-point averages [GPAs]). Available data suggest reasonable predictive validity for the SAT in predicting college performance.10,11 Indeed, traditional intelligence or aptitude tests have been shown to predict performance across a wide variety of settings. But, as is always the case for a single test or type of test, there is room for improvement. The theory of successful intelligence1,3 provides one basis for improving prediction and, possibly, for establishing greater equity and diversity. It suggests, as noted above, that broadening the range of skills tested to go beyond analytical skills to include practical and creative skills as well might significantly enhance the prediction of college performance beyond current levels. Thus, the theory does not suggest replacing but, rather, augmenting the SAT, the MCAT, or similar measures in the university admissions process. A collaborative team of investigators sought to study how successful such an augmentation could be. In the Rainbow Project,7 data were collected at 15 schools across the United States, including eight 4-year colleges, five community colleges, and two high schools. The participants received either course credit or money. They were 1,013 students predominantly in their first year of college or their final year of high school. In this report, analyses only for college students are discussed because they were the only ones for whom the authors had data available regarding college performance. The final number of participants included in these analyses was 793. Baseline measures of standardized test scores and high school GPA were collected to evaluate the predictive validity of current tools used for college admission criteria and to provide a contrast for the current measures. Students’ scores on standardized college entrance exams were obtained from the College Board. Measuring analytical skills. The measure of analytical skills was provided by the SAT plus analytical items of the Sternberg Triarchic Abilities Test (STAT).12 Items assessed learning meanings of words from context, number series, and figural matrices. Measuring creative skills. Creative skills were measured by STAT multiple-choice items and by performance-based items. The multiple-choice items were novel analogies (involving counterfactual presuppositions), novel number operations, and figural series with mapping of terms into new domains. Creative skills also were measured using open-ended measures. One measure required writing two short stories with a selection from among unusual titles, such as “The octopus’s sneakers,” one required orally telling two stories based on choices of picture collages, and the third required captioning cartoons from among various options. Open-ended, performance-based answers were rated by trained raters for novelty, quality, and task-appropriateness. Multiple judges were used for each task, and satisfactory reliability was achieved.7 Measuring practical skills. Multiple-choice measures of practical skills were obtained from the STAT. The multiple-choice items were practical problem solving, practical math, and route planning using maps. Practical skills also were assessed using three situational-judgment inventories: the Everyday Situational Judgment Inventory (Movies), the Common Sense Questionnaire, and the College Life Questionnaire, each of which taps different types of tacit knowledge. The general format of tacit-knowledge inventories has been described by Sternberg et al,13 so only the content of the inventories used in this study is described here. The movies present everyday situations that confront college students, such as asking for a letter of recommendation from a professor who shows through nonverbal cues that he does not recognize the student very well. One then has to rate various options for how well they would work in response to each situation. The Common Sense Questionnaire provides everyday business problems, such as being assigned to work with a coworker whom one cannot stand. The College Life Questionnaire provides everyday college situations for which a solution is required. Unlike the creativity performance tasks, in the practical performance tasks the participants were not given a choice of situations to rate. For each task, participants were told that there was no “right” answer and that the options described in each situation represented variations on how different people approach different situations. An example of a creative item might be to write a story using the title “3,516” or “It’s moving backward.” Another example might show a collage of pictures in which people are engaged in a wide variety of activities, helping other people. One would then orally tell a story that takes off from the collage. An example of a practical item might show a movie in which a student has just received a poor grade on a test. His roommate had a health crisis the night before, and he had been up all night helping him. His professor hands him back the test paper, with a disappointed look on her face, and suggests to the student that he study harder next time. The movie then stops. The student then has to describe how he would handle the situation. Or, the student might receive a written problem describing a conflict with another individual with whom she is working on a group project. The project is getting mired down in the interpersonal conflict. The student has to indicate how she would resolve the situation to get the project done. Administrative details. All materials were administered in one of two formats. A total of 325 of the college students took the test in paper-and-pencil format; 468 students took the test on the computer via the World Wide Web. Participants were tested either individually or in small groups. During the oral stories section, participants who were tested in groups either wore headphones or were directed into a separate room so as not to disturb the other participants during the story dictation. Basic data. When examining college students alone, this sample showed a slightly higher mean level of SAT than that found in colleges across the country. The sample means on the SATs were, for two-year college students, 490 verbal and 508 math, and, for four-year college students, 555 verbal and 575 math. These means, although slightly higher than typical, were within the range of average college students. There is always a potential concern about restriction of range in scores using the SAT when considering students from a select sample of universities, especially when the means run a bit high. Restriction of range means that one tests a narrower range of student skill levels than that which is representative of the entire population that actually takes the SAT. However, the sample was taken from a wide range in selectivity of institutions, from community colleges to highly selective four-year institutions. In fact, statistics assessing range showed that the sample ranged somewhat more widely than is typical for the test. Because there was no restriction of range, there was no need to correct for it. Another potential concern is pooling data from different institutions. We pooled data because in some institutions we simply did not have large enough numbers of cases for the data to be meaningful. Factor structure of the Rainbow measures. Some scholars believe that there is only one set of skills that is highly relevant to school performance, what is sometimes called “general ability,” or g.14 These scholars believe that tests may seem to measure different skills but, when statistically analyzed, show themselves merely to measure the single general ability. Does the test actually measure distinct analytical, creative, and practical skill groupings? Factor analysis addresses this question. Three meaningful factors were extracted from the data. One factor represented practical performance tests. A second, weaker factor represented the creative performance tests. A third factor represented the multiple-choice tests (including analytical, creative, and practical). Thus, method variance proved to be very important. The results show the importance of measuring ability-based styles using multiple formats, precisely because method is so important in determining factorial structure. Predicting college GPA. College admissions offices are not interested, exactly, in whether these tests predict college success. Rather, they are interested in the extent to which these tests predict college success beyond those measures currently in use, such as the SAT and high school GPA. To test the incremental validity provided by Rainbow measures above and beyond the SAT in predicting GPA, a series of statistical analyses (called hierarchical regressions) was conducted that included the items analyzed in the analytical, creative, and practical assessments. If one looks at the simple correlations, the SAT-Verbal (SAT-V), SAT-Math (SAT-M), high school GPA, and Rainbow measures all predict freshman year GPA. But how do the Rainbow measures fare on incremental validity? In one set of analyses, the SAT-V, SAT-M, and high school GPA were included in the first step of the prediction equation because these are the standard measures used today to predict college performance. Only high school GPA contributed uniquely to prediction of college GPA. In Step Two, the analytic subtest of the STAT was added because this test is closest conceptually to the SAT tests. The analytical subtest of the STAT slightly but significantly increased the level of prediction. In Step Three, the measures of practical ability were added, resulting in a small increase in prediction. The inclusion of the creative measures in the final step of this prediction equation indicates that by supplementing the SAT and high school GPA with measures of analytical, practical, and creative abilities, a total of 24.8% of the variance in GPA can be accounted for. Inclusion of the Rainbow measures in Steps Two, Three, and Four represents an increase of about 9.2% to in the variance accounted for over and above the typical of college GPA. the Rainbow measures high school GPA, using only SAT scores as a represents an increase in variance accounted for of about to at in another this means that the Rainbow measures prediction the SAT ability-based styles of make a in predicting beyond measures of traditional general ability. These results suggest that the Rainbow tests to the prediction provided by SATs They also suggest the of high school GPA, in because is an that within including and one important of the present study was to predict success in another important measures that group in mean levels. There has been a as to there are group and whether scores for of groups are or by SATs and related Might be because different on show different ability-based styles of thinking as a of There are a number of ways one can test for group in these each of which a test of the of the of different measures were numbers the of group on test scores (called considering in mean performance levels among the and groups including and for the the measures and the STAT ability the creativity performance tasks, and the performance The indicates the of variance in the that is accounted for by the of the The were for SAT-V, for SAT-M, and for SAT. For the Rainbow ranged from to with a of Thus, the Rainbow measures showed to the SAT. Another test of one to more a of group These results indicate two general in terms of the Rainbow tests seem to have group to traditional assessments of abilities the SAT. in terms of that the students the most from the of group students, showed a in from the mean for most of the Rainbow although a to have been with the practical performance measures. in were for the students to students. Indeed, their was higher for the creative tests. However, the very small sample suggests that about performance be the group were not these suggest that measures can be that and group on standardized for groups and students. These have important for in college admissions. They suggest that different groups do on different patterns of ability-based from The the Rainbow Project apply at other levels of admissions as well. For example, et have shown that the same can be applied in admissions to business also with the of prediction and group well as by including tests of practical thinking in to the et of ability-based styles in the of In this by the Testing and the College they whether the same could be applied to used for college admissions and They tests in and statistics to analytical, creative, and practical skills. is an example in A variety of have been proposed to for people the theory of An theory is an theory of sometimes to as the and this theory and and contrast with the what as the two and two of this theory to the theory might an experiment to test the theory of describe the including the and A that she is on knowledge of what kinds of might her to help her at night in the other the investigators found that by asking such they were to increase the range of skills they tested and to group in test different groups seem to show different patterns of ability-based In with our from a we a of admissions assessments, in to into students’ School Test this school to students’ creative and practical we two assessments of practical and two assessments of creative One students’ to adapt to the new of a school and the new and of a highly and school. In this task, students were to rate a number of to them they a of a practical situation. The included more situations of of student In this a situation was and participants were to the story by with the and the next step in the Creative was also assessed with two different One for a story one of five proposed (1) (2) “The on the (3) the (4) “The was not and the The included different describing various situations related to novel of students were to find a solution using some knowledge of the These were used in analyses predicting freshman GPA for a of students. When into and the tasks the prediction to and the tasks added an to Thus, tests such as the Rainbow do not only of groups. There are students who from the and even from who learn in ways that are different from those assessed by standardized tests. These may well have the abilities they need to succeed in and even in school, but these abilities may not be in scores on tests. Our tests help such students. It is one to have a successful and another actually to implement the in a situation. of these ideas actually make a in Practical The Project has the role of in a to into some of the ideas from the Rainbow Project which represents an of the ideas of Rainbow but beyond that project to include in assessment the of to intelligence, and creativity on the for all of the more than students for undergraduate admissions to and at The were the Rainbow Project was done as a test administered with a the Project was done as a of the of the college It just was not practical to a test such as the Rainbow assessment for admission to one the of is that from the situation in which students must answer in very short of time The was this and students were to answer just a single question. a creative students to write stories with such as “The of or of a Another creative students what the would be if some had for example, if had given up her on the another creative a nonverbal students an to a new or an for a new A practical how students had of an they A students how a they had could be applied a what Some were that numbers of would go they the of There were students in what had been the third of the in terms of of those students, the new not to to more were that average SAT scores would go down and, even they to more than for the first time. The is that the new assessments are not with Rather, they just are not at one or the The of the assessments with SATs were all than In with of were in the the essays had a on But students who had an on the assessments were as to be as those who did The assessments provided a of assessing ability-based styles of thinking in the had been assessed only in a more one year of students who were on did just as well as did students with high school who were for other In these new methods results in the admission of who are more but in a than was in the most were the from large numbers of that they our them a to show themselves for who they a number of in which by were in terms of this year they up In the more students than the year before, and more these those of the Rainbow showed that is to increase and and to do so for an entire undergraduate at a not just for small of students at some Most the university a to students, high school and others that that there is a more to a than the narrow of skills assessed by standardized and that these skills can be assessed in a One might how one answers to that seem so The answer is through For example, we analytical on the basis of the extent to which they are (1) (2) (3) and (4) We creative on the basis of how (1) and (2) they as well as on the basis of their (3) to the with which the students were We practical on the basis of how they are with to (1) (2) and (3) and (4) We on the extent to which they (1) a good by (2) one’s with and (3) over the and short through (4) the of I have described in this how the theory of successful intelligence has been used in a variety of to increase predictive validity and to group of these are medical and so to be whether the theory has to that the basis of the of in which we have successful prediction and of group we are at moderately that the same kinds of results could be obtained for admission to medical schools.