Using Basic Science Subject Tests to Identify Students at Risk for Failing Step 1

Basic science Subject Tests1 are frequently used by medical schools to assess students' performances. More than 30,000 exams are administered annually, most commonly as final exams for courses, but also as a way of monitoring student learning and providing students with experience in taking step-type examinations. In particular, schools would like to be able to use Subject Test scores to identify students at risk for failing U.S. Medical Licensing Examination (USMLE) Step 1. With the introduction of single-day, computer-based tests for USMLE Step 1 in 1999, it is important to reexamine the predictive validity of Subject Tests in basic science for identification of students at risk for poor performance. This research addresses the question of how well Subject Test scores perform as predictors of USMLE Step 1 performance. Method Subjects. Scores for examinees taking selected first-year Subject Tests (gross anatomy/embryology, histology/cell biology, biochemistry, and physiology) for the first time between November 1997 and June 1999 were retrieved from the National Board of Medical Examiners (NBME) Subject Test database. Similarly, scores on second-year Subject Tests [pathology, microbiology/immunology, introduction to clinical diagnosis (ICD), pharmacology, and the comprehensive basic sciences exam (CBSE)] taken between September 1998 and May 2000 were retrieved. To ensure that Subject Tests were taken under comparable conditions at all schools, the sample for each exam (except CBSE) was refined to include only full-class, end-of-course test administrations at Liason Committee on Medical Education-(LCME-) accredited schools. Scores from these administrations were matched with performances from the examinees' first attempts on the computer-based Step 1. Table 1 provides sample sizes and descriptive statistical information.TABLE 1: Descriptive Statistics and Fixed-effects Coefficients from HLM ModelsInstrumentation. Each Subject Test is constructed according to content specifications developed by experts in the subject matter for an end-of-course test.2 Scores on each test form are scaled to a mean of 70 and a standard deviation of 8 in a 1992–93 criterion group; the score reliability of a typical test form ranges from .80 to .85. The USMLE Step 1 is designed to determine whether examinees understand and can apply important biomedical concepts.3 Total scores have a reliability of .92; they are scaled to a mean of 200 and a standard deviation of 20 in a 1991 criterion group of first-time examinees from LCME-accredited schools. As indicated in Table 1, the USMLE Step 1 means and SDs for Subject Test samples are similar to the national mean for examinees from LCME-accredited schools taking the USMLE Step 1 for the first time during 2000. Total group correlations between Subject Test scores and USMLE Step 1 range from .59 to .71. Procedure. For each Subject Test, a series of two-level hierarchical linear models (HLMs)4 were run with USMLE Step 1 scores as the dependent variable. Individual students (nested in schools) are represented at level 1, while schools are represented at level 2. A two-level random-effects analysis of variance (ANOVA) model was run to partition variation in USMLE Step 1 scores into within-and between-school sources. Next, random-coefficients models were developed with recentered Subject Test scores as predictors. The grand mean was subtracted from each Subject Test score before entering it into the prediction equation. This results in a scaling of the level-1 regression intercept for each school that is equal to the adjusted Step 1 mean (controlling for Subject Test scores), much as in an analysis of covariance. Grand-mean centering was also used for level-2 predictors; this preserves the level-1 scaling of the intercepts. This produces a regression of USMLE Step 1 scores onto Subject Test scores for each school, with regression intercepts and slopes varying across schools. Variance components from this model are compared with those from the ANOVA model to quantify the variance in USMLE Step 1 scores predicted by Subject Test scores. This analysis also makes available empirical Bayes estimates of intercepts and slopes for each school. Next, means-as-outcomes models were explored by entering level-2 variables into the model, looking for school/course characteristics that predict school-to-school variation in intercepts. Finally, a hierarchical logistic regression model was estimated to predict probability of passing USMLE Step 1 as a function of Subject Test score. Results Results are presented in some depth for the Pathology Subject Test (because of the large sample size) to illustrate the analytic procedure used and the interpretation of results. Results for other second-year Subject Tests are presented next, followed by results for first-year Subject Tests. Results for the Pathology Subject Test. For pathology, the ANOVA model partitioned the total variance in USMLE Step 1 scores into between- and within-school variance components of 45.5 and 444.0, indicating that 9% of the variance was between schools [45.5/(45.5 + 444)] and 91% was within schools. The square roots of the variance components (6.75 and 21.07) are interpretable as the estimated SD for USMLE Step 1 school means and the pooled within-school SD, respectively. Using the Pathology Subject Test score as a level-1 predictor resulted in the (fixed-effects) equation shown in Table 1: each one-point increase in Subject Test score corresponds to an increase of 2.15 points in the predicted Step 1 score. Between- and within-school variance components for this model were 55.5 and 188.5. Use of the Pathology Subject Test score as a predictor therefore increased the variability in school means: (adjusted) USMLE Step 1 school means became more disparate after controlling for pathology scores. Approximately 58% [(444.0–188.5)/444.0] of within-school USMLE Step 1 variance was explained by pathology scores. The SD of the intercepts was 6.74: schools varied substantially in adjusted USMLE Step 1 school means. The SD of the pathology slopes was 0.13, indicating that there was some school-to-school variability in the strength of the relationship between pathology and USMLE Step 1 scores. In means-as-outcomes analyses, 21% of the variation in intercepts (adjusted school means) was predicted using the timing of the Pathology Subject Test, that is, the time of year when the course ends. Intercepts are larger if the Pathology Subject Test was taken earlier. Timing presumably serves as a proxy variable for underlying differences in curriculum structure and exposure to pathology-related content. Schools that have a pathology course in the fall generally have additional pathology-related experiences that occur in the spring of the second year. In contrast, students attending schools with a spring pathology exam are likely to be at the end of formal pathology-related instruction prior to administration of USMLE Step 1. Thus, it makes sense that a given score on the Pathology Subject Test should be associated with a higher predicted USMLE Step 1 score if the exam is taken in the fall. Each one-month delay in the timing of Pathology Subject Test administration has an associated 2.13-point decrease in USMLE Step 1 score. Thus, for November-versus-May administrations of the Pathology Subject Test, the predicted USMLE Step 1 scores differ by almost 13 points. Figure 1 shows within-school probabilities of passing USMLE Step 1 by Pathology Subject Test score from the random-effects model, grouped by the month in which pathology is taught. Only one school administered the Pathology Subject Test in November. Students achieveing even moderate scores on the Pathology Subject Test this early in the year have a high probability of passing USMLE Step 1 in the following spring. A student who scores a 64 on the Pathology Subject Test (about ten points below the mean) in November has an 81% predicted probability of passing USMLE Step 1. Most schools, however, administer the Pathology Subject Test six months later in May, when a score of 64 is associated with USMLE Step 1 pass rates ranging between 35% and 94%.Figure 1: Probability of passing Step 1 as a function of Pathology Subject Test performance and timing at 41 schools. Based on 8,690 scores from first takers of Pathology Subject Tests administered between September 1998 and May 2000, matched with corresponding first attempts on Step 1. The graph shows predicted probability of passing calculated from empirical Bayes estimates of within-school regression functions from a hierarchical logit model with passing Step 1 as the outcome variable. Each curve represents one school; within-school functions are grouped by timing of the pathology exam. No level-2 covariate is included in this model.Results for other second-year subject tests. Fixed-effect regression results were similar for other second-year Subject Tests. However, there were substantial differences in how well Subject Test scores predicted variations in USMLE Step 1 performances. For microbiology, ANOVA results indicated between- and within-school variance components of 62.1 (12%) and 439.2 (88%). Between- and within-school variance components for the random-coefficients model were 154 and 217; thus, use of the Subject Test score as a predictor increased the variability in school means. The model explained 50% of the within-school variance in USMLE Step 1 scores. For ICD, the between- and within-school variance components from the ANOVA model were 82.7 (16%) and 435.6 (84%). Between- and within-school variance components for the random-coefficients model were 55.7 and 241.6, indicating the model explained 32.7% and 44.5% of the between- and within-school variance in USMLE Step 1 scores. The pharmacology ANOVA model yielded estimated between- and within-school variance components of 38 (8%) and 444.8 (92%). The corresponding variance components for the random-coefficients model were 53.2 and 201. Use of the Subject Test score as a predictor explained 55% of the within-school variance in USMLE Step 1 scores but increased the variability in school means. The ANOVA model for CBSE produced between- and within-school variance components of 65 and 426.5, indicating that 13% of the variance was between schools. The corresponding variance components from the random-coefficients model were 40.2 and 191.7; the model explained 38% and 55% of the between- and within-school variance in Step 1 scores. As with pathology, the timing of CBSE administrations had a significant effect on school intercepts at level 2: Later administrations were associated with lower school means on USMLE Step 1 for a given CBSE score. Timing explained about 12% of the variance in school intercepts. No significant level-2 predictors were found for the other Subject Tests. Results of hierarchical logistic regression models for these second-year Subject Tests were all similar to those for pathology, although the slopes of the resulting risk functions were somewhat flatter than those shown in Figure 1. HLM results for first-year subject test scores. For biochemistry, the ANOVA model partitioned the total variance in USMLE Step 1 scores into between- and within-school variance components of 58.3 and 420.1, indicating that 12% of the variance was between schools and 88% was within schools. Each one-point increase in Subject Test score corresponded to an increase of 1.94 points in the predicted Step 1 score. Between- and within-school variance components for the random-coefficients model were 49.0 and 248.0: the model explained 16% and 41% of the between- and within-school variance in USMLE Step 1 scores. For physiology, the between- and within-school variance components were 64 and 420, indicating that 13% of the variance was between schools and 87% was within schools. The random-coefficients model produced within-school variance components of 61.6 and 233.5; thus, the model explained 3.6% and 44.4% of the between- and within-school variance in USMLE Step 1 scores. For gross anatomy, the level-1 and level-2 variance components were 429.8 and 106.3, indicating that 80% of variance was within schools. The random-coefficients model explained 36% of student-level variation and 13% of school-level variation. For histology/cell biology, ANOVA results indicated that 17% (88.5) of the variance in USMLE Step 1 scores was between schools, while 83% (425.1) was within schools. The random-coefficients model explained 40% of student-level variation but increased the between-school variation. As with the second-year exams, the first-year exam models showed substantial variation across schools in intercepts and slopes. Hierarchical logistic regression models for the first-year exams produced similar results to those for second-year exams, although relationships were weaker. Discussion NBME Subject Test scores are good predictors of performance on the computer-based Step 1, explaining roughly half of the individual-level variation in USMLE Step 1 scores. This is consistent with the results of prior work on the two-day paper-and-pencil Step 1.5 Scores on second-year Subject Tests predict between 45% and 58% of the variance in USMLE Step 1 scores, while scores on first-year Subject Tests predict slightly less (41% to 44%), probably reflecting greater time lag between administration of first-year subject exams and USMLE Step 1. The fixed-effects regression models produced in the HLM analyses are similar across exams: A one-point increase in Subject Test score is associated with roughly a two-point increase in expected USMLE Step 1 score. Across schools, students with low Subject Test scores are at increased risk for poor performance on USMLE Step 1. However, there is substantial school-to-school variation in these relationships. A number of variables have been used in past work to predict students' achievement on the USMLE Step 1, including curriculum type,6–8 stakes,7 and preparation style.9 These factors might be expected to change the nature of the relationship between Subject Tests and USMLE Step 1 across schools. Course timing was found to explain a modest proportion of variation in school regressions for some Subject Tests. This is consistent with prior work showing that the timing of clerkship rotations influences how subject exams predict USMLE Step 2 outcomes.10 Additional school-level explanatory variables are needed to account for the remaining differences among schools. Future work should focus on curriculum content, format, and instructional style as school-level predictors. School-specific prediction equations should be used to identify students at increased risk for failing the USMLE Step 1.