Assessing Post-encounter Note Documentation by Examinees in a Field Test of a Nationally Administered Standardized Patient Test

The large-scale standardized patient (SP) test in this study assessed the clinical skills of fourth-year medical students in a series of clinical encounters targeting history taking, physical examination, communication, and interpersonal skills. Yearly large-scale field tests have been undertaken over the past seven years in preparation for national administration. The study reported here was conducted in 1998. Students are oriented to the test prior to completing up to 12 15-minute SP encounters (cases). Following each encounter, the SP records history elicited, counseling provided, or physical examination performed using an objective checklist developed by expert clinicians. The checklists may be thought of as a process measure, serving as a reflection of actual behaviors demonstrated by the candidate. Interpersonal skills are assessed using the Patient Perception Questionnaire (PPQ), a six-item instrument with a five-point Likert rating scale (uniform for every case). Following each encounter, students are given seven minutes to write a free-response Post-Encounter Note (PEN) (either a list of significant positive and negative history and physical findings or a written chart note documenting findings and counseling). The PEN is specifically tailored to reflect each case. There is no limit to the number of findings students may write. Patient management (diagnosis or therapeutic plans) and interpretation of diagnostic tests are not assessed in these PENs. The PENs potentially reflect a candidate's ability to determine the most significant findings elicited from the encounter and to accurately record them. While numerous studies have examined the use of checklists with respect to fairness, security, and accuracy, there is limited research investigating the psychometric properties of PENs. Previous studies have examined appropriate methods for scoring the PEN. Soliciting global judgments from experts seems appealing because scores are derived from the expertise of practicing physicians,1 but global ratings can be unreliable unless the scoring task is highly structured and extensive standardized training is provided.2,3 From a national testing perspective, recruiting physicians to score the PENs for thousands of candidates may not be feasible. As a result, many researchers have favored the use of analytic keys to score PENs.2,4 A significant advantage of using such scoring keys is the fact that non-physicians can be trained to score the PENs with an accuracy level comparable to that of physicians.5,6,7 Research examining the usefulness of PENs with an SP test has suggested that these scores contribute valuable information to the assessment of clinical skills by providing unique information different from that derived from checklist scores.8 However, other research indicates that the chart audit scores should not replace the checklist entirely, since the information written by candidates in a simulated medical record may not provide a complete picture of events during an SP encounter.9 The inclusion of the PEN in an SP test is appealing. First, it is thought that PENs are relatively immune to within-site and cross-site effects. Also, they do not depend on the accurate recording of checklists by SPs. Additionally, threats to security are minimized because the PEN is a free-response instrument and does not reveal checklist content or other exam material. However, before the PEN can be used in large-scale testing, it is important to determine whether the PEN is a reflection of the checklist or whether the PEN contributes unique information about a student's ability to synthesize and record medical information. The purpose of this study was, therefore, to investigate the relationship between entries recorded in the PEN and actions captured on the checklist. It is hoped that the results of this study will help determine how to best incorporate PEN information into a composite score. Methods Measurement Instruments. History taking, physical examination, and communication skills in this SP test were assessed using six case-specific checklists composed of dichotomously scored items. Each checklist focused on two of the three clinical skills and contained a maximum of 25 items deemed critical for that case by a panel of expert clinicians. The six cases used in the study were based on a test blueprint reflecting the challenges expected to be encountered by a medical student entering residency. Checklists and PPQs were completed by the SPs following each 15-minute clinical encounter. An SP monitor viewed the encounter on a video screen and completed the same checklist as the encounter unfolded. If two or more item responses on the checklist completed by the SP conflicted with the responses on the checklist provided by the SP observer, a review of the videotaped encounter was required. The SP and the SP monitor reviewed the videotape together and discussed what constituted a correct response. The agreed-upon response became the key. A panel of expert clinicians was selected from a range of medical specialties. Each member of the panel had experience teaching third- or fourth-year medical students and experience with developing or implementing SP exams for clerkships or other courses. The panel developed analytic scoring keys for the PENs of the six study cases by using checklists, guides to the checklists, case prescriptions, and videotapes of each clinical encounter. These keys contained a list of significant findings or acceptable synonyms that the clinicians deemed essential for inclusion in a patient note. Examinees. The sample was composed of 80 fourth-year medical students from one northeastern medical school. The students were required to participate in the examination, but the scores did not contribute to their end-of-year grades. A group of six medical chart abstractors (MCAs) was recruited to score the PENs of the 80 students. The MCAs were divided into three pairs and each pair was assigned to score the PENs for two cases. The MCAs were oriented to each of their assigned cases by listening to the case summary and student instructions and by watching a videotaped encounter of the case. The MCAs then scored an initial sample of PENs for each case using the keys developed by the panel. The MCAs were instructed to give credit for each finding (or acceptable synonym) included in the PEN. Following this “practice scoring exercise,” the pairs discussed their decisions and reached consensus. The MCAs then scored the remaining PENs and completed a post-scoring survey regarding their reactions to the process. Analyses. Items on the scoring key were matched to content-equivalent items on the checklist by a test-development staff professional. Items on the scoring key without a matching checklist item and exemplars were not reviewed in the study. For each matched scoring key and checklist item, the proportion-of-agreement rates (the observed agreement rate between a given behavior as measured by the checklist and the PEN) as well as kappa coefficients (an estimate of agreement above and beyond that expected due to chance) were computed. Results The frequency distributions of the proportion-of-agreement rates and kappa coefficients for the six cases are shown in Table 1. A total of 68 item comparisons were examined across six cases. Overall, 69% of the items had proportion-of-agreement rates that fell between.61 and 1.0. At the case level, 75% of the items in Case 1, 92% of the items in Case 2, 69% of the items in Case 3, 56% of the items in Case 4, 100% of the items in Case 5, and 36% of the items in Case 6 had proportion-of-agreements rates that fell between.61 and 1.0. Across the six cases, 31% of the items had rates that fell between.21 and.60, and none of the items had a proportion-of-agreement rate below.20. Kappa values ranged from.12 (Case 6) to.90 (Case 3). Kappa cannot be computed for items that are correctly answered by all examinees, hence, the total number of item comparisons across all six cases was 62. Using classification guidelines proposed by Landis and Koch,10 perfect or excellent agreement was achieved for 15% of the items, good or fair agreement for 42%, and slight or poor agreement for 44%.TABLE 1: Frequency Distribution of Proportions-of-agreement Rates (and Kappa Coefficients) between Checklist and Post-Encounter Note (PEN) Items in a Field Test of an Administered Standardized Patient Test, 1998*The average proportions of discordance between items on the checklists and the PENs are provided in Table 2. Column three in the table represents the average proportion of students who received credit for an item during the encounter (checklist = Y, or yes) but did not write the findings in the note (PEN = N, or no, or errors of omission). At the skill level, the average proportions ranged from.08 to.31 for history taking,.09 to.33 for physical examination, and.09 to.39 for communication skills. Poor documentation in the note does not appear to have been skill-specific. Column five shows the average proportions of students who recorded findings in the note (PEN = Y, or errors of commission) that were not actually pursued in the encounter (checklist = N). With the exception of that for the history-taking skill in Case 6, these average proportions are considerably less than those in column three.TABLE 2: Average Proportions of Checklist-Post-Encounter Note (PEN) Discordance, by Case and Skill, of Examinees' Performances on a Field Test of a Nationally Administered Standardized Patient Test, 1998*Discussion Proportion-of-agreement rates uncorrected for chance were strong for four of the six cases. Cases 4 and 6 exhibited moderate agreement. The proportion-of-agreement rates corrected for chance were poor to moderate. Cases 4 and 6, for which agreement was particularly poor, required that the student use appropriate patient education and counseling techniques and reassure the patient. Thus, students appeared to have difficulty synthesizing and appropriately recording the psychosocial elements of an encounter. It may be that the students simply did not recognize the pertinence of documenting information related to patient education and counseling. Other studies have also reported poor documentation of items related to patient education.8 On the whole, however, poor documentation in the PEN was not confined to cases with communication components. Omitting from the PEN items that were actually pursued during the encounter appeared to some degree in every case. In particular, the physical examination items in Cases 1 and 3 were not documented well, suggesting that the students had difficulty interpreting physical exam findings into the written word. Again, it may be that the students simply did not understand the significance of the information gleaned from the encounter. It is also possible that the students' concept of an adequate note was less comprehensive than that envisioned by the test developers. Revising the student orientation to the test with an emphasis on the importance of the PEN and providing examples of appropriate documentation for a clinical encounter may improve documentation rates. Although it was less likely that the students recorded in the PEN findings that were not actually pursued during the encounter, such items occurred more often in Cases 2 and 6 for history taking, and in Case 4 for communication items. At first glance it appeared that the students might have fabricated findings, but closer examination of the items where this occurred revealed potential problems with the wording of the scoring key. For example, while a checklist item is very specific, a single PEN item attempts to capture all possible synonyms; thus, overlapping concepts may be inadvertently lumped together. Identifying the discrepancies between the checklist and the PEN may be helpful for refining the scoring key. It is important to interpret our findings in light of limitations inherent in the study. First, the analyses were conducted with a small sample (80 students per case) from a single medical school; thus, generalizations should be tentative. Second, the study examined only six cases, which provided a limited opportunity to sample across the test blueprint. Future research should address these limitations by expanding the focus of the study to several diverse schools. Increasing the number of cases and the sample size will also help to minimize measurement error. Finally, excluding items from the scoring key without a matching checklist item may have underestimated the students' abilities to document their findings. Within this study, the PEN scoring key was assumed to represent the “gold standard” for appropriate documentation. Students may have written in the notes items that would not be credited because the clinical experts did not deem them critical. Further content validation of the scoring key may be useful in addressing this shortcoming. Despite these limitations, the results of this study suggest that the PEN provides unique information about students' abilities to document the gathering of information, understand the significance of the information gathered, and translate verbal information into the written word. Thus, poor concordance between the checklist and the PEN suggests that students may have limited skills to properly synthesize, interpret, and record findings from a clinical encounter. Given these results, test developers may be less inclined to treat the PEN score as simply a reflection of the checklist or redundant information, in view of the fact that the ability to record significant findings from clinical encounters and demonstrate understanding of those findings is a critical function not only of medical students but also of practicing physicians.