Developing an Item Pool for CBT

A significant portion of the costs associated with assessment is directly related to development of test questions. Although high-stakes testing programs use a variety of approaches to item development, one of the most common approaches involves committees of faculty (content experts) who write items on assigned topics at home and then travel to the offices of the testing organization to review the items as a group. This approach is commonly used in professional licensure and certification examination programs, and is the one traditionally used by the National Board of Medical Examiners (NBME) for the United States Medical Licensing Examination (USMLE) and other programs. The procedures and the costs associated with item development can remain relatively stable for years (even decades); however, for many the move to computer-based testing (CBT) stimulates considerable interest in exploring other ways to generate items1 as they come face-to-face with the concomitant need to expand item pools in response to item-exposure issues associated with continuous testing. Because of these issues, the move to computer-based testing affects not only those involved in developing tests for national administrations but also the faculty who are developing tests for use at their medical schools. It is clear that pool size requirements depend on many overlapping factors (e.g., test length, number of test forms, repeat testing, and item exposure rules). It is also clear that, in comparison with paper-and-pencil tests that are administered infrequently during the year, computerized tests that are administered many days throughout the year require extremely large item pools.2 While many of the complexities of the move to computer-based testing are well documented, the complexities and costs associated with item development are often ignored or underestimated as attention is directed at issues of psychometrics and test administration logistics. In fact, only a single article was found that compared the quality and quantity of items and the relative costs associated with various approaches to item writing.3 The purpose of this study was to compare the relative merits of three approaches to item development. The three methods described below were used to develop items for USMLE Step 2 in 1997; comparisons are made of item yield, item quality, and cost per item. The year 1997 was selected because it allowed us to follow the status of items forward to their use in the live examinations. Methods Three models of item development were compared: the traditional standing test committee (TC), the ad-hoc task force (TF), and item harvesting (IH). Traditional TC. Each TC consisted of seven committee members and a chair; one editor was assigned to work with each committee. Committee members served a three-year term, with approximately a third of the members rotating off each year. Each new committee member attended a two-day item-writing workshop and then received an item-writing assignment developed by the senior editor. Each committee member was assigned to write 50 to 60 questions at home. Items were submitted on disk by mail; staff edited and annotated the items and returned them to the author, who responded to editorial queries and checked editing to ensure accuracy. Following an interative process, the final version of each item was included in a draft of all questions that was then mailed to the full committee two weeks before the meeting. All writers attended a two-and-a-half-day item-review meeting. Items were read aloud, discussed, and modified by attendees. Items accepted by the full group were eligible for pretesting. Each committee member was paid $1,000 plus travel expenses. Ad-hoc TF. Each TF consisted of 16 item writers, who came to the NBME's offices for a single one-and-a-half-day meeting. The item writers received a 30-minute orientation and examples of “good” and “bad” questions. Good items and, more importantly, the option lists from these items, were used to stimulate development of additional items. During the first day of the meeting, TF members were divided into four subgroups of four members each. The item writers worked in pairs to write items on computer; an editor was assigned to each subgroup. The editor coached each pair on appropriate item-writing techniques and edited their questions as they were written. Pair 1 and Pair 2 exchanged items for review, and modified items as necessary until all four agreed that the items were appropriate. The second day, groups of eight were formed to review the items written by the two subgroups. Items accepted by the group of eight were eligible for pretesting. Each committee member was paid $500 plus travel expenses. Item harvesting. Individual physicians were invited to write 10–15 items. Those who agreed to submit items were sent instructional materials, including an item-writing manual.4 The assigned editor screened submitted items from content and format perspectives, edited acceptable items, and passed these on for final review by the staff medical adviser. Approved items were eligible for pretesting. No honorarium or travel expenses were paid. For all three models, items with acceptable pretest statistics were forwarded to a final review committee for approval to enter the live pool. Items generated by all three methods were treated equally in this review. Analyses. Records of items written in 1997 were examined to provide data for analysis; 1997 was selected because it allowed us to follow these items as they progressed through pretesting into live use. Variables of interest were the number of items written; the number of items accepted for pretesting; the number accepted for use in the live pool; the quality ratings of the items; the amounts of editorial and support staff time required (on a per-item basis); and direct costs such as honoraria and travel/meeting expenses. For items in this study, as well as items in the pool overall, committee members reached consensus on quality using a four-point rating system from most desirable (i.e., this item could be used as an exemplar of items in this content area) to least desirable (i.e., this item should not be used on the examination). Results Table 1 provides comparative information for each model of item writing. While substantially more authors participated in the IH model (TC = 48; TF = 83; IH = 131), the number of items written was much higher for the TC model than for the TF and IH models (2,356, 1,083, and 1,537, respectively). Table 1 also shows the number of items per author that were submitted, approved for pretest use, and approved for subsequent live use. Ultimately, the TC model produced 27 items per author that were approved for live use, while the TF and IH models produced 10 and four items, respectively. The average quality rating of the items approved for live use was highest for the TC model and lowest for the IH model.TABLE 1: Summary of Item Yields and Costs Associated with Three Models of Item WritingCosts associated with item development included committee expenses (travel, hotel, and honorarium) plus staff costs (primarily editor and other staff time). Committee costs were higher per item for TF ($145) than for TC ($111) due to the direct costs such as airfare's being spread across three-day meetings for TCs and two-day meetings for TFs. There were no direct costs associated with the IH model. Staff requirements per live item were surprisingly high for the IH model (2.4 hours) in comparison with the TC and TF models (1.5 hours each). Staff time was calculated by combining editor and support time, with one hour of editor time equal to two hours of support time. Conclusions Between 1995 and 1998, the need to expand the item pool rapidly in preparation for implementation of CBT fostered an environment where creative approaches to item development were essential. The goal was to increase production of high-quality items with a minimum of increased staff effort. Two new item-development models were largely responsible for the increase in item production from 2,300 items submitted in 1995 to an all-time high of nearly 4,500 items submitted in 1997. Analyses showed clear advantages and disadvantages associated with each of the three models. The traditional TC produced the most live items per author, with a higher item quality rating than the two new methods. Direct costs were higher for the TF than the traditional TC, but extending the TF meeting to three days would move these two models into balance in terms of direct costs. In terms of staff time, the TC model required large amounts of pre-meeting work to develop individual item assignments targeted at areas of need and then to edit the items once they were submitted. On the other hand, the TF required little pre-meeting work, but did require significant staff time during the meeting, and then afterward to improve the quick editing that occurred as items were written during the meeting. Finally, IH required relatively small amounts of staff time based on the total number of submitted items, but, because the ultimate yield was so low and because the items (written by untrained authors) required more editorial effort, the amount of staff time per live item was greater than for the other methods. From the perspective of the item writer, the TF and the IH models required much less commitment, making them appealing to item writers who could not promise to write 60 items and attend a three-day meeting each year for three years, as required by the traditional TC model. TF members committed only to attending a single one-and-a-half-day meeting, with no work before or after the meeting. Each IH member agreed only to send in a few items on the topic of choice in a time frame largely dictated by personal schedule. There were other notable issues that arose. First, the TF was useful as a screen for item writers; staff could observe an item writer for two days and judge whether that person would be a good TC member. IH was also somewhat useful to identify potential committee members. Second, IH writers sometimes produced an item that provided a unique approach or an interesting perspective for an old topic. Third, TF and IH activities could be scheduled with more flexibility than TC activities. While TC activities were scheduled almost a year in advance, TF meetings were scheduled on short notice, and IH items could be processed as time permitted because no committee meeting was required. The TC approach produces the highest-quality items, despite item-writing assignments that include the most challenging topic areas. Writers receive substantial training, with a two-day item-writing workshop at the time of their appointment, plus coaching and feedback throughout the writing process. Following pretesting of their items, writers are provided with statistical information about their items and given the opportunity to “learn from their mistakes” and improve upon their work. For programs that can afford the cost and are willing to invest three years in developing writers, this model results in large numbers of high-quality items. The TF process is also a top-quality model. The quality of the items written using this model is high, and, because writers receive intensive one-to-one training, it is a good model to use for screening potential members for the TC. In addition, this model is the best approach to use when questions need to be developed very quickly, since there is minimal up-front work. Increasing the meeting length to three days could decrease committee costs per item (but it is not clear that staff or writers could sustain the item-writing effort this long). Decreasing the editor-to-writer ratio from 1:4 to 1:6 and/or decreasing the number of author reviewers from eight to four could decrease staff time. The IH model is a good choice for the low-budget exam program. Although the cost for staff time per item is highest, there are no direct costs associated with this model. Although half the items are discarded, half are retained, and most of these are of good quality. These items provide a good foundation for the low-budget program and could be useful in supplementing a high-budget program. Item development costs represent a large portion of the budgets associated with high-quality assessment. Results of studies such as this one are useful to those who are charged with building item pools and who must balance item yields, item quality, and costs.