Conceptual and methodological issues in studies comparing assessment formats

Background: There is an extensive literature, dating back over several decades, comparing written items formats (i.e., multiple choice versus free response questions). Most studies conclude that different formats measure different underlying skills or competencies. Purpose: The purpose of this article was to critically examine methodological issues that arise in studies comparing alternate written assessment methods, and identify conditions that must be met before it can be concluded that different formats assess different aptitudes or traits. Methods: Using a hypothetical study as a focus, we critically examined the requirements for an unambiguous conclusion of a format difference. Results: We identified a total of eight different criteria which must be met for research on item formats. Conclusions: Although we made no attempt to systematically review the literature, a reasonable conclusion is that few studies have met these criteria and have sufficient rigor to support the claim that different formats assess different characteristics.