Illuminating the "Black Box" of Standard Setting: An Exploratory Qualitative Study

Standard-setting research has relied very heavily on certain types of information that, although helpful, may be inadequate for judging the quality of standard-setting results. To increase confidence in standard-setting results, the measurement community must understand far more about the panelists' task and about the factors that influence their decisions. Further, the measurement community may be overlooking important sources of data and types of research that could be useful in understanding these processes. To illustrate 1 such type of data, this article will present findings from an exploratory qualitative study of 2 standard-setting meetings. These findings are intended simply to support the potential of such studies to shed light on the standard-setting process. This article is thus to be understood not as an empirical study but as an argument supported by illustrative data, presented within a framework of standard-setting validity.