Assessing the Scientific Knowledge of the General Public

There is considerable controversy in the literature about how to measure the knowledge of the general public. Much of the past work concerns political knowledge and has focused on two issues—whether don’t know (DK) responses should be encouraged or discouraged and whether the items should take a multiple-choice or open-ended format. Similar questions have been raised about the best way to measure the public’s knowledge of basic science facts, which the National Science Board (NSB) has monitored for more than forty years. The NSB battery consists of eleven items, ten of them true-false items. The introduction to the items encourages DK responses. We carried out an experiment that compared true-false and forced-choice versions of the items; the experiment also compared versions of the questions that discouraged DK responses (thereby increasing guessing), encouraged them (decreasing guessing), or simply omitted a DK option. With a few items, the forced-choice format was harder than the true-false (i.e., fewer respondents answered them correctly), but with others that format was easier, with no overall pattern. Similarly, the reliability, unidimensionality, and validity of scale scores did not differ by question format. By contrast, encouraging DKs improved the reliability, unidimensionality, and validity of the battery relative to the other two DK conditions. We present a model that shows that discouraging DKs (thereby encouraging guesses) improves the measurement of knowledge only when it increases educated guesses more than it increases blind guesses. That apparently is not true for the science knowledge items we examine.