A mixed-item approach to measuring political knowledge

Common formats for measuring political knowledge include multiple-choice and open-ended identification items. The former allows respondents to guess the correct answer, potentially leading to overestimation, while the latter often results in a high percentage of invalid responses due to cognitive demands on working memory, leading to underestimation. These biases – overestimation from guessing in multiple-choice items and underestimation from low response rates in open-ended items – can distort inferences about political knowledge. This paper explores a measurement approach that integrates both multiple-choice and open-ended items and proposes a Bayesian item response model designed to accommodate this mixed-item format for estimating latent traits. The analysis of survey data from Taiwan yields several key findings. First, the mixed-item format reduces item nonresponse rates compared to the open-ended format, addressing the issue of underestimation. Second, it differentiates correct responses in multiple-choice items from those in open-ended items, mitigating overestimation. Third, incorrect responses to open-ended items signal a lower level of knowledge than incorrect responses to multiple-choice items. Finally, researchers can still apply the conventional counting procedure when the proposed IRT model is not utilized.