Unveiling the Black Box of Item Difficulty: An Interpretable Decomposition Approach Using LLM-Based Option Plausibility

Large Language Models (LLMs) achieve strong performance on standardized examinations, yet the language-mediated mechanisms through which they assess answer options in Multiple-Choice Questions (MCQs) and diverge from human difficulty judgments remain poorly understood. We argue that predicting the difficulty of MCQs provides a lens for studying how LLMs represent plausibility, uncertainty, and error across competing response options. reviewer1comments1mlAlthough recent deep machine learning approaches achieve competitive accuracy through large feature sets and complex architectures, their limited interpretability reduces their value for understanding model behavior. We propose an interpretable framework that decomposes item difficulty into LLM-based plausibility estimates over response options. These estimates are elicited through direct prompting and pairwise contrastive comparisons, and then integrated into a rational model that expresses item difficulty as a ratio between the plausibility of distractors and the plausibility of the correct option. We evaluated this approach on two high-stakes datasets. reviewer1comments1usmleUsing the United States Medical Licensing Examination (USMLE) dataset, reviewer1comments1rmsethe model achieved a Root Mean Squared Error (RMSE) of 0.277, comparable to previous approaches, while reducing the representation of the underlying elements from hundreds of features to only three parameters. Under Spearman rank correlation, the model reached ρ=0.427 on USMLE, representing a 70.8% relative improvement over previously reported results, and ρ=0.488 on ICFES, a new dataset. A complementary ranking analysis further reveals a systematic inversion between LLM-based difficulty judgments and those of experts, exposing divergences between model-internal assessments and human response patterns. These findings position option plausibility based on divide-and-conquer prompting as a principled framework for probing LLM decision processes, their rank-order misalignment with human response patterns, and their challenges in educational settings.