We present Blind Spot, a benchmark for measuring metacognitive awareness in large language models, defined as the ability to know where one's own knowledge breaks down. Unlike existing calibration benchmarks that vary question difficulty (obscure versus familiar), Blind Spot uses expert agreement as its independent variable: a gradient from arithmetic (agreement = 1.0) through medical diagnosis (approximately 0.60) and legal interpretation (approximately 0.45) to philosophy of mind (approximately 0.05). This variable has a formal definition (Conceptual Invariance, CI) and has been empirically validated at r greater than 0.99 across five independent measurement methods. The benchmark tests four escalating metacognitive abilities, namely confidence calibration under forced commitment, prospective failure prediction, error detection with centroid awareness and false-positive controls, and behavioural variance measured across multiple completions, and aggregates them into a single 0 to 100 score, the Blind Spot Metacognition Score (BMS). Tested on Claude Opus 4.6, Gemini 3, and ChatGPT 5.3, the benchmark produces clean separation, with BMS scores of 59.5, 52.8, and 54.4 respectively. All three models calibrate well on factual questions (16 to 17 of 20) but compress confidence into the 55 to 100 percent range on maximally contested questions, scoring 6 to 8 of 25 on the confidence-floor component. Two external validation experiments confirm the pattern: rerunning the benchmark in randomised order with domain labels stripped preserves the gradient (Claude r = 1.000, ChatGPT r = 0.857, Gemini r = 0.714), and across 60 contested questions all three models give the same directional answer 65 percent of the time, a direct empirical signature of centroid convergence. Task 4 (behavioural variance) is, to our knowledge, the first metacognition measure that cannot be gamed through self-report, because it measures the geometry of the model's output space directly.
Shreya Bhattacharya (Sat,) studied this question.