MedGauge: A Metrological Framework for Measuring Medical AI Agents by Program-Graded Work Products

Evaluation of medical AI has inherited its instruments from the evaluation of people: examinations, expert judgement of free text, and task value denominated in the hours a practitioner would need. This working paper asks which of those instruments survive when the thing measured is a silicon producer (a model running inside a harness with tools) and builds a measurement framework for the parts that do not. It is the argument for the construction of the MedGauge Methodology, version 0.1.0 (https://medgauge.org/methodology/text/), which is the normative text. It tests the claim that silicon producers differ from human producers against the literature and finds twelve properties of silicon producers that bear on measurement, each stated with its counter-evidence; a difference from people is supported for two of them by an opened source, three hold by construction of the subject, and for the remaining seven no source states the human side and no difference is asserted. It shows that human-anchored evaluations divide into anchors on human time (which transfer as a predictor of difficulty but not as a unit), on a human product or rule (which transfer once the product is converted into a key of field checks set independently by two experts) and on human judgement of free text or preference (which do not transfer). On that basis it builds the measurement: following the international vocabulary of metrology, the reference of each quantity moves from a unit to a versioned grading program applied to a fixed artifact contract. It gives scale rules for labels, levels and counts; proposes base and derived quantities with their scale type, traceability and cross-harness comparability; maps them onto ten measurement dimensions and ten task labels; and sets out an examination regime in which a subject passes only if every task passes in all three runs with no safety veto, the strength of a pass being a one-sided 95% lower bound on reliability, with execution errors counted as failures, paired comparison with step-down multiplicity control, and no overall score. Version 0.1.0 of the methodology sets no level threshold and issues no level. The paper proposes a four-layer task typology with its literature base, explains why a closed, uncalibrated, generated task set licenses an attribute-sampling regime but not a response-model regime, and sets out the uncertainty budget for the chain from a published score to its public source clause. Standardised semantics for products, scenarios and task types, and the validation of those standards, are set out as a research plan, not as results. The paper reports no measurement result. At the date of publication the public site is live, no benchmark round has been run, no result record exists and every leaderboard is empty.