Meta-evaluation framework assesses AI ethics benchmarks, revealing biases and suggesting improvements.
AI systems are increasingly evaluated on ethics benchmarks to ensure alignment with human values. However, a critical question remains unexamined: who evaluates the evaluators? Benchmarks themselves may encode hidden assumptions, cultural biases, and malformed framings that AI systems inherit during training or evaluation — a phenomenon termed drift in, drift out. We introduce the Transcendent Invariant Kernel (TIK) framework, a systematic methodology for meta-evaluating AI ethics benchmarks. We model AI as LLM ⊕ CogOS (Language Model ⊕ Cognitive Operating System) and show, via Gödel's incompleteness applied to the deductive closure of CogOS, that undecidable ethical statements exist, and that the resulting undecidability trap is resolvable only by external grounding. We formalize this as the Gödel-trolley: ethical dilemmas conditioned on undecidable statements have no honest internal resolution, but an external kernel Φ provides principled resolution via the "do not do wrong" principle. Through a 3+1+1 judge architecture combined with an external kernel grounded in three ethical traditions (Christ-Teachings, Kantian, Ubuntu), we detect ontological holes (hidden assumptions) and forbidden fruits (malformed questions). We develop a learned TIK predictor architecture and demonstrate adversarial robustness methodology via perturbation analysis. We evaluate 9 major benchmarks (Moral Machine, ETHICS, TruthfulQA, Social Chemistry, Moral Stories, CommonsenseQA, MMLU-Ethics, Scruples, GAIA). Upon completion of data collection, we plan to release BenchmarkMeta, a dataset of annotated questions with TIK scores. Note: This preprint presents the methodological framework with illustrative empirical results. Empirical findings are preliminary and should not be treated as definitive. Feedback welcome: artiomkovnatsky@pm.me
No takes yet. Share an insight, caveat, or question.
Artiom Kovnatsky (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: