This technical note demonstrates ranking inversion in cognitive profiles, indicating the importance of evaluation protocols.
This technical note presents a minimal synthetic stress test on the legitimacy of global cognitive ranking under admissible protocol variation. Two synthetic cognitive agents with complementary performance profiles are evaluated across three task families — local-immediate, distributed-integrative, and mixed — while the evaluation protocol varies by changing the relative emphasis assigned to these families without removing them from the shared evaluative architecture. The central result is protocol-dependent ranking inversion: one profile ranks higher under protocols privileging locally segmentable performance, while the other ranks higher under protocols privileging distributed integration. A balanced global score can still produce an apparently unified ranking, but that ranking is shown to depend partly on the evaluative interface rather than on a protocol-independent dominance relation between the two profiles. The purpose of the note is diagnostic and structural. It does not argue against evaluation in general, does not address human psychometrics directly, and does not propose a new theory of cognition. Its narrower aim is to show, in a controlled synthetic setting, that a global score may partly rank interface compatibility rather than profile structure alone. The same diagnostic issue is also directly relevant to benchmarked AI systems, where heterogeneous task families are often compressed into single global rankings.
No takes yet. Share an insight, caveat, or question.
Danilo Tavella (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: