This framework evaluates multi-model orchestration accuracy in auditable conditions, suggesting improvements for future implementations.
We present ACAR (Adaptive Complexity & Attribution Routing) as a measurement framework for studying multi-model orchestration under auditable conditions.ACAR uses self-consistency variance (σ) computed from N=3 probe samples toroute tasks across single-model, two-model, and three-model execution modes,implemented atop TEAMLLM, a deterministic substrate with immutable artifactsand complete decision traces. We evaluate across 1,510 tasks spanning four benchmarks (MathArena, Reasoning Gym, LiveCodeBench, SuperGPQA) with ClaudeSonnet 4, GPT-4o, and Gemini 2.0 Flash, producing 7,550+ auditable runs. Whatholds: σ-based routing achieves 55.6% accuracy, exceeding the two-model baseline(54.4%) while avoiding full ensembling on 54.2% of tasks; the mechanism is modelagnostic and requires no learned components. What does not hold: (1) Retrievalaugmentation decreased accuracy by 3.4 percentage points—median retrieval similarity was only 0.167, demonstrating that experience injection without semanticalignment introduces harmful noise rather than grounding. (2) When models agreeon incorrect answers (σ=0), no downstream ensemble can recover; this “agreementbut-wrong” failure mode is intrinsic to self-consistency and bounds achievableaccuracy at 8pp below full ensembling. (3) Attribution estimates based on proxysignals (response similarity, entropy) showed weak correlation with ground-truthleave-one-out values; practical attribution requires explicit counterfactual computation. This paper documents what assumptions fail in practice, providing falsifiablebaselines for future work on routing, retrieval, and multi-model attribution.
No takes yet. Share an insight, caveat, or question.
Ramchand Kumaresan (2026) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: