An Empirical Comparison of Agentic AI Frameworks Across Six Language Models: Token Overhead, Interoperability, and Robustness

Developers choosing between agentic AI frameworks rely mostly on vendor documenta-tion and informal blog comparisons, which rarely fix the model, repeat runs, or publishcode. We benchmark four widely used frameworks—LangGraph, CrewAI, AutoGen,and the OpenAI Agents SDK—plus a Model Context Protocol (MCP) tool-servingvariant, across six language models spanning three providers (OpenAI, Groq, and localOllama), including ALLaM, the Saudi Data and AI Authority’s Arabic-centric model.The full matrix is 5 frameworks × 6 models × 3 tool-use tasks × 5 repetitions =450 runs, all with a pinned model per cell, temperature 0, identical tool implemen-tations, and deterministic programmatic success checks. We find that task success isdetermined almost entirely by the model, not the framework: on a fixed model, all fiveframeworks nearly always land on the same success rate. The frameworks instead differon axes that informal comparisons miss: token overhead varies by an order of mag-nitude (CrewAI consumes 10–20× the tokens of AutoGen for identical tasks, a ∼10×cost multiplier on paid APIs); latency is dominated by the provider rather than theframework; MCP tool routing adds no measurable overhead once the server connectionis reused; and frameworks differ sharply in how they degrade when a model lacks nativefunction calling—on ALLaM, four frameworks fail every request, while CrewAI silentlyfalls back to prompt-based tool use and succeeds. All benchmark code, raw per-runrecords, and analysis scripts are released for reproduction.