The Scaffold Effect: Agent Harnesses Introduce Larger Cost Variance Than Model Upgrades in Terminal Benchmark Evaluation

This preprint presents a controlled empirical study of how agent harness scaffolding affects terminal benchmark evaluation independently of model choice. We evaluate two frontier models, Qwen 3.6 Plus and MiniMax M2.5, across three agent harnesses: Goose, OpenCode, and OpenHands-SDK on 50 tasks from Terminal-Bench Pro. The study finds that harness choice produces up to a 40× difference in tokens per solved task, while pass-rate differences remain within 2–10 percentage points. The results suggest that agent benchmark reporting should treat harness-model pairs, rather than models alone, as the unit of comparison. The accompanying GitHub repository contains the selected task list, harness configurations, raw trial logs, aggregate snapshots, schemas, and analysis scripts. Code and data: https://github.com/namanvats/scaffold-effects