From Mexico, I’ve watched a lot of coding-agent launches lately where the pitch isn’t a new model, it’s a better harness: the system prompt, the tools and the context rules wrapped around the model. A paper from two researchers at Shandong University, “What Does a Harness Buy? Tokens, Mostly.”, tested that pitch, and VentureBeat covered it on October 23. The short answer in the title is pretty much the finding.
How they tested it
They took three production harnesses as they ship, Claude Code, OpenCode and the minimal mini-SWE-agent, pinned their versions, and ran the same models through all three on SWE-bench Verified with network access off. The models were Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview, with Claude Opus 5 inside Claude Code as a strong reference point. They used two task sets: the 45 hardest tasks and 447 others. The part I like most is that they also reran identical setups, so they could compare the harness effect with plain run-to-run noise.
What the harness didn’t buy
On the 447 tasks, with the two Qwen models, Claude Code and mini-SWE-agent, the heaviest and the lightest harness, came out equivalent within five points. On the 45 hard tasks, switching harnesses flipped about 13% of outcomes, and so did rerunning the same harness with nothing changed. The only harness effect that cleared the noise was a loss: OpenCode trailed by up to 9 points on the larger pool, and on one model about half of that gap came from runs its output cap cut short. Claude Code hit the same cap but has a step that asks the model to continue.
What it did change: the bill
With the same model, the same tasks and one price list, cost per task differed by up to 3x across harnesses. The authors trace most of it to the first call. Each harness sends a fixed preamble of system prompt, tool schemas and message wrapper with every step: 16,581 tokens in Claude Code, 7,025 in OpenCode and 829 in mini-SWE-agent. Multiply that by the number of steps and you get the gap. Prompt caching lowers the bill, but in their numbers it didn’t change which harness was most expensive.
There’s also a useful warning about benchmarks in general. In early web-enabled runs, 312 of 439 trials fetched the original fix from GitHub, so the final runs were done offline. And the statistics are sobering: by the authors’ math, a 45-task comparison catches a gap of about 13 points only half the time, and even 447 tasks only resolve about 5 points. Plenty of harness release notes brag about smaller gains than that.
The limits
It’s one benchmark, plus a small transfer check on Terminal-Bench 2.1, and the closed frontier model only ran inside Claude Code, so the paper can’t say whether a model like that shows the same pattern across harnesses. Harnesses also do things SWE-bench doesn’t score, like permissions, UX and integrations. The Decoder and XenoSpectrum also wrote it up if you want other summaries.
My take
I don’t read this as “harnesses don’t matter.” I read it as: they matter mostly in how they fail and what they cost, not in some hidden accuracy boost. The authors put it well: the numbers to carry between vendors are the first call, the growth per step and the step count.
If I were choosing a coding agent for a team, I’d log those three numbers on our own repos and run the baseline twice before believing any 2- or 3-point difference. The paper recommends exactly that second run. It’s the cheapest control there is, and most comparisons skip it.
Featured image: “Kummet Meierhof-Museum Herberstein 01” by Liuthalas, Wikimedia Commons, CC BY-SA 4.0. Cropped, resized to 1400×900 and lightly sharpened; the modified image is shared under the same license.
