I may have an unhealthy thing for experimenting and benchmarking. I wrote Morph because I thought the only way to know which model you should use is to test it on your own traces. (If you want another example, I also have an extensive spreadsheet on all the permutations of pizza dough I’ve tried and tweaked through the years.)
Fast-forward to part one of an experiment I recently did to understand the base-level performance of the current coding harnesses — Claude Code, Codex, Kimi Code — and how they perform with their default models. Specifically, what happens if you swap in open-weight models? How much of a performance score is the harness, and how much is the model? And the one that I’m really chasing: Can several coding agents working together beat the best single coding agent alone? (Because if that’s true, imagine using less powerful local models, composited together, or things like that.) This is only the first phase — I know I’m not done here, and am still running experiments.
The finding I kept coming back to? The serving path — who actually answers when you ask for a model by name — moved my results as much as swapping the model itself.
The Exam
I needed a test, something to benchmark everybody on, so I chose SWE-rebench-V2. The nice thing about that benchmark is that each task is a real moment from a real open-source project — an actual ticket that a human filed, plus the codebase from that day. The agent gets the issue and the repo, and a timer set at 30 minutes. Grading is simple: When time’s up, run the project’s own test suite, including the tests written to catch the bug.
(I used rebench because the original, SWE-bench Verified, has been public since 2023, which means that the answers are probably in everybody’s training sets. Rebench refreshes continuously.)
The shootout: The three harnesses, each on its default model (claude-opus-5, gpt-5.6-sol, kimi-k2.7-code). Then model swaps inside Kimi Code (k2.6, then k3). Then one model name, kimi-k3, across four serving paths. And one open model, GLM-5.2, hosted through Codex at two more providers. I could have been a bit more rigorous, but I was just doing a bit of sampling and experimentation to get started.
In all these permutations, I recorded cost per task (failures included) and solves — shown everywhere as solves/attempts, because the numerator matters. My sample is 30 tasks; at this size, only gaps of roughly 20 points separate skill from luck.
One label needs flagging up front: My Codex arm ran gpt-5.6-sol at high reasoning effort — pinned to match the public leaderboards — while the CLI’s out-of-box default is medium. Read that row as “leaderboard configuration,” not factory settings.
Defaults first:
Claude leads, but I can’t actually tell Claude and Codex apart at this sample size (exact McNemar on the paired tasks, discordant 5:3, p = 0.727 — and the intervals are wide). Kimi’s dash is a blank, not a zero: that harness reports neither tokens nor cost, so it ranks on quality but can’t enter a cost argument. (At some point, I’ll put a counting proxy in front of it.)
Two Dials
Now let’s turn some dials. In Mozilla’s State of Open Source AI report, we said the harness is where the action is — it can move a model by a lot. So let’s separate the harness from the model. I held Kimi Code fixed and swapped only the model: The coding-tuned default solved 7/28 (25%), the general k2.6 solved 8/26 (31%), and k3 solved 12/27 (44%). Nineteen points from a model swap, harness untouched. And yes, the coding-tuned variant was the worst of the three on this sample.
(For future work, I want to cross the full matrix — every model in every harness. Claude Code speaks Anthropic’s wire protocol, so hosting a foreign model inside it means inserting my own proxy into the thing being measured.)
Shopping for a Model
There are a lot of open-weight models you can run yourself. Most people aren’t going to have the hardware, nor even want to rent the hardware, for K3 and GLM-5.2, so I went shopping. One model ID is available across as many as 21 provider endpoints (as of my runs), at advertised quantizations from 4-bit to full precision. Prices vary several-fold for the same weights, with availability that comes and goes — behind a router that may pick per request, stick a session to one endpoint, or fail over mid-conversation. I measured four identical calls served by two providers at two different advertised precisions. I’m trying to guard against the unpinned benchmark. That number, if it remains unpinned, is just a sample from a distribution.
So I built a small proxy that injects a provider pin and records which deployment answered every call — 959 calls on one arm, zero violations. Then: same harness, same tasks, one model name (kimi-k3), four serving paths:
Nineteen points between the worst path and the best.
Provider and precision change together in that table, and the precision labels may not mean what they seem. Moonshot released K3 with quantization-aware training baked in: The official checkpoint is natively MXFP4 weights with MXFP8 activations. There is no higher-precision K3 sitting behind it. 4-bit is what Moonshot trained and what Moonshot evaluates. So an endpoint labeled “fp8” can’t simply mean “the same weights, less compressed,” and “Moonshot direct” can’t be assumed to sit at the top of a precision ladder; it’s presumably serving the native 4-bit checkpoint, too. Which makes the sharpest comparison in the table the one I’d been under-weighting: Moonshot’s endpoint through the aggregator (7/28) versus Moonshot’s own API, direct (12/27). If both serve the native checkpoint, roughly 19 points sit in the translation layer between me and the same weights.
There’s a named suspect for how. K3 is trained in “preserved thinking history” mode — it expects its own reasoning content handed back intact on every turn — and Moonshot’s own documentation warns that harnesses or paths that truncate or modify that history cause significant quality degradation. If the serving path drops or alters K3’s earlier reasoning, you’d expect it to get worse as a task goes on — which is consistent with what I saw.
My 30-minute wall clock adds a third suspect: A slower path buys fewer inference cycles before the buzzer, so speed converts into solves. Precision, protocol fidelity, latency — what I’ve measured is their sum: the serving path. Isolating the mechanism comes next, starting with the cheapest test, a transcript-fidelity check on every path: Does what the model said come back to it intact?
The Economics Are Backwards
BaseTen and Modal publish identical per-token rates for this model. Same sticker. The BaseTen path still spent 19% less in total and solved 43% more tasks — $2.33 per solved task against $4.12 — because a degraded path burns turns failing: re-reading files, re-running tests, retrying tool calls, and billing you for every token of the flailing. Per-token price couldn’t distinguish these two sellers at all; cost per solved task could — the Modal path cost 77% more per solved task. Anything cheap that makes an agent fail more is more expensive than the price list says.
It compounds. The aggregator’s displayed price for a model tracks its cheapest endpoint. One provider’s rate doubled in 14 hours, mid-run. And on two independent runs, roughly 80% of the output tokens the billing meter charged me for never appeared in the usage fields the API returned — reasoning tokens, billed but not reported. (Next audit: reconcile per-generation IDs against the invoice.) A price here is a reading with a timestamp; the numerical recipe you’re actually served isn’t part of any spec you can hold a seller to; the only bill you can trust is the meter. Renting is fine — most of us will rent inference forever — but this is renting without a lease.
The Team Question
Back to why I even started this: I wanted to know whether a team of agents can beat a single agent on the biggest one. I’m still tinkering with the orchestration framework. The honest status? Nothing is landing yet.
I’ve tried a bunch of arrangements — Claude as the coder, Codex as the reviewer, and so on. The best of them gained +7 points in my sandbox and exactly zero on a held-out task set the ideas had never touched. (I keep one precisely so a lucky sandbox result can’t seduce me. This one tried.) The sharpest result is set arithmetic, not a percentage. For a selector over already-generated patches to beat its best member, the other members must solve something the best one misses. Against solo Claude: GLM-5.2 at one vendor contributed zero unique solves. GLM-5.2 at a second, independent vendor — run to rule out a degraded path masking the effect — zero. Kimi-k3 at its best: zero. The open-model candidates never solved a task that Claude missed. So no matter how smart the selector was, it had nothing better to choose.
That kills one class of system on my sample: independent generation plus a picker. It says nothing yet about real collaboration — a reviewer pointing at a bug so the coder can revise creates patches that never existed in either solo set. That, plus candidates chosen for genuine complementarity (which might mean a different model family, or might just mean resampling the strongest model) is what I’m testing next. I couldn’t run the frontier-plus-open cell honestly until the pinning machinery existed. Now I can.
So the summary, for now: The model swap moved about 19 points. The serving path moved about 19 points. Swapping the harness, with everything else pinned, moved about 3. No strategy has survived the held-out set. My sample can’t rank those first two against each other — they’re the same size at this resolution — so the operating rule is simpler: Before buying orchestration, benchmark the model and the endpoint that actually serves it. Both are cheap; each serving-path rung above cost me under $35 to run.
Next…
If you build with agents: Measure the model. (Don’t trust the “code-tuned” label.) Pin the serving path, log what it advertises, and judge sellers on cost per solved task, never per token. Demand held-out evidence before buying any multi-agent pattern.
If you buy inference at scale: Benchmark your own workload across paths, because the market quotes tokens and you consume outcomes.
And if you allocate capital: Served well, open-weight models were pretty close to the closed frontier on my sample, at about half the cost per solved task.
So: 19 runs, 1,001 graded attempts — an attempt being one agent configuration on one task, across a pool that includes a 100-task held-out set, which is how 19 runs makes 1,001. I’m going to keep tinkering. Next up: that transcript-fidelity test, the same ladder with the provider held constant, and swapping models into Claude Code and Codex.
Tell me what you’re working on in the comments. I’d love to chip in.
This is exactly why I think the evaluated unit has to be model + harness + serving path + verifier, not the model name. The same nominal model can see a different transcript, get a different response budget, or complete fewer cycles before timeout depending on the path. Cost per token hides all of that; cost per accepted solution exposes it. I also like that the multi-agent result failed on the held-out set. In our own work, we are trying to treat topology as an independent variable and hold the acceptance contract constant across single and decomposed runs. For the next pass, will you also pin total inference budget and tool-call budget, or is wall clock the governing constraint?
Hey Raffi - I love the stuff you’re working on. Help me with some terminology, explain: “Then one model name, kimi-k3, across four serving paths. And one open model, GLM-5.2, hosted through Codex at two more providers”. Thanks.