Putting Kimi K3 to the (Claude Code) Test

A successful test-drive — with caveats.

Kimi K3 dropped, so I got curious.

Thirty runs later, against two tasks pulled out of my own recorded sessions, I can honestly say: no difference. Every Kimi setup solved every task. On one, every permutation I tried wrote a byte-identical function, matching the change that shipped. Opus 5 was the control, and it wrote the same function. I’ve been talking about doing personal evals for months, and here is the shot.

That means if you are writing code in Claude Code, you could pick up and switch tomorrow. Caveats below — and there are a lot of them.

Moonshot’s Kimi came in cheaper: 2.77x on one task and 4.45x on the other. (Though if you are on a Claude Code or a Kimi Code subscription, that cost difference may be hidden inside the flat fee.). And Kimi is wayyyy less chatty.

At this point, my Claude Code setup is getting pretty intricate. So it seemed like a healthy time to look at alternatives anyway. The question was never whether to try K3. It was how much of my setup I could keep and still make the switch.

The Swap

Four lines, and Claude Code was running on Kimi K3:

export ANTHROPIC_BASE_URL=https://api.moonshot.ai/anthropic
export ANTHROPIC_AUTH_TOKEN=$MOONSHOT_API_KEY
export ANTHROPIC_MODEL=kimi-k3
claude

/status confirms it: model kimi-k3, endpoint api.moonshot.ai. Slash commands still worked. Hooks still fired. CLAUDE.md still loaded. (The harness doesn’t really care about the model below it, sort of.) Models and harnesses were jointly optimized, though, so this is where I found out how much that joint optimization was doing for me.

Four Ways In

I ran three of the combinations, and both K3 cells twice, once with the metered API calls and once on a subscription.

Kimi Code CLI is their own harness. MIT licensed. One static binary. On a subscription:

kimi login                          # OAuth device code, opens a browser
kimi -m kimi-code/k3 --yolo         # --yolo auto-approves tool calls

You have to spell out -m kimi-code/k3 or you get kimi-for-coding, which is K2.7.

Claude Code metered against K3 is the three lines at the top of this section: $3 per million in and $15 out, against Opus 5 at $5 and $25.

Claude Code on a Kimi Code subscription is the combination people miss. It works, and your existing membership pays for it:

export ANTHROPIC_BASE_URL=https://api.kimi.com/coding/
export ANTHROPIC_API_KEY=<console key from kimi.com/code>
export ANTHROPIC_MODEL=k3

Use a console key, not the token kimi login writes. The kimi login one lives for 900 seconds, and only the Kimi CLI refreshes it, so if you wire it into Claude Code, then it 401s partway through a session that was working. Bearer wins on metered and x-api-key wins on the subscription.

Before You Point It Anywhere Real…

Big warning: Moonshot’s API platform privacy policy makes both your prompts and your code available for training, with no documented opt-out path. For EEA and UK users, the stated basis is legitimate interests rather than consent. It was last updated 2025-04-30, before K3 existed, and both the controlling entity and the servers are in Singapore.

Anthropic’s commercial and API traffic is excluded from training, with seven-day log retention. But Claude Code on a Pro or Max subscription is a consumer account, and consumer coding sessions have trained Claude by default since September 2025 (unless you turned it off). When it is on, retention stretches to five years.

So the thing deciding whether your code becomes training data is not which vendor you picked. It’s which billing path you are on. The metered API is the protective one on both sides of this comparison.

Nothing I did touched a real repository at first, and I recommend the same for your first experiment. Because of this, I created a fresh repo, no history, and a grep sweep for anything that could name a client.

The Hardest Part

The swap took 90 seconds. Benchmarking it took a day.

What you need is a hermetic task: one that lives entirely inside a codebase and never reaches outside it. No database, no network, no hardware. Run it twice and it scores the same both times. We want no external calls, because then it cannot really be part of a benchmark. (Rerunning it may change due to that external dependency.)

I’ve been talking about personal evals for months — it’s one of the reasons I built Morph. I also have it wired into my ~/.claude so that all my sessions are recorded (as well as emitting OpenTelemetry traces). So now is its chance to shine… I can just search to find traces in my own history.

But generally, the only eval that answers your question is one built out of your own work. Your tasks, your repositories, your definition of done. Not because public benchmarks are wrong, but because they were never measuring you.

I found two good candidates: an LLM routing policy matcher that I had to widen from matching on model alone to matching on model and the calling client. And second: a panorama renderer that was decoding the same JPEG once per tile instead of once per panorama, and that I wanted to make more efficient.

Scoring is the exit code of a pytest file the agent never sees, copied in after it stops. Nothing else counts, which means any implementation that behaves correctly passes, not only one that matches the original.

30 Runs, 30 Passes

Long story short: every configuration (Claude Code vs. Kimi Code vs. Anthropic APIs vs. Kimi K3 APIs) solved both tasks every time. On routing, every one of them produced a byte-identical code: same signature, same two-guard clauses, same order, matching the original commit.

I didn’t need it to be byte-identical, so that was not required. The scorer only checks behavior, so any passing implementation scores the same. They converged anyway.

So, they operated equivalently. Where they differ is in how much they write to get there, and most of that writing is not the diff at all: It is thinking, tool arguments, and edits that later get overwritten. For example, Opus completely rewrote the markdown documenting how rule matching works for the routing task. Every other configuration added a single line. Comment density inside the code is identical across all five, so this is not padding. It is a different idea about what finishing a task means — which is the limit of what this measures. Passing the same tests is not the same as being equally good. What separates the configurations here is cost and verbosity.

The Whole Thing in One Place

Two tasks, three runs each per setup. Cost is what those tokens bill at each provider's own rate sheet.

Subscription rows are what the tokens would have billed at metered rates. The subscription itself is flat cost, so the real marginal cost of those runs was zero. The same is true of the Opus row.

Think Rate, Not Volume

$4.20 for the control against $1.44 and $1.32 for the two metered K3 configurations, across all 30 runs. With the caveat that for most use cases, Claude Code and Kimi Code have subscriptions, so you never really see the cost difference.

Take the tasks separately, because the pair explains itself. On panorama, the code paths and token volumes are near-identical: 13 requests against 14, and the gap is 2.77x, of which 89% is the price sheet. Same work, same tokens, different rate card. On routing it is 24 requests against 15 and 11,743 output tokens against 3,205, so volume piles on rate and the gap opens to .45x.

They disagree because they are measuring different things: one task isolates rate, the other adds volume on top of it.

Caching is doing enormous work on both sides. Every configuration ran at 92-97% hit rate, a sevenfold saving on the preamble, identical on both providers.

Which leaves volume as the only question that matters. Two tasks a day on a subscription costs nothing extra. Two hundred a day and these list prices start deciding your infrastructure.

What's on the Wire

Most of what you pay for is not your code. Across all five setups, 63 to 81% of every input token is the harness describing itself: tool schemas, system prompt, standing instructions about how to behave. Around 20,000 tokens every request, before you type a character. The spread across five configurations is 11%, so this is not a Kimi number or a Claude number. There is something to be said about harnesses being tuned to their underlying models.

Back to the tool list, because I read that as an input-cost story and moved on. Wrong! Cutting the same run from 28 tools to 11 — same model and same task — cut output tokens 20.3% and thinking tokens 41.9%. More tools is a wider decision space, and the model reasons across all of it before doing anything. Pass rate was the only measure that held still, three of them either way.

So if you compare two harnesses on token counts, hold the tool payload constant or say that you did not.

Smaller and funnier: Before your task starts, Claude Code makes one call to name the session. At home it routes to Haiku for $0.000617. Pointed elsewhere, it runs on whatever flagship you configured, and the identical 525-token prompt costs $0.002175. Flagship rates for a string you will never read. The Kimi CLI never makes the call at all.

One note that cuts the other way: Those doubled bytes cost only 9% more tokens, because Moonshot’s tokenizer packs 4.0 characters per token against Anthropic’s 2.6.

One Provider Becomes Eight

I was watching OpenRouter and four inference providers every 15 minutes, starting the day before the weights went up.

For a day and a half, Moonshot was the only place you could get K3. Then seven other companies lit up routes within three hours of each other. That is the proliferation open weights buy you, and it moved faster than I expected.

What it does not buy you is identical choices. On the previous generation, K2.7-Code sits on 13 providers where input price spans 2.7x and quantization runs int4, fp4, fp8, and three reporting nothing at all. Same model name, different arithmetic underneath. The right to leave is real. What you leave to is not the same thing you left, and nothing in the model ID will tell you.

What I Would Do With This

You will never run the 1.4TB of weights on anything you own. That is not what ownership means here. Ownership is that eight operators stood the thing up by Tuesday, and that the harness you already use points at any of them with three environment variables.

As I’ve said before, the harness is where the power sits, not the weights. I did not expect to measure it this bluntly. 63 to 81% of every request is the harness talking about itself, and the largest cost effect in 30 runs came from a client-side default flipping on a hostname. Years of intricate setup, and a stranger walked in and used all of it on the first try. Both halves of that are the same fact.

None of this came from a leaderboard. It came from having a record of my own work with real outcomes attached, and something new to run it against. That record costs five environment variables and a collector, and it is the whole difference between an opinion about a model and a measurement of one. Whatever ships next month, you will want your own.

Swap it tonight. Watch the wire while you do. Swap back if you want, which is the entire point. Let me know what you discover and I’ll share it here.