carmen-kernels 0.1.0

An AI writes GPU kernels for Apple Silicon. A judge it can't fool decides what's real.
The winners go into a real model.

Why

AI can write GPU kernels now, and the papers are full of "3× faster". A lot of it doesn't survive a second look:

• A one-shape allclose check, the KernelBench standard, misses 78.6% of precision bugs (Measuring the Checker).
• A speedup reported as 1.43× was 0.88× on inputs the model hadn't seen (KernelBench-Verified).
• On Apple Metal, 30% of LLM "wins" broke on a config nobody tested (Gaming Without an Attacker).

Writing kernels is cheap now. Trusting them is the hard part. I built carmen to learn how kernels work and to answer one question honestly: can an AI make a model on my Mac faster, with proof?

What happened

On a real model, yes, by a little. carmen's AI wrote two fused 4-bit kernels that replace 8 MLX kernels with 2 in every layer of Qwen 2.5 0.5B. Decode got 3% faster, with identical output.

_{30 paired turns, 95% bootstrap range. carmen e2e --modes stock,compile,mlp+compile --repeats 30 --gen 256}

On single kernels, it wins where MLX runs several kernels and ties where Apple wrote one.

The judge caught all 75 bugs planted in seven kernels. A KernelBench-style check let 35 of them through. How the judge works →

What I learned

• AI kernels don't beat hand-tuned code. They beat code nobody fused.
• A kernel that wins alone can lose inside a model. One scored 1.15× in the judge and made Qwen 18% slower: it was timed at the wrong shapes, against a weak baseline, one launch at a time. Now carmen run <op> --for qwen0.5b judges at the model's real shapes and number format, against mx.compile, and checks the winner inside the model at the end (ideas from SoL-Pi).
• Measuring honestly took more work than writing kernels. Running versions one after another, instead of taking turns, produced two "wins" (+40% prefill, 1.09× decode) that weren't real. A browser in the background swung speed by ±27%.
• Small models on a Mac are launch-bound. Qwen 0.5B reads 0.28 GB of weights per word, so the M4 could do ~346 tok/s. Stock MLX reaches 51%. The rest is hundreds of tiny kernel launches per word. Bigger fused kernels are the way forward.

New to kernels? How it all works explains LLMs, GPUs and kernels from scratch.

How it works

• Carmy (the model) writes the kernel. It's trusted to be smart and never trusted to grade. It can only return text.
• The judge is plain Python. It checks every kernel against a float64 answer key on hundreds of inputs, including secret ones drawn fresh every round, races it against MLX, and says where a kernel is wrong, not just that it is.
• The loop: three drafts in parallel, the best becomes the champion, the next round attacks one bottleneck. Lessons are kept only when the judge proved them.

flowchart LR
    C["Carmy × 3<br/>parallel drafts"] -->|kernel text| J{{"judge<br/>compile · float64 · invariants<br/>fresh fuzz · hidden draw · timing"}}
    J -->|"located error<br/>+ speed profile"| C
    J -->|verified & faster| K(["champion"])
    K -->|"one structural change"| C
    J -->|"proven lessons"| P[("playbook")]
    P --> C
    J -.->|"hidden results<br/>(never shown to Carmy)"| R["report"]

Try it

On an Apple Silicon Mac, with uv:

uv tool install "carmen-kernels[metal,models]"   # or: pip install "carmen-kernels[metal,models]"
export ANTHROPIC_API_KEY=...        # or put it in a .env file where you run carmen
carmen                              # the app
carmen speedup                      # Qwen 2.5 0.5B, stock vs carmen's kernels, on your Mac (no API key)

From source: git clone https://github.com/utsav1033/carmen, then pip install -e '.[metal,models,dev]', and put your key in .env (see .env.example).

Built on the shoulders of

Measuring the Checker · Gaming Without an Attacker · Metal-Sci · Hacker-Fixer Loops · Test-Input Generation for Tensor Programs · Reward Hacking in Self-Improving Code Agents · Meta-Harness · Auditing Harness Tampering · SAGE · Prime Agent · KernelBench-Verified · Contract-Grade Verifier · Building a C compiler with parallel Claudes

_{Named after Carmen "Carmy" Berzatto. Yes, chef. Banner set in Space Grotesk and JetBrains Mono (SIL Open Font License).}