crit 0.10.0

Summary

crit is a toolkit for optimizing or building inference engines tailored to your hardware and desired model.

Quick start

Give one of these prompts to your coding agent and specify your target LLM or target engine:

Use github.com/crit-labs/crit to build an optimized inference engine for <huggingface_url>.

Use github.com/crit-labs/crit to optimize my own inference engine in <engine_reference>.

What crit solves

Less time building an engine. More performance from your hardware.

The boring stuff has already been built ⏱️

Requests, memory management and cleanup eat up development time before optimization even starts. crit supplies a clean foundation so you can get to the work that makes your engine fast.

Curated high-impact inference strategies 📊

Don't rebuild inference strategies from scratch. We studied established inference engines and distilled their core speedup strategies into reusable code, so you can apply what works to your model and hardware quickly.

Physics-based evaluation suite 🔬

A benchmark score alone doesn't tell you what's still left on the table. crit measures your hardware's limits and shows how close your engine gets, with quality and correctness checks along the way.

Case Studies

We gave Claude Opus 5.5 a target LLM along with the crit repository and let it cook autonomously.

How crit works

1. The workflow

An agent, or you, builds the engine as a series of candidates. Each candidate is a complete, servable engine, measured the same way, and each next one is chosen from the last one's numbers.

flowchart LR
    recon["Reconnaissance<br/>model, hardware,<br/>crit guides"] --> limits["crit eval<br/>the hardware's limits"]
    limits --> c1["Candidate 1<br/>a working engine"]
    c1 --> measure["crit eval<br/>scores vs limits"]
    measure --> decide{"Decision<br/>algorithm"}
    decide -->|"build: the biggest<br/>gap first"| next["Candidate 2 … X"]
    next --> measure
    decide -->|"stop: nothing left<br/>beyond the noise"| final["Final candidate,<br/>served by crit serve"]

1. Reconnaissance. Gather everything the engine depends on before writing code: the user's model, hardware, workload, quality and context needs; the model's files (crit run ... recon reads its config, tokenizer, chat template, sampling defaults and draft layers); its model card and reference implementation; crit's agent instructions, guides and docs; and any crit-hub engine to start from.
2. Learn the hardware's limits. crit eval probe measures the GPU, and Eval turns the measurements into the fastest decode, prefill and time to first token this model could reach on it.
3. Build candidate 1: a working engine on Core, measured by Eval. It sets the baseline.
4. Decide what the next candidate changes. The decision algorithm reads the last Eval and names the KPI furthest from its limit, the levers that close the gap, and a forecast crit checks after the build.
5. Build candidate 2, 3, …, each measured by Eval and compared with its forecast.
6. Stop at the final candidate once neither decode nor prefill can still move beyond its noise. It is selected, sealed with its source and evidence, and served through an OpenAI-compatible API.

Every step is recorded through the CLI: each candidate freezes its source by hash, each measurement points at that exact package, and the run's cost is counted from the agent's own transcripts (run records).

2. crit Eval → Full guide

Eval runs one fixed suite against every engine: the same prompts, output lengths, stop handling and quality references, through the same crit serve API a client uses. Results from different runs, agents and machines therefore compare.

Limits, not just tok/s. "252 tok/s" alone does not say whether an engine is good or how much is left. Eval measures the GPU's memory bandwidth and matrix-multiply rates per weight format, and from the model's files computes what each phase costs at best: decode reads the weights (and the KV cache) once per pass, so bandwidth sets its floor; prefill is arithmetic, so the GEMM rates set its ceiling. Every score is reported as a fraction of its limit, with its run-to-run noise, and the gaps say which term (model pass, draft steps, draft width, drafter acceptance, prefill, first token) would give how much at its floor. That is what makes "speed of light" a number an agent can aim at.

Gates decide whether a result counts at all: output repeats, the same greedy requests give the same tokens after a long request history, follow-up turns answer the same from a prompt cache, facts at the start, middle and end of long prompts are retrieved, speculative output equals plain greedy when required, sampled tokens follow the model's distribution, and quality stays inside the budget.

Quality against the checkpoint you chose. Eval measures drift against the checkpoint as stored, for example an NVFP4 model, scored once in FP32 with the model's official code, not against its full-precision original. The user chose that checkpoint and its quantization loss; the engine cannot change it. Comparing with full precision would mix the publisher's loss with the engine's own, so a careless engine could hide behind the quantization and a careful one would be charged for it. What Eval reports is exactly what the engine adds, and the budget (perplexity ratio ≤ 1.05, mean KL ≤ 0.05, p99 KL ≤ 0.5 by default) is there to be spent on speed: lower drift than needed is not a gain.

3. The decision algorithm → Plan records

Before any code for the next candidate, the agent records a plan, and crit checks it against the last Eval, which it freezes:

1. Rank. crit computes the priorities from the fractions of the limits: of decode and prefill, the one further from its limit comes first, both when they are within noise of each other, and time to first token never leads alone. Where Eval gives no limit, crit computes one with Eval's cost model.
2. Compare strategies. The plan sets the current strategy's ceiling against the best alternative's, for example plain decoding against drafting with the model's own draft layer, the alternative held to what its drafter is measured to accept.
3. Name levers and a forecast. Each lever states its projected change per KPI; the plan gives a low, expected and high forecast.
4. Refuse bad plans. crit rejects a plan that stops while a KPI is still needed, leaves a gap above the noise unanswered, forecasts no gain beyond the noise or only chases time to first token.
5. Check the outcome. After Eval, crit sets the forecast next to the measured result and records each lever as built, pruned or deferred.

The run stops only when neither decode nor prefill can still move beyond its noise. The agent cannot drift, overclaim or quit early, and every candidate has a recorded reason to exist.

4. crit Core → Full guide

We studied about 30 inference engines and kept what every one of them has to build before it can be fast: request scheduling, memory admission and tracking, output delivery, stop strings, sampling, state and prefix caches, model files and config, structured output, and process cleanup after a crash. That minimum became Core, a native SDK with a Python package. Your engine owns everything that makes it specific (the model, its kernels, its tensor layouts and its execution strategy) and calls Core for the plumbing. Try the Core + Inference walkthrough.

5. crit Inference → Full guide

We studied about 50 inference engines, from general-purpose runtimes to engines tuned for one model on one GPU, and extracted what makes each of them fast into separate techniques: speculative decoding with draft layers, trees and device-side rounds, expert caches and host-resident experts, quantized projections and KV caches, paged and partitioned attention, prefix checkpoints, CUDA graph reuse and more, 36 entries in all. Each says when it helps, what is tested and what the engine supplies, so an agent can adopt it quickly instead of rediscovering it.

The collection keeps growing the same way. The engines stay pinned at exact revisions, and salvage passes read what changed in them down to individual pull requests, merged or not. A finding enters crit only with its mechanism and evidence, a crit consumer and the gap it closes, the smallest useful adaptation and the three contribution criteria: value that can be shown, nothing tuned to one model or GPU, useful across architectures and GPUs. Upstream numbers are never carried over as crit's, and rejected findings are kept with their reasons.

More parts

Miscellaneous

Documentation