crit 0.11.0

Goal: Get to a fast, working engine for a dense model as quickly as possible. Limit: A fixed candidate budget.

mtp7-device-confidence runs Qwen3.8-27B in its original NVFP4 weights on one RTX 5090, with the model's full 262,144 tokens of context. One coding agent built it from scratch on crit 0.5.0, writing its own kernels on Core without using another inference engine as a reference. It drafts up to 7 tokens per round with the model's own built-in draft layer and stops once the drafter's confidence in its latest draft falls below 0.5; the GPU makes that decision inside one CUDA graph per round, so the host never waits mid-round.

The last three columns are running totals from the run's first command (01:16:25 UTC) to each candidate's seal, and for candidate 7 to its selection, so its row is the whole run. Agent tokens and API equivalent count the agent's messages in that span, from its Claude Code session transcript: Claude Opus 5.5 for every message, priced at Anthropic's API list prices ($4/MTok input, $20/MTok output, $8/MTok 1-hour cache writes, $0.20/MTok cache reads; 99% of the tokens are cache reads). The agent writes the next candidate while the previous one is measured, so most of candidate 7 was built before candidate 6 was sealed: from there to its selection added only 16 minutes and 1.1M tokens.

Measured with crit Eval (v6) at the full 262,144-token context. Decode is greedy (temperature 0), the geometric mean over six prompts, from a chat question to a 14K-token agent session; Eval decodes each engine's own output, so candidates 3 and 4 have the same round time but accept different numbers of drafts.

• Long context: a prompt that fills all 262,144 tokens gets its first token after 79 s, then decodes at 244 tok/s. On prompts outside Eval's speed suite, in 8 languages and code, it decodes at 163 tok/s.
• Quality: perplexity stays within 0.1% of the reference, 5.8% of the drift budget, and drafting never changes the output: it is token for token what plain decoding produces.
• Sampled requests: this engine drafts only on greedy requests. With sampling (temperature 0.6 to 1.0, including the defaults crit serve applies) it decodes one token per pass, at 59–60 tok/s on the same six cases, measured outside Eval. crit Eval v7 measures sampled decode too.

TBD. We are rebuilding this case study with crit 0.5.0; its engine and figures follow shortly.

TBD. We are rebuilding this case study with crit 0.5.0; its engine and figures follow shortly.

TBD. We are rebuilding this case study with crit 0.5.0; its engine and figures follow shortly.

Core is the plumbing every engine needs, distilled from about 30 bespoke inference engines.

We took apart how each engine handles requests, memory, text and model files, kept the base features they all rebuild, and turned them into one native SDK. Your engine keeps what makes it fast: the model, its kernels, its layouts and its execution strategy.

Try the Core + Inference walkthrough.

Inference is our register of speed-up strategies, curated from about 50 inference engines.

We studied what makes each engine fast and turned every strategy that holds beyond one model or GPU into an entry. Your agent picks the entries that match its engine's bottleneck and builds them into the engine on Core.

All 37 entries are in the full guide, or run crit inference search <query>.

Every entry has the same layout

What gets in: the contribution criteria

crit-eval is a speed and quality measurement tool to compare inference engines honestly.

There is no shared way to measure a bespoke inference engine.

Different bespoke inference engines report vastly different types of numbers. An honest comparison requires downloading and running the engines yourself. crit-eval finally makes apples-to-apples comparison possible.

Physics-based evaluation lets you see what performance is left on the table.

crit-eval measures your GPU's memory bandwidth and matrix-multiply rates and turns them into a limit for each score. Its evaluation report can tell your coding agent how much performance can still be squeezed out of your inference engine.

Speed that can't hide a quality loss.

Many inference engines push decode and prefill speed with no regard for quality. crit-eval treats quality as a first-class KPI. It measures not only inference speed but also quality regressions.

A hardware report rounds out the evaluation.

The same model on the same inference engine but on two different machines can also lead to different inference speeds, even if the GPU is the same. We make this transparent by also reporting the runner's hardware configuration.

Kernels helps you find and assess existing kernels for your data format and hardware, including the Hugging Face Kernel Hub. It checks declared compatibility, shows mismatches and gaps, and retains chosen artifacts with exact versions and license notices. Your engine then measures whether the kernel helps.

Models produces transformed weights when a run allows them, starting with post-training quantization to FP8 or NVFP4 (round-to-nearest or GPTQ). It exports a checkpoint engines can read, runs it in your engine through Eval against the original model, and records the recipe, calibration data, seed, file hashes and cost.

crit serve loads a sealed candidate, keeps it resident and answers OpenAI-compatible chat requests, including streaming, tool calls, stop strings and structured output. The same engine also speaks the Anthropic Messages, OpenAI Responses and Ollama APIs, so Claude Code, Codex and local chat UIs need only a base URL, and GET /metrics serves Prometheus metrics. Eval measures every candidate through this same server, so the numbers you read are the numbers you get. Engines implement the adapter contract.

One crit command drives everything: crit run keeps the run's records (plans, candidates, measurements, selection, cost), crit eval runs the common suite, crit serve serves an engine, and crit kernels, crit models and crit inference reach the other parts. Records are written only through it, so each plan, candidate, Eval and seal is checked as it is recorded.

crit-hub collects bespoke inference engines in one repository to make discovery super easy. Each engine pins its runnable source, the exact checkpoint it was built on, what was measured, its known limits and a link to the repository of whoever contributed it. Use them as jump-off point for further optimization.