crit 0.13.0

Goal: Get to a fast, working engine for a dense model as quickly as possible. Limit: A fixed candidate budget.

mtp4-nvfp4-mma runs Qwen3.8-27B in its original NVFP4 weights on one RTX 5090, with the model's full 262,144 tokens of context, in the as_quantized quality mode: Eval judges it against the checkpoint run the way it was quantized. One coding agent built it on crit's current main as a Python engine on Core: cuBLASLt for the prompt's 4-bit and 8-bit matrix multiplies, flash-linear-attention for the gated-delta layers, and its own Triton and CUDA kernels for attention and decode. Each round drafts 4 tokens with the model's own built-in draft layer and verifies them with rejection sampling on the GPU, so sampled requests draft too. Its own tensor-core NVFP4 GEMV reads the 4-bit weights at 1.2–1.4 TB/s, against 0.9–1.0 TB/s for cuBLASLt at the same shapes.

The last three columns are running totals from the run's first command (06:28 UTC) to each candidate's seal, and for candidate 5 to its selection, so its row is the whole run. The run spent its budget of three candidates on 1 to 3, then resumed for two more; development time leaves out the 3h 26m in between. Candidates 1 to 3 were built from crit and crit-eval alone; before planning 4 and 5 the agent had read the notes of earlier runs on this model. Agent tokens and API equivalent count the agent's messages in that span, from its Claude Code session transcript: Claude Opus 5.5 for every message, priced at Anthropic's API list prices ($4/MTok input, $20/MTok output, $8/MTok 1-hour cache writes, $0.20/MTok cache reads; 99% of the tokens are cache reads).

Measured with crit-eval (crit-eval-v0.2.1) at the full 262,144-token context. Decode is sampled at the model's own settings (temperature 1.0, top-p 0.95, top-k 20), the geometric mean over six prompts, from a chat question to a 14K-token agent session; Eval decodes each engine's own output.

• Long context: a prompt that fills all 262,144 tokens gets its first token after 144 s, then decodes at 110 tok/s. On prompts outside Eval's speed suite, in 8 languages and code, it decodes at 131 tok/s.
• Quality: the worst path uses 70% of the as_quantized drift budget. Greedy decoding reaches 194 tok/s, and drafting never changes the output: it is token for token what plain decoding produces.
• What is left: a verify pass takes 15.1 ms against the 12.5 ms its weight reads need, and a draft step 1.24 ms against 0.62 ms. Eval projects 251 tok/s if both reached their floors.

Goal: Show what each quality mode buys, on the same model, GPU and agent. Limit: Five candidates per run.

crit judges an engine's quality in one of two modes, chosen per run. preserve, the default, holds it to the checkpoint's own arithmetic with exact activations, within a 5% drift budget: the weights are used as stored, but the prompt's activations stay in 16 bits unless quality shows they can go lower. as_quantized holds it to the checkpoint run the way it was quantized, with 4-bit and 8-bit activations where the checkpoint declares them, within a 10% budget. We built Qwen3.8-27B on one RTX 5090 once in each mode: mtp4-nvfp4-mma from the case study above, and mtp-adaptive-width, which moves between 2 and 8 drafts per round: wider after a round that kept every draft, narrower after one that kept fewer.

• Decode is a tie. A decode pass reads every weight once, and both engines read the same stored 4-bit and 8-bit weights, so the quality mode barely moves it.
• Prefill is where the modes split. A long prompt is compute-bound. as_quantized may feed 4-bit activations to the GPU's 4-bit tensor cores; preserve keeps the 4-bit layers' activations in 16 bits and widens their weights to match. Eval prices each mode's ceiling accordingly: preserve reaches 77% of its 4,676 tok/s, as_quantized 46% of its 17,841 tok/s.
• Quality follows the reference. preserve stays within 0.4% of the exact model's perplexity. as_quantized is measured against the quantized checkpoint, whose own run-to-run spread on long prompts is wide, so Eval judges each of its texts against twice that spread.
• Both runs were honest about failures. Two of preserve's five candidates failed before Eval: one read log-probabilities at the wrong stride, and one kept its first engine in memory when Eval reopened it at the full context. Each fix became the next candidate.

Pick preserve when answers must match the original model; pick as_quantized when prompt throughput matters more, as with agents that send long prompts.

Measured like the case study above. The preserve run was built from crit and crit-eval alone, from its first candidate to its last.

Core is the plumbing every engine needs, distilled by studying multiple inference engines.

We took apart how each engine handles requests, memory, text and model files, kept the base features they all rebuild, and turned them into one native SDK. Your engine keeps what makes it fast: the model, its kernels, its layouts and its execution strategy.

Try the Core + Inference walkthrough.

Inference is a curated collection of high-performance strategies.

We studied bespoke inference engines and what makes them fast and turned every strategy that holds beyond one model or GPU into an entry. Your agent picks the entries that match its engine's bottleneck and builds them into the engine on Core.

Examples:

All entries are in the full guide, or run crit inference search <query>.

Every entry has the same layout:

What gets in: contribution criteria

Kernels is a register of kernel collections.

crit hosts no compute kernels. Your agent takes each operation's kernel from an established provider, or writes its own when none fits, then measures it against its floor in Eval's kernel profile and against the best other provider for the same operation.

Kernel workbench

When the agent writes a kernel, or adapts a provider's, crit kernels new gives it a workbench folder with the checkpoint's own shapes, an FP64 reference and a yardstick such as cuBLAS. crit eval workbench check then compares the kernel with the reference on every shape and requires the same bits on a repeat, and bench times it against the yardstick and the GPU's floor. See the workbench.

crit-eval is a speed and quality measurement tool to compare inference engines honestly.

There is no shared way to measure a bespoke inference engine.

Different bespoke inference engines report vastly different types of numbers. An honest comparison requires downloading and running the engines yourself. crit-eval finally makes apples-to-apples comparison possible.

Physics-based evaluation lets you see what performance is left on the table.

crit-eval measures your GPU's memory bandwidth and matrix-multiply rates and turns them into a limit for each score. Its evaluation report can tell your coding agent how much performance can still be squeezed out of your inference engine.

Speed that can't hide a quality loss.

Many inference engines push decode and prefill speed with no regard for quality. crit-eval treats quality as a first-class KPI. It measures not only inference speed but also quality regressions.

A hardware report rounds out the evaluation.

The same model on the same inference engine but on two different machines can also lead to different inference speeds, even if the GPU is the same. We make this transparent by also reporting the runner's hardware configuration.

crit serve loads a sealed candidate, keeps it resident and answers OpenAI-compatible chat requests, including streaming, tool calls, stop strings and structured output. The same engine also speaks the Anthropic Messages, OpenAI Responses and Ollama APIs, so Claude Code, Codex and local chat UIs need only a base URL, and GET /metrics serves Prometheus metrics. Eval measures every candidate through this same server, so the numbers you read are the numbers you get. Engines implement the adapter contract.

One crit command drives everything: crit run keeps the run's records (plans, candidates, measurements, selection, cost), crit eval runs the common suite, crit serve serves an engine, crit pull installs a crit-hub engine, and crit kernels, crit models and crit inference reach the other parts. Records are written only through it, so each plan, candidate, Eval and seal is checked as it is recorded.

crit-hub collects bespoke inference engines in one repository to make discovery super easy. Each engine pins its runnable source, the exact checkpoint it was built on, what was measured, its known limits and a link to the repository of whoever contributed it. Use them as jump-off point for further optimization.

crit pull crit-labs/nvfp4-gemv-int8kv installs one: it checks the card's requirements against your GPU and CUDA, fetches the source at its pinned commit, builds it, fetches and checks the checkpoint, and then crit serve --engine crit-labs/nvfp4-gemv-int8kv serves it (Installed engines).