inferencekit is a plugin for Claude Code, Codex and OpenCode, plus a command-line tool (CLI). Give your agent a model, hardware and a goal. It finds, benchmarks and tunes supported inference engines for your workload. You can stop with the best tested configuration or go further with custom optimizations, including GPU kernels.

How it works

inferencekit gives your agent the playbook and the tools of a senior inference engineer. On every run it works the problem the same way.

1. Understand the task. The agent first asks about your goal, deadline, budget and how much the results may change, reusing answers you already gave. It identifies the model, hardware and real inputs, then estimates memory, compute and launch costs per request. Text, image, audio and video use their own metrics and quality checks.
2. Find and tune an engine. It researches current implementations, then tunes and benchmarks supported engines (vLLM, SGLang, llama.cpp, TensorRT-LLM and others) for your workload on your hardware. The fastest tested configuration that meets your quality requirements can be the result, or the tuned baseline for further work.
3. Find the bottleneck. It selects the profiler for your hardware: Nsight on NVIDIA, ROCm tools on AMD, or Instruments and Metal on Apple. It measures the critical path and investigates memory, compute and launch costs, checking current vendor docs when support is uncertain.
4. Investigate and implement, if requested. Search current code, papers, compiler docs and engineering talks; inspect dispatch, IR and assembly when useful. Choose graph execution, fusion, matrix kernels, packing, state reuse or other techniques from measured bottlenecks. Independent implementations can run in parallel; integration and shared-device measurements stay isolated.
5. Prove every step. Every change gets a fast correctness check and a timed screen; milestones get the full proof: interleaved rounds against the tuned baseline, the anti-cheat gates and quality against a reference model. Changes that miss the agreed performance or quality requirements are reverted.
6. Ship it. You get the fastest proven configuration, its launch command, and a report with the commands that reproduce every number.

When it asks you

The agent works on its own and stops only when it needs you:

• Missing details. If you didn't say which model, GPU or workload, it asks before it starts.
• Your preferences. Choose exactly the same results, small differences with the same quality, or speed with an agreed quality tradeoff. Set your goal, spending, deadline and check-ins; it reuses answers you already gave.
• Model changes. Permission to simplify the model is separate from permission to use stronger compression. Plans are optional: ask to see one if you want it.
• The budget cap. When the run reaches the cap you set, it stops and asks.
• Things only you can give. For example, permission to read the GPU's profiler counters, or a Hugging Face token for a gated model.

Everything else runs without asking. For a hands-off run, approve a budget with a cap up front (non_interactive: true and budget.approved: true in the intake file); the agent then writes its questions and the choices it made to QUESTIONS.md instead of waiting for you.

Results

Pushing speed on AMD: with quality secondary, MOSS-Image-SFT reached 2.46× faster image generation than the tested tuned vLLM-Omni baseline on a 16 GB RX 9070 XT.‡

Single-user decode for the language models; one clip at a time for video; single-user speech generation for Drama3.

Engine	GPU	Speedup	Agent	Agent time	Tokens	Cost
Gemma 4 26B-A4B (NVFP4)	RTX 5090	1.33× vs tuned vLLM	Claude Opus 5.5 (high)	19 h	630M	$199
LTX-2.5 text-to-video	RTX PRO 6000	1.25× vs ComfyUI	Claude Opus 5.5 (high)	3.6 h	47M	$18
Qwen3-8B (FP8)*	RTX 5090	1.25× vs vLLM	Claude Opus 5.5 (xhigh)	about 10 h	about 240M	about $125
Drama3 speech†	Apple M4 (16 GB)	1.112× vs tuned MLX-Audio	GPT-6.1 Sol (high)	~70 h elapsed	~2.07B	~$475 standard-rate equivalent
MOSS-Image-SFT (FP8)‡	RX 9070 XT (16 GB)	2.46× vs tuned vLLM-Omni	GPT-6.1 Terra (high)	~19.6 h elapsed	~1.80B	~$206 standard-rate equivalent

_{* Against vLLM with its default settings; neither side uses speculative decoding. † Drama3 used identical quantized weights and matched PCM audio, codes, chunks and completion in 144 fresh output pairs. Four paired timing rounds gave a 95% CI of 1.059×–1.156×. Comparison details.}

_{‡ Lossy speed-first experiment: FP8/24 steps vs BF16/50 steps; paired 95% CI 2.453–2.465×, conservative lower bound 2.453×. Outputs, timing and accounting.}

Time and tokens include setup, benchmarking, optimization, validation and reporting. Costs are API list-price estimates; most tokens are cached conversation reads. Drama3’s estimates cover direct agent work (accounting). How each engine was built and measured: inferencekit-engines.

Speed you can trust

Fast is easy. Fast and still correct is the hard part, and it's the part agents skip.

Agents are good at finding speed. They're also good at finding fake speed: a cached answer, a shorter reply, a different video. inferencekit keeps only the speed that's real.

Without these checks, one agent thought its video engine was 2% faster, but it was rendering a completely different video. Another was sure it had a 1.12× speedup; measured again on a clean machine, its engine was 5% slower than where it started.

A speedup counts only when both engines pass every check that applies.

A way to game a benchmark	What catches it
Answer repeated prompts from a cache	caching: a replayed prompt may not be over 10% faster; a prompt changed by one character may not get the old answer
Report more tokens than were made	token_count: the reported count must match the streamed tokens and the re-tokenized text
Stop early or cut answers short	output_length: with end-of-text ignored, every answer must be exactly the requested length
Drop rare tokens from the vocabulary	rare_tokens: random CJK text, emoji, long numbers and code names must come back exactly
Ignore sampling settings or fake probabilities	hidden_probes: temperature must change answers; probabilities must be valid
Use another chat template so prompts get shorter	Render check: every prompt must become the same number of tokens as on the baseline
Speculative decoding that changes answers	spec_exact: greedy answers must match the same engine with speculation off
Lower quality for speed	Quality vs a reference, GSM8K, per-step video check
Numbers that only hold on the agent's machine	Every run writes manifest.json with each engine's install and launch commands, so anyone can re-run it on a clean machine, on prompts sealed with inferencekit prompts seal that the agent never saw

Each check ends PASS, FAIL, UNPROVEN (not enough evidence) or NOT RUN. A missing check is never a pass.

Features

• 9 skills, 4 commands, 5 read-only subagents and an MCP server, the same in every supported agent.
• Speed-of-light ceilings: measured memory bandwidth (ceiling bw) and exact bytes per token read from the checkpoint's safetensors headers (roofline), including NVFP4, FP8 and mixture-of-experts models.
• Engine triage and tuning: vLLM, SGLang, llama.cpp, TensorRT-LLM, ComfyUI, Diffusers and others. Only settings that the engine's own logs show in effect are counted.
• Kernel work: profiling routed by hardware, CUDA/HIP, Triton and Metal kernels, and torch.compile. The included engine skeleton targets NVIDIA CUDA.
• One timer for every engine: interleaved rounds in a balanced order, medians with 95% confidence intervals.
• Anti-cheat gates: caching, token counts, output length, rare tokens, sampling probes, prompt rendering, speculative-decoding exactness.
• Quality checks: reference quality and GSM8K for language models; latent and final-clip checks for video. Images and speech use client timing and structural checks plus an external semantic quality protocol; see media benchmarks.
• Budget control: spending authorized up front, with caps in hardware hours, dollars and elapsed time. No repeated plan approval inside the agreed scope.
• Reproducible output: a ranked table labelled PASS, FAIL, UNPROVEN or NOT RUN, a launch command, a review page and the commands that reproduce every number.

Tools behind each step

Step	Tools
Size the problem	intake, doctor, ceiling bw, roofline
Start from the best engine	bench rounds, gates run, quality
Find the bottleneck	doctor routes profiling; Nsight commands on NVIDIA, vendor tools on AMD/Apple
Pull the right levers, prove every step	correctness, bench rounds, gates run, quality
Ship it	ledger, review, report

Install

Requirements: uv and Python 3.10 or newer. Install the engine runtime for your hardware. NVIDIA capture commands require CUDA and Nsight; AMD and Apple profiling uses vendor tools with separate evidence files. See supported hardware.

Claude Code

claude plugin marketplace add kittenml/inferencekit
claude plugin install inferencekit@inferencekit

Codex

codex plugin marketplace add kittenml/inferencekit
codex plugin add inferencekit@inferencekit

OpenCode

git clone --depth 1 https://github.com/kittenml/inferencekit ~/.local/share/inferencekit
python3 ~/.local/share/inferencekit/plugin/opencode/install.py

Agent	Commands	Check the install
Claude Code	/inferencekit:optimize-inference, /inferencekit:bench, /inferencekit:profile, /inferencekit:perf-report	claude plugin list; /mcp lists the inferencekit server
Codex	same stages; subagent files are followed one at a time where Codex has no subagents	codex plugin list; codex mcp list
OpenCode	/optimize-inference, /bench, /profile, /perf-report, and each skill as /<skill-name>	opencode mcp list

Updating, removing, per-project installs and OpenCode's options: docs/install.md.

Try it

/inferencekit:optimize-inference Qwen/Qwen3-8B-FP8 on 1x RTX 5090, fastest single-user decode, by 6h

To tune existing engines without custom kernels:

/inferencekit:optimize-inference Tune vLLM and SGLang for my model and GPU using their built-in settings only; do not write custom kernels.

In OpenCode the command is /optimize-inference. It asks only for missing preferences and spending authorization; a plan is optional. For a repeatable setup, start from an intake file: inferencekit intake template --out intake.yaml, then /inferencekit:optimize-inference intake.yaml. For unattended runs, set non_interactive: true and budget.approved: true with a cap.

More things to ask for: docs/examples.md.

CLI and MCP server

uv tool install "inferencekit-cli==0.1.0"
inferencekit doctor                                         # GPU, driver, CUDA, profiler access
inferencekit gates run --base-url http://127.0.0.1:8000     # anti-cheat gates on any OpenAI-compatible server
uvx --from 'inferencekit-cli[mcp]==0.1.0' inferencekit mcp serve   # MCP server (stdio) for any MCP client

Every CLI command is also an MCP tool with the same arguments. Text benchmarks, gates and quality checks talk HTTP to OpenAI-compatible servers. Image and streamed speech measurements use the media adapter contracts.

Supported hardware and agents

	Status	Run end to end
NVIDIA GPUs (CUDA)	Full tool set	Yes: RTX 5090, RTX PRO 6000
AMD ROCm GPUs, Apple silicon	Hardware discovery and vendor profiling guidance; HTTP benchmarks; vendor bandwidth/captures collected externally	RX 9070 XT and Apple M4 case studies above
x86 CPUs	Skills and HTTP benchmarks; platform profiling collected externally	No
ARM CPUs	Not covered	No
Language models	Benchmarks, gates, GSM8K, quality vs a reference	Yes
Text-to-video	Per-clip timing, per-step check, final-clip PSNR	Yes
Text-to-image	Client-timed decoded images and structural checks; external quality proof required	No
Speech models	Client-timed streamed PCM, first-signal latency, RTF and underruns; external quality proof required	No
Claude Code	Supported	Yes
Codex, OpenCode	Install paths and CI install checks	No

Profiles and logs can contain environment variables, host names and paths: scan them before you share them.

Roadmap

• More hardware: H100, H200, B200, RTX 4090, Intel GPUs and Jetson, with the hardware tools (doctor, ceiling bw, profiling) for each.
• More models and modalities: speech (TTS and STT), image, more video models.
• Multi-user serving: published results for many users at once (load sweeps), not just one user at a time.
• OpenCode end to end.

Contributing

We welcome bug reports, engine settings and new skills. See CONTRIBUTING.md. Develop with uv sync --all-extras, then uv run --all-extras pytest -q (no GPU needed). Report security problems privately: SECURITY.md.

License

Apache-2.0: LICENSE and NOTICE. Copyright 2026 Stellon Labs. inferencekit is an independent project, not affiliated with NVIDIA, any engine project or any model publisher. Product and model names belong to their owners. Every result holds for the GPU, driver, engine version, model and workload it was measured on; change any of them and the numbers can change. The software comes with no warranty, as the Apache-2.0 licence states, and you are responsible for the cost of any GPU time you use with it.