arbi-serve

Why arbi-serve

Long-context inference that's actually viable on consumer GPUs: full model context, virtually lossless quality, concurrency built for real traffic — not a demo. Charts backing each of these land in Benchmark and value below as they're published.

The wedge between datacenter inference (vLLM, SGLang) and edge tools (Ollama): self-hosted LLM serving with datacenter-grade concurrency and VRAM efficiency, one-command-easy to run, monitor, and configure. Built for businesses that want real production characteristics on their own GPUs — RTX included — without a datacenter budget, and without depending on someone else's cloud to run their own models. Distributed inference, in the sense that matters: the power to run this yourselves stays with you, not a central provider.

It's a serving engine for a documented, explicit set of model architectures (Feature status below), not a generic fallback for every checkpoint on the Hub.

Quickstart

Needs uv, one NVIDIA GPU, Python 3.12+:

uvx arbi-serve run Qwen/Qwen3.5-0.8B

Installs itself, boots, drops you into a chat prompt. Everything below is detail you don't need yet.

Details

arbi-serve is the native serving engine for Turbo Attention (tkv) — the 2–8-bit asymmetric KV-cache codec (WHT rotation + calibrated per-channel scaling, dequant fused inline into both prefill and decode). Calibration, per-layer bit allocation, hot-swap, profiling, and memory accounting are first-class rather than bolted on. Where SOTA kernels fit (flash-attn, CuTeDSL, FlashInfer, Triton, vendor kernels) the engine adopts them; where the serving path needs tighter control — fp8 paged-KV attention on consumer RTX cards the standard kernels refuse to run, hybrid GDN/Mamba/ShortConv state, full-duplex audio — it carries its own.

• Calibrated, near-lossless quantization. The TKV codec (2–8 bit, WHT rotation + per-channel calibration bundles) is built on our own attention backend (turbo-attn) specifically to hold quality at a fraction of the KV-cache memory — not a generic INT8 cast.
• VRAM efficiency and management. A deterministic budget across every pool (weights, activations, graphs, KV, LoRA, recurrent state, codec scratch — docs/memory-accounting.md), per-component attribution across ~35 gauges held to a checkable invariant, and a boot that fails loud with a component-level reason instead of OOMing under load.
• Concurrency and iteration, live. Continuous batching and CUDA graph capture for throughput; swap the attention backend, reload the model, or reload calibration at runtime with zero restart; compare numerical paths head-to-head; switch between resident models by naming a different one in the standard API. Flag correctness is CI-governed.
• Easy to monitor and configure. Prometheus at /metrics, an admin console, and structured boot/flag logging — the compile/cache paths and memory budget stay inspectable instead of opaque.
• Real-time audio. Full-duplex speech-to-speech primitives include VAD, barge-in, turn-taking, and TTS output for the supported audio models.
• Fail loud, never subtly wrong. Unsupported model/backend/quant combinations raise with a precise boundary instead of falling through to a slow or silently incorrect path.

The trade is deliberate: this is an explicit model and hardware surface, not a generic Transformers fallback or a promise of feature parity with other servers. See the feature analysis for boundaries.

arbi-serve consumes tkv through its public API only: tkv.runtime.*, tkv.codec, and tkv.core. It never imports tkv.integrations.* or private/internal turbo-attn modules.

Install

uvx arbi-serve run <model> (above) needs nothing installed ahead of time. Into your own project or venv instead:

uv pip install arbi-serve                 # lightweight generated Python client
uv pip install "arbi-serve[server]"       # standard NVIDIA inference server
uv pip install "arbi-serve[ui]"           # remote admin console + client
uv pip install "arbi-serve[server,ui]"    # local server + admin console
uv pip install "arbi-serve[fa]"           # no-op extra (kept for resolve compat)
uv pip install "arbi-serve[exl3]"         # compatibility no-op; see EXL3 note below
uv pip install "arbi-serve[dev]"          # + tests + lint tooling

(plain pip install works identically if you don't have uv)

One-command bare-metal run

arbi-serve run <model> — see Quickstart — installs arbi-serve[server] with uv (falling back to pip) if it isn't there yet, boots, waits for readiness, and drops into a chat REPL. Run it again with a different model while the first is still up and it hot-swaps the new model into the same resident server (POST /v1/admin/model, the same fast park/wake path used for A/B iteration) instead of booting a second process — pass --new to force a separate instance on another --port. Extra flags for a fresh boot go after -- (e.g. arbi-serve run demo -- --tp-size 2). Add --console to also boot the admin console (installs [ui,otlp] too, and leaves OTLP push at the engine's own default instead of forcing it off).

Companion commands:

arbi-serve pull Qwen/Qwen3.5-0.8B   # fetch weights without booting a server
arbi-serve list                    # locally cached models
arbi-serve ps                      # running instances + their active model
arbi-serve stop                    # stop a tracked instance

run/pull/ps/stop are Python-only orchestration around the existing bare-metal entrypoint (python -m arbi_serve) and the generated admin client — they never fall back to Docker Compose on their own. If the native CUDA/kernel stack can't be made to work bare-metal, they say so and stop; use the Docker Compose quickstart below instead.

arbi-serve doctor diagnoses a broken/incomplete install (works even without torch/tkv — that's exactly when it's most useful) — driver/CUDA/compute-capability, disk and cache writability, Hugging Face reachability, and, once the server deps are present, turbo-attn/tkv compatibility and environment sanity. arbi-serve plan --model <path> estimates VRAM/context/concurrency for a config before booting, without loading any weights — an estimate, not the persisted boot-confirmed sizing cache described in docs/memory-accounting.md. arbi-serve config explain <flag> / config flags / config resolve inspect the same RuntimeFlags/ServerConfig surface the admin console does. arbi-serve --version, arbi-serve --help (lists every subcommand, including openapi/convert/calibrate/warmup/bench), and each subcommand's own --help are the full reference.

Talking to a server directly

The base install (no CUDA/PyTorch needed) is also a useful command-line client for any already-running endpoint — one started with arbi-serve run above, the Docker Compose path below, or a remote deployment:

arbi-serve chat --url http://localhost:8000 --model Qwen3.5-0.8B \
  "What is 2 + 3? Answer with only the number."

The model can be omitted when the endpoint advertises exactly one model, and the prompt can come from stdin. Use --token (or ARBI_SERVE_TOKEN) for an authenticated endpoint, --temperature and --max-tokens for sampling controls, and --json when a complete OpenAI-compatible response is needed:

printf '%s\n' 'Name three primary colors.' | arbi-serve chat --url http://localhost:8000

GDN and structured generation are part of the standard server runtime, not model-specific extras. arbi-serve does not use Transformers for serving; the official container omits it. For bare-metal installs, upstream xgrammar currently declares Transformers transitively even though arbi-serve supplies its own tokenizer metadata. Transformers is otherwise test/reference-only.

EXL3 serving is complete in the official server image. For a bare-metal installation, install the pinned weights-only fork separately without its unused model-stack dependencies (the published [exl3] extra remains an empty compatibility marker because Python package metadata cannot express --no-deps):

pip install --no-deps \
  "exllamav3 @ git+https://github.com/arbi-dev/exllamav3.git@cb76869b47eecb30a8ed788d9218c4bf2cb7f09e"

See pyproject.toml for the full extras list and version floors. For source installs and a development setup, see CONTRIBUTING.md.

Docker Compose quickstart: one NVIDIA GPU

The Quickstart above (uvx arbi-serve run) is the fastest path for one model on one machine. This is the Docker Compose path instead — same GPU requirement, more explicit control over backend/codec, and what's actually running behind the scenes if you want to see it. It explicitly selects auto, which uses bf16 paged KV through the tkv-bypass kernels and needs no calibration bundle. The general Compose file defaults to tkv, so keep this override in the command. Qwen/Qwen3.5-0.8B is the small GDN smoke model used for quick bring-up and high-concurrency testing. It is not the project's performance or quality reference: that is the Qwen3.8-27B GDN path below. See the supported model matrix before choosing another checkpoint.

Prerequisites: Docker Compose v2, the NVIDIA Container Toolkit, one CUDA GPU, and enough VRAM for the selected model, context, and CUDA graphs.

git clone https://github.com/arbicity/arbi-serve
cd arbi-serve

ARBI_MODEL=Qwen/Qwen3.5-0.8B \
ARBI_KV_CACHE_DTYPE=auto \
ARBI_MAX_CONTEXT=2048 \
docker compose up -d --build

The first boot builds the image and downloads the hub checkpoint into the persistent Compose cache. Follow detail with docker compose logs -f arbi-serve. Wait for the serving gate, not just the web server binding: /health is liveness and can return 200 while the model is building; /health/ready is 503 until a request can be admitted. Probe both (as the Compose healthcheck does):

until curl -fsS http://localhost:8000/health/ready >/dev/null; do sleep 2; done
curl -fsS http://localhost:8000/health
curl -fsS http://localhost:8000/health/ready

Discover the exact model id before making a request:

curl -fsS http://localhost:8000/v1/models

For a hub id, the default served id is the repository name (Qwen3.5-0.8B in this example; an @revision suffix is retained). For a local checkpoint it is the final path component. Send that served id, not necessarily the original host path. The lightweight CLI uses the generated arbi client and auto-discovers the model when exactly one is advertised:

arbi-serve chat --url http://localhost:8000 --model Qwen3.5-0.8B \
  --max-tokens 24 "What is the capital of France?"

Applications can use the official OpenAI Python SDK directly:

pip install openai

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
answer = client.chat.completions.create(
    model="Qwen3.5-0.8B",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
    max_tokens=24,
)
print(answer.choices[0].message.content)

ARBI_MODEL accepts a Hugging Face id (org/name or org/name@revision) or a container-side local path. Set HF_TOKEN for gated/private repos; ARBI_HF_OFFLINE=1 forbids hub traffic. Local checkpoints require ARBI_MODELS_ROOT=/host/path/to/models and an ARBI_MODEL path under /models.

Canonical reference: Qwen3.8-27B GDN

Qwen3.8-27B is the primary development, performance, and quality reference. Two checked-in recipes serve it on one 24-GB card, both EXL3 weights with calibrated TKV and explicit memory/capture budgets: compose.serve.yaml with the checkpoint's own native MTP head at depth 5, and compose.dflash.yaml with a separate DFlash2 draft checkpoint at depth 7 — the latter is the faster single-card path. compose.serve.yaml also documents a separate TP=2 variant. The 0.8B quickstart above is for smoke and high-concurrency coverage, not representative benchmark numbers.

For the single-card reference, place the matching EXL3 checkpoint and curated calibration bundle under the mounted directories, then run (replace the bundle filename with the one matched to that exact checkpoint):

ARBI_IMAGE=arbi-serve:latest \
ARBI_MODELS_ROOT=/path/to/models \
ARBI_MODEL=/models/Qwen3.8-27B-exl3-4.0bpw \
ARBI_CALIBRATIONS_ROOT=/path/to/calibrations \
ARBI_CALIBRATION_FILE=/cal/YOUR_MATCHING_QWEN38_27B_BUNDLE.json \
docker compose -f compose.serve.yaml up -d

ARBI_MAX_BATCH is left at the recipe's own default (8) on purpose: pinning it to 1 gives a single-stream server whose KV pool, capture ladder and acceptance figures are not the reference ones.

ARBI_IMAGE is required here rather than defaulted: compose does not fail on an image it cannot find — with a build: stanza present it silently builds one, which reads as a several-minute hang. Name the image you mean (the tag the quickstart's docker compose up --build produced, or a published one), and add --build when building is what you want. compose.yaml, the quickstart above, is the one recipe that still builds by default.

Read the recipe's TP1/TP2 notes before changing batch, context, GPU-memory, or MTP settings; those knobs jointly determine whether the reference fits and whether a comparison remains apples-to-apples.

Calibrated TKV notes

Use this arm when you have a calibration bundle matched to the checkpoint and want the Turbo Attention KV codec. The general Compose file currently does the following by design:

• ARBI_KV_CACHE_DTYPE defaults to tkv (the quickstart overrides it to auto for evaluation).
• TKV_BITS is a literal Compose value of "4", meaning uniform K4V4. It is not an operator override; exporting TKV_BITS on the host does not change this file. Smart-mix/per-layer allocation uses a float such as 4.0 and a matching bundle, and requires an intentional recipe edit.
• ARBI_CALIBRATION_FILE, when set, is forwarded as TKV_CALIBRATION_FILE inside the container. It must be a container path (for example /cal/qwen3.5-0.8b_k4v4.json) under the read-only ARBI_CALIBRATIONS_ROOT mount. A matching bundle discoverable under /cal or the persistent calibration cache can also be selected by the runtime.

On a single 24-GB GPU, do not rely on in-process first-boot calibration: the temporary calibration swap can need roughly 12 GiB in addition to loaded weights and may OOM. Supply a pre-built matching bundle, or use the bf16 quickstart. An explicit TKV boot does not silently change to bf16; with no bundle it stages calibration (and may fail from VRAM pressure), or refuses when that path is not available.

Example that exercises TKV plumbing with the 0.8B smoke/high-concurrency model (not the canonical performance reference):

ARBI_MODEL=Qwen/Qwen3.5-0.8B \
ARBI_KV_CACHE_DTYPE=tkv \
ARBI_CALIBRATIONS_ROOT="$PWD/calibrations" \
ARBI_CALIBRATION_FILE=/cal/qwen3.5-0.8b_k4v4.json \
docker compose up -d --build

Admin console and API

The optional UI is a separate process and does not sit in the request path:

ARBI_MODEL=Qwen/Qwen3.5-0.8B ARBI_KV_CACHE_DTYPE=auto \
docker compose --profile ui up -d --build

Open http://localhost:8899; the API remains on http://localhost:8000. OpenAI-compatible routes include /v1/completions, /v1/chat/completions (including SSE), /v1/models, /v1/embeddings, /v1/rerank, /v1/files, and /v1/batches. Metrics are available at /metrics.

Observability

The optional local Prometheus/Grafana/Tempo/Loki stack is a separate Compose project:

docker compose -f docker-compose.observability.yaml up -d

Open http://localhost:3000 for the provisioned dashboards. The server also serves Prometheus text at /metrics; disable OTLP export with ARBI_SERVE_ENABLE_OTEL=0 when the collector is not wanted.

Bare metal

Install the server dependency set first:

pip install "arbi-serve[server]"
arbi-serve --model /path/to/checkpoint --kv-cache-dtype auto --max-context 2048

For calibrated TKV, use --kv-cache-dtype tkv --calibration /path/to/bundle.json. --max-context states the deployment cap explicitly; choose it to fit the model, backend, and GPU budget. The same auto/tkv semantics apply outside Compose.

The console runs in its own container and dials the engine by address. The recipes state two things for it and nothing else: --engine, the engine's service name on the compose network, and --admin-token-file, a path to the operator credential the engine mints on first boot and both containers read off the shared cache volume. The credential is what lets the console reach /v1/admin/* from another container — or another machine — without the loopback fallback and without ever setting ARBI_ALLOW_UNAUTHENTICATED_ADMIN. See docs/admin-auth.md. Add further servers — including remote ones, each with its own credential — from the console's server picker; they are kept server-side and persist across restarts, and the browser only ever names one by id.

To run only the admin console from a lightweight machine:

pip install "arbi-serve[ui]"
python -m arbi_serve.admin_console --host 0.0.0.0

Open http://localhost:8899 and add a server in the picker. Pass --engine and --admin-token-file when it runs beside one; the console then refuses to serve until it has verified that engine answers, that the port it bound is one its container publishes, and that it is running the same arbi_serve source as that engine — each of which used to fail silently.

Benchmark and value (receipts pending)

The repository does not publish universal speed, context, or cost numbers here. Record results on the target GPU, model, context, and workload before making a deployment decision. These are intentionally explicit placeholders:

Metric	Value	Conditions / receipt
Time to first token (TTFT)	TBD	GPU, model, prompt/output lengths, warm/cold state: TBD
Decode throughput (tokens/s)	TBD	concurrency and sampling settings: TBD
Steady-state VRAM	TBD	weights, KV mode/bits, context, batch: TBD
Maximum validated context	TBD	model + GPU + backend: TBD
Quality/parity result	TBD	dataset, seed, and reference path: TBD

For measurement pitfalls and a reproducible-measurement checklist, see docs/bench-measurement-footguns.md. Methodology and raw benchmark receipts for these public values: TBD (to be published; no stable link yet).

Feature status

This is a forward-looking feature map, not a backwards-compatibility matrix. The server intentionally owns the seams that matter for fast self-hosted iteration: load, profile, capture, swap, schedule, offload, and observe. If a doc disagrees with the implementation, the implementation wins.

Shipped

Area	Status
OpenAI API	/v1/completions, /v1/chat/completions, /v1/models, /v1/embeddings, /v1/rerank, /v1/files, /v1/batches; streaming for generation.
Native turbo-attn	TKV is the default center of gravity: KV-cache quantization, calibration bundles, per-layer bit allocation, hot-swap, profiling, and compare tooling.
Other attention kernels	tkv-bypass (the Turbo prefill CuTeDSL kernel + the split-K decode kernel, the default bf16 path), FlashInfer prefill backend, and vendored kernels where the engine needs tighter control.
Scheduling	Continuous batching, chunked prefill, prefix-aware admission, interactive-vs-batch priority, batch prefill caps, MTP K-bucketing, and pool-aware admission.
Prefix cache	Radix prefix cache with per-request opt-out, tenant namespace isolation, admin flush, and recurrent savepoints for hybrid models.
Fast load / reload	Safetensors loading, flat-dump cache, model reload, backend reload, calibration reload, post-load compaction, and memory-attribution checks.
Backend hot-swap	Per-StateKind active backend map; admin hot-swap drains, swaps, and rebuilds per-rank state coherently.
Tensor parallel	TP=2 serving under torchrun, including rank-0 scheduling, worker-rank forward replay, TP collectives, hot-swap, CUDA graph capture, and bundled MTP.
Speculative decoding	Bundled MTP heads, block-diffusion draft models (DFlash2 — the drafter the reference single-card recipe serves, see compose.dflash.yaml), mixed-K admission bucketing, greedy and stochastic rejection sampling, graph-safe seed buffers, captured drafter/verify paths, and external draft-model mode for single-GPU paged-KV models.
Capture / compile	CUDA graph decode, piecewise capture, MTP drafter/verify capture, sleep/resume graph survival, and optional torch.compile integration.
Hybrid architectures	GDN, Mamba/Mamba2, ShortConv, MLA, SWA/GQA dense paths; recurrent rollback/savepoint plumbing where needed.
Suspended-job offload	Batch-job KV snapshots to host RAM and disk, fault-in on resume, persisted disk index, scheduler integration, and hybrid recurrent-state opt-in.
Weight quantization	AWQ, EXL3, FP8, NVFP4 detection and load paths behind a backend-agnostic quant registry.
LoRA	Runtime LoRA load/list/unload, mixed-adapter batches, BGMV apply, capture-aware state.
Structured output	xgrammar-backed JSON schema, regex, grammar, and JSON-object constraints with batched bitmask application and in-memory/disk grammar cache.
Multimodal	OpenAI image input parsing, Qwen-VL preprocessing, M-RoPE positions, vision/text merge, and Gemma/Qwen vision seams.
Observability / DX	OTel, Prometheus metrics, NVML/process gauges, Grafana dashboards, request timelines, memory diagnostics, allocator-history dump endpoints, and backend comparison harnesses.
Admin operations	Critical-section drain/mutate/resume for backend swap, model reload, calibration reload, cache flush, dynamic config, sleep/release/resume, and flat-weight dump/resume.

Roadmap / partial

Area	Current boundary
Arbitrary TP	TP=2 is shipped. Current roadmap priority is general TP across 2/4/8/16 and non-power-of-two layouts where the model geometry allows it.
LoRA at TP>1	Refused at driver construction until per-row adapter state has a worker broadcast path.
Multi-group at TP>1	Single-rank multi-group hot-swap exists; TP composition is not wired.
Expert parallel / MoE all-to-all	Dense and some MoE model code exists, but EP serving and missing all-to-all paths are not a production surface yet.
Draft models	Bundled MTP is the production multi-rank speculative path. On a single card the reference recipe serves a separate DFlash2 draft checkpoint (compose.dflash.yaml); external draft models more broadly are an active expansion area.
Prefix-cache warm/cold tiers	Suspended batch-job KV offload is shipped. Reusable prefix-cache page migration across GPU/host/NVMe tiers is a separate follow-up.
Broader architecture matrix	Qwen3 and Qwen3.5/3.6/3.8 GDN, DeepSeek/Kimi MLA, LFM2, Gemma4, Nemotron-H, Granite, and Llama paths are wired. More architectures are added through the documented model seam rather than a broad Transformers fallback.
Hardware validation	Ada / sm_89 / RTX 4090 is the reference architecture. Hopper and Blackwell paths are present, with feature-specific live gates and ongoing validation.

Out of scope

Area	Reason
Generic Transformers fallback	The server owns model code and kernel/capture contracts directly; unsupported architectures fail loudly instead of running slow or subtly wrong fallback paths.
Cross-repo private TKV imports	arbi-serve intentionally consumes only public tkv.runtime.*, tkv.codec, and tkv.core surfaces.
Silent compatibility aliases	First-party consumers are updated in lockstep; stale shims are avoided.
Returning plausible data from stubs	Unwired model/backend/quant paths raise NotImplementedError with a precise boundary.
Distributed prefix cache across server instances	Current cache isolation is per-process/per-tenant. Cross-instance cache sharing is a separate product architecture.

• Public release scope — files retained in or excluded from the public distribution.
• Supported models — architecture coverage and tiering.
• Runtime flags — python -m arbi_serve.runtime_flags --doc lists every ARBI_* / TKV_* runtime config knob (name, type, default, allowed values, impact). Auto-generated from arbi_serve.runtime_flags.
• API schemas — request / response shapes.
• Slim image — production-tuned Docker image.
• Container privilege — the engine serves as a non-root uid with every capability dropped; what was measured to establish that nothing here needs privilege, and the one stated exception.
• Benchmark measurement guidance — controls and pitfalls for collecting comparable receipts.

Repo layout

• arbi_serve/ — the importable package (backends, scheduler, server, models, …). See CONTRIBUTING.md for the per- module breakdown.
• client/ — generated OpenAI-style client.
• Dockerfile + compose.yaml — container build and general-purpose stack.
• docker/ — model-specific Docker Compose recipes.
• docs/, examples/ — user-facing reference and runnable snippets.
• tests/ — pytest suite (CPU, GPU, and live tiers).

Contributing

Issues and PRs welcome. See CONTRIBUTING.md for development setup, test tiers, and style.

Security reports: please email security@arbi.city rather than filing a public issue. See SECURITY.md.

License

Apache-2.0. See LICENSE.