Point a coding agent or harness at one anvil-serving endpoint. Per request, the router resolves a workload intent, chooses a fast-local, heavy-local, or opt-in cloud tier from a measured quality profile, and runs structural verification where the profile says a local answer must be checked before it reaches the agent.

anvil-serving is not a generic token proxy. It is a local-first routing layer that answers the question a proxy cannot answer: is this local model trusted for this kind of work?

For OpenClaw and agent-assisted operations, anvil-serving also exposes a structured control plane: anvil-serving mcp serve for same-host stdio MCP, and anvil-serving controller serve for a token-authenticated private/tailnet controller that lets a gateway host operate a separate router, serve, or voice host without making raw SSH the product contract.

Why It Exists

Local models can be cheap and fast for bounded coding work, but they are not uniformly reliable. The planning eval that shaped anvil-serving found local outputs were usually structurally valid, while dependency and ordering reasoning lagged far behind frontier models. Static routing by model name, regex, or cost cannot catch that.

anvil-serving routes with evidence:

How It Works

Callers send a workload intent in the model field:

planning   quick-edit   review   chat   chat-fast   long-context   ocr   vision

The router maps that intent to candidate tiers, filters them by hard constraints, ranks by the quality profile, and optionally verifies the response before returning it. If the caller cannot set an intent, the Tier-0 classifier infers the work class from the request.

flowchart LR
    H["agent / harness"] --> FD["front door<br/>Anthropic + OpenAI"]
    FD --> INT["intent<br/>preset or classifier"]
    INT --> POL{"quality profile<br/>filter, then rank"}
    POL -->|"allow"| L["local tier"]
    POL -->|"allow-with-verify"| V["verify"]
    POL -->|"deny"| E["exhaustion or opt-in cloud"]
    L --> R["response"]
    V -->|"pass"| R
    V -->|"fail"| E

Every routed request is logged as a metadata-only decision record — work class, tier attempts, verify outcomes, token counts — retrievable from the router's /v1/decisions endpoint.

Slow model replacement is coordinated without a cluster scheduler. The guarded promotion transaction quiesces only the affected tier, waits for its active generations, keeps an unrelated GPU tier resident for policy-approved fallback, and requires the replacement to advertise the exact configured model name before it can re-enter rotation. See ADR-0018.

Evaluate Quickly

The only prerequisite is Python >= 3.11 — the runtime is standard-library only. Docker (and a GPU) matter only when you stand up real local model serves.

Install from a clone when evaluating the current main documentation and control-plane commands:

pip install -e .

pip install anvil-serving installs the latest published package, which can lag main; use it only when you do not need unreleased commands such as MCP/controller operations.

First prove the front door with the built-in echo backend. This requires no GPU and no model server:

python -m anvil_serving.router

If port 8000 is already in use, pass --port <free-port> and use that port in the URLs below.

Then, in another shell:

curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"chat","messages":[{"role":"user","content":"hello from anvil-serving"}]}'

To route real local tiers, start compatible OpenAI-style model serves on the URLs named in configs/example.toml (anvil-serving serves manages them as Docker Compose services), validate them with preflight, then run:

anvil-serving router run --config configs/example.toml

Use 127.0.0.1 for local URLs.

Full walkthrough: Getting started.

Command Surface

One CLI covers the router, the local serving tools, the measurement loop that feeds the quality profile, and the control plane. Full flags and examples: CLI reference. Run anvil-serving --help for the grouped command surface, anvil-serving <command> --help for focused action flags, and anvil-serving --version to verify the installed build.

Data plane — run and manage the router:

Local serving tools — stand up and validate the tiers the router routes across:

Quality loop — the measurements behind the routing profile:

Control plane and integrations:

CLI Compatibility Notes

Canonical command changes:

• anvil-serving deploy → anvil-serving serves render
• anvil-serving external-bench → anvil-serving eval benchmark external
• anvil-serving cache-prune → anvil-serving models cache prune
• anvil-serving score → anvil-serving models score

Removed forms fail with migration guidance instead of silently dispatching. See the CLI migration table for every replacement.

Cost And Security Defaults

• Local-only by default: configs/example.toml contains no cloud tier and no cloud API key.
• Opt-in cloud: configs/example-with-cloud.toml shows explicit metered cloud routing. Only work classes listed in [router].metered_cloud can use that tier.
• Credentials by env var: configs name env vars such as ANTHROPIC_API_KEY; they never contain literal secrets.
• Loopback first: the front door binds 127.0.0.1 by default.
• Token before exposure: configure [server].auth_env = "ANVIL_ROUTER_TOKEN" before publishing the router beyond loopback.
• Controller token required: bind anvil-serving controller serve only to 127.0.0.1 or a private/tailnet address and set ANVIL_CONTROLLER_TOKEN through --auth-token-env; unauthenticated loopback is an explicit development opt-out, not the default.

See SECURITY.md for the threat model and vulnerability reporting path.

Status

The source tree is versioned 0.13.1, while published tags and package releases can lag main. The router, local serving tools, host management, router/serve/voice lifecycle verbs, harness sync, and OpenClaw MCP/controller control plane all ship on main. Install from a clone when evaluating those main-only surfaces. The control plane keeps the request data plane clean: OpenClaw's hook plugin handles per-turn intent, the router handles quality and configured fallback/exhaustion, and MCP/controller tools handle explicit operations such as status, voice lifecycle, preflight, benchmarking, and OpenClaw config sync.

Known Limitations

• OpenClaw native failover does not reliably escape a plugin-pinned provider for local-preferred classes. Use ANVIL_CLOUD_CLASSES or anvil-serving's opt-in cloud tier for at-risk classes.
• Most shipped promotion verdicts are seed verdicts, pending operator-promoted real-traffic calibration. The planning-class deny decision has hard eval evidence; other classes should be remeasured on your served models.
• The local-tier quickstart requires compatible model serves already running at the configured base_url values. Use the echo-backend path above for a no-GPU evaluator smoke test.

Documentation

Start with the path that matches you:

• Evaluating anvil-serving? This README → Getting started (no-GPU smoke test) → Architecture → the full Quality-gated router design reference.
• Operating a deployment? Getting started (real tiers) → Configuration reference → CLI reference → Operator playbooks → Troubleshooting; examples/fakoli-dark/ contains an offline Dark/Mini topology reference alongside the existing machine-specific two-GPU operational artifacts.
• Contributing? CONTRIBUTING.md (module map and extension recipes) → Architecture → ADRs.

Contributions welcome — see CONTRIBUTING.md and the code of conduct.

MIT licensed.