coding-guardrails

A proxy that sits between your coding agent and a local LLM. It makes local models more reliable at calling tools and stops them from doing dangerous things. Nothing else — no model hosting, no agent orchestration, no magic.

It works with any agent that speaks the OpenAI API (Pi, Claude Code, OpenCode, Aider, Continue, Cline, Roo) and any OpenAI-compatible local backend (llama-server, Ollama, Llamafile, vLLM).

What it is — and what it isn't

It is two layers between your agent and the model:

1. Forge (Layer 1) — a reliability layer for tool calling. Rescue-parses malformed tool calls (local models produce them constantly), retries with corrective nudges, validates responses, captures and reinjects thinking tokens, and compacts context when it grows too long.
2. Coding Guardrails (Layer 2) — 15 composable rules that block or nudge tool calls before they reach the machine. Path traversal, destructive commands, network egress, secrets, and more.

It isn't:

• Not a model. It doesn't improve answer quality. A 9B model still reasons like a 9B model — it just calls tools correctly and can't rm -rf / your disk while doing it.
• Not an agent harness. Your agent stays in charge. The proxy is transparent to the agent; it only makes the traffic safer and more reliable.
• Not a fix for every local-model quirk. The measured results below include known weaknesses, because there are some.

Why it exists

Running coding agents on local models is practical and private, but local models have real, measurable failure modes:

• They emit malformed tool calls (broken JSON, hallucinated arguments).
• They loop on the same call when things go wrong.
• They answer in prose when a tool call is required — or call a tool when a plain answer was enough.
• They will happily run sudo rm -rf / if the task asks for it.

Layer 1 fixes the first three mechanically. Layer 2 fixes the last one by policy. Neither requires rewriting your agent.

Quick Start

pip install coding-guardrails

coding-guardrails server build                                          # builds cg's llama-server (pinned commit: llama.cpp 0.9.0-dev + DSpine draft support)
coding-guardrails server start --model Qwen3.8-27B-UD-Q3_K_XL           # backend on :8080 (default — DSpine draft model attached; 100% completion / 99% accuracy)
coding-guardrails serve --backend-url http://localhost:8080 \
  --model Qwen3.8-27B-UD-Q3_K_XL --port 8081                            # proxy on :8081

# Point your agent at http://localhost:8081/v1

Your agent sees a standard OpenAI-compatible API. Already running your own llama-server? Skip server build/start and point --backend-url at it.

server start uses model profiles that bake in the right sampling and boot flags (KV cache quantization, speculative decoding, context size) per model — no flag archaeology.

Key proxy flags

The 15 guardrail rules

Every rule is independently configurable (disable, change severity, tune limits) via configs/guardrail-config.yaml. See docs/rules.md.

Hard blocks (the call never reaches the machine)

Input scanning (pre-generation — message content, not tool calls)

Soft nudges (advisory — the call proceeds, the model is told)

A blocked call returns a text explanation to the agent, not an empty tool call — agents retry empty tool calls forever and never see a nudge that comes back as a tool result.

How well does it actually work? — measured

We run the Forge 30-scenario tool-calling eval (150 runs) through the proxy against each supported model. Every number below is from a recorded run — timestamps and run IDs included — and the eval is reproducible (eval/scripts/run_forge_eval.py, proxy mode).

All numbers are recorded runs on the same local GPU (proxy mode). The Qwen3.8-27B row is v0.26.0 (2026-11-09, shipped profile with the DSpine draft model); the rest are the v0.16.1 re-eval (2026-08-07/08) after the respond() fix:

What the numbers mean. Completion = the run finished through the terminal tool without exhausting retries. Accuracy = the final answer was correct. A run can complete and be wrong (the accuracy gap) or fail to complete at all (the completion gap).

Known weaknesses, stated plainly:

• Data-heavy recovery was weak across every model — until Qwen3.8-27B. The data_gap_recovery_extended, argument_transformation, inconsistent_api_recovery, and grounded_synthesis scenarios scored 0-60% accuracy on Qwen3.5-9B, Ornith, and Qwen3.6-27B alike, and 0% on LFM2.5. Qwen3.8-27B is the first model we measured that closes that gap: 100% on all four families — which is why it is now the default worker. Multi-source synthesis into a precise report is still frontier-hard for anything smaller.
• LFM2.5-2.6B is genuinely not suited to agentic coding — its 0/10 on tool_selection is real (it never calls the terminal tool; the proxy log shows zero terminal calls across all runs), matching the model card's own caveat. Use it for bounded, structured tasks or not at all.
• Completion ≈100% ≠ correctness — except for Qwen3.8-27B (100% / 99%). For every other model we measured, the correctness ceiling is 92-95%.

We measure ourselves too — the respond() bug

These numbers are trustworthy for a specific reason: the eval has already caught our own bug. In v0.7.4 (2026-06-01) the proxy began converting terminal respond() calls to plain text even when the agent had explicitly declared a respond tool. Every model's tool_selection score dropped to 0/5 overnight. For a month we misdiagnosed it as a model quirk — "the model answers in prose." The eval logs told the truth:

# buggy proxy (v0.7.4 era):
L1 done  (4.9s, respond -> text: Alice has read, write, and admin permissions on the `forg...

# fixed proxy (v0.16.1):
L1 done  (respond() passed through — declared in request)

The model had been calling respond() correctly the whole time; the proxy was eating the call. v0.16.1 passes declared respond() calls through (plus a terminal-tool nudge for the prose path), and tool_selection went back to 5/5. Full re-evaluation: 138/150 → 149-150/150 across the lineup.

We publish this story for the same reason we publish the eval tables: the numbers are only useful if they're honest, and they've already proven they can catch our own regressions.

Supported models and profiles

Model profiles live in src/coding_guardrails/models/profiles.py — sampling defaults, boot flags, VRAM requirements, and context limits per GGUF.

Speculative decoding: DSpine (why the default got ~2x faster)

The Qwen3.8-27B default ships a draft model. DSpine (Tencent Hunyuan) is a five-layer parallel drafter that reads hidden states from five target layers, drafts a 16-token block, and injects each position's predicted predecessor code into its successor at every layer. It rides llama.cpp's existing draft-dflash implementation, so a profile only has to name the drafter:

# models/profiles.py
drafter="Qwen3.8-27B-DSpine-b16-Q4_K_M",   # key into server/sources.py DRAFTERS
spec_n_max=4,                              # tokens drafted per verify step

cg server download <profile> fetches the drafter with the target; cg server start appends -md … --spec-type draft-dflash --spec-draft-n-max 4 and budgets the drafter's VRAM in the gate. A missing drafter logs a warning and boots the target without speculation rather than failing.

Measured 2026-11-09 on an RTX 3090 Ti (UD-Q3_K_XL target, same harness both ways):

Short-context decode goes from 46.3 to 94.5 tok/s (n-max 4); code and math hit ~120 tok/s. --spec-draft-n-max is hardware dependent — 4 is the peak on a Q3 target, while the model card's 15 targets a Q4_K_M/3090 setup; sweep it yourself with llama-benchy or the eval before trusting a number.

DSpine is not upstream in llama.cpp yet, so cg's pin points at a fork (PINNED_REPO in server/version.py) whose branch is upstream master plus one audited commit. Everything else in the pin is stock llama.cpp; switch the repo back to ggml-org once DSpine lands upstream.

Vision (multimodal)

Qwen3.8-27B is natively multimodal (images; hour-scale video is vLLM/SGLang territory — unsupported on llama.cpp). server start auto-attaches a sibling mmproj-*.gguf next to the model, and the proxy captions inbound images ([image: caption] text blocks) so vision works through the text-only guardrail pipeline — verified end-to-end through the proxy on 2026-08-15 (image → caption → correct answer). Disable captioning with --no-vision-captioning; text-only backends degrade to a placeholder.

More than one --model turns the proxy into a single-port multi-model proxy: one port serves several models and swaps them with unload-before-load (frees VRAM/turn over one GPU, then loads the requested model). E.g. cg server start --manage-backend --model Qwen3.8-27B-UD-Q3_K_XL --model MiniCPM5-2B-Q4_K_M --port 8081. The proxy routes each request by its model id and advertises all served ids from /v1/models.

Full details, boot commands, and per-model caveats: docs/models.md.

Real use — a delegated coding task, end to end

The proxy is exercised daily by delegating real repo work to a subagent (cg-worker) routed through it. A representative run — update the model table in this repo's README — produced 0 blocks and 1 advisory nudge across the whole session:

[coding_guardrails.layer2] DEBUG:   edit | allowed (nudged: lint)
[coding_guardrails.layer2] DEBUG: 1 nudged | 1 allowed

The work (a 5-row markdown table + prose rewrite) landed correct on the first try, verified by diff. The lint nudge was advisory; the agent applied the trivial fix and moved on. That's the intended shape of every session: the model does the work, the guardrails do the policing, neither slows the other down.

Limitations & footguns

• Shared-GPU allocator fragmentation. On a desktop GPU shared with a compositor and other processes, repeated large model load/unload cycles can fragment the driver's memory allocator: llama-server then fails boots with cudaMalloc failed: out of memory on small buffers while nvidia-smi shows GBs free. A reboot clears it. The Qwen3.6-27B profile (48K q8_0 KV, -ub 256) is sized to fit the fragmented state, not the ideal one.
• 24 GB VRAM is the practical ceiling. Qwen3.6-27B is capped at 48K context (56K+ needs a single KV allocation the fragmented allocator can't provide). Qwen3.8-27B (UD-Q3_K_XL + q4_0 KV + DSpine draft) runs 120K on the same card (128K at 3 drafted tokens) — its hybrid architecture keeps the KV cache cheap. 200K context is still only realistic for the 9B-class models. A speculative drafter also costs ~150 MB of recurrent state per drafted token, which comes out of that same budget.
• Local models are local models. The eval tables above are the honest picture: excellent at tool plumbing, weak at multi-source synthesis.
• The lint gate runs on every edited file. If you don't lint your code, the proxy will start noticing. You can lower its severity in config.

Configuration

configs/guardrail-config.yaml — per-rule enable/disable, severity, and limits; workspace root for path safety; secret patterns. Full reference in docs/rules.md. Server and fleet-operations docs: docs/server.md, docs/agents.md.

Development

source .venv/bin/activate
pytest tests/unit/ -q          # 794 tests (~25s)
python eval/scripts/run_forge_eval.py --mode proxy --runs 5   # the eval behind the tables above

Architecture notes: docs/architecture.md.