coding-guardrails

A proxy that sits between your coding agent and a local LLM. It makes local models more reliable at calling tools and stops them from doing dangerous things. Nothing else — no model hosting, no agent orchestration, no magic.

It works with any agent that speaks the OpenAI API (Pi, Claude Code, OpenCode, Aider, Continue, Cline, Roo) and any OpenAI-compatible local backend (llama-server, Ollama, Llamafile, vLLM).

What it is — and what it isn't

It is two layers between your agent and the model:

1. Forge (Layer 1) — a reliability layer for tool calling. Rescue-parses malformed tool calls (local models produce them constantly), retries with corrective nudges, validates responses, captures and reinjects thinking tokens, and compacts context when it grows too long.
2. Coding Guardrails (Layer 2) — 13 composable rules that block or nudge tool calls before they reach the machine. Path traversal, destructive commands, network egress, secrets, and more.

It isn't:

• Not a model. It doesn't improve answer quality. A 9B model still reasons like a 9B model — it just calls tools correctly and can't rm -rf / your disk while doing it.
• Not an agent harness. Your agent stays in charge. The proxy is transparent to the agent; it only makes the traffic safer and more reliable.
• Not a fix for every local-model quirk. The measured results below include known weaknesses, because there are some.

Why it exists

Running coding agents on local models is practical and private, but local models have real, measurable failure modes:

• They emit malformed tool calls (broken JSON, hallucinated arguments).
• They loop on the same call when things go wrong.
• They answer in prose when a tool call is required — or call a tool when a plain answer was enough.
• They will happily run sudo rm -rf / if the task asks for it.

Layer 1 fixes the first three mechanically. Layer 2 fixes the last one by policy. Neither requires rewriting your agent.

Quick Start

pip install coding-guardrails

coding-guardrails server build                                          # builds cg's llama-server (pinned commit; includes the Gemma 4 tool-call fix)
coding-guardrails server start --model Qwen3.8-27B-UD-Q3_K_XL           # LLM backend on :8080 (default — measured 100% completion / 99% accuracy)
coding-guardrails serve --backend-url http://localhost:8080 \
  --model Qwen3.8-27B-UD-Q3_K_XL --port 8081                            # proxy on :8081

# Point your agent at http://localhost:8081/v1

Your agent sees a standard OpenAI-compatible API. Already running your own llama-server? Skip server build/start and point --backend-url at it.

server start uses model profiles that bake in the right sampling and boot flags (KV cache quantization, speculative decoding, context size) per model — no flag archaeology.

The 13 guardrail rules

Every rule is independently configurable (disable, change severity, tune limits) via configs/guardrail-config.yaml. See docs/rules.md.

Hard blocks (the call never reaches the machine)

Soft nudges (advisory — the call proceeds, the model is told)

A blocked call returns a text explanation to the agent, not an empty tool call — agents retry empty tool calls forever and never see a nudge that comes back as a tool result.

How well does it actually work? — measured

We run the Forge 30-scenario tool-calling eval (150 runs) through the proxy against each supported model. Every number below is from a recorded run — timestamps and run IDs included — and the eval is reproducible (eval/scripts/run_forge_eval.py, proxy mode).

All numbers are recorded runs on the same local GPU (proxy mode). The Qwen3.8-27B row is v0.17.0 (2026-08-15); the rest are the v0.16.1 re-eval (2026-08-07/08) after the respond() fix:

What the numbers mean. Completion = the run finished through the terminal tool without exhausting retries. Accuracy = the final answer was correct. A run can complete and be wrong (the accuracy gap) or fail to complete at all (the completion gap).

Known weaknesses, stated plainly:

• Data-heavy recovery was weak across every model — until Qwen3.8-27B. The data_gap_recovery_extended, argument_transformation, inconsistent_api_recovery, and grounded_synthesis scenarios scored 0-60% accuracy on Qwen3.5-9B, Ornith, and Qwen3.6-27B alike, and 0% on LFM2.5. Qwen3.8-27B is the first model we measured that closes that gap: 100% on all four families — which is why it is now the default worker. Multi-source synthesis into a precise report is still frontier-hard for anything smaller.
• LFM2.5-2.6B is genuinely not suited to agentic coding — its 0/10 on tool_selection is real (it never calls the terminal tool; the proxy log shows zero terminal calls across all runs), matching the model card's own caveat. Use it for bounded, structured tasks or not at all.
• Completion ≈100% ≠ correctness — except for Qwen3.8-27B (100% / 99%). For every other model we measured, the correctness ceiling is 92-95%.

We measure ourselves too — the respond() bug

These numbers are trustworthy for a specific reason: the eval has already caught our own bug. In v0.7.4 (2026-06-01) the proxy began converting terminal respond() calls to plain text even when the agent had explicitly declared a respond tool. Every model's tool_selection score dropped to 0/5 overnight. For a month we misdiagnosed it as a model quirk — "the model answers in prose." The eval logs told the truth:

# buggy proxy (v0.7.4 era):
L1 done  (4.9s, respond -> text: Alice has read, write, and admin permissions on the `forg...

# fixed proxy (v0.16.1):
L1 done  (respond() passed through — declared in request)

The model had been calling respond() correctly the whole time; the proxy was eating the call. v0.16.1 passes declared respond() calls through (plus a terminal-tool nudge for the prose path), and tool_selection went back to 5/5. Full re-evaluation: 138/150 → 149-150/150 across the lineup.

We publish this story for the same reason we publish the eval tables: the numbers are only useful if they're honest, and they've already proven they can catch our own regressions.

Supported models and profiles

Model profiles live in src/coding_guardrails/models/profiles.py — sampling defaults, boot flags, VRAM requirements, and context limits per GGUF.

Vision (multimodal)

Qwen3.8-27B is natively multimodal (images; hour-scale video is vLLM/SGLang territory — unsupported on llama.cpp). server start auto-attaches a sibling mmproj-*.gguf next to the model, and the proxy captions inbound images ([image: caption] text blocks) so vision works through the text-only guardrail pipeline — verified end-to-end through the proxy on 2026-08-15 (image → caption → correct answer). Disable captioning with --no-vision-captioning; text-only backends degrade to a placeholder.

Full details, boot commands, and per-model caveats: docs/models.md.

Real use — a delegated coding task, end to end

The proxy is exercised daily by delegating real repo work to a subagent (cg-worker) routed through it. A representative run — update the model table in this repo's README — produced 0 blocks and 1 advisory nudge across the whole session:

[coding_guardrails.layer2] DEBUG:   edit | allowed (nudged: lint)
[coding_guardrails.layer2] DEBUG: 1 nudged | 1 allowed

The work (a 5-row markdown table + prose rewrite) landed correct on the first try, verified by diff. The lint nudge was advisory; the agent applied the trivial fix and moved on. That's the intended shape of every session: the model does the work, the guardrails do the policing, neither slows the other down.

Limitations & footguns

• Shared-GPU allocator fragmentation. On a desktop GPU shared with a compositor and other processes, repeated large model load/unload cycles can fragment the driver's memory allocator: llama-server then fails boots with cudaMalloc failed: out of memory on small buffers while nvidia-smi shows GBs free. A reboot clears it. The Qwen3.6-27B profile (48K q8_0 KV, -ub 256) is sized to fit the fragmented state, not the ideal one.
• 24 GB VRAM is the practical ceiling. Qwen3.6-27B is capped at 48K context (56K+ needs a single KV allocation the fragmented allocator can't provide). Qwen3.8-27B (UD-Q3_K_XL + q4_0 KV + MTP + mmproj) runs 128K on the same card — its hybrid architecture keeps the KV cache cheap. 200K context is still only realistic for the 9B-class models.
• Local models are local models. The eval tables above are the honest picture: excellent at tool plumbing, weak at multi-source synthesis.
• The lint gate runs on every edited file. If you don't lint your code, the proxy will start noticing. You can lower its severity in config.

Configuration

configs/guardrail-config.yaml — per-rule enable/disable, severity, and limits; workspace root for path safety; secret patterns. Full reference in docs/rules.md. Server and fleet-operations docs: docs/server.md, docs/agents.md.

Development

source .venv/bin/activate
pytest tests/unit/ -q          # 564 tests, ~25s
python eval/scripts/run_forge_eval.py --mode proxy --runs 5   # the eval behind the tables above

Architecture notes: docs/architecture.md.