Saddle

Break your local model to harness.

Saddle is a deterministic execution harness that turns a non-deterministic local LLM into a dependable coding tool: constrained DAG planning, parallel proof-gated workers, machine-checked gates. No retries on unverified foundations.

Status: early. The agent-plus-auditor loop below is built and measured on a small benchmark (details under Status below). Don't trust it unattended yet: on harder tasks the auditor has accepted wrong work (#109).

Phase 2: agent plus auditor

Phase 2 adds:

• saddle auto: one task, run autonomously in a git worktree, with a time and token budget. The result is a branch.

• A tiered auditor whose findings reach the model as tool results. finish is refused while the audit fails.

• A lane chip in the chat: Ask (the default, read-only: the model gets only read, list and search tools), Edit (opt-in, edits your folder directly, unaudited) and Task · Small, which starts the same audited run from the browser and shows an evidence packet compiled from the ledger.

• Run-state signals so a run finds you: tab title, favicon, an opt-in browser notification, and a state pill per session in the sidebar.

• Two questions a run can ask once each: whether tests may be edited when the auditor needs a test, and whether to extend a budget at 80%. Unanswered, each takes the old behaviour as its default.

• saddle verify --anchor, which checks each run's outcome against the Saddle-Outcome trailer on its branch.

• saddle audit --tiered, which runs the same battery on any diff.

• docs/USING-SADDLE.md: install, the Ask / Edit / Task lanes, run-state signals, the two questions, how runs end, reading the packet, saddle verify --anchor.

• docs/AUDIT-TIERS.md: the gates by tier, reason classes, caching.

• docs/CLI.md: every flag and exit code.

Status as of 2026-09-28:

• Measured, no harm: on two benchmark tasks, six seeds each, the audited agent's false-done rate was 0 of 6 on both, the same as the plain agent's, at 0.95x and 0.69x of its wall time to a finished evidence packet.
• Measured, detection: a rate on planted defects, not a pass. Re-audited offline on 48eb6d6, the auditor refused 65 of 65 planted-bug trees, 49 of 65 for the right reason. Every right-reason refusal came from the agent's own tests failing; on one task the other gates refused correct and wrong trees alike.
• Observed, not yet measured: in single unscored runs on two harder tasks, the auditor accepted a tree that failed the task's hidden checks (#109). Measuring that rate is next.
• Not measured: detection of defects that were not planted.
• Not built: only the Small lane exists. Contracts are not sealed before implementation. The only questions a run asks are the two above.

Layout

• docs/ARCHITECTURE.md — full architecture specification.
• src/saddle/{dag,gates,evidence,runner,slice,scheduler,journal,vllm,cli}.py — the vertical slice, in layering order dag → gates → evidence → runner → slice:
  • dag.py — DAG data model and validation.
  • gates.py — pure gate predicates.
  • evidence.py — subprocess policy and gate-runner adapters.
  • runner.py — node execution.
  • slice.py — end-to-end vertical slice orchestration.
  • scheduler.py — Kahn scheduler.
  • journal.py — proof journal.
  • vllm.py — vLLM client.
  • cli.py — command-line interface.
• transcript.py, timeline.py, ux.py, tools.py, chat.py — reporting and CLI support (chat.py present; see #23).

Quickstart

The CLI as built (cli.build_parser; every command and flag in docs/CLI.md): saddle doctor, saddle dag, saddle run, saddle auto, saddle audit, saddle tail, saddle verify, saddle explain, saddle chat (alias saddle web), saddle up. Configure the vLLM API key via SADDLE_VLLM_API_KEY or VLLM_API_KEY. Run the suite with:

PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest -q

Requirements

• Python 3.12+
• Self-operated vLLM ≥ 0.28 server with guided decoding (XGrammar) + KV offloading — the proven setup is qwen38-27b-rtx3090
• pytest, coverage.py, ruff, mutmut (sampled per node in Tier 1; the merge-time Tier 2 is not built)
• Linux with bwrap (bubblewrap) that can start: Task runs refuse to run without it. On Ubuntu 24.04 unprivileged user namespaces are restricted by AppArmor, so bwrap needs the upstream bwrap-userns-restrict profile or an equivalent.
• A user systemd manager (systemd-run --user --scope works) for the per-command memory cap; without one saddle falls back to a weaker per-process address-space ceiling. SADDLE_MEMORY_MAX sets the cap (bytes, or a number with K, M, G or T; default 6G).

Security model

Saddle runs code a model wrote, on your machine, as you. It confines some of that and not all of it. Run it only on code you would be willing to run yourself.

It needs a model server: a local OpenAI-compatible chat-completions endpoint (vLLM with structured outputs; --base-url, default vllm.DEFAULT_BASE_URL, http://127.0.0.1:18020/v1). Your prompts and the files the model reads go to that server. Point --base-url somewhere else and they go there instead.

What is confined (sandbox.py, memcap.py):

• File tools stay in the workdir. Every path a tool reads or writes is resolved, symlinks included, and refused if it lands outside the working directory (sandbox.resolve_within).
• Commands run under bwrap when it can start (Sandbox._argv). The root is built up from the system directories and the interpreter, read-only (SYSTEM_DIRS); HOME and /tmp are empty; the workdir is the one writable place, and its .git is read-only (Sandbox._git_binds). Each command gets its own namespaces (sharing the host network unless the lane takes it away), its own session and process group, and its process group is killed when it ends.
• The environment is an allowlist (sandbox.ENV_KEEP), so the model key in saddle's own environment does not reach a command.
• A Task run (saddle auto) requires isolation and has no network. It refuses to start when bwrap is missing or cannot start (require_isolation=True, IsolationUnavailableError), and its commands see loopback only (network="none").
• Every command the model runs has a memory cap (memcap.cap): a systemd user scope with MemoryMax, no swap and at most 4096 tasks, or a per-process prlimit --as ceiling where no user systemd manager is reachable. The cap also covers the audited tree's test command and mutmut run (evidence.tree_memory_limit). Default 6 GiB; SADDLE_MEMORY_MAX changes it. A command the cap kills is recorded as a failure.
• The git calls of an autonomous run and of the packet's branch actions run with hooks and core.fsmonitor off (sandbox.HOST_GIT_GUARD, used in auto.py and web/branch_actions.py), so config planted in .git does not run there. saddle's other git calls, such as the gates' diffs (evidence.py), do not pass it.
• The chat server binds to loopback by default; any other bind address requires a token (web.app.needs_token).

What is not confined:

• The auditor's gates. pytest, coverage and mutmut run as you on the host, with your read access and your network, under the memory cap only (#104). A test the model wrote runs there during an audit.
• Ask and Edit in the chat without a working bwrap. Those lanes use bwrap when it starts; otherwise commands run as you and the sandbox reports isolation: none (Sandbox.isolation). They keep the host network either way. Ask offers the model read-only tools only (tools.tools_for_mode); Edit writes your folder directly, unaudited.
• git, ruff and coverage bookkeeping calls have no memory cap.

Reading the comments

Comments and docstrings state the claim they rely on, and the evidence behind it, in plain words. Some name a benchmark task and seed (T5, t8-s1) or a round of runs ("round 3e"); docs/BENCHMARK-RECORD.md describes the tasks and the rounds. The raw run records behind those measurements are the author's own and are not public.

License

Copyright (C) 2026 Eliza Halpern and Eryn Lipkowitz. See AUTHORS.

Saddle is free software under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See LICENSE.