LocalHarness

Run AI agents on the models you already run locally.

LocalHarness is the agent layer on top of your inference engine: vLLM, Ollama, LM Studio, or llama.cpp. It doesn't serve models. It gives the model you already serve real agents, with tools, memory, and deny-first permissions, all defined in YAML instead of Python.

It's model-agnostic. Point it at any OpenAI-compatible endpoint and the same agent runs. Agents are hierarchical: an orchestrator routes work to subagents, each with its own fresh context, tools, and memory.

The bet: the harness, not the model, is where most of the capability lives. The same model can swing tens of benchmark points depending on the harness around it.

Five things it does that are hard to find anywhere else:

• Read documents bigger than the context window — losslessly. Every section is actually read, never truncated, and every number in the answer traces back to the source text it came from. Built for long filings, contracts, and reports, on hardware you control.
• Structural defense against prompt injection. Untrusted web content can never share an agent with host-mutating tools like bash, write, or edit. The boundary is enforced in the agent topology and fails closed — not left to the model to refuse.
• Memory that consolidates while idle. Lessons auto-capture from real failure→recovery moments at zero extra model calls; recurring ones are promoted into the prompt during idle "sleep" passes; superseded facts are never deleted, and search routes through a gist/schema hierarchy — gists route the search, leaf records anchor the answer. All SQLite: no vector DB, no second resident model.
• Sittings that remember each other. Every run records a session; on close it gets one topical summary line (what you asked, or the error it resolved) — derived from events, zero model calls. The next sitting opens with the recent timeline already in its prompt (relative times, newest first, hard-capped at 8 lines) and can answer "what did we do last time?" without a single lookup.
• A write gate that predicts. Per-tool statistical priors — computed in pure SQL, zero model calls — score every tool outcome against that tool's own history. A normally-reliable tool that suddenly errors becomes a captured memory, and a user correction ("no, I meant…") writes a quarantined, reversible one; a failure the tool's history already expected stops being news. Each of these lands below the prompt's visibility line — captured and searchable, but not steering the agent until an idle consolidation pass confirms it, with one config lever to revert to motif-only capture.

localharness init auto-detects your running endpoint (here, vLLM serving Qwen) and probes its tool-calling. Then localharness start is zero-config: it creates the orchestrator — a general-purpose root agent — and drops you straight into the REPL. Ask it a real question and watch the agent work — here it chains web_search → web_fetch across several iterations to research the best open-source model for a 128 GB machine, the tool-call loop visible the whole way.

Why local

Frontier coding agents are great when you're driving them. But metering and rate limits make them an awkward fit for the recurring jobs you'd actually want an agent to own: the nightly report, the scheduled cleanup, the watch-and-react task. LocalHarness keeps the Claude Code / OpenCode workflow you already know, pointed at a model running on hardware you control.

• No metering. A job that fires every hour runs on hardware you already own, with no per-token bill.
• Your data stays put. Code, files, and prompts never leave the machine.
• Always on. No quota or rate caps to budget around for unattended runs.
• Familiar. Same agent, tool, and permission model as the cloud tools, just local.

A frontier agent like Claude Code is still the easy way to set the harness up and compose a bespoke subagent for a task. The split that works: frontier to design, local to run.

Migrating existing headless work? LocalShift is the companion project. Point Claude Code at a cron job, skill, or bare prompt and it builds a per-workload quality eval, proves the local model is good enough (or honestly says keep-frontier), then cuts the job over to run claude-free on LocalHarness.

Features

• YAML-defined agents — add an agent, division, or tool policy without writing Python
• Event-bus core — components communicate via a typed event stream, persisted as append-only JSONL per agent
• Memory that learns from use — per-agent SQLite memory with an automatic write gate (lessons captured from failure→recovery signals, zero extra model calls), activation-ranked recall in pure SQL, cancellable idle consolidation, and a persisted gist/schema hierarchy over document analyses; conflicting facts supersede, never overwrite
• Workspace layers — a .localharness/ folder in a project layers its own agents and config over the machine-wide one (nearest wins, deny patterns union so a project can never widen them); localharness config show names the file behind every effective key, and doctor names both layers and every key the project overrides
• Per-project memory — where a workspace applies, memory, sessions, history, and the audit log live with the project, so many projects on one machine stop pouring their lessons into each other's context; /memory promote moves a single fact to the global store, deliberately
• Deny-first permissions — policies inherit down the hierarchy and can only narrow
• Tool-call fallback — native function calling where the model supports it, XML/Hermes fallback where it doesn't
• MCP support — connect Model Context Protocol servers and expose their tools to agents
• Built-in tools — read, write, edit, glob, grep, bash, python, web search/fetch, and subagent delegation
• Benchmark suite — scenario corpus in bench/ for measuring harness changes against your own model
• Autoresearch loop — propose → gate → promote mutation archive for harness self-improvement experiments
• Pluggable channels — terminal by default, or localharness start --channel discord to drive a session from Discord (needs the dispatch extra, uv sync --extra dispatch, plus LOCALHARNESS_DISCORD_TOKEN and LOCALHARNESS_DISCORD_ALLOW)

How it compares

LocalHarness is an agent layer — not an inference engine, and not a cloud SaaS. It sits on top of whatever serves your model and gives that model agents, tools, memory, and permissions.

If you already serve a model with Ollama or vLLM and want to run real agents against it — with tools, isolated memory, and deny-first permissions — that's the gap LocalHarness fills.

Supported runtimes

Every backend is reached over one OpenAI-compatible client, so the request path is provider-agnostic. What differs per runtime is lifecycle (whether the harness can start and stop the server itself), introspection (exact token counting, context-window discovery), and how much has been validated on real hardware. All four also work as a plain attach target — point init at an already-running endpoint. Setup pages: llama.cpp · Ollama · LM Studio · vLLM (see reference architectures).

✅ validated — footnotes note setup caveats, not gaps:

1. Token counting is exact for every runtime — whole-request exact, not just content. vLLM applies its chat template server-side (/tokenize messages-mode) and llama.cpp renders it via /apply-template before /tokenize, so message counts — including the rendered tools block — equal the real call's usage.prompt_tokens to the token (verified live on both). Ollama and LM Studio serve no tokenize endpoint, so the harness loads the served model's own GGUF vocab + chat template in-process (llama-cpp-python vocab-only — the exact-tokenizer extra) and counts to the token — verified equal to each server's own count (message structure exact; the tools block is not rendered on these two). Older vLLM/llama.cpp builds without messages-mode / /apply-template keep exact content counts and estimate message overhead, disclosed at start; without the extra (or with no local model files), Ollama/LM Studio fall back to a labeled approximate estimate.
2. Context window is read at server level for vLLM (max_model_len) and llama.cpp (/props), and from the loaded model for Ollama (/api/ps context_length) and LM Studio (loaded_context_length) — the latter two are known once the model is resident (the harness warm-loads it) and fall back to config, disclosed, before then. Never a silent guess; the served window, not a model ceiling that would over-report and 400 mid-session.

Live-validated means a real end-to-end run on the DGX Spark reference machine (detect → serve → tool-call → verified stop). Recorded bench runs currently exist for vLLM only; the other runtimes ship opt-in bench.yaml entries you run against your own model. Per-runtime live markers: LOCALHARNESS_LIVE_{VLLM,OLLAMA,LLAMACPP,LMSTUDIO}=1 uv run pytest -m live_<name>.

Requirements

• Python ≥ 3.12 and uv
• A local LLM server with an OpenAI-compatible API (vLLM, Ollama, LM Studio, or llama.cpp)
• On Windows: Git for Windows — bash_exec runs under git-bash (see Platform support)

Quick start

git clone https://github.com/ahwurm/localharness.git
cd localharness
uv sync

uv run localharness init    # probes vLLM :8081/:8000, Ollama :11434, LM Studio :1234, llama.cpp :8080
uv run localharness start   # interactive session

init detects your endpoint and models, probes tool-calling capability, and writes ~/.localharness/config.yaml. No server running? init walks you through setup: pick your hardware (reference architecture) and it provisions the local server (pulling the vLLM container where Docker is available) — start then reuses it. Inside the REPL, /model lists served + downloaded models and swaps between them; localharness start --model <name> picks one for a single session without touching config, and --list-models lists without starting a session. localharness model --download <repo_id> (optionally --file <name> for one GGUF quant out of a multi-quant repo) pulls a model from Hugging Face ahead of time. Non-standard setup: localharness init --endpoint http://host:port/v1. To give one project its own agents, config, and memory, run localharness init --workspace inside it — the resulting .localharness/ directory is discovered by walking up from wherever you start, and layers over the global config.

Got it running? If LocalHarness saved you an API bill, a star helps other local-LLM folks find it.

Running the harness on a different machine than the model

The harness and the model server are separate processes talking HTTP — they don't need to share a machine. A laptop can run agents against a model served elsewhere on your network: localharness init --endpoint http://<server-ip>:8000/v1. Two things to know:

• Tools run where the harness runs. bash/file tools execute on the client machine; the model server only sees text in, text out. Pointing a harness at a server doesn't let anyone act on the server.
• Secure the endpoint. Inference servers ship with no authentication by default. On a network with untrusted devices, start the server with an API key (e.g. vLLM --api-key) and set provider.api_key to match; for access from outside your LAN use a private overlay network (Tailscale/WireGuard). Never port-forward a bare endpoint to the internet.

Platform support

Tools run where the harness runs, and bash_exec always launches a real bash — never /bin/sh, never WSL. What differs per platform is how that bash is found.

Linux

• bash_exec runs bash -c with the bash on PATH (falling back to /bin/bash), so brace expansion, [[ ]] and arrays behave as written even where /bin/sh is dash.
• CI runs on Ubuntu; the Linux path is the one exercised by every test run and the DGX Spark live validation. The doctor GPU checks are Linux-only.

Windows

• Native, no WSL required. Install Git for Windows. The harness looks for Git\bin\bash.exe under %ProgramFiles%, %ProgramFiles(x86)% and %LocalAppData%\Programs, and deliberately skips the WSL launchers (System32\bash.exe and the Store alias under WindowsApps): without a distro they print UTF-16 garbage, and with one they act on a different filesystem than the native file tools.
• It picks the Git\bin\bash.exe wrapper over Git\usr\bin\bash.exe on purpose: the wrapper puts /usr/bin on PATH, so coreutils (mkdir, ls, cp, …) resolve no matter which shell started the harness. Git\usr\bin\bash.exe launched directly inherits a PowerShell PATH with no coreutils on it. To use a different bash, set LOCALHARNESS_BASH to a wrapper-style executable.
• A bash_exec command that could not RUN is a tool error on every platform, and its output is forwarded to the model inside the error so it can react (e.g. command not found). That is exit 127 (not found), 126 (not executable), and abnormal termination — a signal on POSIX, an NTSTATUS crash code on Windows. 126/127 are the shell's convention rather than a reserved range, so a program that picks those codes for its own reasons is reported as a failure it did not have; that is the deliberate side to err on. A command that ran and returned any other non-zero code is an ordinary result with exit code N on the first line: grep with no match and test -f on a missing file are answers, not faults. A timeout kills the command's whole process tree (a job object on Windows, the process group on POSIX).
• Paths: the file tools accept Windows or POSIX paths, relative to the harness working directory. /tmp/... maps to %TEMP%, which is where git-bash mounts /tmp, so the file tools and bash_exec agree on one tree. Inside bash_exec commands, use forward slashes — bash strips backslashes as escapes.
• Running the model on another machine (e.g. a DGX over Tailscale) and the harness on a Windows laptop is a supported setup; see the previous section.

CLI

Testing

uv sync --extra dev
uv run pytest                                          # hermetic — no model server needed
LOCALHARNESS_LIVE_VLLM=1 uv run pytest -m live_vllm    # opt-in tests against a live endpoint

Some bench scenarios read fixture files from /tmp/bench_fixtures/. Both pytest and localharness bench stage these automatically from tests/fixtures/bench/, so no manual copy step is needed from a repo checkout.

Reference architectures

LocalHarness is developed against two maintainer-tested hardware targets. Both must meet the practicality bar — 64k of KV-cache headroom and ≥9.5 tok/s single-stream. Four tested configs on architecture A (Qwen 3.8 / 3.6 and DeepSeek V4 Flash across llama.cpp and vLLM); the current default:

Start at docs/reference-architectures/. Per-hardware setup and tuning notes — timeouts, context budgets, runtime parity — are in gaps.md.

Documentation

• docs/reference-architectures/ — supported hardware targets and setup notes
• docs/specs/ — component specs

Status

Early stage (v0.13.1, pre-1.0). Interfaces and config schema may change without notice.

License

MIT