Proofhouse

A local workspace for revising prompts with evidence you can trace: every PASS names the exact prompt, outputs, checks and judgements it came from.

I built Proofhouse because I kept rewriting the same prompt for different models and never had a good answer for which version was better, or whether a revision had quietly dropped a requirement. You give it an objective and a target model; it asks its clarifying questions in one batch and turns your answers into constraints that every later revision must keep. You record each prompt revision and the outputs a model actually produced, declare the checks that matter, and Proofhouse tells you which revision passes, which constraint lacks evidence, and what changed between revisions.

It runs on your machine with no API key. Proofhouse writes the prompts it needs as packets; you run them in the agent or model you already use and paste the results back. It is a local tool, not a hosted service, and I make no benchmark or quality claims for it.

https://github.com/user-attachments/assets/17ec6d22-1ba7-4952-bec0-d3f24c9b29ce

44-second demo of the standalone app. You type one sentence, pick a model, answer seven questions, and get a compiled prompt where each rule is traced back to the answer that produced it. The scene is an authored example, not a recorded run. Remotion draws it; the source is in videos/readme-hype/studio.

Portfolio: km-it-ops.github.io

Try it in two minutes

uv tool install proofhouse
proofhouse-compiler demo

That runs the reference workflow end to end, offline, from files shipped in the package. It replays recorded outputs: a security-advisory summary whose first revision invents an exploitation claim and fails its checks, and whose second revision passes without losing a constraint. It does not call a model. On Windows, set $env:PYTHONUTF8='1' first for reliable console output.

With pip: python -m pip install proofhouse, then proofhouse-compiler demo.

From a clone, the same check is uv run python scripts/reference_workflow.py (or python -m pip install -e ".[test]" without uv). The by-hand steps later on this page copy examples/reference-advisory, which is in the repository, not in the wheel.

How it works

proofhouse-compiler optimize renders the framework's clarify / compile / self-heal prompts as packets you run in the agent or model of your choice, and checks what you paste back against the criteria you declare. It does not call a model and it does not rate quality.

These are the reference workflow's steps, and they run as written. pc below is proofhouse-compiler (uv run proofhouse-compiler from a clone). Work in a scratch folder holding copies of the example inputs:

mkdir -p build/try && cp examples/reference-advisory/* build/try && cd build/try

1. Start a case and answer one batch of questions. Proofhouse writes a clarify packet, advisory-case/01-clarify.md. You run it in your own agent and save the answers as advisory-case/answers.json (for this example, copy answers.json there). Compile turns each answer into an accepted constraint (K1-K3) that every later revision receives.

pc optimize new --case advisory-case --objective-file objective.txt --model "Claude Sonnet 5" --preset efficient
pc optimize compile --case advisory-case

2. Say how each constraint will be checked. Checks can look at the prompt text or at what a model returned (--target output), and some can be a human judgement (--manual).

pc optimize constraints add --case advisory-case --text "Cite the advisory section for each claim"      # K4
pc optimize criteria add --case advisory-case --id AUD --must-contain "SOC analysts"
pc optimize criteria add --case advisory-case --id UNK --target output --must-contain UNKNOWN
pc optimize criteria add --case advisory-case --id NOGUESS --target output --must-not-contain "exploited in the wild"
pc optimize criteria add --case advisory-case --id LEN --target output --max-words 120
pc optimize criteria add --case advisory-case --id CITE --target output --regex "\(s\d\)"
pc optimize criteria add --case advisory-case --id ACC --target output --manual "Every claim matches the advisory"
pc optimize constraints link --case advisory-case --id K1 --criterion AUD
pc optimize constraints link --case advisory-case --id K2 --criterion UNK
pc optimize constraints link --case advisory-case --id K2 --criterion NOGUESS
pc optimize constraints link --case advisory-case --id K3 --criterion LEN
pc optimize constraints link --case advisory-case --id K4 --criterion CITE
pc optimize constraints link --case advisory-case --id K4 --criterion ACC

3. Record a revision and what it produced, judge the manual check, then check it.

pc optimize record --case advisory-case --prompt-file prompt-v1.txt
pc optimize output add --case advisory-case --revision 1 --output-file output-v1.txt --input-id EC-2026-017 --input-file advisory.txt --model "Claude Sonnet 5" --source "pasted from host agent"
pc optimize verdict --case advisory-case --revision 1 --criterion ACC --run R1 --result fail --note "claims active exploitation; the advisory does not say that"
pc optimize check --case advisory-case        # exit 3: v1 FAIL  UNK NOGUESS CITE ACC fail

4. Revise without losing anything. The revise packet carries your feedback, the original answers and every accepted constraint. Run it in your agent, save the new prompt, and record revision 2 the same way.

pc optimize revise --case advisory-case --feedback "It claimed active exploitation, which the advisory never states."
pc optimize record --case advisory-case --prompt-file prompt-v2.txt
pc optimize output add --case advisory-case --revision 2 --output-file output-v2.txt --input-id EC-2026-017 --input-file advisory.txt --model "Claude Sonnet 5" --source "pasted from host agent"
pc optimize verdict --case advisory-case --revision 2 --criterion ACC --run R2 --result pass
pc optimize check --case advisory-case        # exit 0: v2 PASS, K1-K4 satisfied

5. Compare, report and hand it on. compare shows what newly passes or regresses between revisions, report writes the evidence as Markdown, and export / import move a case to another workspace, refusing any file that does not match the bundle's manifest (the manifest is not signed).

What a PASS proves, and what it does not

Details: docs/architecture.md, decision 0002, evidence format.

Two ways in

Proofhouse is two products in one repo, at different levels of maturity. The PromptOps skill and framework (v1.3), the conversational meta-optimizer with current frontier model profiles, is the most mature part of the project. The headless compiler's maturity is listed under Experimental and prototype parts.

Both entries use the same framework (proofhouse-framework.json) and the same model notes. The framework also covers Efficient / Balanced / Thorough token presets, loops for recurring agents (trigger, body, exit, escalation), prompt audits with missing-context labels, and agent design (permission maps, tool boundaries, stop conditions). The skill is installed to ~/.cursor/skills/proofhouse (or ~/.claude/skills/proofhouse with --host claude) and the installer checks it (the bundle ships inside the package; tests/test_skill_bundle.py keeps it identical to skills/proofhouse/); without the console script, python -m zipfile -e skills/proofhouse/proofhouse.skill ~/.cursor/skills (or ~/.claude/skills) does the same. The skill can research an unfamiliar model with the host agent's tools; the command line never goes online, and uses your own notes (models remember) or a generic profile labeled fallback.

Supported models (framework v1.3)

Built-in modelNotes cover prompting quirks, API ids, and cost and caching levers for each model:

Profiles last reviewed 2026-09-03; a profile older than 90 days is flagged stale. None of the 18 profiles cites sources yet, so every profile is labeled unverified. They are prompting guidance, not vendor specifications. proofhouse-compiler models list shows per-model ids, aliases, and dates.

Full profiles: proofhouse-framework.json · human-readable proofhouse-framework.md

Install and verify

One Python package, version 0.4.2, ships two console scripts:

Supported: the optimize case workflow, models, install-skill, demo, and validate / compile / closed-loop. The experimental commands are listed under Experimental and prototype parts.

uv sync --extra test                                    # package + pytest into .venv
uv run proofhouse-compiler doctor                       # doctor: success
uv run proofhouse-compiler closed-loop tests/compiler/fixtures/closed_loop_requirements_minimal.json --repair-budget 1
                                                        # closed-loop: PASS (structural)
uv run proofhouse validate --dataset evals/datasets/prompt_audit_cases.jsonl
                                                        # Dataset validation passed: ...
uv run pytest -q                                        # all passed, 1 deselected (live tests are opt-in)
uv run proofhouse-compiler demo --quiet                 # reference workflow: OK
uv run python scripts/reference_workflow.py --quiet     # same run via the script wrapper

CI runs the test suite on Linux, macOS and Windows with Python 3.11 to 3.14, runs the reference workflow against the built wheel, and runs proofhouse-compiler demo from that wheel outside the checkout. Everything above is offline: no network, no API key. Nothing here is a benchmark.

The compiler's closed-loop PASS is structural: the prompt compiles offline with the fake adapter and passes security gates, and its evidence lists what it did not measure. Its approved headless profiles are structured_minimal_v0 and structured_developer_v0.

Experimental and prototype parts

These ship in the repository but are not the supported path. Treat them as previews.

The headless requirements compiler's maturity is still PARTIAL. Full map with versions and diagnostic codes: docs/surfaces.md.

Design rules

• Stay lightweight by default. Tighten only for safety, agentic execution, or missing context.
• Never invent repository or project facts.
• Use exact missing-context labels: UNKNOWN, NOT SPECIFIED, NOT FOUND IN PROVIDED MATERIAL.
• Keep cybersecurity and sensitive-data work defensive, authorized, and privacy-preserving.
• No private chain-of-thought dumps. Concise rationales only.

It was designed for coding agents, Custom GPTs, Cursor skills, and security-adjacent AI work, where inventing facts or skipping a safety boundary is not acceptable.

Repository map

proofhouse-framework.*   Portable meta-optimizer spec (v1.3 model profiles)
skills/proofhouse/       Cursor / Claude Code skill bundle + artifact JSX
src/proofhouse/          Eval harness, headless compiler, optimize (cases, constraints, runs, checks, export)
examples/                Demo inputs, including reference-advisory/ for the reference workflow
scripts/                 Generators (model surfaces, skill bundle, contracts) and reference_workflow.py
docs/                    Scope, architecture, surfaces, evidence format, decisions, assurance statements
prompts/                 Core prompt, modes, modules, templates, Custom GPT pack
evals/                   JSONL datasets, YAML rubrics
apps/                    Experimental artifact and dashboard prototypes
tests/                   Test suite, contract schemas and fixtures

Internal mission reports and review corpora are not published in this repository. Public decisions are summarized in docs/decisions/, and each assurance statement in docs/assurance/ lists the exact revision, commands, results and exclusions.

Start here

• Reference workflow: one failed check to a verified fix, offline
• Product scope · Architecture · Surfaces and contracts · Evidence format
• Decision records · Assurance statements · Troubleshooting
• Quickstart · Showcase · Custom GPT setup
• Changelog · Contributing · Security policy

_{Custom GPT surface: PromptOps Architect powered by Proofhouse · MIT License}