gauntlet

Your agent passed the eval. But did it actually improve — or did it remember the test?

Gauntlet is an integrity layer for evaluating stateful AI agents. Traditional benchmarks assume every trial starts clean. Stateful agents break that assumption: their memories, transcripts, and result directories quietly become the study material for their own next exam.

uvx --from gauntlet-guard gauntlet demo

Ten seconds, $0, no LLM, deterministic — and it asserts its own story (CI runs it):

ACT 1  baseline: memory-equipped v2 fails like v1        → verdict: PROVISIONAL
ACT 2  v2 "learns from feedback": task+answer → LESSONS.md
       re-grade the SAME task                             → verdict: KEEP (+1.0!)
ACT 3  guard scans the agent's memory vs the sealed manifest
       [EXACT] LESSONS.md :: frostgate-A (13/13 shingles) — hashes, not content
       instance RETIRED → fresh holdout instance → same agents, same memory
                                                          → verdict: PROVISIONAL
The +1.0 was memory, not intelligence.

How an agent can accidentally cheat your eval

None of this requires malice. It requires a good feature and a reused task set:

• MEMORY.md / LESSONS.md / now.md — "what I learned this run" writes, verbatim, the task you just evaluated it on;
• saved trajectories & transcripts — the full prompt survives in a log directory the next session retrieves from;
• result/eval output directories — fixtures copied into run artifacts stay agent-readable;
• vector stores / retrieval memory — the paraphrase you thought was safe is 6 shingles away from a near hit;
• generated helper artifacts — the "summary" file that quotes the task to explain it.

If your agent stack has any of the above, your evals are measuring memory and capability as one number. gauntlet separates them:

   without gauntlet                                  with gauntlet
   ─────────────                                     ─────────────
   agent → eval → "improved!" → ship                 agent → sealed run (isolated slot,
              │                                     env fingerprint, tamper-evident seal)
              ▼                                              → blind grade (keep/revert/
        writes task text into                                 provisional, bootstrap CI)
        memory / transcripts                                  → guard scans memory vs sealed
              │                                                 manifest (content-free report)
              ▼                                              → leaked instance RETIRED → fresh
        next eval "passes better"                              holdout regenerated → verdict
              │                                                you can actually ship on
              ▼
        production surprise

Quickstart

pipx install gauntlet-guard        # or: uv tool install gauntlet-guard
uvx --from gauntlet-guard gauntlet guard selftest   # prove the scanner before trusting it

# the loop, end to end:
gauntlet run init exp --tasks tasks.json --arms harness,raw --repeats 3
gauntlet run prep   exp --slot t-xxxxxxxx --fixtures fx/
gauntlet run exec   exp --slot t-xxxxxxxx -- python3 my_agent.py
gauntlet run blindpack exp --out pack              # pseudonyms only; judges see no arms
gauntlet run status exp
gauntlet grade    exp --primary harness --baseline raw
# exit code IS the verdict: 0 keep · 1 revert · 2 provisional · 3 integrity failure

# the memory audit:
gauntlet manifest add ~/.private/eval/manifest.jsonl --instance task-0001 --seed s-77 prompt.txt
gauntlet guard scan   ~/.private/eval/manifest.jsonl --store ~/agent/LESSONS.md --store ~/agent/logs/
# findings cite hash references, never content — safe to paste into issues

We ran our own gauntlet first

The first real scan target was the author's actual agent infrastructure: 12 sealed task instances, scanned against long-term memory files, journal logs, working-state directories, and evaluation run results. 271 files, 3.1 seconds, zero runtime dependencies. The agent's memory files were clean — but task fixtures had persisted verbatim into run-result directories: real, previously uncounted contamination vectors, calibrated correctly (NEAR for fixture content, TRACE for shared boilerplate). A control scan of unrelated notes: zero false positives. The machine-readable report contains zero task text — verified by assertion in CI.

Design principles

1. The scanner must not leak. Findings are file, position, counts, hash references — never matched text. The report is safe to publish even when the task is secret. (ADR-0002)
2. Manifests are private keys. Task content lives only in the manifest; keep it out of every agent-readable surface. (ADR-0007)
3. Deterministic first, judge second. Shingle/hash evidence outranks LLM opinion; judges enter only as external blind score files. (ADR-0006)
4. Leaks are lifecycle events, not warnings. Detection → retire → regenerate from an unused seed. A holdout that leaked once is dead.
5. Verdicts over scores. keep / revert / provisional with ≥N repeats and reported spread; provisional means the claim failed to be proven. (ADR-0005)
6. Decisions are written before they are implemented. The verdict engine's rules were frozen in docs/GRADE_DESIGN.md before the code existed; the demo asserts its own narrative in CI.

Status

Built from a battle-tested private protocol: doctrine changes stay PROVISIONAL until they beat the previous version on repeated, holdout-guarded evals. gauntlet is that gauntlet, made a tool.

The repo is the receipt

Decisions precede code and both are public: docs/adr/ records every frozen call (10 ADRs, including one that killed a version-inflating release), ROADMAP.md pins non-goals as firmly as targets, SECURITY.md treats "silent untrustworthiness" as the vulnerability class, CONTRIBUTING.md states the gates honestly (unmasked exit codes, dual-version verification), and the negative-result issue template is what measurement culture should look like.

Development

uv sync
uv run pytest                          # 41 tests, incl. the demo's self-assertion
uv run ruff check && uv run ruff format --check
uv run ty check src/
uv run gauntlet demo                   # the story must hold, or CI goes red

License

MIT