agent-build-gates

Quality gates for agent-built code. Five of them, all pure standard library, plus an optional sixth.

When an agent writes code at volume, the failure that costs most is not a bug. It is work that looks complete and never touched the service it claims to demonstrate: a class that stands in for a queue, a validator that returns ALLOW because the API was unreachable, a fallback chain whose models never got called. Asking for honesty does not catch that. A script that fails the build does.

pip install agent-build-gates

Zero dependencies. ship-gate needs the [strands] extra, because it drives real agent runs and costs money; the other five are pure standard library.

The gates

no-sim-check

Flags substituted integrations: code that fabricates a success for a call it could have made.

no-sim-check $(git ls-files '*.py')

It looks for substitute-object vocabulary (mock, stub, fake, dummy, hardcoded), returns that fabricate a result, "in production this would" deferrals, and a bare return True straight out of an except. Two scoping rules keep the signal usable, both learned from classifying every hit in a 275-file repo:

• Identifier-aware boundaries. A plain \b stops at underscores and CamelCase humps, so class MockSQSQueue, mock_client and _simulate_human_response are invisible to it while their docstrings trip it. This uses boundaries that break on both.
• Prose is exempt. The two vocabulary rules never fire on comments or docstrings, because a comment cannot fake an integration. Only code can. The deferral rule still reads comments, since a deferral comment is the marker for a call that was never made.

Escape a justified line with a trailing # nosim:ok <reason>.

The question the tool cannot answer for you, and the one that separates a real hit from noise: was a real call available and skipped? A helper that genuinely raises to drive a recovery path is fault injection and legitimate. Code that fabricates a success is not.

check-no-aws-ids

Blocks AWS account identifiers from entering any tracked text file (binaries skipped): 12-digit account ids, SSO admin profile strings, sso_account references, and ARNs carrying an account field.

check-no-aws-ids $(git ls-files)                  # explicit files
check-no-aws-ids                                  # every tracked file in the current repo
check-no-aws-ids --frozen-regex REGEX $(git ls-files)   # skip lines REGEX matches (append-only history)

Paths resolve against your working directory, so it behaves the same installed from a wheel or vendored into a repo. Extend the known-safe literals at the call site rather than editing the module: scan(files, allow=ALLOW | {"123456789012"}).

Escape a single line with noaws:ok, matched anywhere on it, so it works as # noaws:ok reason in Python and <!-- noaws:ok reason --> in Markdown. Anything that must legitimately contain account-shaped strings needs this: this gate's own tests, and documentation like the paragraph above. The marker covers only its own line, never the rest of the file.

check-no-local-paths

Blocks local machine paths from files bound for a public repo: absolute home directories (/Users/<name>/, /home/<name>/), home-relative project folders (~/Code, ~/Documents, ~/Desktop, ~/Downloads, ~/Docs), /Volumes/, and macOS per-user temp (/private/var/folders, /private/tmp). Plain /tmp is not flagged.

check-no-local-paths $(git ls-files)
check-no-local-paths --frozen-regex '"id": "obs-(0717|0750)"' log.jsonl

Binary files are skipped. Each hit prints as file:line:text, the text cut to 160 characters. Escape a single line with localpath:ok, as # localpath:ok reason in code or <!-- localpath:ok reason --> in Markdown. For an append-only file whose old lines were public before the check existed and cannot be edited, --frozen-regex skips the lines it matches, so the old lines are named instead of the whole file being skipped and new lines are still checked.

check-denylist

Blocks literal strings that must not reach a public repo, such as another team's project name. Those cannot be built into a published gate without publishing the names being protected, so they live in a file your repo does not track, one term per line (blank lines and # comments ignored).

check-denylist $(git ls-files)                       # reads ./.git-denylist
check-denylist --denylist ~/terms.txt --ignore-case $(git ls-files)

Matching is case-sensitive unless you pass --ignore-case. A hit prints denylist: '<term>' found in <file> and never the surrounding line. A missing denylist file exits 2, not 0: a gate that silently scans nothing is worse than one that fails. A caller that wants "no file, no check" tests for the file first. There is deliberately no per-line escape marker, because one in a tracked file would let a protected name through with a comment.

eval_harness

Multi-run evaluation with confidence intervals and significance testing, so a single lucky run cannot be reported as a result.

from agent_build_gates import Case, run_suite, gate, label_match, wilson, case_means

cases = [Case("input text", expected="positive")]  # at least 6 cases to compare to a baseline

result = run_suite(cases, my_run_fn, {"correct": label_match}, n=5)
passed, reasons = gate(result, baseline=previous, min_quality=0.8, metric="correct")
print(wilson(case_means(result, "correct")))

run_fn(input) -> (output, tokens) is yours, so the harness never assumes a framework. gate combines a quality floor, a cost ceiling and a paired sign-flip test against a frozen baseline, and returns every reason it failed rather than the first.

The case is the unit of evidence, not the run. The n runs of each case are averaged into one score before any interval or test, because repeats of one prompt are not independent samples. A paired test on k cases cannot return p below 2 * 0.5**k, so with fewer than 6 cases the baseline comparison cannot detect a regression; gate then fails with "regression undetectable" rather than passing. label_match scores an answer correct only when the whole normalised output is the expected label, so "positive, not negative" does not count for both. wilson gives the interval on the rate; pass it case_means, not the flat run scores.

Each result carries cases_sha256, a fingerprint of its case set, and whatever you pass as run_suite(..., meta={...}) (model, temperature, a prompt hash). Both are saved with a baseline, and gate refuses to compare against a baseline measured on a different case set.

ship-gate, behind an extra

ship_gate composes the harness into one auditable GO or NO-GO verdict over real agent runs, and writes the verdict plus the underlying runs to JSON so the decision can be re-examined. It drives a framework and it spends money, so it is optional:

pip install 'agent-build-gates[strands]'

Importing agent_build_gates never pulls that in. Import agent_build_gates.ship_gate directly when you want it; without the extra it raises an ImportError naming the install command.

Testing your gates

These gates are themselves tested, which is the point. A tripwire with no test proving it fires is an unverified claim about the thing that verifies everything else. When this package's own anti-simulation gate was finally given paired positive and negative controls, it turned out to be wrong in both directions at once: it fired on comments describing deliberate fault injection, and it missed the substituted classes it existed to catch. Fixing it took one repo from 133 reported hits to 0 and surfaced nine real substituted integrations that had been invisible.

Measure the precision and recall of your own gates before you trust a green run.

Changes

• 0.3.0 (2026-10-19): added check-no-local-paths and check-denylist.

License

MIT.