nightward

A regression gate for AI-made changes. Your AI agent can run the gate, but it can't approve it.

When an AI agent fixes A, it often moves B, C and D too, and the test suite stays green because no test pins down B, C or D. You find the side effects later, fix them, and the fixes move something else. An agent loop has no reliable way to know when it is done.

nightward gives that loop a definition of "done" that the agent can check but not change:

1. Your tests capture what the code does today with a behavior(...) pytest fixture. You don't write expected values.
2. A human reviews those captures and approves them once. That baseline is a set of JSON files committed next to your code.
3. After every change, nightward captures again and lists everything that moved, grouped by feature (the blast radius). The gate stays closed until a human approves each change as intended or the code is fixed.

The agent gets run and status, on the command line or as MCP tools. Approving is a human CLI command, and its result is a file diff that goes through code review.

nightward run       # capture behaviors and compare them with the approved baseline
nightward review    # a human reads the diffs
nightward approve   # the human accepts what review showed (intended changes only)
nightward gate      # exit 0 only when nothing unapproved remains (CI, agent loops)

nightward is not a test generator, and it has no idea what is correct. It only makes sure that no change to a captured behavior passes silently.

What's new in 0.3.0, and what changes for 0.2 users: see the CHANGELOG.

Install

nightward needs Python 3.10+. Installing it also registers its pytest plugin, so the behavior fixture is available in every test.

pip install nightward
pip install "nightward[mcp]"      # + MCP server for AI agents

Other extras: [judge] (the Anthropic SDK, for the LLM judge) and [docs] (the PDF, Word and Excel adapters).

Quickstart (5 minutes)

In an empty directory, create a tiny app and a test that captures its behavior. app.py:

def checkout_total(items):
    subtotal = sum(i["price"] * i["qty"] for i in items)
    tax = round(subtotal * 0.1, 2)
    return {"subtotal": subtotal, "tax": tax, "total": round(subtotal + tax, 2)}


def user_login(username):
    return {"user": username, "status": "ok", "roles": ["member"]}

test_app.py writes no expected values. It captures what the code returns:

from app import checkout_total, user_login

CART = [{"price": 10.0, "qty": 2}, {"price": 5.5, "qty": 1}]


def test_checkout(behavior):
    behavior("checkout_total", checkout_total(CART), group="billing")


def test_login(behavior):
    behavior("user_login", user_login("alice"), group="auth")

1. Approve the current behavior as the baseline.

git init                    # skip in an existing repo
nightward init              # creates .nightward/ and git-ignores its per-run files
nightward run .             # runs pytest and captures; lists what moved (names only)
nightward review            # shows each diff; approve covers only what review showed
nightward approve --all
git add .gitignore .nightward   # commit baseline/, rejected/, judge/ (init ignores per-run files)
git add app.py test_app.py && git commit -m "nightward baseline"

run finds two NEW behaviors, review prints their JSON, and approve makes them the baseline:

Boundary: breached (2 unapproved)
unchanged=0 changed=0 new=2 removed=0
...
approved checkout_total (c9ff6ed9)
approved user_login (b86be5a5)

Boundary: intact
unchanged=2 changed=0 new=0 removed=0

2. Make the kind of change an agent makes. In app.py, change the tax rate 0.1 to 0.2 (in bash: sed -i 's/subtotal \* 0.1/subtotal * 0.2/' app.py). Then:

nightward run .             # the tests still pass
nightward gate              # exit code 1: not done
nightward status --json     # the same verdict, for a script or an agent
nightward approve --all     # refused: nobody has reviewed this change yet
nightward review            # the diff a human decides on

Boundary: breached (1 unapproved)
unchanged=1 changed=1 new=0 removed=0

group: billing
  - [CHANGED] checkout_total

boundary breached (1 unapproved) (as of the last run, ...)

{"boundary": "breached", "unapproved": 1, "changes": [{"name": "checkout_total", "kind": "CHANGED", "group": "billing"}], ...}

error: checkout_total changed since your review (the capture now in the store was taken by `nightward run` at ...). Run `nightward review checkout_total`, then approve what it shows.

[CHANGED] checkout_total
--- approved
+++ received
@@ -1,5 +1,5 @@
 {
   "subtotal": 25.5,
-  "tax": 2.55,
-  "total": 28.05
+  "tax": 5.1,
+  "total": 30.6
 }

3. Decide. If the change is a regression, fix the code: put 0.1 back, run nightward run . again, and it reports Boundary: intact, so nightward gate exits 0. If the change is intended, run nightward approve --all (it now succeeds, because review showed you this diff) and commit the updated .nightward/baseline/checkout_total.approved.json.

4. Optional: see the blast radius in a browser. nightward view builds a static, read-only dashboard and serves it on localhost. nightward view --no-serve only builds nightward-site/, which init already git-ignored.

For a bigger cascade (one tax tweak moves three customer-facing behaviors), run the petshop example from a clone of this repo.

What to commit

Commands

gate and status report the last run, so run nightward run again after every code edit. The full reference (removals, stale reports, merge conflicts, locking, and using the plugin inside an existing pytest job) is in docs/commands.md.

AI agents (nightward mcp)

An agent loop needs a definition of "done" that it can check but not change. nightward mcp is a stdio MCP server with two tools: nightward_run (run the tests, capture, recompute the verdict) and nightward_status (read the last verdict). approve and reject are not exposed: if the agent that makes a change could also approve it, the gate would turn into a changelog. The judge and the store are the humans' committed choices, so the agent can't pick a lenient judge or point the server at another store.

pip install "nightward[mcp]"                      # the mcp 1.x SDK
claude mcp add nightward -- nightward mcp         # e.g. Claude Code; run it in the project root

Both tools return the status --json shape. The agent is done only when boundary is intact and stale is false:

Setup for other MCP hosts, every argument and returned field, and the rules for the loop are in docs/mcp.md.

Gating AI text: the semantic judge

Free-text AI output changes its wording on every run, so an exact comparison breaches every time. Mark such a behavior semantic=True and commit a judge in pyproject.toml: [tool.nightward] judge = "anthropic:claude-haiku-4-5", or the key-free persona:editor. The judge only decides whether two outputs mean the same thing; approval stays human. Every ruling is recorded in the committed ledger .nightward/judge/, and a judge that can't rule falls back to the exact comparison. Details are in docs/semantic-judge.md.

What "intact" means

"Boundary intact" means no captured behavior changed, nothing more:

• The gate sees only what behavior() captures, so capture the behaviors you care about.
• The gate is a convention, not a sandbox. An agent with a shell could run approve or edit the baseline, but every bypass leaves a trace in git. Review baseline, ledger, scrub-rule and test diffs in every PR, protect your main branch, and let CI run nightward run && nightward gate from source.
• A semantic judge can be wrong, so its rulings are recorded for review.
• Normalization defines "the same": timestamps and UUIDs are masked by default (opt out with scrub=False), and dict key order is ignored.

Read the threat model and what nightward normalizes before you rely on a green gate.

Documentation

• CHANGELOG: what's new in 0.3.0, and upgrading from 0.2
• Commands and workflow: every command, removals, stale reports, merge conflicts, and the plugin in an existing pytest job
• What nightward normalizes: scrubbing, custom scrub rules, names, non-JSON values, and doctor
• Semantic judge
• AI agents (MCP)
• Dashboard
• Document & artifact inputs: PDF, Word, Excel, text and raw bytes
• Threat model
• CI recipe: GitHub Actions

Scope

In scope: pytest capture, the blast-radius diff, the gate, the loop signal, scrubbing and doctor, the static dashboard, the MCP agent gate (run and status, never approve), and the semantic judge. Out for now: PR-comment summaries, call-graph grouping, and other languages. See CONTRIBUTING.md.

MIT licensed.