Nemisis

CrashCheck proves an AI patch survives a real crash, not just that its tests pass.

Nemisis is the command-line tool; CrashCheck is the check it runs. Your tests call the handler once, in one process, with nothing going wrong. CrashCheck runs the patch in a real worker process, kills it the moment a write becomes durable, restarts it, replays the same event, and reads what actually survived. One command, one verdict, an exit code CI can block on.

A coding agent fixes "retries sometimes credit an order twice". Its patch passes the test suite. It passes an ordinary call-it-twice check. Then a worker is SIGKILLed right after the $25 credit hits disk, the retry runs in a fresh process, and the account holds $50.

$ uv run nemisis check --base fixture:sqlite-credit-v1/buggy \
    --candidate fixture:sqlite-credit-v1/misleading-green --corrected fixture:sqlite-credit-v1/atomic

verdict: PATCH_FAILED_STILL_REPRODUCES
timeline: $25.00 durable -> SIGKILL -> fresh worker -> $50.00

Exit 1 blocks the merge. The same frozen crash replayed against the real fix exits 0.

Try it

Needs Python 3.12+, uv, and a POSIX machine (macOS or Linux).

uv tool install "git+https://github.com/Alex-lop/Nemisis@main"
# or, once 0.2.0 is published: uv tool install nemisis
nemisis check --base fixture:sqlite-credit-v1/buggy \
  --candidate fixture:sqlite-credit-v1/misleading-green \
  --corrected fixture:sqlite-credit-v1/atomic

About two seconds. Then replay the frozen crash against the fix (the capsule: path is printed by check):

nemisis replay .nemisis/repros/double-credit/*/capsule.json \
  --source fixture:sqlite-credit-v1/atomic --role corrected

The checkout is for developing Nemisis and for Verify the project; every uv run nemisis below is that checkout running the same command an installed nemisis runs.

git clone https://github.com/Alex-lop/Nemisis.git && cd Nemisis
uv sync --frozen --dev

What it does

CrashCheck runs the exact patch in a real worker process and treats every durable write as a place the process could die.

1. Hunt on the base. Two fixed kill points are tried on the buggy tree before the candidate is even read; the one that reproduces the duplicate is frozen into a content-addressed capsule.
2. No-crash control. The base is delivered the same event twice with no kill. It ends exactly once, so the duplicate needs the crash: this is a crash/retry bug, not a broken handler.
3. Kill, restart, replay. Five fresh worlds per tree. Each one seeds a database, waits until the credit is durably committed, SIGKILLs the whole process group, confirms exit -9, starts a fresh worker, replays the byte-identical event, and reads the database through an independent read-only connection.
4. Sweep every commit of a claimed fix. A patch that passes step 3 is then killed once after each store commit it makes. A handler that marks first and credits second passes step 3 and loses the credit here.
5. Decide from durable state and process receipts. Balance, ledger rows, marker count, PIDs, exit codes, worker nonces, tree digests. Five worlds must agree or there is no verdict.

Try to fool it

The checker was red-teamed by hand until it stopped losing, and CI now red-teams it on every run: nemisis redteam renders handlers from a grammar over store operations (guard, credit, mark, the atomic call, the same calls inside try/except or a retry loop, and the writes around the store that hostile reviews wrote by hand: a file beside, above, under ~ or $TMPDIR, a file tidied away before returning, a raw SQL write, a table, a pragma, a re-pointed row, a world-detection attempt; a quarter of them through a helper function), runs check on each, and compares the verdict with an oracle computed from the operation sequence alone, in either scenario's vocabulary (--scenario). Ten fixed-seed cases run in the normal suite; a nightly workflow runs three hundred per scenario. Three hand-written handlers that fooled an earlier engine ship as fixture refs, so the claim above is one flag away for anyone:

uv run nemisis check --base fixture:sqlite-credit-v1/buggy --candidate fixture:sqlite-credit-v1/mark-first

Or generate a hundred and read the disagreements (there should be none; one is a checker or oracle bug and the most valuable thing you can send):

uv run nemisis redteam --cases 100 --seed 1 --out ./redteam

Or write your own in thirty seconds:

uv run nemisis export fixture:sqlite-credit-v1/buggy ./my-candidate
$EDITOR ./my-candidate/app/credits.py
uv run nemisis check --base fixture:sqlite-credit-v1/buggy --candidate ./my-candidate

A handler that writes outside the store still runs. It forfeits the verdict instead of earning one. raw-sql is the textbook fix written as one raw SQL transaction on the store's database: correct, and unjudgeable, because a write the store did not make has no kill point. CrashCheck names the write and the store call that expresses the same fix (store.credit_and_mark(...), see the store API) instead of guessing a verdict. shadow-table keeps its dedup flag in a table it creates inside the store's own database: an earlier engine blessed it while a crash between that write and the credit left the customer unpaid forever. tail-bytes is the guarded atomic fix (processed, then credit_and_mark) followed by sixteen bytes appended past the database file's last page: the nightly red team caught the engine issuing verdicts on it for five nights, and blessing it twice (SQLite's own close had tidied the bytes away before the engine looked); the engine now reads the file before the worker may exit, pins the file whole during a delivery, and holds the write-ahead log to its frames. Attribution now reads the whole database file (schema, the header fields a commit never changes, every row with its rowid) and the whole world the worker runs in (its working directory and the two above it, HOME, TMPDIR, the bound tree entry by entry, with every entry's permission bits, flags, attributes, and modification time), before and after every delivery, and the raw header and length of the database file itself. Three hostile reviews found forty-eight handlers an earlier engine blessed; twenty-eight of those shapes are pinned as refusals. What it cannot read it does not claim; the boundary lists the channels that remain.

Let Nemotron write the patch

The hackathon story, made literal. NVIDIA's Nemotron on Nebius Token Factory plays the coding agent: it gets the bug report, the buggy module, and the store API, and nothing about how CrashCheck kills or judges. Its module is accepted only after deterministic checks (signature, imports, no private attributes), becomes an ordinary candidate tree, and is judged like any other.

The --issue path below is inside the checkout. export copies a fixture tree, not the issue text, and no other command prints it, so this section needs the clone from Try it.

export NEBIUS_API_KEY=...   # without it: exit 2, nothing written
uv run nemisis propose-patch --issue src/nemisis/fixtures/sqlite_credit_v1/issue.md \
  --base fixture:sqlite-credit-v1/buggy --out ./nemotron-candidate
uv run nemisis check --base fixture:sqlite-credit-v1/buggy --candidate ./nemotron-candidate

The report gains a Candidate author card with the model's receipt. It is labelled LIVE only for a real Token Factory call; injected clients are MOCKED and say so. This tree has no key, so no LIVE receipt exists yet. init --nemotron is the second, smaller model job: proposing the contract's catalog binding, candidate-blind. See docs/LIVE_SETUP.md.

Point it at your code

Read this before init. The alpha judges one handler shape and nothing else: a top-level synchronous def handler(store, event) with exactly two positional parameters, no defaults, no *args, no **kwargs, no alias or re-export. Every durable write goes through the store CrashCheck injects as the first argument (CreditStore: processed, credit, mark_processed, credit_and_mark; InventoryStore: reserved, reserve, mark_reserved, reserve_and_mark; the store API), against the scenario's schema, on the scenario's event. Your own connection, your own tables, your own payload are outside it. init reads the signature and nothing more, so apply_credit(conn, event) that runs SQL on conn mints a contract and then ends at EVIDENCE_INCOMPLETE, exit 2, because the handler raised AttributeError before the durable checkpoint. That is a narrow shape and most handlers are not already in it: porting one means rewriting its storage calls as store calls, and what survives is the part with the crash window in it. Inside that shape, the handler body is anything you like.

The contract pins the base tree digest, so the fix lives on a branch while the base branch stays at the tree the contract was accepted for. Commit the fix on main and check --base main exits 2: contract originating base digest differs from the supplied base. Committing the contract is safe; .nemisis/ is outside the digest.

nemisis init --issue issue.md --target app.credits:apply_credit --base main
nemisis init --issue issue.md --target app.credits:apply_credit --base main \
  --accept-contract PASTE_PRINTED_DIGEST
git checkout -b fix-double-credit   # the fix is committed here, not on main
nemisis check --base main --candidate HEAD --scenario .nemisis/config.json

Commit the accepted .nemisis/config.json on the base branch, then copy the example workflow into .github/workflows/ to run it on every pull request. Without a checkout, take it from main:

curl -o .github/workflows/crashcheck.yml \
  https://raw.githubusercontent.com/Alex-lop/Nemisis/main/.github/examples/crashcheck.yml

Every wall-clock wait in the kernel is one budget, ten seconds re-armed for each phase of each world (the worker's hello, reaching the next store commit, finishing the delivery), and its expiry is an EVIDENCE_INCOMPLETE whose message names the phase, the commits seen so far, and the budget. On a slow or loaded machine, NEMISIS_WORKER_TIMEOUT_SECONDS=30 raises it; no receipt depends on the value, and a value outside 1 to 600 is refused rather than clamped.

Two scenarios, one kernel

The kernel is written once; a scenario is one object (schema, seed, store, probe, predicate, words). sqlite-inventory-v1 is the second: an order reserves two units of a SKU, stock goes 10 to 8, and a crash between the decrement and its marker oversells to 6. It is the bug the original differential verifier could only mark UNRESOLVED, decided:

uv run nemisis check --base fixture:sqlite-inventory-v1/buggy --candidate fixture:sqlite-inventory-v1/mark-first

The scenario is inferred from a fixture base ref; init --scenario sqlite-inventory-v1 binds your own app.inventory:reserve_inventory the same way the credit contract does.

What it never does

• Never pushes, merges, comments, or touches your git history. It writes .nemisis/ and exits.
• Never upgrades a label. Local runs are LOCAL, the packaged case is FIXTURE, injected model clients are MOCKED, LIVE needs a genuine provider receipt. Missing evidence fails closed.
• Never lets a model near the verdict. Models write patches or propose catalog IDs; deterministic code owns probes, kill points, and decisions.
• Not a sandbox. Local mode is for a trusted checkout; the GitHub Action refuses fork PRs until the kernel runs inside a Token Factory Sandbox. Kill points are store commits, so durable state a handler keeps outside the store (a dedup file, another database) has windows CrashCheck cannot reach, and says so.

Verify the project

From the checkout in Try it; an installed tool ships no tests to run.

uv run ruff format --check src tests && uv run ruff check src tests
uv run mypy src tests
uv run pytest

The claims below name the tests that prove them, and tests/test_readme_truth.py fails if a link, an image, or a test count here goes stale:

The terminal captures are regenerated by the vhs tapes in docs/assets/screenshots/.

Docs

Product contract · Architecture · Security boundary · Proof ledger · Benchmark · two-and-a-half-minute demo · Pitch · Live setup · Status · Decisions

The original differential verifier (nemisis verify --fixture idempotency-retry) is still shipped; it is where CrashCheck's crash-retry row comes back UNRESOLVED and CrashCheck begins.

Apache-2.0. See LICENSE.