consequence

See what your code would do before it does it.

That script really ran. It read its config, branched on it, opened the database, built its SQL. Nothing reached the disk, the network, or the table.

Contents

• Why this exists
• Install
• Three modes
• Worked example: an agent tidies up
• Worked example: a real migration tool
• What it sees
• Policies
• Using it in tests
• Using it in CI
• Python API
• Command reference
• How it works
• Check it yourself
• The limit — read this one
• Troubleshooting
• Reporting a problem
• Contributing
• Related work

Why this exists

terraform plan works because Terraform owns every effect a run can have. It knows what a resource is, so it can tell you what is about to change.

Python owns none of them. shutil.rmtree is a function call like any other, and the same question — what is this about to do — has no answer. The usual substitutes are a --dry-run flag somebody has to remember to implement correctly on every code path, or reading the code and hoping.

That got harder to live with once code started arriving faster than anyone could read it. An agent hands you forty lines that look reasonable. There is an rmtree in there somewhere, on a path assembled three functions away from where it is used. You can read it carefully every time, or you can run it and find out.

This gives you the third option.

Install

pip install consequence

or

uv add consequence

Python 3.10 and up. Linux, macOS and Windows, all tested in CI. The only dependency is tomli, and only on 3.10 — from 3.11 the standard library has tomllib. A tool you reach for because you are nervous should not bring a dependency tree with it.

Three modes

plan — nothing reaches the outside world

import consequence

with consequence.plan() as run:
    deploy()

run.print()

Writes land in a copy-on-write overlay, so when the program reads back the config it just wrote, it gets what it wrote. This matters more than it sounds. Programs write-then-read constantly, and a dry run that simply swallows writes sends the program down a branch the real run would never take, then describes that branch to you as if it were the plan.

guard — effects happen, but only the ones a policy allows

with consequence.guard("safe.toml"):
    agent.run(task)

Anything the policy does not allow raises Denied at the line that tried it, naming the rule that refused. Everything before that point really happened, which is the honest shape of a policy failure and not something to paper over.

audit — everything happens, and everything is written down

consequence audit --log run.jsonl deploy.py
consequence log run.jsonl --destructive

Worked example: an agent tidies up

examples/cleanup_agent.py is the kind of script you get when you ask something to tidy up a project. It is not malicious. It is plausible, and it is wrong in one specific way, and you cannot see which way by reading it quickly:

def main() -> None:
    root = Path.cwd()

    config = root / "config" / "app.toml"
    config.write_text("[app]\ndebug = false\nworkers = 8\n")

    shutil.rmtree(root / "config")  # <- there it is
    (root / "keep-me.log").unlink()

    connection = sqlite3.connect(root / "app.db")
    connection.execute("DROP TABLE IF EXISTS sessions")
    connection.execute("DELETE FROM audit_log WHERE created_at < date('now', '-30 days')")
    connection.commit()

    subprocess.run(["git", "push", "--force", "origin", "main"], check=False)

It writes the config, then deletes the directory containing it. Run it and you find out. Plan it and you are told — see the screenshot at the top of this page.

Try it:

git clone https://github.com/aviseth/consequence
cd consequence
uv run python examples/setup_demo.py
cd examples/demo
uv run consequence plan cleanup_agent.py

Worked example: a real migration tool

The demo above was written alongside this package, which makes it weak evidence. Here it is against sqlstep, a migration runner that has never heard of consequence, applying three real migrations to a real SQLite database:

$ consequence plan -m sqlstep -- --url sqlite:///app.db up

  - app.db              drop table legacy_sessions      drivers.py:175 in run_sql  DESTROY
  + app.db              create table schema_migrations  drivers.py:147 in ensure_table
  + app.db              create table users              drivers.py:175 in run_sql
  + app.db              create index idx_users_email    drivers.py:175 in run_sql
  + app.db              insert into schema_migrations   drivers.py:164 in record
  + app.db              create table orders             drivers.py:175 in run_sql
  + app.db              insert into users               drivers.py:175 in run_sql
  + app.db              insert into orders              drivers.py:175 in run_sql
  ...

Plan: 11 to create, 1 to destroy.
1 destructive effect(s). Nothing was performed.

One line in one migration drops a table. That is the thing you wanted to know before running it against production, and it is the thing that is easiest to miss in a directory of .sql files. Afterwards, app.db does not exist.

What it sees

Every effect carries the line that caused it, filtered down to code you wrote — not the stdlib frame twelve levels down where the write actually happened.

Effects are graded, and the report sorts by grade so the line that deletes a directory is never below the line that wrote a log file:

Policies

TOML, and small enough to read in one go.

# What this program may do. Anything not listed is refused.
default = "deny"

[filesystem]
read   = ["**"]
write  = ["./build/**", "./config/*.toml"]
delete = ["./build/**"]

[process]
allow = ["git status*", "git diff*"]

[network]
allow = ["api.github.com"]

[database]
allow = ["delete from audit_log", "insert into *"]
allow_schema_changes = false

Three rules, all load-bearing.

Deny beats allow. A path matching both is denied, so the meaning of a policy never depends on the order the rules happen to be written in. Nobody can review order-dependent safety.

There is no implicit default. You write allow or deny, and a policy without one is an error rather than a guess. allow is a monitoring posture; deny is a containment one. Choosing on your behalf is how a policy ends up meaning something nobody intended.

* does not cross a /; ** does. Otherwise /etc/* matches /etc/nginx/nginx.conf and the policy is far more permissive than it reads.

Filesystem rules are path globs and take the verbs read, write and delete. Every other domain takes allow and deny, matched against the target, the detail, or both together — so git names the program and git status* names the command, and you do not have to know which one the interceptor recorded where.

Using it in tests

def test_rendering_is_pure(no_effects):
    render_invoice(rows)  # fails the test if it touches anything


def test_the_build_stays_put(effects, tmp_path):
    build(tmp_path)
    effects.assert_only_under(tmp_path)


def test_nothing_shells_out(effects):
    parse(document)
    effects.assert_no("process.spawn")

Available on the effects recorder: assert_none(), assert_only_under(*roots), assert_no(kind), assert_nothing_destructive(), plus .effects and .writes() if you want to look yourself.

Tests that quietly write outside their tmp_path pass alone, pass in CI, and then fail six months later because two of them raced on the same file in somebody's home directory.

Only the test's own effects count. pytest is busy during a test — creating tmp_path, setting PYTEST_CURRENT_TEST, writing __pycache__ entries — and none of that is the test's doing. An effect counts if the test file appears somewhere in its call stack, so helpers you call are included and fixtures acting on their own behalf are not.

no_effects checks at teardown rather than blocking each write as it happens, so a failure shows you everything the test wanted to do instead of stopping at the first one. The cost is that pytest reports it as an error on a passing test. Call effects.assert_none() yourself if you would rather it fail in the body.

Using it in CI

check plans and exits non-zero if anything would be destroyed.

- run: pip install consequence
- run: consequence check --policy ci.toml scripts/migrate.py

--allow-destructive narrows check to policy denials only, for a program whose job genuinely is to delete things.

Python API

import consequence

# --- sessions -------------------------------------------------------------
consequence.plan(policy=None, network="block")  # nothing escapes
consequence.guard(policy)  # policy, or a path to one
consequence.audit(policy=None)  # watch, do not stop

# --- reading a run --------------------------------------------------------
with consequence.audit() as run:
    do_the_thing()

run.effects  # every Effect, in order
run.destructive  # the ones you cannot undo
run.denied  # the ones policy refused
run.where("file.write", min_severity=consequence.Severity.MODIFY)
run.counts()  # {"file.write": 3, "net.request": 1}
run.to_json()  # plain dicts, for a log
run.report()  # the text the CLI would print
run.print()

An Effect carries kind, target, detail, severity, allowed, reason, performed, origin (the call site) and frames (the trimmed stack).

Denied is raised at the offending line in guard mode. Blocked is raised when plan mode is asked to do something it cannot honestly simulate — reaching the network is the only one.

Command reference

consequence plan   [--policy P] [--network block|allow] [--show-reads] [--limit N] [--json] TARGET
consequence run     --policy P  [--show-reads] [--limit N] [--json] TARGET
consequence check  [--policy P] [--allow-destructive] [--network ...] [--json] TARGET
consequence audit  [--policy P] [--log FILE] [--json] TARGET
consequence log     FILE [--kind KIND] [--destructive] [--json]

TARGET is a script path or -m module. Arguments for the program go after --:

consequence plan -m mypkg.cli -- --force --output build/

run requires a --policy. Running for real with nothing able to say no is what audit is for, so it refuses rather than quietly doing the harmless thing.

How it works

Interception happens at the lowest layer that still knows what is going on. Patching os.unlink catches Path.unlink and shutil.rmtree for free, because they both call it; patching Path.unlink as well would count the same deletion twice. The exceptions are subprocess and sqlite3, where the interesting detail — the argv, the statement — only exists at the top layer and is gone by the time anything reaches a syscall.

SQLite goes through the authorizer callback, not through string matching on your SQL. SQLite calls it while preparing each statement, with the action code and table name already parsed. That cannot be fooled by unusual SQL, it sees statements issued through any API including executescript and triggers, and refusal happens inside the engine rather than in a wrapper somebody could bypass. In plan mode your connection is an in-memory copy of the real database, seeded with backup(), so INSERT then SELECT returns the row and the file on disk never opens for writing.

Failure is loud. If an interceptor cannot be installed, install() raises rather than continuing. An earlier version suppressed that, and reported a clean plan for a program that dropped a table.

Everything is restored on exit, including when the body raises. Nested sessions share one set of patches and are reference counted, so an inner session leaving does not strip the interceptors out from under an outer one.

Check it yourself

You should not take the last section on faith. examples/fidelity_check.py runs your own program twice from identical copies — once planned, once for real — and compares:

python examples/fidelity_check.py --seed ./project -- -m mypkg up

planned 15 effect(s); the real run had 15

safety    the plan changed nothing on disk
fidelity  every one of the 13 real effect(s) was predicted

safety is a content hash of every file before and after the plan. fidelity is whether the plan predicted everything the real run actually did. Point it at a copy: the second run is real.

The limit — read this one

This is not a sandbox. It is in-process. The interceptors are patched into builtins, os, shutil, subprocess, socket and sqlite3 in the same interpreter as your program. Code that goes around them is neither seen nor stopped:

• a C extension calling unlink(2) directly
• ctypes into libc
• a fork that re-execs
• anything that captured a reference to the original function before the session started

So: run it on code you are unsure about, not on code you believe is hostile. For hostile code you want a container, a VM, or seccomp — and those will not tell you what the code was trying to do. That is the trade, and it is exactly what lets plan mode hand a program its own writes back.

Two smaller limits worth knowing:

Plan mode refuses network calls rather than inventing replies, because a fabricated response sends the program somewhere the real one would not go. Pass network="allow" (or --network allow) when a read has to succeed for the plan to reach anything interesting.

Plan mode copies a SQLite database into memory, and will not do so above 512 MB. Above that, use audit mode against a restored snapshot.

Troubleshooting

The plan is empty. The program probably did not run. Check for a the program raised: line under the report — a module that fails to import produces a truthful plan of nothing at all.

A write I expected is missing. Was it done by a C extension, or by a library holding a reference to open captured at import time? See the limit.

Blocked: plan mode cannot simulate. Something reached for the network. Either pass --network allow, or stub the client.

InterceptionFailed. Something has already replaced one of the functions this patches, or an object refused the assignment. It is raised rather than swallowed on purpose: a missing interceptor means a plan that understates what your code does.

A policy allows nothing on Windows. Fixed in 0.1.0 — patterns are written with forward slashes on every platform. If you still see it, please open an issue with the pattern and the path.

Everything is denied. A policy's default is required. If you meant to watch rather than to stop, that is default = "allow".

Reporting a problem

Open an issue.

The most useful bug report for this project is a plan that lied — an effect that happened for real but was missing from the plan, or one that was reported but never happens. Those are the bugs that matter here, and they are the ones worth interrupting your day to write up. If you can, include:

• the smallest program that shows it
• consequence plan --json output, and what the real run did
• your OS and python -V

examples/fidelity_check.py produces most of that automatically.

Security-relevant reports: please use private vulnerability reporting rather than a public issue. Note that "code escaped interception" is a known and documented limitation rather than a vulnerability — see the limit.

Contributing

Contributions are welcome, and the bar is honest work rather than perfect work. See CONTRIBUTING.md for the full version. The short one:

git clone https://github.com/aviseth/consequence
cd consequence
uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check . && uv run mypy

Good first contributions, in rough order of how much they would help:

• A new interceptor. os.link, os.symlink and shutil.chown are covered; mmap, os.sendfile and pathlib on future versions are not. The pattern is in intercept.py and each one is about fifteen lines and two tests.
• Another database. SQLite is done properly through the authorizer. Postgres and MySQL would need a different approach — probably at the driver's execute — and the design conversation is worth having in an issue first.
• A fidelity bug, with a failing test. The most valuable kind.
• Policy expressiveness, carefully. Every feature added here is a new way for a policy to mean something its author did not intend.

Two conventions worth knowing before you send a patch. Tests are named as sentences describing the behaviour, not test_foo_returns_bar. And comments explain why, especially where the obvious approach was tried first and did not work — several of the ones in intercept.py and databases.py are tombstones for real bugs, and they are there so nobody re-introduces them.

Related work

Licence

MIT. See LICENSE.