diffgenome

Runtime evidence for code changes. diffgenome runs a repository's existing tests against a change, records what actually executes, and reports it as a small, versioned JSON contract (diffgenome-runtime/1) that a reviewer or a review tool can consume.

It answers questions static analysis and models can only guess at:

• Which changed functions did the existing tests actually execute, and in exactly which tests and subtests?
• Through which call chains did execution reach the changed code, including dynamic dispatch, registries and decorators the static call graph cannot follow?
• Which outcomes of each changed condition were observed (true / false / never evaluated)?
• How did the changed code exit (returned, returned-error, raised <type>), and with which argument type shapes?
• Where did observation stop: mocks, fakes, eager task runners, external systems?
• Which changed behavior did no existing test exercise?

No model is involved. Everything reported was observed, or is a static decision site needed to place an observed outcome on a changed line.

Install

pip install diffgenome

Python 3.12+ (the Python tracer uses sys.monitoring; the target's own interpreter too). No runtime dependencies. Language support:

Tests run inside a disposable copy of the repository (never the checkout itself), under an OS sandbox chosen per platform:

Either way: no network beyond the sandbox, writes only inside the workspace, a scrubbed environment, and child processes stay confined. --allow-loopback lets tests use localhost servers they start themselves; on macOS it also reaches services on the host's loopback (a local database), on Linux it does not. Without a usable sandbox diffgenome refuses to run target tests and says why.

Use

# Python: the change between two revisions, the tests that cover it
diffgenome change --runtime python --repo . --python .venv/bin/python \
  --source-root src --test-root tests --diff main..HEAD --out .diffgenome/run

# Go
diffgenome change --runtime go --repo . --test-root api --tests ./api/ \
  --mock-dir db/mock --diff main..HEAD --out .diffgenome/run

# analyze a revision other than the checkout, without touching the checkout
diffgenome change ... --rev <head> --diff <base>..<head>

Python traces keep only what bears on the change: calls to changed functions, every frame above them back to the test, and --down levels below them. Unrelated work in a test (fixture setup, bulk data) is dropped as the test finishes, so large suites stay affordable. Use --full-traces to keep every call.

The result is .diffgenome/run/diffgenome-change.json. Its runtime section is the contract:

{
  "format": "diffgenome-runtime/1",
  "universe": { "test_scope": "tests/", "executions": 55, "passed": 55, "failed": 0, "tests": [...] },
  "changed_functions": [{
    "symbol": "py:app.rows.handler.RowHandler.restore_row", "file": "...", "line": 3221, "origin": "repo",
    "executed": true, "calls": 3, "tests": ["...::test_editor_can_redo_create_row", "..."],
    "exits": { "returned": 3 }, "raised_inside": {}, "arg_shapes": [[["user", "User"], ["row_id", "int"]]],
    "callers": [{ "symbol": "...DeleteRowActionType.undo", "calls": 2, "tests": 2 }],
    "stand_ins": {},
    "changed_sites": [{ "line": 3240, "predicate": "view is None", "true": 0, "false": 3 }]
  }],
  "edges": [{ "caller": {...}, "callee": {...}, "executions": 2, "tests": [...], "tests_total": 2 }],
  "boundaries": [{ "kind": "external", "caller": "...", "target": "celery.canvas._chord" }],
  "gaps": [
    { "kind": "function_not_executed", "symbol": "...TrashHandler.restore_item" },
    { "kind": "branch_outcome_not_observed", "line": 217, "predicate": "not fields", "outcome": "true" },
    { "kind": "stand_in_reached", "target": "...FormulaFieldType.run_periodic_update" }
  ],
  "not_reported": ["values (only type shapes and opaque fingerprints are recorded)",
                   "except-handler coverage (handlers are not decision sites)", "per-line coverage"]
}

Every "not executed" is relative to universe.test_scope: the tests that were run. Locations carry origin (repo, test, external) so a consumer can tell application frames from test frames.

The artifact's other sections (nodes, edges, executions, boundaries, facts) describe the observed behavioral neighborhood of the change; report.md and behavioral-map.md render it for people.

Consumers

Sydes uses the runtime contract for change verification: observed call edges join its call graph for reachability, the exact tests per changed function become supporting test evidence, the gaps become verification gaps, and AI recovery is only asked about what the runtime evidence cannot answer. Sydes depends on the contract alone, not on diffgenome internals.

Optional: the Behavioral Genome (research)

diffgenome genome builds a semantic layer on top of the runtime evidence: a model proposes behavioral rules (variables, decisions, transitions), and a deterministic checker keeps only the rules the observed executions verify or support. It is optional research code: nothing in diffgenome change or the runtime contract uses it, and it needs no extra packages (an OpenAI key only for live proposals, read from OPENAI_API_KEY or ~/.config/diffgenome/openai_api_key). Install with pip install "diffgenome[genome]" to state the intent. See docs/research-overview.md and docs/runtime-evaluation.md for what has and has not been shown to work.

Development

uv sync
uv run pytest
uv run ruff check src tests && uv run mypy src

License

Apache-2.0. See LICENSE.