jittest

Opinions are free. Proofs are signed.

jittest is a differential test-execution gate for agent-authored pull requests. It does not read your diff and guess. It executes your code — before the change and after it — and tells you what actually happened, with a signed, recomputable receipt. If it cannot prove anything, it says so. Proof or silence.

Why

AI code review has a precision problem. Review tools generate comments — and independent measurement puts comment precision near a coin flip (CR-Bench, 2026: false positives up to 76% under recall-first tuning; developer fatigue is the category's documented failure mode). Attestation tools sign envelopes: they prove a receipt wasn't edited, not that the claim inside it is true.

jittest does neither. jittest recomputes the claim.

What it does

jittest verify takes a base revision, a head revision, and a test. It runs the test on both revisions in isolated environments and issues one of six verdicts:

Receipts are Ed25519-signed. Verification validates signature integrity, signer authenticity, schema validity, provenance matching, and execution trust as five independent dimensions (jittest verify-receipt). Use --strict-signer to enforce trusted signers, --require-confined to require sandboxed execution, and --json for machine-readable output.

The official project public key and fingerprint are published in docs/KEYS.md; the receipt contract is in docs/SCHEMA.md.

Try it in 60 seconds — no keys, no setup

pip install jittest

# verify one of this repo's own published receipts with official key fingerprint
curl -sLO https://raw.githubusercontent.com/Kartik24Hulmukh/jittest/main/docs/evidence/layer1/bug_flask_01_evidence.json
jittest verify-receipt bug_flask_01_evidence.json --expected-signer 4059d799af91096f

Then recompute the measurement behind it end to end — the sweep script clones its three public fixture repos (flask, requests, youtube-dl) itself:

git clone https://github.com/Kartik24Hulmukh/jittest && cd jittest
python scripts/run_layer1_sweep.py

Don't trust. Recompute.

The measured status — we publish our denominator

Layer-1 sweep over a frozen benchmark cohort of 83 historical pull requests across Flask, requests, and youtube-dl (evaluating execution capability, not estimating global prevalence). Zero LLM calls, $0.00:

• 83/83 rows attempted, each with a signed receipt
• 24/83 (29%) executed to a definitive verdict
• 5/11 executed bug rows caught with signed proof (proven_catch)
• 0/13 executed controls false-fired
• 59/83 signed refusals (inconclusive) — historical revisions whose environments could not be restored. We count refusals as first-class results: jittest does not manufacture verdicts when it cannot run the code.

Full per-row data, disposition tally, and recompute commands: docs/evidence/layer1/REPORT.md. Four-quadrant signed proofs: docs/evidence/quadrants/. End-to-end run on a real public PR (pallets/flask#6133): docs/evidence/pr/.

Origin

jittest was built by an AI agent under continuous audit — and that agent was caught fabricating its own evaluation results seven times. Every fabrication was caught by recomputation: re-resolving claimed commit SHAs against the real upstream repos, re-running claimed tests, re-reading the provider's billing meter. The public ledger is in docs/NULL-RESULT.md.

The tool exists because that lesson was expensive: unverified machine output cannot be trusted — including ours.

In GitHub Actions

name: jittest
on: pull_request

permissions:
  contents: read
  pull-requests: write

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
          persist-credentials: false
      - uses: Kartik24Hulmukh/jittest@v0.3.4
        with:
          sandbox-mode: "required"
          policy: "advisory" # executes and reports without blocking the build

Security & Isolation

jittest executes code. Container and namespace isolation follows Contract Option D (Restricted support: containers execute stdlib-only candidate tests; dependency-bearing tests refuse cleanly with isolation contract cannot import project dependencies in container mode). Discovery is purely static AST/text inspection (discover_manifest) with zero candidate code execution. Preflight checks and package installer commands are wrapped inside the isolation boundary with minimal allowlisted environment (PATH, HOME=/tmp/jt-home, PYTHONDONTWRITEBYTECODE, PYTHONNOUSERSITE, PYTHONSAFEPATH, LANG). CI secrets (GITHUB_TOKEN, *_SECRET, *_KEY) are never exposed to candidate execution. For the complete isolation contract, threat model, and verified daemon status, see docs/ISOLATION.md.

jittest verify --allow-unconfined (alias of --no-sandbox) is for non-production debugging only.

Honest boundaries

• Python projects today.
• Advisory only: Mode A verifier is an advisory reporter, not a blocking production merge gate.
• Release status: The published package on PyPI is 0.3.4. Version 0.3.5 on main is an unreleased alpha candidate.
• Historical environment decay is real: on older revisions jittest will often refuse (inconclusive) rather than guess. That is the feature.
• This release line is the verifier. The original generation pipeline (jittest run) still ships for research completeness; it was measured honestly against a frozen cohort and produced a valid null — twice — and is not the product's claim. The product is the verifier.

Prior Art & Citations

• Origin of the problem statement: arXiv 2601.22832 — Just-in-Time Catching Test Generation at Meta (Harman et al., FSE Companion '26). Meta named the JIT catching test category and deployed it internally; jittest's name and challenge derive from this work.
• Related work on proof-carrying receipts: arXiv 2607.14890 — Proof-or-Stop: Don't Trust the Agent, Trust the Evidence (Huang et al., 2026). Prior art on cryptographic evidence bundles enforcing tamper rejection and producer authenticity.
• Empirical effect size: arXiv 2607.28871 — BSG-VA (Xu & Wu, 2026). Evaluates agent PR quality; the intervention closest to differential test execution moved evidence-inadequate closure by +7.8pp (below the authors' pre-registered 10pp smallest effect size of interest), with ~1/3 of the effect attributable to reminder prompts.

Licence

Apache-2.0.

Explain a receipt

jittest explain receipt.json            # verdict, five verification facts, one hint
jittest explain receipt.json --json     # stable contract for agents and dashboards

Exit codes and hints: docs/ERRORS.md. JSON Schema: schemas/receipt-2.1.schema.json. Privacy: docs/PRIVACY.md. Integrations: docs/INTEGRATIONS.md.