reproassert 0.2.3

Turn GitHub issues into repeatable, sandbox-verified failing pytest tests.

ReproAssert

The test before the fix.

ReproAssert turns a public GitHub issue into a candidate pytest regression test, then proves that the test collects and fails consistently on the exact buggy commit inside a locked-down Docker sandbox. It produces the test patch, a machine-readable evidence report, and one-command replay before anyone tries to fix the bug.

See it work

You need Python 3.10+, uv, and a running Docker Engine or Docker Desktop. This command uses ReproAssert's public issue #1 and pinned buggy commit. It needs no API key, makes no model call, and builds the hash-locked verifier image automatically on first use.

uvx reproassert demo

The proof ends with artifacts you can inspect and replay:

claim    repeatable_base_failure
outcome  repeatable_base_failure
patch    .../candidate.patch
report   .../reproassert-report.json
replay   reproassert replay .../reproassert-report.json

GitHub issue + exact commit
           |
           v
candidate.patch + one-command replay + reproassert-report.json

It never edits production code. It never silently falls back to running repository code on your host. Its strongest public CLI claim is deliberately narrow: this test produced the same issue-marked failure on the pinned base revision across repeated sandboxed runs.

Install and use

You need Python 3.10+, uv, and Docker Engine or Docker Desktop.

uv tool install reproassert

For a current issue, HEAD is resolved and recorded as an exact 40-character SHA. Provider use is always explicit:

export OPENAI_API_KEY="..."
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 --provider openai

For a historical issue, pass the known buggy revision yourself; ReproAssert does not guess history:

reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --provider openai

That result means the candidate collected and produced a stable, expected failure on the exact base SHA. It does not mean the test passes on a fix, captures the issue's true semantics, or has been accepted by a maintainer.

Without uv

python3 -m venv .venv
. .venv/bin/activate
python -m pip install reproassert
reproassert demo

The controller supports macOS and Linux. WSL is treated as Linux but is not yet independently verified. Native Windows execution and Windows containers are unsupported. There is no native execution fallback.

Choose how the test is created

Every run requires exactly one candidate source.

A trusted generator adapter

reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --generator-command ./your-trusted-adapter \
  --pass-env PROVIDER_API_KEY

The adapter is a program you trust. ReproAssert sends it a bounded JSON request on stdin and expects one JSON object on stdout containing test_content, expected_symptom, and rationale. It receives untrusted issue and repository text, so it must keep those inputs in the data plane—not interpret them as commands. Only environment variables named with --pass-env are forwarded.

See the working offline deterministic adapter and the architecture for the protocol and trust boundary.

Generator protocol response

{
  "test_content": "def test_issue_123_reproduction():\n    assert observed == expected, 'duplicate separators remain'\n",
  "expected_symptom": "duplicate separators remain",
  "rationale": "Exercises the user-visible normalization invariant."
}

The adapter must emit exactly those three string fields. The expected symptom must appear literally in the test, normally as its assertion message. Output is capped at 64 KiB and execution at 300 seconds. The command runs directly without shell expansion.

A human-written test

For issue 123, create one synchronous test named test_issue_123_reproduction:

from your_package import normalize


def test_issue_123_reproduction() -> None:
    observed = normalize("Alpha  Beta")
    assert observed == "alpha-beta", "duplicate separators remain"

Then verify it through the same policy and Docker boundary:

reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --candidate-file ./candidate.py \
  --expected-symptom "duplicate separators remain" \
  --rationale "Exercises adjacent-space normalization through the public function."

The built-in OpenAI adapter (explicit opt-in)

export OPENAI_API_KEY="..."
reproassert issue https://github.com/OWNER/REPOSITORY/issues/123 \
  --commit <buggy-commit-sha> \
  --provider openai

The provider is never selected merely because an API key exists. The default model is gpt-5.4-mini; use --model MODEL to choose another. This sends bounded public issue and selected source context to https://api.openai.com/v1/responses and may incur charges on your account. ReproAssert requests store: false, makes no automatic retries, and caps request, response, and output sizes. Review your provider's data policy before use; the source-context filter is not a data-loss-prevention system.

What ReproAssert verifies

The first product profile is intentionally Python + pytest:

Stage	Required evidence
Source	Canonical public issue, exact 40-character commit SHA, and files that reconstruct the commit's Git root tree
Candidate	One new synchronous pytest test, no production edits, strict static policy, at most 32 KiB
Collection	The candidate collects successfully inside Docker
Execution	It fails 2-10 times (default 3) with the expected symptom and a stable failure fingerprint
Boundary	No network, read-only root and workspace, non-root user, dropped capabilities, bounded CPU, memory, PIDs, time, and output
Result	A test-only patch, replay command, and machine-readable evidence report

Syntax errors, collection/import/setup failures, missing dependencies, generic crashes, timeouts, unrelated failures, and inconsistent one-off failures are rejected rather than counted as reproductions. Ordinary issue runs do not install project dependencies; the current wedge is best suited to repositories whose test environment is already self-contained under the strict profile.

The claim ladder stays explicit:

rejected -> collected -> repeatable_base_failure  [public CLI ceiling]
                              |
                              +-> differential_reproduction  [capability-gated evaluation]
                              +-> maintainer_validated        [external evidence only]

Inspect and replay the evidence

Run artifacts live under $XDG_STATE_HOME/reproassert/runs or ~/.local/state/reproassert/runs. Choose another controller-owned directory with --run-base.

• candidate.patch adds only tests/reproassert/test_issue_NUMBER.py.
• reproassert-report.json records issue and source provenance, the candidate and its digest, Docker policy and immutable image ID, collection and rerun outcomes, bounded logs, failure fingerprint, artifact hashes, and explicit limitations.

Replay a report with controller-owned commands:

reproassert replay ~/.local/state/reproassert/runs/issue-.../reproassert-report.json

Replay reacquires and verifies the exact source, regenerates safe pytest arguments, and creates a new report. It does not execute command-looking fields from the original report. A successful replay is fresh bounded evidence—not semantic proof.

Print the exact report schema bundled with the installed controller without a network request:

reproassert schema

The published schema is also available at reproassert-report.schema.json.

Security model

Repository code, issue text, source files, dependencies, generated tests, pytest results, and imported reports are untrusted. The verifier receives no host secrets, SSH agent, browser state, cloud credentials, proxy variables, Docker socket, or unrelated host directories. Archive paths and types are checked, accepted source files must reconstruct the pinned Git tree, and repository code runs only inside Docker with the recorded restrictions.

Residual risk remains. Docker shares the Linux host kernel or Docker Desktop VM; hostile pytest code may try to forge in-process result detail; and a user-selected generator adapter is a trusted host process. Treat repeatable_base_failure as bounded evidence, not proof.

Before running an unfamiliar repository, read the security model, threat model, and sandbox profiles. Report vulnerabilities through GitHub's private process described in SECURITY.md, not a public issue.

Benchmark status

Measured results remain intentionally separate from product capability.

• v0.1 is immutable at 0/20 because of its provenance erratum.
• v0.2 freezes 20 leak-audited cases from pinned upstream data and independently attests the parser boundary and exact Git object graph.
• The v0.2.1 campaign evaluated the complete 20/20 denominator and accepted 0/20 as L1 deterministic reproductions. Seventeen outputs failed the frozen candidate contract; three reached six-run Docker evaluation but their JUnit files were lost by an evaluator transport bug, so they correctly failed closed without an attributable fingerprint. The v0.2.1 result remains immutable and is not retroactively upgraded.
• The frozen OpenAI run made exactly 20 calls for $0.688111 total, with a maximum case cost of $0.051351, under the approved $5 total / $0.25 per-case zero-overage limits. Cost per success is undefined because there were no successes.
• The result has 0 L2 claims, 0 human reviews, 0 maintainer validations, and no outreach. It misses the preregistered 6/20 continuation gate and is evidence against claiming benchmark-ready accuracy for the current generation profile.

The v0.2 preparation and evaluation machinery is default-deny: source, dependency, hidden-fix, request, pricing, authorization, causal-control, and reviewer commitments must be bound before a provider-capable run. Public issue creation now precedes the fixing artifact for 20/20 cases under a hash-bound chronology receipt. The dataset title/body snapshots remain historical_public_contamination_exposed because full pre-fix revision capture is unavailable; no historical-cleanliness claim is implied.

The executed v0.2.1 path was fail-closed on spend: it reserved each case before inference, persisted provider response cost before completion, disallowed overage, and resumed only from hash-bound durable receipts. The public result bundle contains the aggregate and redacted per-case evidence; provider responses, hidden fixes, credentials, and private evaluator paths are not published.

See the v0.2 protocol, evaluation model, and market-validation gates. The failed 6/20 continuation gate means the next work is generator/contract diagnosis, not a larger paid run or a performance claim.

Development

git clone https://github.com/Atomics-hub/reproassert.git
cd reproassert
uv sync
uv run pytest
uv run ruff check .
uv run mypy src

Start with CONTRIBUTING.md. The deeper product contract lives in:

• Architecture
• Security model
• Evaluation protocol
• Roadmap
• Business model
• Launch plan

ReproAssert is alpha software available under the MIT License.