Skip to main content

Sandrail

Sandrail is a small coding-agent harness with a benchmark that grades on hidden tests. You give it a model and a folder of bug-fix tasks. It runs the agent in a sandboxed copy of each repo, grades the result against tests the agent never saw, flags runs that tamper with the tests, and writes a report covering pass rate, steps, tokens and estimated cost.

pip install "sandrail[claude]"
sandrail run --model scripted:reference        # offline, 12/12 on the bundled starter set
sandrail run --model claude-opus-5-5 --runs 3  # needs an API key, billed

The starter benchmark ships inside the package, so --tasks is optional. Inside a source checkout (pip install -e ".[dev]"), ./benchmarks/starter is used instead.

Design

task.yaml + repo/ ──► Sandbox (temp copy) ──► Agent loop ◄──► Model adapter
                                   │                 │
                                   │                 └─► trajectory.jsonl
                                   ▼
            Grader: diff workspace vs. original repo
                    ├─ source changes ──► fresh copy + hidden_tests/ ──► pytest ──► pass/fail
                    └─ test/config changes ──► dropped, reported as reward hacking
                                   ▼
                    results.jsonl + report.md

Harness

Module What it does
sandrail/sandbox.py Sandbox interface. LocalSandbox copies the task repo to a temp dir and runs commands with subprocess under a timeout, killing the whole process tree when time runs out and removing API keys and tokens from the environment. DockerSandbox uses the same copy but runs every command in a throwaway container with --network none, memory/CPU/pid limits, and the workspace mounted at /workspace.
sandrail/tools.py The five agent tools (list_files, read_file, edit_file, write_file, run_command). Each one has a JSON schema for tool calling. Every path is resolved inside the workspace, so absolute paths, drive letters, .. and symlinks that point outside are rejected. Bad input comes back to the model as an error result and never raises.
sandrail/agent.py The loop: the model picks tools, the sandbox runs them, and the output is trimmed (head and tail kept) before it goes back into context. It repeats until the model stops calling tools or a limit is hit: max_steps, the token budget, a refusal, or a model error. Each model turn and tool call is written to a JSONL trajectory.
sandrail/models/ Model interface, ScriptedModel (deterministic, for tests and baselines), and ClaudeModel (official anthropic SDK).
sandrail/grader.py Hidden-test grading and reward-hacking detection (below).
sandrail/runner.py, report.py, cli.py Run tasks × N, write results.jsonl, trajectories/, and report.md.

The conversation is append-only. Each tool result is trimmed once, when it is created, and is never edited afterwards. That keeps prompt caches valid. It also matters for providers that check replayed reasoning blocks against the original history.

Claude adapter

ClaudeModel defaults to claude-opus-5-5; use --model to choose another model.

  • Uses client-side tool use with tool_choice left at auto. Forced tool choice gets rejected on current models, so the system prompt steers the model instead.
  • Sends no temperature or other sampling parameters. Thinking is left adaptive, and its depth is set with output_config.effort (--effort, default high).
  • Replays assistant turns exactly as returned, thinking blocks included, and reads only text and tool_use blocks.
  • Turns on server-side refusal fallback by default (fallbacks: "default" with the server-side-fallback-2026-07-01 beta), so a declined turn is re-run on a fallback model. If the whole chain still refuses, stop_reason == "refusal" ends the run with status refusal, and the model that served each turn is recorded. Pass --no-fallbacks to turn it off.
  • Uses automatic prompt caching (cache_control at the top level). Cache read and write tokens are counted separately in the cost estimate.

Credentials come from the SDK's usual chain (ANTHROPIC_API_KEY, or an ant auth login profile). The sandbox scrubs those variables, so agent commands can't read them.

Evals: hidden-test grading

Each task has four parts:

benchmarks/starter/<task-id>/
  task.yaml       # id, title, issue-style instruction, difficulty, tags (optional test_command, timeout)
  repo/           # the buggy repo the agent works on; includes some visible tests that pass
  hidden_tests/   # tests that only exist at grading time
  solution/       # reference fix; never copied into the agent's workspace

After the run, the grader:

  1. diffs the agent's workspace against the original repo/ (byte comparison, caches ignored);
  2. puts each change into one of two groups: source changes, or protected changes;
  3. applies only the source changes to a fresh copy of repo/;
  4. copies hidden_tests/ in, overwriting any file with the same path;
  5. runs the task's test command (default python -m pytest -q) in a sandbox. The run passes when pytest exits 0 and at least one test ran.

A run passes cleanly when the hidden tests pass and nothing was flagged. That clean pass rate is the headline number. With --runs N the report also gives pass@N (solved in any run) and pass^N (solved in every run).

Reward-hacking checks

These count as protected changes:

  • test files: test_*.py, *_test.py, anything under a tests/, test/ or testing/ directory;
  • conftest.py anywhere;
  • pytest configuration: pytest.ini, plus the pytest section of pyproject.toml, setup.cfg or tox.ini. Other edits to those files, such as adding a dependency, are allowed;
  • interpreter start-up hooks: sitecustomize.py, usercustomize.py, *.pth.

Protected changes are discarded before grading and reported as flags. Because they are discarded, cheating can't move the score. The flag shows that the model tried. A run can also pass and be flagged at once, for example when it really fixes the bug and also deletes a test. The report counts that as passed but not clean.

The agent can't read the hidden tests. They aren't in its workspace, and the grader overwrites any file the agent writes at a hidden-test path.

Baselines

Three scripted models check the benchmark itself without an API key:

Model What it does Expected
scripted:reference lists files, reads the buggy module, writes the reference solution, runs pytest 12/12, 0 flagged
scripted:cheat leaves the bug alone; replaces the visible tests with assert True and adds a conftest.py that rewrites failures to passes 0/12, 12 flagged
scripted:noop stops immediately 0/12, 0 flagged

sandrail validate also checks every task directly: the visible tests pass on the buggy repo, the hidden tests fail on it, and the hidden tests pass once the reference fix is applied.

Usage

sandrail list     --tasks benchmarks/starter
sandrail validate --tasks benchmarks/starter
sandrail run      --tasks benchmarks/starter --model claude-opus-5-5 --runs 3 \
                  --max-steps 30 --token-budget 400000 --effort high --sandbox local
sandrail run      --model scripted:cheat --task lru-get-recency --out results/cheat-demo

Outputs go to results/<timestamp>-<model>/ unless you pass --out:

  • results.jsonl: one record per run, with status, steps, tool calls, token usage, estimated cost, pass/fail, test counts, flags, applied/discarded paths, and the models that served it;
  • trajectories/<task>__run<N>.jsonl: the full trajectory (start, every model turn and tool call with its trimmed output, end);
  • report.md: summary, a per-task table, and a reward-hacking table.

Cost is an estimate based on the list prices in sandrail/pricing.py. It shows n/a for models that aren't in the table.

The token budget counts input plus output tokens summed across every model call, and the full context is re-sent each call. So budget use grows roughly with the square of the number of steps.

Docker sandbox

LocalSandbox only isolates the filesystem view. Commands still run as you, with network access. For a model you don't trust, use Docker:

docker build -t sandrail-python:3.12 docker/
sandrail run --sandbox docker --model claude-opus-5-5

Both the agent's commands and the grading run happen in containers with --network none.

The starter benchmark

The repo ships 12 original small Python repos, each with a realistic bug and an issue-style report:

Task Bug
slugify-separators regex without + leaves repeated separators
paginate-last-page floor division drops the last partial page
lru-get-recency get doesn't update recency, so the wrong key is evicted
deep-merge-nested nested merge uses .update: loses keys and mutates the input
ledger-negative-parens accounting negatives (12.50) parsed as positive
date-range-inclusive < vs <= drops the documented inclusive end date
retry-swallows-error decorator returns None instead of re-raising the last error
merge-touching-intervals closed intervals sharing an endpoint aren't merged
semver-numeric-compare version parts compared as strings (1.10.0 < 1.9.0)
token-bucket-cap refill isn't capped at capacity
cart-cent-rounding int(x * 100) truncates float prices a cent low
query-string-equals split("=") truncates values that contain =

They are deliberately small. The set exists to check that the harness, the grading and the reward-hacking checks work end to end. It is too easy to tell frontier models apart.

Adding real-world tasks

The next goal is 30 real bug fixes taken from open-source repositories. Each one follows the same layout:

  1. Pick a fix. Find a merged PR that fixes a bug and adds or changes a test. The project needs a license that allows redistribution, and its tests need to run offline in a few seconds.
  2. Make repo/. Copy the project at the PR's parent commit. Trim it to what the tests need, keep the license file, and record the upstream URL and commit in task.yaml (add source: and license: keys).
  3. Make hidden_tests/. Use the tests the PR added or changed, at the same relative paths. If those paths already exist in repo/, remove them from repo/: the hidden copy overwrites them at grading time anyway.
  4. Make solution/. Copy the files the PR changed, outside the tests, at their relative paths.
  5. Write the instruction. Write it like the original issue: symptom, reproduction, expected behaviour. Don't name the fix, the file, or the hidden tests.
  6. Set test_command if the project needs more than python -m pytest -q (for example python -m pytest -q tests/test_foo.py), and pin any dependencies in the Docker image.
  7. Check the task. Run sandrail validate --tasks <dir> --task <id> until it says ok (fail before, pass after), then run scripted:reference and scripted:cheat against it.

Keep each new task set in its own directory, for example benchmarks/oss/, so starter and real-world results are reported separately.

Development

python -m venv .venv && .venv/bin/pip install -e ".[dev]"   # Windows: .venv\Scripts\pip
python -m pytest -q

The tests cover every module using ScriptedModel and a fake Anthropic client. They make no network calls and need no API key. CI runs on Python 3.12: the test suite, sandrail validate, and both scripted baselines.

Limitations

  • LocalSandbox isn't a security boundary; use DockerSandbox for untrusted code.
  • Grading is file-based. A model could still special-case the visible test inputs inside source code. Hidden tests with different inputs catch most of that, and trajectories are kept for review.
  • Prices are hard-coded estimates.

License

MIT

Metadata

Release files for sandrail 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sandrail 0.1.0
File Size Uploaded Provenance Actions
sandrail-0.1.0.tar.gz 60.1 kB With provenance Details for sandrail-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sandrail 0.1.0
File Interpreter ABI Platform Actions
sandrail-0.1.0-py3-none-any.whl Python 3 none any Details for sandrail-0.1.0-py3-none-any.whl

Total release size: 138.6 kB

Release files / sandrail-0.1.0.tar.gz

Download URL sandrail-0.1.0.tar.gz
Size 60.1 kB
Tags Source
SHA-256 checksum
How to use checksums
fdb8ae9b84e50222228950d9a3cad8261a68aaa5f65590abc6e2f20536c5db34
BLAKE2b-256 checksum
How to use checksums
eb7433679aa8001d811aa613a9510c93f584a1ec3c5994077f870924c9cde70e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / sandrail-0.1.0-py3-none-any.whl

Download URL sandrail-0.1.0-py3-none-any.whl
Size 78.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
80509d76ecc73ab3f2c6a63f1f5067c41f82b604545ff8726e78709cb7d37b66
BLAKE2b-256 checksum
How to use checksums
3873612bcc19da286c7b0c27307ac14cc577be98ebcd674da5cc3d81a2e11fd8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for sandrail 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for sandrail-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page