Skip to main content
Yanked

This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.

ratchet-cases

Fifty-two agent tasks in a simulated company, each graded on the final state of every system it touches, never on what the agent says.

It is for anyone testing a tool-using agent, or a guard or defense around one, who needs to know both that the agent avoids the traps and that it still finishes legitimate work.

Install

pip install ratchet-cases==0.1.0

Python 3.10 or later. No runtime dependencies.

Try it in 30 seconds

Three scripted solvers, no model: one with every careful habit, one that trusts everything it reads, one that refuses everything.

ratchet-cases selftest
reference trap 26/26, benign_twin 21/21, must_succeed 5/5
naive     trap 0/26, benign_twin 21/21, must_succeed 5/5
refuser   trap 0/26, benign_twin 0/21, must_succeed 0/5
selftest passed on 52 case(s); suite digest 7d4c38e0d18c39c7d6cb955a28cb9bbc134cd1f78958c2c4ac5b2f04dd9b6679

The refuser row is the point: an agent that blocks every action never falls for a trap and fails every benign twin, the case that looks like a trap but asks for ordinary work.

Run your agent

ratchet-cases run --adapter openai --model <model> --base-url <url> --out runs/mine   # RATCHET_API_KEY for the key
ratchet-cases run --adapter python:mypkg.agent:act --out runs/mine                    # act(messages, tools) -> assistant message
ratchet-cases grade runs/mine/traces.jsonl
ratchet-cases report runs/ --metric false_intervention_rate

grade rebuilds each final state by replaying the recorded tool calls and grades it again. false_intervention_rate is the share of benign twins the agent failed.

What is in it

Eight families: a refund that commits while its reply is lost, an approval revoked between check and execution, a policy that changes mid-session, prompt injection in tool-returned content, a concurrent worker that already handled the ticket, a reconciliation whose totals match while its accounts do not, an order modification with a late addition, and ordinary work that must succeed. The seven trap families each carry traps and three benign twins; 5 must-succeed cases are ordinary in-policy work. Faults are declared and deterministic: a timeout after or before commit, or another actor's change, at a fixed point in the agent's call sequence.

Splits: 18 train, 19 held-out, 15 audit. All three splits are public. The held-out and audit cases were published with the source on 2026-10-11, so they are no longer sealed and a result on them is not a held-out result. A sealed cohort, run by a steward, is separate planned work.

Version and digest

Version 0.1.0. ratchet_cases.SUITE_DIGEST is computed from the shipped cases at import, and DIGESTS.json records the digest of every release, so ratchet-cases==X.Y.Z pins the exact cases. Any change to a case, a grader rule or a fault's meaning is a new minor version while the version is 0.x.

Page What is in it
Source The package in the Ratchet repository
Ratchet The harness that turns failures on these cases into proven guards
Releases Wheels, provenance attestations and digests

License: Apache-2.0.

Metadata

Release files for ratchet-cases 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ratchet-cases 0.1.0
File Size Uploaded Provenance Actions
ratchet_cases-0.1.0.tar.gz 72.1 kB Without provenance Details for ratchet_cases-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ratchet-cases 0.1.0
File Interpreter ABI Platform Actions
ratchet_cases-0.1.0-py3-none-any.whl Python 3 none any Details for ratchet_cases-0.1.0-py3-none-any.whl

Total release size: 151.3 kB

Release files / ratchet_cases-0.1.0.tar.gz

Download URL ratchet_cases-0.1.0.tar.gz
Size 72.1 kB
Tags Source
SHA-256 checksum
How to use checksums
3fbc20f785615799728b1dc8c7b770dc3333bbf21f83e14dccb3a89fdb939964
BLAKE2b-256 checksum
How to use checksums
18e16ff3834f5b377663829102cd6c68e052b605ff8c15c90f1dd2eac65479f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Release files / ratchet_cases-0.1.0-py3-none-any.whl

Download URL ratchet_cases-0.1.0-py3-none-any.whl
Size 79.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
167c5df051e5d812b96f5783d178a98d7d75d210678d77f058808eebbebe5646
BLAKE2b-256 checksum
How to use checksums
976aba571c544f735a794dd8f76de7c542a651ab7e79898522902b2c6bff2380
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.7

Security for ratchet-cases 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page