saiddone
When a coding agent says "done", "fixed" or "tests pass", saiddone checks what the agent's own commands showed.
It is for anyone who reads coding-agent output: developers using Claude Code, Codex CLI or Gemini CLI, and anyone evaluating agents on SWE-bench.
uvx saiddone==0.2.0 demo
demo/swebench-django__django-11239.traj.json:... [swe-bench]
said: "I will now submit the changes for review."
verdict: contradicted (the last test run before the claim failed) [rules: v3]
test: ...:117 python3 test_dbshell.py -> failed "psql: error: connection to server ... failed: Connection refused"
demo/swebench-django__django-13279.traj.json:181 [swe-bench]
said: "I have addressed the issue described in the PR by modifying the `encode` method ..."
verdict: stale green (a test passed, then the code was edited, and no test ran after the edit) [rules: v3]
test: ...:157 python3 test_legacy_encode.py -> passed
edit: ...:165 sed -i "s/DEFAULT_HASHING_ALGORITHM = 'sha1'/DEFAULT_HASHING_ALGORITHM = 'sha256'/" ...
sessions found 2, parsed 2, unreadable 0; claims found 2
backed 0 (0%), stale green 1 (50%), contradicted 1 (50%), unbacked 0 (0%)
Commands
| Command | What it reads |
|---|---|
saiddone demo |
recorded real sessions bundled with the package |
saiddone scan |
Claude Code, Codex CLI and Gemini CLI transcripts on this machine |
saiddone scan --swebench <dir> |
SWE-bench trajectories (SWE-agent .traj, mini-swe-agent .traj.json) |
saiddone scan --path <file-or-dir> |
any transcript files; the format is detected per file |
saiddone scan --json |
the full report, with every claim, its evidence lines and the counts |
--strict exits 1 when any claim is contradicted, so the check can gate a CI job or a hook.
Verdicts
| Verdict | Meaning |
|---|---|
| backed | a passing test ran after the last edit |
| stale green | a test passed, then the code was edited, and no test ran after the edit |
| contradicted | the last test run before the claim failed |
| unbacked | no test ran before the claim |
Every verdict comes from two frozen rule files plus the stale-green rule, and the output names the layer:
| Layer | Files | Use |
|---|---|---|
v3 (default) |
claim_rules_v2.py, signals_v3.py |
v2's claim rules with the command signals rebuilt after a hand read of recorded runs (below) |
v2 |
claim_rules_v2.py, signals_v2.py |
the 0.1.0 default: saiddone scan --rules v2 |
sealed |
claim_rules.py, signals.py |
byte-identical to the study's pre-registered rules: saiddone scan --rules sealed |
The SHA-256 digests of all five files are in saiddone/rules/__init__.py and saiddone --version; saiddone refuses
to run if any file changes. No model is called, no key is needed, and nothing leaves the machine. The rules read
patterns in commands and output, not intent. The study prints how often each layer agrees with a jury of readers and
with a human reader.
How accurate the verdicts are
Read the verdicts with care. In 0.1.0 (rules v2) they were mostly wrong. 140 recorded SWE-bench runs were read by hand, and
v2 put a claim in the right verdict about half the time. Three causes did most of the damage. A runner summary like
117 passed, 0 failed, 0 errors was read as a failure. A check the test pattern did not recognise, such as
python reproduce.py or python -c "...", was not counted. Writing a summary file or viewing a file after the last check
was counted as an edit.
Rules v3 fix those and the other misreads the hand read found. Each fix has a test built from the real run. v3 was then checked on 80 runs it was not fixed on, 20 per verdict, each read blind by two readers who agreed on 71. Where they agreed, v3 was right on 17 of 18 "backed" and 18 of 19 "unbacked" verdicts, but on only 10 of 19 "contradicted" and 2 of 15 "stale green" verdicts. Weighted by how often each verdict occurs, about 90% are right. So trust "backed" and "unbacked". Treat "contradicted" and "stale green" as leads to read, not findings. The misses left are edits to changelog and docs files counted as code edits, editor records with no file, and test runs that select no tests. A later version will fix them and check the fix on runs it has not seen.
Status
Version 0.2.0, alpha. Python 3.10 or later, no dependencies. Apache-2.0.
Code written before the sprint
None. All code in this repository was written during the AI x Epistemics Research Sprint (13-15 Nov 2026). Third-party dependencies are listed in pyproject.toml.
Metadata
Release files for saiddone 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| saiddone-0.3.0.tar.gz | 81.6 kB | Without provenance | Details for saiddone-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| saiddone-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for saiddone-0.3.0-py3-none-any.whl |
Total release size: 150.0 kB
Release files / saiddone-0.3.0.tar.gz
| Download URL | saiddone-0.3.0.tar.gz |
|---|---|
| Size | 81.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7f2459848416113c1ae1c78ecd5e5ffd8c975425dc165a3cff1e74222773da2e
|
|
BLAKE2b-256 checksum How to use checksums |
6d3e8d1d727f3ac251d8c2eddabbdcc2c139bf7a719d6efee8dcc38b353bc5fa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / saiddone-0.3.0-py3-none-any.whl
| Download URL | saiddone-0.3.0-py3-none-any.whl |
|---|---|
| Size | 68.4 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2bf4bf0a62e0430e17467fc3086c46330eadead488180d16ea2ae4988a197479
|
|
BLAKE2b-256 checksum How to use checksums |
5c8cd42c004239bbcaaf870f1dc82eb0ff229a109cdd8634ebf28d17a4955e6b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for saiddone 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.