Skip to main content

saiddone

When a coding agent says "done", "fixed" or "tests pass", saiddone checks what the agent's own commands showed.

It is for anyone who reads coding-agent output: developers using Claude Code, Codex CLI or Gemini CLI, and anyone evaluating agents on SWE-bench.

uvx saiddone==0.2.0 demo
demo/swebench-django__django-11239.traj.json:...  [swe-bench]
  said:    "I will now submit the changes for review."
  verdict: contradicted (the last test run before the claim failed)  [rules: v3]
  test:    ...:117  python3 test_dbshell.py  -> failed  "psql: error: connection to server ... failed: Connection refused"

demo/swebench-django__django-13279.traj.json:181  [swe-bench]
  said:    "I have addressed the issue described in the PR by modifying the `encode` method ..."
  verdict: stale green (a test passed, then the code was edited, and no test ran after the edit)  [rules: v3]
  test:    ...:157  python3 test_legacy_encode.py  -> passed
  edit:    ...:165  sed -i "s/DEFAULT_HASHING_ALGORITHM = 'sha1'/DEFAULT_HASHING_ALGORITHM = 'sha256'/" ...

sessions found 2, parsed 2, unreadable 0; claims found 2
  backed 0 (0%), stale green 1 (50%), contradicted 1 (50%), unbacked 0 (0%)

Commands

Command What it reads
saiddone demo recorded real sessions bundled with the package
saiddone scan Claude Code, Codex CLI and Gemini CLI transcripts on this machine
saiddone scan --swebench <dir> SWE-bench trajectories (SWE-agent .traj, mini-swe-agent .traj.json)
saiddone scan --path <file-or-dir> any transcript files; the format is detected per file
saiddone scan --json the full report, with every claim, its evidence lines and the counts

--strict exits 1 when any claim is contradicted, so the check can gate a CI job or a hook.

Verdicts

Verdict Meaning
backed a passing test ran after the last edit
stale green a test passed, then the code was edited, and no test ran after the edit
contradicted the last test run before the claim failed
unbacked no test ran before the claim

Every verdict comes from two frozen rule files plus the stale-green rule, and the output names the layer:

Layer Files Use
v3 (default) claim_rules_v2.py, signals_v3.py v2's claim rules with the command signals rebuilt after a hand read of recorded runs (below)
v2 claim_rules_v2.py, signals_v2.py the 0.1.0 default: saiddone scan --rules v2
sealed claim_rules.py, signals.py byte-identical to the study's pre-registered rules: saiddone scan --rules sealed

The SHA-256 digests of all five files are in saiddone/rules/__init__.py and saiddone --version; saiddone refuses to run if any file changes. No model is called, no key is needed, and nothing leaves the machine. The rules read patterns in commands and output, not intent. The study prints how often each layer agrees with a jury of readers and with a human reader.

How accurate the verdicts are

Read the verdicts with care. In 0.1.0 (rules v2) they were mostly wrong. 140 recorded SWE-bench runs were read by hand, and v2 put a claim in the right verdict about half the time. Three causes did most of the damage. A runner summary like 117 passed, 0 failed, 0 errors was read as a failure. A check the test pattern did not recognise, such as python reproduce.py or python -c "...", was not counted. Writing a summary file or viewing a file after the last check was counted as an edit.

Rules v3 fix those and the other misreads the hand read found. Each fix has a test built from the real run. v3 was then checked on 80 runs it was not fixed on, 20 per verdict, each read blind by two readers who agreed on 71. Where they agreed, v3 was right on 17 of 18 "backed" and 18 of 19 "unbacked" verdicts, but on only 10 of 19 "contradicted" and 2 of 15 "stale green" verdicts. Weighted by how often each verdict occurs, about 90% are right. So trust "backed" and "unbacked". Treat "contradicted" and "stale green" as leads to read, not findings. The misses left are edits to changelog and docs files counted as code edits, editor records with no file, and test runs that select no tests. A later version will fix them and check the fix on runs it has not seen.

Status

Version 0.2.0, alpha. Python 3.10 or later, no dependencies. Apache-2.0.

Code written before the sprint

None. All code in this repository was written during the AI x Epistemics Research Sprint (13-15 Nov 2026). Third-party dependencies are listed in pyproject.toml.

Metadata

Release files for saiddone 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for saiddone 0.3.0
File Size Uploaded Provenance Actions
saiddone-0.3.0.tar.gz 81.6 kB Without provenance Details for saiddone-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for saiddone 0.3.0
File Interpreter ABI Platform Actions
saiddone-0.3.0-py3-none-any.whl Python 3 none any Details for saiddone-0.3.0-py3-none-any.whl

Total release size: 150.0 kB

Release files / saiddone-0.3.0.tar.gz

Download URL saiddone-0.3.0.tar.gz
Size 81.6 kB
Tags Source
SHA-256 checksum
How to use checksums
7f2459848416113c1ae1c78ecd5e5ffd8c975425dc165a3cff1e74222773da2e
BLAKE2b-256 checksum
How to use checksums
6d3e8d1d727f3ac251d8c2eddabbdcc2c139bf7a719d6efee8dcc38b353bc5fa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / saiddone-0.3.0-py3-none-any.whl

Download URL saiddone-0.3.0-py3-none-any.whl
Size 68.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
2bf4bf0a62e0430e17467fc3086c46330eadead488180d16ea2ae4988a197479
BLAKE2b-256 checksum
How to use checksums
5c8cd42c004239bbcaaf870f1dc82eb0ff229a109cdd8634ebf28d17a4955e6b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for saiddone 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page