Skip to main content
Yanked

This release has been yanked by its maintainers, and will be ignored by installers, except when explicitly specified.
Consider using release 0.3.0 instead.
Reason given by maintainers: Withdrawn; not for use

saiddone

When a coding agent says "done", "fixed" or "tests pass", saiddone checks what the agent's own commands showed.

It is for anyone who reads coding-agent output: developers using Claude Code, Codex CLI or Gemini CLI, and anyone evaluating agents on SWE-bench.

uvx saiddone==0.2.0 demo
demo/swebench-django__django-11239.traj.json:...  [swe-bench]
  said:    "I will now submit the changes for review."
  verdict: contradicted (the last test run before the claim failed)  [rules: v3]
  test:    ...:117  python3 test_dbshell.py  -> failed  "psql: error: connection to server ... failed: Connection refused"

demo/swebench-django__django-13279.traj.json:181  [swe-bench]
  said:    "I have addressed the issue described in the PR by modifying the `encode` method ..."
  verdict: stale green (a test passed, then the code was edited, and no test ran after the edit)  [rules: v3]
  test:    ...:157  python3 test_legacy_encode.py  -> passed
  edit:    ...:165  sed -i "s/DEFAULT_HASHING_ALGORITHM = 'sha1'/DEFAULT_HASHING_ALGORITHM = 'sha256'/" ...

sessions found 2, parsed 2, unreadable 0; claims found 2
  backed 0 (0%), stale green 1 (50%), contradicted 1 (50%), unbacked 0 (0%)

Commands

Command What it reads
saiddone demo recorded real sessions bundled with the package
saiddone scan Claude Code, Codex CLI and Gemini CLI transcripts on this machine
saiddone scan --swebench <dir> SWE-bench trajectories (SWE-agent .traj, mini-swe-agent .traj.json)
saiddone scan --path <file-or-dir> any transcript files; the format is detected per file
saiddone scan --json the full report, with every claim, its evidence lines and the counts

--strict exits 1 when any claim is contradicted, so the check can gate a CI job or a hook.

Verdicts

Verdict Meaning
backed a passing test ran after the last edit
stale green a test passed, then the code was edited, and no test ran after the edit
contradicted the last test run before the claim failed
unbacked no test ran before the claim

Every verdict comes from two frozen rule files plus the stale-green rule, and the output names the layer:

Layer Files Use
v3 (default) claim_rules_v2.py, signals_v3.py v2's claim rules with the command signals rebuilt after a hand read of recorded runs (below)
v2 claim_rules_v2.py, signals_v2.py the 0.1.0 default: saiddone scan --rules v2
sealed claim_rules.py, signals.py byte-identical to the study's pre-registered rules: saiddone scan --rules sealed

The SHA-256 digests of all five files are in saiddone/rules/__init__.py and saiddone --version; saiddone refuses to run if any file changes. No model is called, no key is needed, and nothing leaves the machine. The rules read patterns in commands and output, not intent. The study prints how often each layer agrees with a jury of readers and with a human reader.

How accurate the verdicts are

Read the verdicts with care. In 0.1.0 (rules v2) they were mostly wrong. 140 recorded SWE-bench runs were read by hand, and v2 put a claim in the right verdict about half the time. Three causes did most of the damage. A runner summary like 117 passed, 0 failed, 0 errors was read as a failure. A check the test pattern did not recognise, such as python reproduce.py or python -c "...", was not counted. Writing a summary file or viewing a file after the last check was counted as an edit.

Rules v3 fix those and the other misreads the hand read found. Each fix has a test built from the real run. v3 was then checked on 80 runs it was not fixed on, 20 per verdict, each read blind by two readers who agreed on 71. Where they agreed, v3 was right on 17 of 18 "backed" and 18 of 19 "unbacked" verdicts, but on only 10 of 19 "contradicted" and 2 of 15 "stale green" verdicts. Weighted by how often each verdict occurs, about 90% are right. So trust "backed" and "unbacked". Treat "contradicted" and "stale green" as leads to read, not findings. The misses left are edits to changelog and docs files counted as code edits, editor records with no file, and test runs that select no tests. A later version will fix them and check the fix on runs it has not seen.

Status

Version 0.2.0, alpha. Python 3.10 or later, no dependencies. Apache-2.0.

Metadata

Release files for saiddone 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for saiddone 0.2.0
File Size Uploaded Provenance Actions
saiddone-0.2.0.tar.gz 64.1 kB Without provenance Details for saiddone-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for saiddone 0.2.0
File Interpreter ABI Platform Actions
saiddone-0.2.0-py3-none-any.whl Python 3 none any Details for saiddone-0.2.0-py3-none-any.whl

Total release size: 117.7 kB

Release files / saiddone-0.2.0.tar.gz

Download URL saiddone-0.2.0.tar.gz
Size 64.1 kB
Tags Source
SHA-256 checksum
How to use checksums
56c5ce07a8f6174cfa7bb15f7eac5214271631689c32320156a8956a24951e68
BLAKE2b-256 checksum
How to use checksums
1f5d3756e012711b200b29e2265ee15ce15c8e8798dd5c9abbbddeb7f7fc50da
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / saiddone-0.2.0-py3-none-any.whl

Download URL saiddone-0.2.0-py3-none-any.whl
Size 53.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b1c3f5a58f22d96f25fc50091fb50cf1f82d8fc70257fd00a934ac04a537bc06
BLAKE2b-256 checksum
How to use checksums
9688e6e97cca76ef44e0ae99915793f225b7a560635d2317a5186fe3bd6c3e8f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for saiddone 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page