Skip to main content

SwearBench

Which AI coding model made you swear the least, and still shipped?

Public benchmarks measure what models can do. SwearBench measures how they made you feel: it reads your own local agent logs, finds every message you sent in reaction to a model's reply, has an LLM judge label how mad you were and why, checks whether the work ended up accepted, and ranks the models.

Example from the author's own logs (~4,600 messages, July–October 2026):

swearing vs. result

leaderboard card

Run it

uvx swearbench

(or pipx run swearbench, or pip install swearbench)

It prints a leaderboard and writes swearbench-out/report.md, chart.svg (swearing vs. result), card.svg and results.json. Before anything leaves your machine it tells you how many messages it will send to the judge and asks.

Flag
--dry-run show what logs were found and how many messages would be judged
--judge claude-cli|codex-cli|anthropic|openai|command who labels your messages (default: first of claude, codex, ANTHROPIC_API_KEY, OPENAI_API_KEY)
--judge-model ID model for the judge
--judge command --judge-cmd "ollama run qwen3" any command that reads a prompt on stdin and prints the reply; keeps everything local
--since 2026-09-01 only recent reactions
--exclude MODEL drop a model from the ranking
--no-quotes leave the hall of shame out of the report

Labels are cached in ~/.cache/swearbench/, so re-runs only judge new messages.

Where it looks

Tool Location
Claude Code ~/.claude/projects/**/*.jsonl
Codex CLI ~/.codex/sessions/**/*.jsonl
OpenCode ~/.local/share/opencode/opencode.db
T3 Code ~/.config/t3/userdata/state*.sqlite (used for exact per-turn model attribution when present)

Only messages you typed count. Subagent transcripts, headless runs (claude -p, codex exec) and messages the judge flags as agent-written are skipped. Models that only ever ran as subagents are listed, never ranked.

How it scores

Every message you send is charged to the model whose reply you were answering. The judge labels it:

  • anger 0–4, and who it's aimed at: the model, or something external (swearing at your cloud provider is not the model's fault; "this looks fucking cool" is not anger)

  • how you got mad, weighted by how bad it is:

    Mode Weight Mode Weight
    fabrication (claimed success that wasn't) 3 incomplete / lazy 1.5
    overreach (did things you didn't ask) 3 insult 1.5
    giving up on it 3 profanity 1
    regression (broke what worked) 2.5 shouting 0.5
    ignored an instruction 2 sarcasm 0.5
    made you repeat yourself 2 slow / verbose 0.5
    taste (critique while iterating on looks) 0.5
  • satisfaction −2 (rejects the work) to +2 (praise)

  • whether it blames earlier work (something shipped before the last reply is broken or missing)

Rage for a message is built to match how it felt, not how often it happened:

  • Severity beats frequency. Rage = anger² + mode weights, so one 4/4 blowup (16) outweighs four 1/4 grumbles (4). The report counts 4/4 blowups per model.
  • Taste isn't failure. "That looks lame" while iterating on a design, with no broken rule, lie, regression or ignored instruction, counts a quarter.
  • Regret goes to whoever caused it. A complaint about earlier work ("why did X disappear") is charged to the models that worked in the same repo in the week before, split by how many turns each had there, not to the model that happens to be fixing it.
  • Each interrupt adds 1.5.

The headline is half friction, half result:

Friction   = 100 − 0.25 × rage per 100 turns + 10 × average satisfaction
Ships      = accepted ÷ (accepted + rejected) sessions
SwearBench = (Friction + Ships) / 2

Friction deliberately ignores how often you were annoyed (a model you use for lots of quick "merge it" turns would look calm by volume alone); it only counts how much rage piled up per turn. A session's verdict is your last reaction in it. Sessions that stop or move to another model without a verdict (usage limits, "pick up the work" in a new thread) are left out of Ships rather than counted as failures; switching away in anger still counts as a rejection. Friction alone rewards a model that is pleasant but never finishes; Ships alone ignores what it cost you to get there. The chart plots the two against each other.

Intervals are a 90% bootstrap over sessions; models with fewer than 40 reactions aren't ranked. With T3 Code, the report also shows the share of sessions that ended in a merged PR. It is informational only, since T3 records PRs only from when it started tracking them.

Per token of work. If the logs carry token usage, the report adds a second ranking: rage per million output tokens the model produced in sessions you drove. A model that does twice the work per message gets credit for it. Subagent token use is shown separately.

Caveats

  • n = 1. It measures you, your tasks and your mood as much as the models. Models used in different months did different work; the report tells you counts, not causes.
  • The judge is a model too. If it belongs to a family being ranked, SwearBench says so; re-run with another --judge and compare.
  • Regret attribution is a heuristic: it blames whoever worked in the same repo during the previous week, not the session that actually introduced the problem.
  • The weights are opinions. Rage, taste and severity weights live at the top of score.py; change them and re-run, labels are cached.
  • Deleted or rotated logs mean missing data, especially for the per-token view.
  • report.md quotes your own messages. Read it before you share it. The card and chart have no quotes.

License

MIT

Metadata

Release files for swearbench 0.2.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for swearbench 0.2.1
File Size Uploaded Provenance Actions
swearbench-0.2.1.tar.gz 21.2 kB With provenance Details for swearbench-0.2.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for swearbench 0.2.1
File Interpreter ABI Platform Actions
swearbench-0.2.1-py3-none-any.whl Python 3 none any Details for swearbench-0.2.1-py3-none-any.whl

Total release size: 44.3 kB

Release files / swearbench-0.2.1.tar.gz

Download URL swearbench-0.2.1.tar.gz
Size 21.2 kB
Tags Source
SHA-256 checksum
How to use checksums
7cfbb9f418e023cfc30961f03949b35f4f3d5e601f58435182f22f4ae03206e8
BLAKE2b-256 checksum
How to use checksums
65695b8474bd5e765caaf9367a1c1affb633f12fd219ff4e3680262f475093e7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 5, 2026.

Transparency log

Release files / swearbench-0.2.1-py3-none-any.whl

Download URL swearbench-0.2.1-py3-none-any.whl
Size 23.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3d2d184e7679b57ef4a99d1af02eff4efe82b999bfc91fb5b3bd27a044bcceb1
BLAKE2b-256 checksum
How to use checksums
db5234736393014547d441e214b288c1f5a4a09e326a746d1a7a875ce53ea82a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 5, 2026.

Transparency log

Security for swearbench 0.2.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for swearbench-0.2.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 5, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page