bluepencil

A quality gate for prose. It fails 46.9% of machine-written documents and 6.3% of human-written ones, and both of those numbers were measured rather than hoped for.

pip install bluepencil
bluepencil check draft.md

Live demo, nothing to install: huggingface.co/spaces/wolfvswhale/bluepencil

Exit code 1 when something fires at error severity, so it drops into CI, a pre-commit hook, or an agent loop without wrapping.

Why the numbers are the point

Most style linters ship a list of banned phrases and a threshold someone picked because it felt about right.

You install it, it fires on your good writing, you uninstall it.

Every threshold here was set by measuring the distribution of that statistic across 2,602 human-written documents spanning three registers: informal forum answers, formal scientific abstracts, and multi-paragraph encyclopedia articles. Each cadence gate is placed at the percentile that makes it fire on 5% of human writing. That rate is a property of the tool, not a hope, and bluepencil gates will print it back to you.

The companion repo prose-eval is the harness that produced those distributions.

What it catches

$ bluepencil check draft.md
draft.md
        3:1  error            throat-clearing
             "In today's fast-paced world"
             -> Delete it. Start with the first real sentence.
       3:44  error            negative-parallelism
             "is not just a nice-to-have, it's"
             -> State the correct thing. Leave the wrong thing out.
       4:24  error            editorializing
             "It's important to note"
             -> Just write the note.
          -  warn             flat-cadence
             Sentence lengths vary less than human writing typically does.
             Measured 0.198, below the calibrated threshold of 0.237.
             -> Rewrite one sentence in every three-sentence run to a
                different length.

1 file checked, 0 clean, 3 errors, 1 warning

Two kinds of finding, and they need different responses.

Pattern findings point at a span with a line and column. The fix is unambiguous and takes seconds.

Cadence findings describe the document. There is no line to fix; the draft needs a pass. These are the ones that catch prose which has had every flagged word swapped out and still reads as machine-written, which is the failure mode a phrase blocklist cannot see.

Measured performance

Fire rates across 1,500 human and 1,500 machine documents, held out from calibration. "Lift" is how much more often a gate fires on machine text than human text.

Where it doesn't work, and why

Nine of the eighteen gates fire more often on human writing than on machine writing here. All of them:

The obvious reading is that those rules are wrong.

The likelier reading is that the corpus is old.

The corpus used here, HC3, was collected around December 2022. Em dashes are the signature of current models: a pre-registered study of 69,632 medRxiv preprints (arXiv:2606.29540) puts em-dash prevalence in Discussion sections at 4.23% before ChatGPT, 8.0% in 2024, and 20.3% in 2025. HC3 predates almost all of that rise, so it cannot contain the behaviour the gate looks for. Newer paired corpora built on current models do exist, including MIRAGE, MAGA-Bench, and AITDNA, and recalibrating against one of them is the obvious next step. Until then these gates stay on and stay labelled unvalidated rather than disproven.

uniform-paragraphs is a different problem: the machine half of the corpus is single-paragraph, so the statistic is unmeasurable there and its 0.0x is an artifact of the data, not a result.

One gate shipped disabled. no-short-sentences could not be calibrated: more than 5% of human documents contain no short sentence at all, so no threshold below the target false-positive rate can ever fire. Calibration detects that and disables it rather than shipping a gate that silently never triggers.

Running it where you already are

Pre-commit. Add to .pre-commit-config.yaml:

repos:
  - repo: https://github.com/wolfvswhale/bluepencil
    rev: v0.1.0
    hooks:
      - id: bluepencil

GitHub Actions. --format github emits annotations that render inline on the pull request diff:

- run: pip install bluepencil
- run: bluepencil check . --format github --fail-on never

Drop --fail-on never when you want it to block merges.

Claude Code and other agent tools. Copy skill/SKILL.md into your skills directory. It tells the agent when to run bluepencil, how to read the two kinds of finding, and, importantly, when to overrule one.

Anything else. --format json gives you the findings with line numbers, measurements, thresholds, and suggested fixes.

Configuring it

bluepencil.toml, discovered from the file upward:

[bluepencil]
fail_on = "error"        # error | warn | never
min_words = 120          # documents shorter than this skip cadence gates

[bluepencil.gates]
em-dash-habit = "off"    # off | warn | error
promotional = "error"

[bluepencil.budgets]
transition-crutch = 6.0  # occurrences per 1000 words tolerated

[bluepencil.thresholds]
flat-cadence = 0.30      # override a calibrated value

Roughly one human document in twenty gets a cadence finding a good editor would overrule. That is the 5% target working as designed. Turn the gate off rather than mangling the sentence.

Recalibrating for your own writing

The shipped thresholds describe general human prose. If your house style is genuinely different, point the calibration at your own corpus:

pip install "bluepencil[calibrate]"
python scripts/calibrate.py --target-fpr 0.05
python scripts/measure.py --limit 1500

The first writes bluepencil/thresholds.json. The second regenerates the tables above.

What it will not tell you

Whether the text is true, whether the argument holds, or who wrote it. It measures the surface. A document can pass every gate and still be empty, and a good writer will sometimes trip three gates on purpose.

It is a gate, not a classifier. A document failing means it contains constructions the standard rejects, which is true of plenty of human writing, as the 6.3% says.

Sources

Rules derived from Wikipedia's Signs of AI writing, maintained by WikiProject AI Cleanup, plus a house editing standard. Calibration corpora: HC3, AI-and-Human-Generated-Text, and wikitext-103.

MIT licensed. Built by J. Alderman Lyell (@wolfvswhale).