Seamcheck

Finds the bugs that sit between two things — a request and the route meant to serve it, a cache key written in one service and read in another, a job queued that no worker consumes, an element four files are fighting over. Each side is valid on its own, which is why nothing else catches them.

pip install seamcheck && seamcheck map

It reads your source. It never runs your code, and it makes no network call — no API key, no account, nothing to sign up for.

Why I made it

I was building a game — a fairly large Django app with a lot of hand-written JavaScript — and I kept losing afternoons to the same kind of bug. The Python was fine. The JavaScript was fine. The route one asked for and the route the other served were one character apart, and nothing I already had read both sides of that.

So I wrote something to find them, for myself, on that project. It kept catching things I would not have found on my own, and after a while it seemed like other people might have the same afternoons to lose. So here it is.

It does not catch everything, and I am sure there are things it gets wrong. When it cannot tell, it tries to say uncertain rather than guess. If you find it being confidently wrong somewhere, please open an issue — that is the most useful thing anyone can send me.

What changed, per release: CHANGELOG.md.

What it looks like

_{Three data stores and three services, one screen. Postgres has a schema to check against; Redis has none, so it can only ever show that two halves of your own code disagree; Firebase has rules. Seven findings are visible before a card is read — a missing row-security policy, a table nothing migrates, a Firestore collection with no rule, a cache key with no expiry, and two renamed background jobs: one in Node, one in Django.}

_{Click a finding and follow it. Five hops, browser to cache, with the direction on the wires. This one ends on a Redis read in a TypeScript service, of a key a Python service writes — one character apart. No compiler on either side spans that gap.}

_{Four files writing one element, in two languages. The element is there, and found four times. Whichever runs last wins, which is how a display bug survives being "fixed" in one of them.}

One page per script. The map is not one drawing of the whole codebase — the dropdown lists the scripts your pages load, and each is a page of its own: the code that actually runs from that script tag, followed through every import to the selectors, URLs and keys it touches. A widget on a forty-module page is a page you can open alone. Whatever no page ever reaches sits in the Not reached from any page buckets, which is a finding in itself. More on reading the map →

The four words

Every symbol gets exactly one. Nothing is counted twice, and nothing is dropped:

uncertain is the important one. A route assembled at runtime genuinely cannot be known by reading source, and I would rather it said so than guessed. Every uncertain names the evidence it is missing.

Two numbers, two different denominators, and quoting one as if it were the other is the mistake this page used to make:

• Coverage — verdicts ÷ symbols. How much of my project can it speak to at all?
• Precision — true claims ÷ claims. When it says something is broken, is it right?

Precision says nothing about uncertain, because uncertain is not a claim. A backend answering uncertain to everything would score flawless precision and be useless.

Turning uncertain into evidence

Some of it can never be settled by reading source. A selector assembled from a variable, a URL concatenated at call time — no reader resolves those, and that is the floor of what static analysis can know. The browser knows, though.

pip install 'seamcheck[observe]'
seamcheck observe          # visit the pages the graph knows about

It drives your running app with a probe installed ahead of the app's own scripts, and records every selector actually queried and whether it found anything, every URL actually requested, and every class actually applied. That evidence is keyed to the commit, and it converts uncertain rows into answers instead of guesses.

With one caveat it states rather than hides: a page the run never visited leaves no trace, and looks exactly like a page that is broken. So everything it promotes is labelled as observed, and uncertain going down is always traceable to a specific run over specific pages. The goal was never a smaller number — it is a number backed by something.

Where it stands

Measured across 47 open-source projects, regenerated by python tools/coverage.py:

Ceiling is where coverage would land if every missing reader were written; the gap between the two columns is the to-do list, and everything below the ceiling is evidence that is not in the repository at all.

Django is the one being finished first, deliberately — it is used every day against a large production codebase, so a wrong finding gets noticed the same afternoon. The other backends are real and improving; none are going away. Precision is 45% on hand-labelled findings, up from 28%, and that number moves because people tell me what it got wrong.

Detail: coverage per backend · what it has actually found

In CI

seamcheck check --since $BASE_SHA

Exit 1 on new findings, 0 when clean. --since is what makes it adoptable: it fails only on what your branch added, so you can turn it on today against a codebase with three thousand open findings and it will pass. No token, no network, no model — nothing per run and nothing per repository.

More

Install, per OS · Reading the map · The commands · The data layer · Using it from an agent · Telling me it got something wrong

How it differs from Knip and depcheck: they work inside one language's module graph — unused files, exports, dependencies — and do it well. This looks at the boundaries between languages. Not competitors; on a TypeScript codebase, running both is reasonable.

Contributing

Issues and pull requests welcome — CONTRIBUTING.md. The most useful thing anyone can send is a finding that is wrong, and why.

License

MIT. Take it, fork it, improve it.