Your coding agent is stateless, and your codebase doesn't fit in its context window. So when it writes a new function, it can't see that you already wrote that helper three modules over — it writes the copy. Over a year of AI-assisted commits, duplication stops being an accident and becomes the default.

find-dup-defs is the gate that catches it.

• 🔍 Clusters duplicate and near-duplicate definitions — functions, methods, classes, constants, type aliases, TS interfaces, Rust traits — across Python, TypeScript and Rust.
• 🧬 Matches through renames and edits, not token streams: everything is compared as an alpha-renamed AST canonical.
• 🎯 Grades each cluster by how much a refactor would actually pay off, so the biggest win is on top — not a flat wall of hits.
• 🎛️ Calibrates its own noise filters to your tree and hands you the config to commit.
• ⚡ 2–5× faster than PMD CPD and 8–18× faster than jscpd while doing strictly more semantic work than either.

Install

Run it without installing anything:

uvx find-dup-defs ./src

Prebuilt wheels for Linux, macOS and Windows are on PyPI — no Rust toolchain needed:

pip install find-dup-defs        # or: uv tool install find-dup-defs

Or build from source via crates.io — or grab a binary from the Releases page:

cargo install find-dup-defs

What it looks like

The tool run on its own source. No flags, no config:

$ find-dup-defs crates --only rs

--- duplicate functions (cross-file, AST sim warn=0.5 error=0.85) ---
DUPLICATE FUNCTION [ERROR]: callable_facts  [ast sim 0.90, T=0.65, n=2, loc=25, args=7]
  crates/rs-canon/src/lenses.rs:336
  crates/ts-canon/src/lenses.rs:396

DUPLICATE FUNCTION [WARNING]: keyword_start  [ast sim 0.69, T=0.75, n=3, loc=25, args=3]
  crates/py-canon/src/defs.rs:62
  crates/rs-canon/src/defs.rs:47
  crates/ts-canon/src/defs.rs:68

--- duplicate methods (cross-name, exact AST-normalized) ---
DUPLICATE METHOD [ERROR]: Python::kinds/Rust::kinds/TypeScript::kinds  [normalized-exact, T=0.48, n=3, loc=7, args=1]
  crates/py-canon/src/frontend.rs:53
  crates/rs-canon/src/frontend.rs:39
  crates/ts-canon/src/frontend.rs:39

# summary: 21 ERROR, 24 WARNING groups

Read one line: T=0.75 is the thickness — how much deleting this would pay. n=3 places, loc=25 lines each. And the last one has three different names, so no grep and no token matcher would have found it.

Exit code is non-zero when an ERROR survives, so it drops into CI as-is.

Contents

Why does this happen?

GitClear's 2025 report measured 211M changed lines: copy-pasted lines grew from 8.3% to 12.3% of all changes between 2021 and 2024, while refactored lines fell from 25% to under 10%. For the first time on record, copy/paste exceeded reuse.

That isn't a coincidence, it's a mechanism. A human who half-remembers writing something greps for it. An agent can't — it holds a few thousand lines of your repo at once, your _helpers.py isn't among them, and emitting a fresh copy is locally the path of least resistance.

Every copy is individually reasonable; the aggregate is a codebase that says the same thing five ways. A linter won't flag it, because each copy is valid code. You need something that looks across files at the definitions themselves.

Quickstart: calibrate, commit, gate

1. Calibrate

Start here. It never gates anything — it reads your tree and reports back:

$ find-dup-defs ./src --calibrate
=== thickness calibration (ERROR): 76 clusters analyzed ===
  T [0.2, 0.3)  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 25
  T [0.3, 0.4)  ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 27
  T [0.4, 0.5)  ▇▇▇▇▇▇▇▇▇ 8
  …
suggested thresholds (p50/p75/p90):
  balanced   --error-thickness 0.34  →  21 ERROR remain  (median dup: 14 loc, 2 args)

=== inferred directives (auto-detected noise patterns) ===
  → -D 'de-escalate:*@*/{test,tests,__tests__}/*=test parametrize/fixture candidates'
    rationale: 21 clusters live entirely in test paths
    affects: 21 total (10 ERROR, 11 WARNING, 0 INFO)

Three things come out:

• a histogram of how refactor-worthy your duplication actually is;
• threshold suggestions at p50/p75/p90, each with a real code sample at the cut, so you see what you'd be gating on;
• inferred directives — ready-to-paste -D strings for the noise found in your tree, each with its rationale and blast radius. Twenty-one clusters living entirely under tests/? It hands you the de-escalation rule for exactly that.

2. Commit what you agree with

$ cat find-dup-defs.directives
de-escalate:*@*/{test,tests,__tests__}/* = test parametrize/fixture candidates
suppress:*@*_pb2*                        = generated protobuf

3. Gate CI on the rest

find-dup-defs ./src --error-thickness 0.5 -D @find-dup-defs.directives --errors-only

Nothing is filtered until a directive says so. Calibration suggests; the committed file decides. No hidden heuristics, and the why stays next to the rule.

What it finds (and why not just CPD)

Three passes, all from the same single parse per file:

Three more are opt-in and advisory — patternology, lenses and converge — and none of them ever raises an ERROR.

What token-based clone detectors structurally can't do is the middle two rows. jscpd and PMD CPD match token streams; rename the variables or edit a line and the match is gone.

find-dup-defs clusters on an alpha-renamed AST canonical — every bound local rewritten to _v0, _v1, …, the def's own name blanked to _fn — so a function and its renamed-and-edited twin collapse to the same shape. The Type-3 pass goes further: it builds IDF-weighted per-line vectors and runs them through difflib-fast's simjoin, an exact L2AP weighted-cosine join (every pair with cos ≥ θ, no LSH approximation, asserted bit-identical to brute force), then single-linkages the survivors.

So the answer to "why not CPD" isn't one feature, it's the stack: cluster by meaning not tokens, calibrate the noise ourselves, rank by refactor-payoff instead of dumping a flat list — and do all of it faster anyway.

What it deliberately doesn't flag

Method receivers (self, cls, &self) are stripped, so a method matches the equivalent free function. And the shapes that look like duplication but aren't never form clusters in the first place:

• Python / TS — @overload / @abstractmethod / Protocol stubs (... / pass / docstring bodies), raise NotImplementedError, dispatch overrides that just return None / False / 0 / self, and @property setter/deleter accessors (suffixed so a getter never matches its setter).
• Rust — one-line write! / writeln! Display/Debug impls, matches! predicates, todo! / unimplemented! / panic! / unreachable! stubs; and #[cfg(...)]-gated same-name siblings (#[cfg(unix)] fn x + #[cfg(windows)] fn x) collapse to one logical item.

Each surviving cluster lands in a tier: ERROR gates CI, WARNING is for review, INFO is hidden unless you ask (--show-info, or --json where it's always present).

Does it actually work?

Speed

hyperfine --warmup 1 --runs 3, macOS arm64, against jscpd@4 and PMD CPD 7.24, both in Python mode on the same trees:

It does more semantic work than either — alpha-renamed canonicals, an exact IDF cosine join, severity grading, calibration — and is still faster, because it's Rust + rayon over single-parse frontends with no JVM or Node startup to amortize. Throughput on django (426K SLOC) is ~422K SLOC/s, against PMD's ~205K and jscpd's ~44K.

Signal, on real repos

Ten production TypeScript repos (vscode, the TS compiler, vue, angular, svelte, nest, astro, prisma, next.js, excalidraw; ≈6M SLOC), with --calibrate, the inferred directives, and the balanced thickness cut — raw ERROR count drops 94% on average:

Twenty-eight large Python repos (≈8M SLOC), auto-applied directives, 67% average cut:

What's left at the top is the kind of thing a human reviewer would also flag. pip's Version __lt__…__gt__ ×6 collapse into one _compare helper, −130 lines. scipy's dct/dst/idct/idst ×4 want a factory, −330 lines. scikit-learn's BaseSGD{Classifier,Regressor}._fit is a sibling-estimator dupe waiting for a shared impl. The vendored snapshots, test fixtures, .d.ts and Storybook noise is gone before you read a line.

Thickness: the number you sort by

What moves a cluster between tiers is its thickness — a normalized [0, 1] estimate of how much deleting the duplication would pay:

T = 0.7 · sat(volume, 30) + 0.1 · sat(args, 5) + 0.2 · sim       sat(x, k) = 1 − exp(−x/k)
volume = (n_members − 1) · loc        # lines a refactor would actually delete

Volume dominates on purpose — a 60-line function copied four times outranks a 3-line one copied six, whatever the similarity scores say. Wide signatures and higher similarity nudge it up.

Three flags move the cut:

Each defaults to 0.0 — off — until calibration tells you a number. Sort by T and the biggest refactor is on top.

Directives: tuning you can commit

The tool is meant to tune itself once, then be gated by an explicit, committed config — never by hidden heuristics.

The rule language is directiva, one rule per line:

ACTION : [<KIND>] NAME [@PATH] [=NOTE]

suppress drops a finding, de-escalate / escalate move it one tier (stepped and clamped), note annotates without touching severity, and set carries pipeline config. The note travels with the rule, so the why is still there when someone reads the file a year later:

-D 'de-escalate:<methods>Plugin.get_*_hook=intentional plugin no-op API'
-D 'suppress:<functions>spawn@*lib-rt/*=bootstrap copy, cannot import'
-D 'escalate:<methods>Lock.*@*/storage/*=must share impl before v1.0'

# keep them in a committed file and point CI at it (one per line; # comments; @- reads stdin)
-D @find-dup-defs.directives

Globs support {a,b,c} alternation, so one paste covers a whole convention family. --json reports what each directive matched (directives[], with origin and a matched count) — matched: 0 is a directive that no longer suppresses anything.

Pipeline settings ride the same channel:

Beyond duplicates

Three opt-in passes, all advisory, none of them ever an ERROR gate. They answer questions the duplicate passes structurally can't reach.

Patternology

The passes above answer "are these two definitions the same?". Patternology answers the next question: "this shape that recurs across seven functions — should it be one helper?"

It's the same engine carried one step further — same alpha-renamed canonical forms, same Finding / severity / directive pipeline — not a separate tool bolted on. WARNING for a tight family, INFO otherwise. A refactor map, not a CI failure.

$ find-dup-defs ./crates --only rs --patternology     # the tool on its own code
--- helper candidates in functions (patternology — collapsible duplication) ---
DUPLICATE FUNCTION [WARNING]: analyze_impl_fn/analyze_item_fn  [ast sim 1.00, n=2, loc=3, args=1]
  # helper: fn _fn(_v0: &?) -> AnalyzedFn { analyze(&_v0.sig.ident.to_string(), &_v0.sig, &_v0.block) }
  #         (1 param); collapses 2 sites, ~3 loc saved

Two granularities:

• whole-function — families sharing an entire shape, found by structural tf·idf cosine over node-type q-grams and a greedy maximal-clique cover (not connected components, which would single-linkage a whole dense neighborhood into one blob).
• sub-block — a recurring statement-window idiom embedded inside otherwise-different functions, mined by support — how many functions contain it — not pairwise similarity, which is the case whole-function cosine structurally cannot reach. A fetch-one idiom shared across seven unrelated repository methods comes out as ? = await _v0.execute(?); return ?.scalar_one_or_none() (3 params).

Knobs: --pattern-theta (whole-fn cosine floor, default 0.85), --pattern-support (sub-block support floor, default 3), and -D settings:pattern-min-thickness=<F> to drop the thin two-site tail (--calibrate suggests the value).

for pkg in $(ls ~/.cargo/registry/src/*/); do
  find-dup-defs "$pkg" --patternology --json
done | jq -s 'map(.groups[] | select(.pattern)) | group_by(.pattern.signature)'

Lenses

Every pass above canonicalizes one text — the definition's body — and varies only how much identity it strips. That is one axis. A lens varies the text instead: it projects the same definition onto a different question and throws the rest away.

The case that motivates it: two caches, one storing a JSONB blob and one storing typed columns, sharing no identifier anywhere. Same architecture, written twice.

$ find-dup-defs ./cache_a ./cache_b
No cross-file duplicates.

Nothing — and the reason is exact: the model's own name is a free name in the body canonical, so session.get(JsonCache, k) and db.get(ThumbEntry, i) never meet. Erase what the module itself introduced (imports, sibling definitions, class fields — renamed in attribute position too, which the local set never reaches) and what survives is the grammar of talking to things the module did not define:

$ find-dup-defs ./cache_a ./cache_b --kinds lenses
DUPLICATE LENSES [WARNING]: cache_get/thumb_lookup    [normalized-exact, T=0.60, n=2, loc=7]
    votes[5]: control×3 effects×1 outgoing×1 scope×1 signature×1
DUPLICATE LENSES [WARNING]: cache_put/thumb_store     [normalized-exact, T=0.65, n=2, loc=8]
    votes[5]: control×2 effects×2 outgoing×2 scope×1 signature×1
DUPLICATE LENSES [WARNING]: cache_evict/thumb_purge   [normalized-exact, T=0.73, n=2, loc=2]
    votes[4]: effects×3 outgoing×3 scope×1 signature×1

Ten lenses, each answering one question:

Opt in with --kinds lenses — Python, Rust and TypeScript. Directives address it like any other kind: -D 'suppress:<lenses>*@*/legacy/*=deliberate parallel port'. The kind exists exactly when asked for, so the section list, the default report and the default JSON are byte-identical without it.

Agreement is the signal

All ten stitch into one record, each fact tagged with the lens it came from (control:if, outgoing:.commit, schema:col Text nullable). The Type-3 pass's IDF-weighted cosine over those lines then is the vote — nothing new had to be built. Agreeing through several lenses raises the score, agreeing through one barely moves it, and a fact the whole corpus shares (control:return) is weighted to nothing without anyone declaring it noise.

Measured on one production tree, mean thickness climbs with the vote count — 0.71 at one vote, 0.79 at three, 0.89 at five, 0.92 at six — even though the score is computed from corpus IDF and knows nothing about votes. Two independent estimates of the same thing, which is the best evidence the weighting works that could be had without tuning it to fit.

A finding carried by a single lens is weak by construction — the vote count is there to be read.

Converge

Every pass above answers are these two the same. This one answers "these two are about the same thing — where do they stop agreeing?" — and reports the step rather than the cluster.

$ find-dup-defs ./src --converge
--- divergences in functions (converge — one thing done twice, and where they part) ---
DUPLICATE FUNCTION [INFO]: _apply_source_rate_limit / _enforce_submit_budget
  votes[4]: text×0 shape×62 subject×6 fork×12
  A only: _v2 = utcnow()
  B only: _v1 = now(UTC)
  ~ _ = _ - _(_=_)
    A: _v3 = _v2 - timedelta(hours=1)
    B: _v2 = _v1 - timedelta(seconds=HOUR_SECONDS)
  ~ if _ > _:
    A: if _v6 > _v1.source_limit_per_hour
    B: if _v4 > landing.lead_submit_limit_per_hour
  subject: domain.rate_limit.repo (reached by 11 definitions)

text×0 — those two share no line. One rate limiter written four times across a codebase, and the four disagree about where "now" comes from.

Opt in with --converge; findings are INFO and never gate. --converge-top keeps the strongest 50 of each kind by default, which is the one place this differs from every other pass: they report a set, where cutting off would drop findings as true as the ones kept, and this reports a ranking with no threshold in it, where the tail is what the ordering exists to push away. --converge-top 0 prints all of them (198k lines on a mid-sized tree — the reason for the default).

Families

Many places around one subject, all one shape, said once:

--- families of functions (converge — many places, one subject, one shape) ---
DUPLICATE FUNCTION [INFO]: chart_week_plusminus / chart_week_plusminus_wave / chart_daily_unsubscribers
  votes[2]: shape×73 members×3   # 3 definitions around domain.report._chart_helpers…
  src/domain/report/tg_charts.py:173
  src/domain/report/tg_charts.py:233
  src/domain/report/tg_charts.py:298

A group of N definitions produces N(N−1)/2 pairs, and reporting those says one fact once per pair while burying it. In a read of fifty findings on a real tree, four slots went to pairs among one family of six sibling functions and three more to a set of chart builders — the finding was never "these two are alike", it was "this family exists".

Grouped by greedy maximal clique, not by connected component: under single-linkage 1-2-3-4 is one group whose ends share no edge. The clique proposes and the evidence disposes — members are dropped while dropping raises the score, so a clique that grew past the real family shrinks back to it, with no threshold and no notion of which member is the odd one.

score = (E_text + E_shape + E_subject) · D · sharpness · novelty / members

For agents

The JSON output is built so an agent never has to round-trip to the filesystem. Each finding ships:

• groups[].snippet — the full source of one member;
• members[] — every location as file:line;
• the thickness for prioritization, plus kind / severity / similarity;
• notes[] — any directive annotations.

Pattern findings additionally carry a structured pattern object — template, signature, params, granularity, support, loc_saved — so a consumer groups by signature without parsing prose. Lens findings carry facets — [[lens, shared facts], …], strongest first — so a consumer can rank by how many perspectives agreed rather than by similarity alone. The field is omitted when a run produces no tagged facts, so the default document is unchanged.

# calibrate → JSON, then scan with the chosen tuning + inferred directives
find-dup-defs ./repo --calibrate --json > calib.json
find-dup-defs ./repo \
  --error-thickness <calib> \
  $(jq -r '.inferred_directives[].directive | "-D \"" + . + "\""' calib.json) \
  --errors-only --json > findings.json

Architecture

Five crates, layered so the engine never depends on a language:

              dup-defs-core       ← the shared ground: the Def / KindSpec / Facets / Frontend
                  ▲                 contract, the kind vocabulary, the alpha-rename, the lens
        ┌────┬────┼────┬────┐       machinery and the dotted-path form.  No deps.
      py-   rs-   ts-  find-dup-defs
     canon canon canon  (engine + CLI: the 3 passes + patternology + converge +
        └────┴────┴───────▲          severity + directives + calibration + reports)
                          │
                    the engine depends on the contract and on each frontend,
                    and on no frontend's internals

find-dup-defs is the engine and CLI; it clusters a Vec<Def> and never names a language. dup-defs-core is everything both sides share: the contract (Def, Facets, KindSpec, Analysis, the Frontend trait), and the pieces the frontends would otherwise each keep a copy of — the kind vocabulary, alpha_rename, count_loc, the lens vocabulary and stitching, the dotted-path form and its prefix walk. py-canon, ts-canon and rs-canon are the frontends (Ruff, oxc, syn).

Adding a language is one more <lang>-canon crate implementing Frontend — plus a Dialect impl if it wants patternology — and no engine changes.

The similarity engine underneath is difflib-fast, an exact Ratcliff–Obershelp + L2AP cosine-join port.

And the tool eats its own cooking: this workspace gates to 0 ERROR under find-dup-defs crates -D @find-dup-defs.directives. (The file crates/find-dup-defs/src/simgraph.rs exists because an earlier run flagged the cosine/union-find helpers that type3 and patternology had each copied — so they were extracted into one module.)

CLI reference

USAGE:  find-dup-defs [OPTIONS] <PATHS>...

LANGUAGES
  --only <CODES>            Restrict to frontends (py,ts,rs). Default: all found in PATHS.
  --kinds <K,…>             functions,methods,classes,interfaces,constants,type-aliases
                            + `lenses` (opt-in; py, rs, ts) — see Lenses.
                            Closed vocabulary: an unknown name exits non-zero.

SEVERITY (thickness ladder)
  --error-thickness <F>     Demote ERROR → WARNING if T < F   (default 0.0 = off)
  --warning-thickness <F>   Demote WARNING → INFO  if T < F   (default 0.0 = off)
  --escalate-thickness <F>  Promote anything → ERROR if T ≥ F (default 0.0 = off, applied last)

SIMILARITY
  -t, --threshold <F>       Name-gated cluster floor   (default 0.5)
  -e, --error-threshold <F> Name-gated ERROR floor     (default 0.85)
  --type3-theta <F>         Type-3 cosine floor        (default 0.7)
  --max-name-group <N>      Skip name-gated clustering for (kind,name) groups > N

LENSES (opt-in · py, rs, ts)
  --kinds lenses            Cluster by perspectives other than the body; each finding reports
                            which lenses agreed (`votes[n]: control×3 outgoing×2 …`)

PATTERNOLOGY (opt-in · advisory, never ERROR)
  --patternology            Surface collapsible-duplication helper candidates
  --pattern-theta <F>       Whole-fn structural cosine floor (default 0.85)
  --pattern-support <N>     Sub-block idiom support floor     (default 3)

CONVERGE (opt-in · advisory, never ERROR · py, rs, ts)
  --converge                Where two definitions about the same thing stop agreeing, and the
                            families of many that do the same thing around one subject
  --converge-top <N>        Keep the strongest N of each kind (default 50; 0 = every one)

FILTERS / MODES
  -D, --directive <S>       ACTION:[<KIND>]NAME[@PATH][=NOTE], repeatable. ACTION ∈
                            suppress / de-escalate / escalate / note / set:KEY=VALUE.
                            `@PATH` reads a directive file (# comments; @- = stdin).
  --min-size <N>            Only clusters with ≥ N members (default 2)
  --errors-only             Filter output to ERROR
  --show-info               Include INFO in the human report
  --calibrate               Histogram + threshold suggestions + inferred directives
  --json                    Machine-readable output
  --repo-root <PATH>        Root for relative paths in the report (default `.`)
  --no-cross-name / --no-type3   Skip pass 2 / pass 3

SETTINGS (via -D 'settings:KEY=VALUE')
  max-name-group=<N>        Cap for name-gated clustering
  pattern-min-thickness=<F> Drop patternology's thin tail
  converge-cap=<N>          Converge seeding cap (default 60)
  gpu=on|gpu+cpu|gpu|off    Metal offload (requires --features gpu)

Limitations

The honest ledger:

• Languages. Python, TypeScript and Rust today. A new language is a <lang>-canon sibling crate.
• Rust patternology is the youngest of the three: rs-canon splices statement bodies as node children rather than lists, so long-body alignment is prefix-only, and macro internals are opaque.
• TypeScript patternology sees top-level function declarations and arrow / function-expression consts. Class methods don't participate — their slice doesn't re-parse as a standalone function, so they carry no patternology canonical. The duplicate passes still cover them.
• The use lens is Python-only — it needs the tree-wide mention index that py-canon builds. The other nine read the definition's own tree and run on all three frontends.
• use assembles by name with no import resolution, which holds while top-level names are effectively unique (measured: 2435 distinct across 2444 classes in one production tree) and degrades on trees where one name covers hundreds of definitions — the case --max-name-group exists for.
• Type-4 clones (same logic, different syntax) are out of scope.
• Token-level sub-expression duplication is out of scope too; pair with jscpd or PMD CPD if you need it.
• The thickness constants were tuned on the benchmark corpora above. Your codebase may want different ones — that's what --calibrate is for.