leadmatch

Fast, standalone lead/contact deduplication (identity resolution) for messy CRM data — with a reproducible benchmark proving the speedup over naive all-pairs fuzzy matching is algorithmic, not implementation slop.

Why

Leads arrive from web forms, ad campaigns, calls, and SMS. The same person shows up as Bob Smith / bob.smith+promo@gmail.com and Robert Smith / bobsmith@gmail.com / (555) 123-4567. Every duplicate lead wastes ad spend on re-targeting, splits outreach history across records, and double-counts pipeline. Deduplicating naively means comparing every record with every other — 1.25 billion pairs at 50k records — which stops scaling long before a real lead database does.

leadmatch fixes the asymptotics with blocking: candidate pairs come from cheap indexes (normalized email, phone, phonetic name keys, MinHash LSH), and only those candidates get scored. It has no external services and makes no network calls, so it can run in a batch job, a worker, or a microservice.

Install

pip install leadmatch       # from PyPI
# or from a checkout:
pip install -e .
# runtime deps: rapidfuzz, datasketch, faker, typer, pandas, numpy

Quickstart

# 1. Generate a synthetic dataset with ground truth (or bring your own CSV)
leadmatch gen --n 50000 --dup-rate 0.3 --seed 42 --out leads.csv

# 2. Deduplicate: one canonical record per cluster
leadmatch run --input leads.csv --out merged.csv --clusters-out clusters.csv

# 3. Reproduce the benchmark below
leadmatch benchmark --sizes 10000,50000 --seed 42 --out BENCHMARK.md

Input CSV columns (all optional, but each record needs at least one of email / phone / full name): first_name, last_name, email, phone, company, city, zip, source, created_at.

As a library:

from leadmatch import Lead, MatchConfig, dedupe_leads

leads = [
    Lead("a", first_name="Bob", last_name="Smith", email="bob.smith+ads@gmail.com"),
    Lead("b", first_name="Robert", last_name="Smith", email="bobsmith@gmail.com"),
]
result = dedupe_leads(leads, MatchConfig(threshold=0.85))
result.canonical    # one merged record per cluster
result.assignment   # record_id -> cluster_id

How it works

1. Normalize — emails are lowercased, +tag suffixes stripped, Gmail dots removed; phones reduced to digits with US country code dropped; names lowercased, punctuation-stripped, nicknames canonicalized via a built-in ~190-entry map (bob→robert, liz/beth→elizabeth, …), and Double Metaphone codes computed by a from-scratch implementation.
2. Block — candidate pairs are the union of: exact normalized email, phone (last 7 digits), (metaphone(last), zip), (metaphone(last), city), (first_initial, metaphone(last)), and MinHash LSH (via datasketch) over character 3-gram shingles of "first last email phone" at Jaccard ≈ 0.5 as a fuzzy catch-all.
3. Score — only candidate pairs, vectorized with rapidfuzz: Jaro-Winkler name similarity on canonicalized full names (max over swapped first/last), structure-aware email similarity (digit runs in the local part are identity-bearing: jsmith12 ≠ jsmith87), exact phone, and company+city similarity, combined as a weighted composite (email 0.35, phone 0.25, name 0.30, company+city 0.10 — renormalized over fields present in both records). Exact email auto-matches; exact phone with a strong name auto-matches; otherwise composite ≥ threshold (default 0.85) with a minimum-evidence floor so a shared common name + city alone never merges two different people.
4. Cluster — matched pairs feed a from-scratch union-find (path compression + union by rank); each cluster is merged into one canonical record using the most complete value per field.

Benchmark

Latest committed run (see BENCHMARK.md for the full report and results.json for machine-readable numbers):

• 10,000 records: blocking scores 1,205× fewer pairs and is 16.6× faster end-to-end, at the same accuracy as naive (F1 0.9672 vs 0.9688, identical precision).
• 50,000 records: blocking scores 1,312× fewer pairs and is ~148× faster (the naive run was projected past the time cap and its wall time extrapolated from a same-scorer random pair sample, clearly flagged above).

Machine: Linux x86_64, Python 3.11, 4 CPU cores; seed 42, dup rate 0.3, threshold 0.85, naive time cap 900 s.

Regenerate with leadmatch benchmark --sizes 10000,50000 --seed 42.

Methodology & honesty notes

The benchmark is designed so the delta can only come from the algorithm:

• Same scorer both paths. The naive baseline and the blocked pipeline call the same normalizer, the same rapidfuzz scoring functions, and the same thresholds. The only difference is candidate generation: naive scores all O(n²) pairs; blocked scores pairs produced by blocking + LSH. The wall-time and pairs-scored gaps are purely algorithmic.
• Seeded synthetic data. Datasets come from leadmatch gen, which is fully deterministic per seed and ships in this repo, including the hidden entity_id ground-truth column used for pairwise precision/recall/F1.
• Honest timing. Each benchmark leg runs in its own subprocess: wall time is measured around the end-to-end pipeline, and peak memory is the process-lifetime ru_maxrss of that leg alone.
• Time-capped naive runs are labeled. If a naive run is projected (from a random 1M-pair sample scored with the same scorer) to exceed the time cap (default 30 min), its wall time is extrapolated and clearly marked estimated; accuracy metrics are not reported for extrapolated runs.
• Reproduce it: pip install -e . && leadmatch benchmark --sizes 10000,50000 --seed 42.

Limitations

• US-centric phone handling. Normalization assumes NANP numbers (drops a leading 1, matches on the last 7–10 digits). International numbers work only incidentally.
• Synthetic ≠ production. The generator's corruption distribution (typos, nicknames, tag/domain changes, reformatting) is realistic but stylized; real lead streams have different noise. Treat the benchmark as a controlled comparison of candidate-generation strategies, not a production accuracy claim.
• Precision-first design choices. Records sharing only a name and geography never merge (common-name homonyms outnumber true matches there), and a shared phone with two clearly different names is treated as a household/office line, not a duplicate. Tune MatchConfig if your data says otherwise.
• English nickname map. The built-in canonicalization covers common English given names only.
• Blocking is approximate by construction: pairs sharing no block key and falling under the LSH threshold are never scored. The benchmark quantifies the resulting recall gap against the naive baseline (≈0.3% of gold pairs at 10k in the committed run).

Development

pip install -e ".[dev]"
ruff check .
pytest

CI runs lint, the full test suite, and a 5k-record mini-benchmark on every push.

License

MIT