poi-dedupe

Deduplicate and conflate points of interest from several sources (OpenStreetMap, Overture Maps, Google exports, government registries, scraped lists) into one set of golden records, with every field traceable to the record it came from.

sources ──► spatial blocking ──► similarity features ──► match score ──► clustering ──► golden records
            (grid / geohash,      (name, address, phone,   (weighted or     (union-find     (survivorship
             exact radius)         website, category,       logistic)        with conflict   rules +
                                   distance decay)                           checks)         provenance)

• Zero required dependencies. Matching, scoring, clustering, merging, CSV and GeoJSON are pure Python. NumPy is only needed to train a logistic model; pandas and pyarrow are optional extras for DataFrames and GeoParquet.
• Complete blocking. No pair within the radius is ever missed, at any latitude, across the antimeridian or over a pole; this is property-tested against brute force.
• Explainable. The default model is a weighted mean whose terms you can print per pair.
• Multilingual normalisation: Turkish dotted/dotless I, German ß and umlauts, accents, legal forms (Ltd. Şti., GmbH, A.Ş., LLC), stop-words in seven languages and cautious abbreviation expansion.

Install

pip install poi-dedupe                 # core: pure Python
pip install "poi-dedupe[numpy]"        # + logistic-regression training
pip install "poi-dedupe[pandas]"       # + DataFrame in/out
pip install "poi-dedupe[parquet]"      # + GeoParquet in/out (pyarrow)
pip install "poi-dedupe[all]"          # all of the above

Python 3.9 to 3.13.

Quick start (CLI)

poi-dedupe run osm.csv overture.csv registry.csv \
    --name name --lat lat --lon lon --id id \
    --category category --address address --phone phone --website website --updated updated \
    --country-code 90 --priority registry,osm,overture \
    -o merged.geojson --clusters-out clusters.csv

Each input file is one source (the source name defaults to the file stem; override it with --sources a,b,c). Record ids become "<source>:<id>". Inputs can be .csv (delimiter is detected), .geojson (Point features) or .parquet (lat/lon columns or a WKB geometry column); output can be .geojson, .csv or .parquet (GeoParquet 1.0).

Useful run options: --radius (metres, default 250), --threshold, --blocking geohash, --category-blocking, --cross-source-only, --clustering components, --max-diameter, --one-per-source, --coordinates mean|medoid|source_priority, --rule phone=most_recent, --taxonomy my_taxonomy.json, --lang de.

Quick start (Python)

from poi_dedupe import POI, Deduper, Survivorship

records = [
    POI("osm:1", "Kahve Dünyası", 41.04380, 29.00940, source="osm", category="amenity=cafe",
        phone="+90 212 555 1234", updated="2024-01-01"),
    POI("overture:1", "KAHVE DUNYASI Besiktas", 41.04383, 29.00945, source="overture",
        category="coffee_shop", phone="0212 555 12 34", website="kahvedunyasi.com"),
    POI("registry:1", "Kahve Dunyasi Ltd. Şti.", 41.04378, 29.00938, source="registry",
        category="Coffee Shop", address="Barbaros Blv. No: 12"),
    POI("osm:2", "Garanti BBVA ATM", 41.04385, 29.00950, source="osm", category="amenity=atm"),
]

result = Deduper(
    radius_m=150,
    survivorship=Survivorship(source_priority=["registry", "osm", "overture"]),
).run(records)

len(result.golden)                     # 2: the cafe (3 records) and the ATM
g = result.golden[0]
g.fields["phone"], g.provenance["phone"]
# ('+90 212 555 1234', {'value': ..., 'record_id': 'osm:1', 'source': 'osm',
#                       'rule': 'majority', 'support': 2, 'candidates': 2})

Reading files instead:

from poi_dedupe import ColumnMap, read_records, write_golden

cols = ColumnMap(name="name", lat="lat", lon="lon", id="id", category="category", phone="phone")
records = read_records("osm.csv", cols) + read_records("overture.geojson", cols)
write_golden(Deduper().run(records).golden, "merged.parquet")

DataFrames: records_from_dataframe(df, cols, source="osm"), golden_to_dataframe(result.golden), pairs_to_dataframe(result.pairs).

How it works

1. Candidate generation (blocking)

grid (default) or geohash buckets with their neighbour cells, then an exact haversine check against radius_m. Bucket sizes come from great-circle bounds: two points d metres apart differ by at most d / R in latitude and by at most 2 asin(sin(d / 2R) / cos φmax) in longitude, where φmax is the highest latitude present. Neighbouring cells are therefore always enough, and the result equals brute force exactly (tested at the equator, 41° N, 60° N, 78° N, 89.995° N and on the antimeridian). Near the poles the bucketing falls back to one longitude column, which is still complete but slower.

Options: category_blocking=True drops pairs whose categories are both known and incompatible; cross_source_only=True never compares two records of the same source.

2. Similarity features

All in [0, 1]; None means "no evidence" (a field missing on either side).

Name normalisation, step by step: case folding (İ ı I i all become i; ß becomes ss; lang="de" turns ü into ue), accent folding (ş→s, é→e, ø→o, ł→l), dotted acronyms joined (A.Ş.→as), apostrophes dropped (McDonald's→mcdonalds), &→and, other punctuation removed, abbreviations expanded, trailing legal forms stripped, stop-words removed (never down to an empty name).

Abbreviations that are ambiguous are expanded only by position: St. Mary → saint mary (leading), Main St → main street (trailing), and a st in the middle stays st. The same goes for Dr. (doctor/drive). Unambiguous ones (ave, blvd, rd, cd → caddesi, sk → sokak, hauptstr → hauptstrasse, ...) are always expanded.

3. Scoring

WeightedScorer (default): a weighted mean over the features that are present, so a missing phone neither helps nor hurts. Default weights: names 0.15 + 0.15 + 0.15, address 0.12, phone 0.12, website 0.06, category 0.10, distance 0.15; default threshold 0.70. scorer.explain(features) returns each term's contribution.

LogisticScorer: trained from labelled pairs with train_logistic (NumPy Newton/IRLS with an L2 penalty, or backend="sklearn"). Each optional feature gets a presence flag, so the model can learn that a different phone number is evidence against a match. Inference is pure Python; models are saved as small JSON files.

Threshold tuning: precision_recall_curve(scores, labels) and best_threshold(scores, labels, min_precision=0.98) (or max F1). Pass n_positive when some true pairs were never scored, so recall stays honest.

4. Clustering

• union_find (default): matches are applied strongest first, and two clusters merge only if the merged cluster stays within max_diameter_m (default 2 × radius), keeps at most one record per source (one_per_source=True) and violates no cannot-link pair. A weak bridging match cannot chain two different places together.
• components: connected components, and any component wider than max_diameter_m is split by removing its weakest edges.

Both return a partition of all records (singletons included). Invariants are tested on random graphs.

5. Golden records

Survivorship sets one rule per field:

Defaults: name, category and phone use majority; address and website use most_complete; every other column (attrs) uses default_rule. Ties go to source priority, then recency, then record id, so results are deterministic. Coordinates use mean (spherical centroid), medoid or source_priority. Every field records its provenance (value, record_id, source, rule, support, candidates), and entity ids are a stable hash of the member ids.

Taxonomy

Taxonomy.default() knows about 35 canonical categories with English, Turkish and German aliases and understands OSM tags (amenity=cafe), Overture-style ids (coffee_shop) and hierarchical labels (eat_and_drink > cafe, food.cafe). To extend it, use a JSON file (--taxonomy):

{
  "categories": {"vending": {"group": "retail", "distance_scale_m": 5, "aliases": ["vending machine"]}},
  "compatibility": [["vending", "convenience", 0.3]],
  "source_mappings": {"google": {"Espresso bar": "cafe"}}
}

Benchmark

poi-dedupe benchmark generates two independent synthetic cities from different seeds. The threshold is tuned and the logistic model trained on the first; all numbers below are measured on the second, unseen one. Three sources (osm, overture, registry) contain each place with probability 0.85 / 0.75 / 0.60, plus 2 % same-source duplicates. The noise includes coordinate jitter scaled by category (5 % of records get three times the jitter), typos, case and accent changes, legal suffixes, district suffixes, dropped words, St./Cd. abbreviations, missing fields, five phone formats, URL variants and a separate category vocabulary per source. Chain brands (Starbucks, Migros, Shell, bank ATMs, ...) give hard negatives: same name and website, different branch. All names and numbers are invented.

Metrics are pairwise against complete ground truth. "Pair" scores the matched pairs, "cluster" scores the pairs implied by the final entities.

Standard city: 1000 entities in 5 × 5 km, 2221 records, radius 250 m. Blocking: 20 273 candidate pairs, pair completeness 0.977, reduction ratio 0.992.

Dense city (--extent 1500 --radius 150): the same 1000 entities in 1.5 × 1.5 km, so far more look-alike neighbours. Blocking: 71 653 candidate pairs, pair completeness 0.947, reduction ratio 0.971.

How to read these numbers:

• The recall ceiling is set by blocking. 2.3 % (standard) and 5.3 % (dense, smaller radius) of the true pairs lie beyond the radius because of the heavy-jitter records, and no model can recover them. Pick the radius from how far apart your sources really put the same place.
• In the dense city the default threshold over-merges: cluster precision (0.83) is well below pair precision (0.93) because transitive chains join look-alike neighbours. Tuning the threshold or training the logistic model on a few hundred labelled pairs from your data closes most of that gap.
• Synthetic data is not your data. The generator is a controlled stress test, not a claim about accuracy on real OSM / Overture / registry data, where coordinate offsets, naming conventions and missing fields follow other distributions. Label a sample of poi-dedupe pairs output and use evaluate / train before you trust a threshold.
• Runtime (pure Python, one core): about 2 s per configuration in the standard city and about 6 s in the dense one (≈ 72 k pairs). Cost grows with the number of candidate pairs, that is with density × radius².

Reproduce with poi-dedupe benchmark and poi-dedupe benchmark --extent 1500 --radius 150 (add --json for the full report).

API overview

Limitations

• Phone handling is deliberately simple (no numbering-plan validation); pass default_country_code when a source uses national formats.
• Registrable domains use a short built-in list of multi-part suffixes (co.uk, com.tr, ...), not the full Public Suffix List.
• Pure Python favours portability over speed: plan on roughly 10 k pair comparisons per second. Very dense inputs benefit from a smaller radius or category_blocking=True.
• Input geometries are points; polygons (malls, campuses) should be reduced to a representative point first.

Development

pip install -e ".[dev]"
pytest --cov=poi_dedupe
ruff check . && ruff format --check src tests && mypy src

Citation

See CITATION.cff.

License

MIT © Yusuf Eminoğlu