Implementation of Fellegi-Sunter's canonical model of record linkage in Apache Spark, including EM algorithm to estimate parameters
Project description
Splink: Super fast data linkage at any scale
splink
is a Python package for probabilistic record linkage (entity resolution), within the Fellegi-Sunter framework.
It's key features are:
-
It is extremely fast. It is capable of linking a million records on a laptop in around a minute.
-
It is highly accurate, with support for term frequency adjustments, and sophisticated fuzzy matching logic.
-
It supports multiple SQL backends, meaning it's capable of running at any scale. For smaller linkages of up to a few million records, no additional infrastructure is needed. For larger linkages, Splink currently supports Apache Spark or AWS Athena as backends.
-
It produces a wide variety of interactive outputs, helping users to understand their model and diagnose linkage problems.
Acknowledgements
We are very grateful to ADR UK (Administrative Data Research UK) for providing funding for this work as part of the Data First project.
We are also very grateful to colleagues at the UK's Office for National Statistics for their expert advice and peer review of this work.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for splink-3.0.0.dev12-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | caa86731b34d51dea360146bffb5cfef83c1216feede8a916441572876789417 |
|
MD5 | 8f9c8aaa9fe32da222132c5c060f6404 |
|
BLAKE2b-256 | d5ff4cd2bf96391f440ad16bc6c8b421c1af6529e437d4bc6c932d5b84ba12b2 |