LinkTransformer
LinkTransformer is a Python package for semantic record linkage, candidate retrieval, row transformation, clustering, and text classification over tabular data.
- Paper: https://arxiv.org/abs/2309.00789
- Website: https://linktransformer.github.io/
- Demo video: https://www.youtube.com/watch?v=Sn47nmCvV9M
Tutorials
- Link records with LinkTransformer: https://colab.research.google.com/drive/1OqUB8sqpUvrnC8oa_1RoOUzV6DaAKL4N?usp=sharing
- Train your own LinkTransformer model: https://colab.research.google.com/drive/1tHitPGjMMI2Nvh4wwA8rdcbYfbLaJDvg?usp=sharing
- Classify text with LinkTransformer: https://colab.research.google.com/drive/1hSh_p8j7LP2RfdtxrPslOfnogC_CbYw5?usp=sharing
- Demo app (Hugging Face Space): https://huggingface.co/spaces/96abhishekarora/linktransformer_merge
- Feature deck: https://www.dropbox.com/scl/fi/dquxru8bndlyf9na14cw6/A-python-package-to-do-easy-record-linkage-using-Transformer-models.pdf?rlkey=fiv7j6c0vgl901y940054eptk&dl=0
More tutorials are coming soon.
Installation
pip install linktransformer
Quick Start
import os
import pandas as pd
import linktransformer as lt
left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})
out = lt.merge(
left_df,
right_df,
on=["CompanyName", "Country"],
model="sentence-transformers/all-MiniLM-L6-v2",
)
print(out[["CompanyName_x", "CompanyName_y", "score"]])
NEW RELEASE: End-to-end linkage Workflow: merge_k_judge (End-to-End Record Linkage)
merge_k_judge is the recommended end-to-end linkage API when you want both retrieval and LLM adjudication with confidence.
- Retrieve top-
kcandidates with embeddings (merge_knn) - Judge each candidate pair with an LLM
- Return match decisions and confidence scores
judged = lt.merge_k_judge(
df1=left_df,
df2=right_df,
on=["CompanyName", "Country"],
k=5,
knn_sbert_model="sentence-transformers/all-MiniLM-L6-v2",
judge_llm_model="gpt-4o-mini",
llm_provider="openai",
openai_key=os.getenv("OPENAI_API_KEY"),
)
# key output columns:
# - score (retrieval similarity)
# - is_match (bool)
# - confidence (float in [0, 1] when available)
You can also combine providers (for example OpenAI embeddings retrieval + Gemini judge) by setting knn_api_model, judge_llm_model, and llm_provider explicitly.
Core APIs
1) Link two dataframes
lt.merge(...): semantic 1:1 / 1:m / m:1 linkage.lt.merge_knn(...): top-kcandidate retrieval.lt.merge_blocking(...): run merge within blocks to do fuzzy merge within exact matches.lt.aggregate_rows(...): map fine rows to coarser labels.
matches = lt.merge_knn(
left_df,
right_df,
on=["CompanyName", "Country"],
model="sentence-transformers/all-MiniLM-L6-v2",
k=3,
)
2) Transform rows with LLM prompts
Use lt.transform_rows(...) to normalize, rewrite, or standardize values in one or more columns. Eg : Fix OCR errors in the Column, Standardize names.
cleaned = lt.transform_rows(
left_df,
on=["CompanyName", "Country"],
model="gpt-4o-mini",
openai_key=os.getenv("OPENAI_API_KEY"),
openai_prompt=(
"Standardize organization names and country strings for record linkage. "
"Return a JSON list in the same order."
),
)
# adds: transformed_CompanyName-Country
3) Cluster and deduplicate
lt.cluster_rows(...): cluster semantically similar rows.lt.dedup_rows(...): cluster + keep representative rows.
deduped = lt.dedup_rows(
left_df,
on="CompanyName",
model="sentence-transformers/all-MiniLM-L6-v2",
cluster_type="agglomerative",
cluster_params={"threshold": 0.7},
)
4) Evaluate matched pairs
lt.evaluate_pairs(...): similarity over known pairs.lt.all_pair_combos_evaluate(...): dense pairwise scoring.
5) Classification
lt.classify_rows(...): classify rows with HF or OpenAI chat models.lt.train_clf_model(...): train a custom row classifier.
6) Train linkage models
lt.train_model(...): train a linkage model from paired or clustered data.
Provider Notes
- OpenAI key: set
OPENAI_API_KEYor passopenai_key. - Gemini key: set
GEMINI_API_KEYor passgemini_key. - API embedding models and local SBERT models are both supported.
- For multi-column API retrieval, LinkTransformer serializes columns safely using
<SEP>.
Test Naming Convention
Tests use test_lt_* naming to mirror the package API surface and make workflows discoverable.
Contributing
Issues and pull requests are welcome.
License
This project is licensed under the MIT License. See LICENSE.
Maintainers
- Sam Jones (
samuelcaronnajones) - Abhishek Arora (
econabhishek) - Yiyang Chen (
oooyiyangc)
Metadata
Release files for linktransformer 0.1.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| linktransformer-0.1.18.tar.gz | 2.8 MB | Without provenance | Details for linktransformer-0.1.18.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| linktransformer-0.1.18-py3-none-any.whl | Python 3 | none | any | Details for linktransformer-0.1.18-py3-none-any.whl |
Total release size: 5.6 MB
Release files / linktransformer-0.1.18.tar.gz
| Download URL | linktransformer-0.1.18.tar.gz |
|---|---|
| Size | 2.8 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
71eee6bad83ff840db15634e7a9e7470bd6d6eb12d7f26711fb0b2f5dc32bc2f
|
|
BLAKE2b-256 checksum How to use checksums |
dfd19e1be1a47463ecc5c3261cde77fb03f38b0ea2c9691dfef7ded1b9c55875
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.7
|
Release files / linktransformer-0.1.18-py3-none-any.whl
| Download URL | linktransformer-0.1.18-py3-none-any.whl |
|---|---|
| Size | 2.8 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1fc70cad55b87c6b9072cbd38fe6d6124f2bf268fd6569561a2205f1be2caaaf
|
|
BLAKE2b-256 checksum How to use checksums |
54a9b0eeccb4059ee717884dd019317eb9a6de7ec171f690310ef7ded5374e33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.7
|
Security for linktransformer 0.1.18
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.