Skip to main content

LinkTransformer

arXiv LinkTransformer demo

LinkTransformer is a Python package for semantic record linkage, candidate retrieval, row transformation, clustering, and text classification over tabular data.

Tutorials

More tutorials are coming soon.

Installation

pip install linktransformer

Quick Start

import os
import pandas as pd
import linktransformer as lt

left_df = pd.DataFrame({"CompanyName": ["Tech Corporation"], "Country": ["USA"]})
right_df = pd.DataFrame({"CompanyName": ["Tech Corp"], "Country": ["USA"]})

out = lt.merge(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
)
print(out[["CompanyName_x", "CompanyName_y", "score"]])

NEW RELEASE: End-to-end linkage Workflow: merge_k_judge (End-to-End Record Linkage)

merge_k_judge is the recommended end-to-end linkage API when you want both retrieval and LLM adjudication with confidence.

  1. Retrieve top-k candidates with embeddings (merge_knn)
  2. Judge each candidate pair with an LLM
  3. Return match decisions and confidence scores
judged = lt.merge_k_judge(
    df1=left_df,
    df2=right_df,
    on=["CompanyName", "Country"],
    k=5,
    knn_sbert_model="sentence-transformers/all-MiniLM-L6-v2",
    judge_llm_model="gpt-4o-mini",
    llm_provider="openai",
    openai_key=os.getenv("OPENAI_API_KEY"),
)

# key output columns:
# - score (retrieval similarity)
# - is_match (bool)
# - confidence (float in [0, 1] when available)

You can also combine providers (for example OpenAI embeddings retrieval + Gemini judge) by setting knn_api_model, judge_llm_model, and llm_provider explicitly.

Core APIs

  • lt.merge(...): semantic 1:1 / 1:m / m:1 linkage.
  • lt.merge_knn(...): top-k candidate retrieval.
  • lt.merge_blocking(...): run merge within blocks to do fuzzy merge within exact matches.
  • lt.aggregate_rows(...): map fine rows to coarser labels.
matches = lt.merge_knn(
    left_df,
    right_df,
    on=["CompanyName", "Country"],
    model="sentence-transformers/all-MiniLM-L6-v2",
    k=3,
)

2) Transform rows with LLM prompts

Use lt.transform_rows(...) to normalize, rewrite, or standardize values in one or more columns. Eg : Fix OCR errors in the Column, Standardize names.

cleaned = lt.transform_rows(
    left_df,
    on=["CompanyName", "Country"],
    model="gpt-4o-mini",
    openai_key=os.getenv("OPENAI_API_KEY"),
    openai_prompt=(
        "Standardize organization names and country strings for record linkage. "
        "Return a JSON list in the same order."
    ),
)
# adds: transformed_CompanyName-Country

3) Cluster and deduplicate

  • lt.cluster_rows(...): cluster semantically similar rows.
  • lt.dedup_rows(...): cluster + keep representative rows.
deduped = lt.dedup_rows(
    left_df,
    on="CompanyName",
    model="sentence-transformers/all-MiniLM-L6-v2",
    cluster_type="agglomerative",
    cluster_params={"threshold": 0.7},
)

4) Evaluate matched pairs

  • lt.evaluate_pairs(...): similarity over known pairs.
  • lt.all_pair_combos_evaluate(...): dense pairwise scoring.

5) Classification

  • lt.classify_rows(...): classify rows with HF or OpenAI chat models.
  • lt.train_clf_model(...): train a custom row classifier.

6) Train linkage models

  • lt.train_model(...): train a linkage model from paired or clustered data.

Provider Notes

  • OpenAI key: set OPENAI_API_KEY or pass openai_key.
  • Gemini key: set GEMINI_API_KEY or pass gemini_key.
  • API embedding models and local SBERT models are both supported.
  • For multi-column API retrieval, LinkTransformer serializes columns safely using <SEP>.

Test Naming Convention

Tests use test_lt_* naming to mirror the package API surface and make workflows discoverable.

Contributing

Issues and pull requests are welcome.

License

This project is licensed under the MIT License. See LICENSE.

Maintainers

  • Sam Jones (samuelcaronnajones)
  • Abhishek Arora (econabhishek)
  • Yiyang Chen (oooyiyangc)

Metadata

Release files for linktransformer 0.1.18

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for linktransformer 0.1.18
File Size Uploaded Provenance Actions
linktransformer-0.1.18.tar.gz 2.8 MB Without provenance Details for linktransformer-0.1.18.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for linktransformer 0.1.18
File Interpreter ABI Platform Actions
linktransformer-0.1.18-py3-none-any.whl Python 3 none any Details for linktransformer-0.1.18-py3-none-any.whl

Total release size: 5.6 MB

Release files / linktransformer-0.1.18.tar.gz

Download URL linktransformer-0.1.18.tar.gz
Size 2.8 MB
Tags Source
SHA-256 checksum
How to use checksums
71eee6bad83ff840db15634e7a9e7470bd6d6eb12d7f26711fb0b2f5dc32bc2f
BLAKE2b-256 checksum
How to use checksums
dfd19e1be1a47463ecc5c3261cde77fb03f38b0ea2c9691dfef7ded1b9c55875
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.7

Release files / linktransformer-0.1.18-py3-none-any.whl

Download URL linktransformer-0.1.18-py3-none-any.whl
Size 2.8 MB
Tags Python 3
SHA-256 checksum
How to use checksums
1fc70cad55b87c6b9072cbd38fe6d6124f2bf268fd6569561a2205f1be2caaaf
BLAKE2b-256 checksum
How to use checksums
54a9b0eeccb4059ee717884dd019317eb9a6de7ec171f690310ef7ded5374e33
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.11.7

Security for linktransformer 0.1.18

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page