Univo

Compare anything. Keep every record clean.

Find duplicates in Python data, from text datasets to structured records. Combine fuzzy matching, local embeddings and field-level rules. Fast, lightweight, no server required.

pip install univo

from univo import Finder

names = ["Northwind Traders", "Contoso", "Northwnd Traders", "NORTHWIND TRADERS", "Fabrikam"]
for match in Finder().find(names, min_score=0.8).matches:
    print(names[match.left_id], "~", names[match.right_id], round(match.score, 2))

Northwind Traders ~ NORTHWIND TRADERS 1.0
Northwind Traders ~ Northwnd Traders 0.94
Northwnd Traders ~ NORTHWIND TRADERS 0.94

No training and no API key: records never leave your machine.

By default, find returns up to 1,000 matches scoring 0.8 or more (limit, min_score), and text is compared by spelling. Text(mode="semantic") compares by meaning, with a local model: the 15 MB multilingual default, or any Model2Vec model from Hugging Face or a folder. More: choosing a mode, choosing a model, every option.

Built for data cleaning, entity resolution, ML dataset deduplication and real-time duplicate checks.

Records and rules

A schema says which fields to compare, how, and which differences rule out a match. Nested fields are reached by path.

from univo import Exact, Finder, Schema, Text

schema = Schema(
    id_field="id",
    fields={"name": Text(), "address.city": Text(), "vat": Exact(conflict="reject")},
)
finder = Finder(schema=schema)
finder.add(
    [
        {"id": 1, "name": "Northwind Traders SAS", "address": {"city": "LYON"}, "vat": "FR401"},
        {"id": 2, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR999"},
    ]
)
query = {"id": 3, "name": "Northwind Traders", "address": {"city": "Lyon"}, "vat": "FR401"}
for match in finder.search(query).matches:
    print(match.right_id, round(match.score, 2), match.reasons)

1 0.94 ['name: trigrams', 'address.city: edit_distance', 'vat: exact']

Record 2 has another VAT number, so it can never match.

Meaning, with the model you choose

from univo import Finder, Schema, Text

finder = Finder(
    schema=Schema(fields={"description": Text(mode="semantic")}),
    model="minishlab/potion-base-8M",  # any Model2Vec model on Hugging Face, or a folder
)
texts = ["online clothing store", "shop selling clothes on the internet", "steel factory"]
report = finder.find([{"description": text} for text in texts], min_score=0.6)
for match in report.matches:
    print(texts[match.left_id], "~", texts[match.right_id], round(match.score, 2))

online clothing store ~ shop selling clothes on the internet 0.74

Without model, Univo uses its 15 MB multilingual default. The model downloads once, is verified, then works offline (models).

Why Univo

• Records, not just strings. Dicts, lists and nested fields, compared field by field, with an explanation for every match.
• Rules before scores. Different VAT numbers, tenants or environments rule out a match, however alike the records look.
• Spelling, meaning and exact values, chosen per field, with a 15 MB multilingual model that runs offline, or any Model2Vec model you choose.
• Fast, exact by default, and light. 100,000 names deduplicated in 4.8 s. Search is exact by default, with explicit limits and optional approximate retrieval. One 2 MiB wheel, no Python dependency.

New records as they arrive

Keep an index and check each new record before you store it.

finder = Finder(
    schema=Schema(id_field="id", fields={"name": Text(), "vat": Exact(conflict="reject")})
)
finder.add([{"id": 1, "name": "Northwind Traders", "vat": "FR401"}])

finder.check_or_add({"id": 2, "name": "Northwnd Traders", "vat": "FR401"}).status  # "review"
finder.check_or_add({"id": 3, "name": "Contoso", "vat": "FR777"}).status  # "inserted"
finder.check_or_add({"id": 4, "name": "Northwind Traders", "vat": "FR401"}).status  # "matched"

review means a similar record exists and nothing was inserted: your application decides.

Checking one new record against 100,000 synthetic company names, in one partition, at the default threshold of 0.8, with the model already loaded (Apple M2, PyPI wheel, script):

Times depend on your data: they grow with the records of a partition and with the number of close candidates. Indexes can be updated, saved and loaded (mutations, snapshots).

Compared with other tools

Against SemHash 0.5.0, with the same model, on an Apple M2:

Univo's search is exact; SemHash's index is approximate. On names alone, neither tool is precise enough to merge automatically, so Univo returns scored candidates, not decisions. Method and every result: benchmarks.

Documentation

Quickstart · Reference · Benchmarks · Architecture · All docs