semaclust

semaclust (semantic + clustering) is a small Python library for clustering similar strings using sentence embeddings and agglomerative clustering. It is useful for deduplicating free-text fields, normalizing user-entered values, and collapsing spelling or formatting variants into a canonical form.

Documentation: https://cobanov.github.io/semaclust/

Installation

pip install semaclust
# or
uv add semaclust

Install from source:

pip install git+https://github.com/cobanov/semaclust.git

Quickstart

from semaclust import TextClusterer

texts = [
    "New York", "NYC", "new york city",
    "Los Angeles", "LA",
    "San Francisco", "San Fran", "SF",
]

clusterer = TextClusterer(distance_threshold=1.0)
clusterer.fit(texts)

print(clusterer.n_clusters_)
# 3

print(clusterer.clusters_)
# {0: ['New York', 'NYC', 'new york city'],
#  1: ['Los Angeles', 'LA'],
#  2: ['San Francisco', 'San Fran', 'SF']}

print(clusterer.transform())
# ['NYC', 'NYC', 'NYC', 'LA', 'LA', 'SF', 'SF', 'SF']

fit_transform is the one-call shortcut:

TextClusterer(distance_threshold=1.0).fit_transform(texts)

API at a glance

Fitted attributes (sklearn convention): labels_, clusters_, representatives_, n_clusters_, texts_.

Choosing a model and threshold

The default encoder is all-MiniLM-L6-v2 (22M params, 384-dim, ~80 MB). It's fast, small, and a sensible drop-in, but it isn't the strongest option for every task. The table below summarizes 9 encoders on three workloads of escalating difficulty: short text with abbreviations (cities), medium text with synonyms (job titles), and long sentences with topical themes (customer feedback). See benchmarks.md for the full methodology, raw thresholds, and per-test breakdowns.

ARI (Adjusted Rand Index) of 1.0 means an exact match with the ground-truth clustering. Bold = perfect.

* Requires a clustering: prefix on each input. The bench applies this automatically; if you swap the model in directly, wrap it in a custom encoder that prepends the prefix.

Practical takeaways:

• Bigger isn't better. all-mpnet-base-v2 and Qwen3-Embedding-0.6B both fail the simplest test (cities) - they merge Los Angeles with San Francisco before merging SF with San Francisco.
• Best small drop-in: BAAI/bge-small-en-v1.5 matches the default's size class and has a single threshold window (0.95-1.00) that works for both cities and feedback.
• Best overall: mixedbread-ai/mxbai-embed-large-v1 - the only model to exceed 0.8 ARI on job titles, with a 0.10-wide cities+feedback overlap at 0.95-1.05. Costs 15x more parameters than the default.
• No model solves the job-titles case. Synonymy across roles (SWE ~ Programmer ~ Software Engineer, Product Manager ~ Product Owner) defeats every encoder we tested.

To swap the encoder, pass a string or a custom Encoder:

from semaclust import TextClusterer

clusterer = TextClusterer(
    encoder="BAAI/bge-small-en-v1.5",
    distance_threshold=1.0,
)

Hardware acceleration

device="auto" is the default. It picks CUDA if a GPU is visible, MPS on Apple Silicon, and otherwise lets sentence-transformers fall back to CPU. Mac users get the GPU for free; no code change needed.

TextClusterer(device="auto")   # default; cuda > mps > cpu
TextClusterer(device="mps")    # force MPS
TextClusterer(device="cuda")   # force CUDA
TextClusterer(device="cpu")    # force CPU
TextClusterer(device=None)     # delegate to sentence-transformers (no MPS auto-pick)

Plugging in your own encoder

Any object implementing encode(texts: list[str]) -> np.ndarray satisfies the Encoder protocol:

from semaclust import TextClusterer, Encoder
import numpy as np

class MyEncoder:
    def encode(self, texts: list[str]) -> np.ndarray:
        return np.random.rand(len(texts), 384).astype(np.float32)

clusterer = TextClusterer(encoder=MyEncoder())

CLI

semaclust ships a small typer-based CLI:

# Cluster lines from a file, write JSON
semaclust cluster items.txt --threshold 1.0 --output clusters.json

# Replace each line with its cluster representative
cat items.txt | semaclust replace --threshold 1.0

Run semaclust --help for the full reference.

Migration from 0.1.x

The 0.3 release is a breaking change. The single cluster(texts) entry point is gone; the new API mirrors scikit-learn:

The representative_selector argument moved from per-call to the constructor.

Development

git clone https://github.com/cobanov/semaclust.git
cd semaclust
uv sync --extra dev
uv run pytest
uv run ruff check src tests
uv run mypy src/semaclust

Install the pre-commit hooks once:

uv run pre-commit install

License

MIT, see LICENSE.