semantic-clusterer

A production-ready Python library for unsupervised semantic text clustering at scale. Ships with a built-in ONNX embedder, an adaptive multi-tier pipeline, and two focused public classes:

Both classes share the same fit / predict / save / load production lifecycle.

More documentation

• USER_GUIDE.md — a task-oriented walkthrough from first run to production deployment, with copy-paste recipes for every feature.
• WORKING.md — how the engine works internally: the pipeline tiers, dimension bands, scoring objective, and determinism model.
• CHANGELOG.md — version history.

Table of Contents

• Installation
• Quick Start
• Production Workflow
• Choosing Your Knobs
• Output Formats
• How It Works
• Benchmarks & Evaluation
• Public API
• Configuration
• Custom Embedders
• Determinism
• Dependencies
• Known Limitations
• Examples

Installation

pip install semantic-clusterer

Python 3.9+ required. umap-learn and hdbscan are hard dependencies installed automatically.

For development:

pip install -e ".[dev]"

Quick Start

SemanticClusterer — discovers clusters automatically

from semantic_clusterer import SemanticClusterer

texts = [
    "How do I reset my password?",
    "I forgot my login password",
    "What are your business hours?",
    "When do you open on weekdays?",
    "My package hasn't arrived",
    "Where is my order?",
]

# One line — zero config
clusterer = SemanticClusterer()
groups = clusterer.cluster(texts)
# [["How do I reset my password?", "I forgot my login password"],
#  ["What are your business hours?", "When do you open on weekdays?"],
#  ["My package hasn't arrived", "Where is my order?"]]

SemanticKSplit — exactly k groups

from semantic_clusterer import SemanticKSplit

ks = SemanticKSplit(k=3)
groups = ks.split(texts)
# always exactly 3 groups, no noise label for valid inputs

Detailed output with topic labels and keywords

clusters = SemanticClusterer().cluster(texts, return_format="detailed")
for c in clusters:
    print(c["topic_label"])     # e.g. "Reset Password & Login Account"
    print(c["keywords"][:5])    # c-TF-IDF keywords
    print(c["confidence"])      # mean cosine similarity to centroid
    print(c["items"])           # original texts in this cluster

Production Workflow

Train once on a corpus, persist to disk, serve predictions in a separate process — no re-training needed.

from semantic_clusterer import SemanticClusterer

# --- Training (run once offline) ---
sc = SemanticClusterer(cluster_granularity="balanced")
sc.fit(corpus)

print(sc.outlier_threshold)    # e.g. 0.34 — auto-calibrated from training data
print(sc.get_topic_labels())   # {0: "Reset Password", 1: "Shipping", ...}

sc.save("./model")

# --- Inference (separate process / deployment) ---
loaded = SemanticClusterer.load("./model")
labels = loaded.predict(new_texts)                      # OOD → -1 automatically
labels = loaded.predict(new_texts, outlier_threshold=None)   # disable OOD

SemanticKSplit has the same fit / predict / save / load lifecycle:

ks = SemanticKSplit(k=8, quality="balanced")
ks.fit(corpus)
ks.save("./ksplit_model")

loaded_ks = SemanticKSplit.load("./ksplit_model")
labels = loaded_ks.predict(new_texts)

What gets saved

model/
  manifest.json    ← schema, dim, mode, config snapshot, auto threshold
  centroids.npy    ← float32 (K, D)  L2-normalised cluster centroids
  labels.npy       ← int32 (N_train,) training labels
  keywords.json    ← c-TF-IDF keywords and topic labels per cluster
  stats.json       ← per-cluster cohesion statistics

The embedding model is never pickled. Re-inject it on load():

loaded = SemanticClusterer.load("./model", embedding_model=my_embedder)

Choosing Your Knobs

cluster_granularity — for SemanticClusterer

Controls how many clusters are produced and how aggressively near-similar clusters are merged.

SemanticClusterer(cluster_granularity="balanced")  # default
SemanticClusterer(cluster_granularity="coarse")    # fewer, bigger clusters
SemanticClusterer(cluster_granularity="fine")      # most granular

What it does under the hood:

• Raises the min_cluster_size floor so tiny splinter clusters can't form
• Runs a post-clustering centroid merge pass (threshold drops as preset gets coarser)
• Adds a fragmentation penalty to the HDBSCAN parameter search score

quality — for SemanticKSplit

Controls how many random restarts are run and how rigorously the best partition is selected.

SemanticKSplit(k=8, quality="balanced")  # default
SemanticKSplit(k=8, quality="best")      # most thorough
SemanticKSplit(k=8, quality="fast")      # single pass

Both knobs default to "balanced". You can start with the defaults and only tune if the output isn't what you expect.

Output Formats

cluster() and split() accept return_format:

"simple" (default)

groups = SemanticClusterer().cluster(texts)
# [
#   ["text A", "text B"],   # cluster 0
#   ["text C", "text D"],   # cluster 1
# ]

"detailed"

clusters = SemanticClusterer().cluster(texts, return_format="detailed")
# [
#   {
#     "cluster_id": 0,
#     "topic_label": "Reset Password & Login Account",
#     "representative": "How do I reset my password?",
#     "keywords": ["password", "reset password", "login", ...],
#     "items": ["How do I reset my password?", "I forgot my login password"],
#     "size": 2,
#     "confidence": 0.91,   # mean cosine similarity to centroid
#   },
#   ...
# ]

confidence is a cohesion metric, not a probability. Values near 1.0 mean tight, coherent clusters.

keywords and topic_label are generated using c-TF-IDF. Disable with extract_keywords=False in config.

Row-aligned labels

labels = SemanticClusterer().cluster_labels(texts)
# int32 array, shape (len(texts),)
# 0, 1, 2, ...  → cluster id
# -1            → filtered (empty / None / NaN input)

# Great for DataFrame integration:
import pandas as pd
df["cluster"] = SemanticClusterer().cluster_labels(df["text"].tolist())

How It Works

SemanticClusterer (Variable-K Pipeline)

graph TD
    A[Input Texts] --> B[Deduplication & Preprocessing]
    B --> C[Embedding Generation <br><i>ONNX MiniLM or Custom</i>]
    C --> D[Dataset Profile & Routing]
    
    D -->|N <= 150| E1[Tiny Tier <br><i>Exact Agglomerative</i>]
    D -->|151 <= N <= 5000| E2[Small Tier <br><i>PCA + UMAP + HDBSCAN Sweep</i>]
    D -->|5001 <= N <= 50000| E3[Medium Tier <br><i>Profile-Guided Sweep</i>]
    D -->|N > 50000| E4[Large Tier <br><i>Sharded K-Means + centroid stitching</i>]
    
    E1 --> F[c-TF-IDF Topic Representation]
    E2 --> F
    E3 --> F
    E4 --> F
    
    F --> G[Clustering Output <br><i>Simple groups or detailed reports</i>]

SemanticKSplit (Fixed-K Pipeline)

graph TD
    A["Input Texts"] --> B["Deduplication & Preprocessing"]
    B --> C["Embedding Generation <br><i>ONNX MiniLM or Custom</i>"]
    C --> D["Dataset Profile & Routing"]
    
    D -->|"N <= 150 <br><i>tiny</i>"| E1{"k value?"}
    D -->|"151 <= N <= 5000 <br><i>small</i>"| E2{"k value?"}
    D -->|"5001 <= N <= 50000 <br><i>medium</i>"| E3["balanced-kmeans"]
    D -->|"N > 50000 <br><i>large</i>"| E4["minibatch-kmeans-assign"]
    
    E1 -->|k == 2| F1["bisecting-kmeans"]
    E1 -->|k >= 3| F2["agglomerative-cut-k"]
    
    E2 -->|k == 2| G1["bisecting-kmeans"]
    E2 -->|"3 <= k <= 10"| G2["spectral-cosine"]
    E2 -->|k > 10| G3["balanced-kmeans"]
    
    F1 --> H["Multi-Restart Loop & Selection"]
    F2 --> H
    G1 --> H
    G2 --> H
    G3 --> H
    E3 --> H
    E4 --> H
    
    H --> I{"Any empty clusters?"}
    I -->|Yes| J["Auto-Repair: Bisect largest cluster"]
    J --> I
    I -->|No| K["Output exactly K groups"]

Adaptive pipeline tiers

SemanticClusterer automatically selects one of four pipeline tiers based on dataset size:

Tier selection and dimensionality reduction are fully automatic - the library routes on dataset size and embedding dimension internally, so there is nothing to configure. See WORKING.md for the exact routing thresholds and per-tier algorithms.

⚠️ Large pipeline status: The large tier (N > 50 000) has been upgraded with multi-reduction search, granularity-aware HDBSCAN, and the full medium- grade post-processing pipeline, but has not yet been benchmarked or tested in v0.1.0. Use with caution on production workloads at this scale. The tiny, small, and medium tiers are fully tested and release-ready.

Dimension bands

Embedding dimensions are grouped into four bands. Each band gets its own UMAP / HDBSCAN / PCA parameter grid, so switching embedding models requires zero retuning:

Built-in embedder

No embedding_model supplied → uses the bundled ONNX MiniLM-L6-v2:

• 384-dimensional embeddings
• ~90 MB model, cached to ~/.cache/semantic_clusterer/ on first use
• Hardware-accelerated (supporting GPU/NPU execution providers) or CPU-optimized inference, no PyTorch dependency

Auto-calibrated outlier thresholds (Schema v3)

When you call fit(), the library measures the cohesion of every training cluster (cosine similarity of each member to its centroid). It records detailed per-cluster statistics (including standard deviation, median, and 95th percentile radius) and uses them to calibrate:

1. "auto" / "adaptive" (default): Dynamic per-cluster thresholds that adjust based on cluster size, density, and neighboring centroid confusion.
2. "global": A single unified global threshold calculated from the 5th percentile of the training similarity distribution.

sc.fit(texts)
print(sc.outlier_threshold)   # e.g. 0.4849 — global auto-calibrated threshold

Choose from four prediction modes:

labels = sc.predict(new_texts)                           # per-cluster adaptive boundaries (default)
labels = sc.predict(new_texts, outlier_threshold="global") # forced global threshold
labels = sc.predict(new_texts, outlier_threshold=None)    # OOD disabled (all items mapped)
labels = sc.predict(new_texts, outlier_threshold=0.5)     # manual hard floor

Benchmarks & Evaluation

To validate performance, the library is benchmarked against the standard 20 Newsgroups (20NG) dataset (20 highly overlapping topics). The benchmarks compare SemanticClusterer (which auto-discovers $K$ unsupervised) and SemanticKSplit (which partitions into exactly $K=20$ classes) against published BERTopic baselines across multiple embedders and dataset scales.

1. Unsupervised Clustering (SemanticClusterer)

Auto-discovers the natural number of classes without prior knowledge of the target $K=20$.

2. Reference Fixed-$K$ Partitioning (SemanticKSplit)

Supervised partition with class count fixed at $K=20$.

3. Head-to-Head vs. BERTopic Baseline

SemanticClusterer consistently outperforms the standard BERTopic baseline on clustering alignment across all evaluation tiers:

4. Production API Generalization (Fit $\rightarrow$ Predict)

Evaluated on an 80/20 train-test split with out-of-distribution (OOD) adaptive threshold filtering:

Key Takeaways for Production Deployments

• Beats BERTopic Baselines: SemanticClusterer beats standard BERTopic ARI across tiny (+20.3%), small (+11.0%), and medium (+7.4%) datasets.
• Positive Generalization Gap: Predict NMI with outlier_threshold="auto" matches or exceeds Fit NMI in 8 out of 9 evaluation cells, demonstrating clean out-of-distribution noise filtering without over-rejection.
• Calibrated Cluster Auto-Discovery: Discovered cluster counts consistently land between $K=19-24$ for 20 ground-truth newsgroup classes.
• Deterministic Reproducibility: The pipeline guarantees bit-perfect reproducibility across repeated runs given the same random_state.

Public API

from semantic_clusterer import (
    SemanticClusterer,        # density-based variable-K clustering
    SemanticKSplit,           # fixed-K partitioning
    SemanticClustererConfig,  # config for SemanticClusterer
    SemanticKSplitConfig,     # config for SemanticKSplit
    ClusteringReport,         # run-report dataclass
    FittedState,              # in-memory fitted-model snapshot
    ClusterStats,             # per-cluster cohesion stats
    SUPPORTED_DIM_BANDS,      # read-only dim-band mapping
    normalize_embedding_model,
    validate_embeddings,
    __version__,
)

SemanticClusterer

class SemanticClusterer:
    def __init__(
        self,
        embedding_model=None,          # custom embedder or None for built-in ONNX
        config=None,                   # SemanticClustererConfig | dict | None
        verbose=False,
        random_state=42,
        *,
        cluster_granularity="balanced",  # "fine" | "balanced" | "coarse"
    ): ...

    # One-shot clustering
    def cluster(self, texts, return_format="simple"): ...
    def cluster_labels(self, texts) -> np.ndarray: ...
    def cluster_with_report(self, texts) -> Tuple[np.ndarray, ClusteringReport]: ...
    def embed(self, texts) -> np.ndarray: ...

    # Production lifecycle
    def fit(self, texts) -> "SemanticClusterer": ...          # returns self
    def predict(self, texts, *, outlier_threshold="auto") -> np.ndarray: ...
    def fit_predict(self, texts) -> np.ndarray: ...
    def save(self, path: str) -> None: ...

    @classmethod
    def load(cls, path: str, *, embedding_model=None, verbose=False): ...

    # Topic accessors (require fit/load first)
    def get_topic_keywords(self, cluster_id=None): ...
    def get_topic_labels(self) -> Dict[int, str]: ...

    @property
    def outlier_threshold(self) -> Optional[float]: ...
    @property
    def cluster_stats(self) -> Optional[List[dict]]: ...
    @property
    def is_fitted(self) -> bool: ...

SemanticKSplit

class SemanticKSplit:
    def __init__(
        self,
        embedding_model=None,
        *,
        k: int,                        # required, >= 2
        config=None,                   # SemanticKSplitConfig | dict | None
        verbose=False,
        random_state=42,
        quality="balanced",            # "fast" | "balanced" | "best"
    ): ...

    # Fixed-K partitioning
    def split(self, texts, return_format="simple"): ...       # always exactly k groups
    def split_labels(self, texts) -> np.ndarray: ...
    def split_with_report(self, texts) -> Tuple[np.ndarray, ClusteringReport]: ...
    def cluster(self, texts, return_format="simple"): ...     # alias for split()
    def embed(self, texts) -> np.ndarray: ...

    # Production lifecycle — identical to SemanticClusterer
    def fit(self, texts) -> "SemanticKSplit": ...
    def predict(self, texts, *, outlier_threshold="auto") -> np.ndarray: ...
    def fit_predict(self, texts) -> np.ndarray: ...
    def save(self, path: str) -> None: ...

    @classmethod
    def load(cls, path: str, *, embedding_model=None, verbose=False): ...

    # Topic accessors
    def get_topic_keywords(self, cluster_id=None): ...
    def get_topic_labels(self) -> Dict[int, str]: ...

    @property
    def outlier_threshold(self) -> Optional[float]: ...
    @property
    def cluster_stats(self) -> Optional[List[dict]]: ...
    @property
    def is_fitted(self) -> bool: ...

ClusteringReport

Returned by cluster_with_report() and split_with_report(). JSON-coercible via to_dict().

@dataclass
class ClusteringReport:
    n_input_texts: int
    n_clustered: int
    n_noise: int
    n_clusters: int
    pipeline_tier: str          # "tiny" | "small" | "medium" | "large"
    embedding_dim: int
    dim_band: str               # "low" | "mid" | "high" | "xhigh"
    dataset_profile: dict
    chosen_params: dict         # includes "cluster_granularity", "quality_n_restarts", etc.
    intrinsic_metrics: dict     # score, coverage, silhouette, davies_bouldin, ...
    phase_timings: dict         # seconds per phase
    warnings: List[str]
    confidence_level: str       # "high" | "low"
    random_state: int
    library_version: str

Configuration

SemanticClustererConfig

from semantic_clusterer import SemanticClustererConfig

config = SemanticClustererConfig(
    # Key knob
    cluster_granularity="balanced",   # "fine" | "balanced" | "coarse"

    # HDBSCAN overrides (power users)
    min_cluster_size=None,            # int >= 2, or None (auto)
    min_samples=None,                 # int >= 1, or None (auto)

    # Output enrichment
    extract_keywords=True,            # bool
    keywords_top_n=10,                # int >= 1

    # Infrastructure
    batch_size=64,
    normalize_embeddings=True,
    random_state=42,
    max_samples=200_000,              # None disables the dataset size cap
    verbose=False,
)

Pipeline routing (tier) and dimensionality reduction are not user options. They are selected automatically from dataset size and embedding dimension. This keeps the public surface small and the results consistent.

SemanticKSplitConfig

from semantic_clusterer import SemanticKSplitConfig

config = SemanticKSplitConfig(
    # Key knob
    quality="balanced",               # "fast" | "balanced" | "best"

    # Output enrichment
    extract_keywords=True,
    keywords_top_n=10,

    # Infrastructure (same as SemanticClustererConfig)
    batch_size=64,
    normalize_embeddings=True,
    random_state=42,
    max_samples=200_000,
    verbose=False,
)

Pass a config instance or a plain dict:

clusterer = SemanticClusterer(config=SemanticClustererConfig(cluster_granularity="coarse"))
clusterer = SemanticClusterer(config={"cluster_granularity": "coarse", "batch_size": 128})

Custom Embedders

Any object with one of these methods works automatically:

class MyModel:
    def embed(self, texts: list) -> np.ndarray: ...           # preferred
    def encode(self, texts: list, **kwargs) -> np.ndarray: ...  # SentenceTransformers
    def embed_documents(self, texts: list) -> list: ...        # LangChain

Or a callable: embedding_model=lambda texts: my_api(texts)

SentenceTransformers

from sentence_transformers import SentenceTransformer
from semantic_clusterer import SemanticClusterer

model = SentenceTransformer("all-MiniLM-L6-v2")
sc = SemanticClusterer(embedding_model=model)

LangChain / Azure OpenAI

import os
from dotenv import load_dotenv
from langchain_openai import AzureOpenAIEmbeddings
from semantic_clusterer import SemanticClusterer, SemanticKSplit

load_dotenv()

embedder = AzureOpenAIEmbeddings(
    azure_deployment=os.environ["AZURE_OPENAI_EMBEDDING_DEPLOYMENT"],
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version=os.environ["AZURE_OPENAI_API_VERSION"],
)

sc = SemanticClusterer(embedding_model=embedder, cluster_granularity="balanced")
sc.fit(texts)
labels = sc.predict(new_texts)

ks = SemanticKSplit(embedding_model=embedder, k=8, quality="best")
groups = ks.split(texts)

text-embedding-3-small produces 1536-dim vectors (the high band). The library picks band-appropriate parameter grids automatically — no retuning needed.

Determinism

Two runs produce permutation-equivalent labels when they share:

• Same library version
• Same Python minor version
• Same OS family
• Same major.minor of numpy, scikit-learn, hdbscan, umap-learn
• Same random_state

Pin the seed:

sc = SemanticClusterer(random_state=42)
ks = SemanticKSplit(k=8, random_state=42)

Dependencies

UMAP fallback: If umap-learn fails to import, small and medium pipelines fall back to PCA-only and emit a UserWarning. Large and tiny pipelines are unaffected.

Missing hdbscan: SemanticClusterer.__init__ raises ImportError with a helpful message.

Known Limitations

• Model download: First run downloads ~90 MB to ~/.cache/semantic_clusterer/.
• Empty / short inputs: Texts that are empty after preprocessing receive label -1.
• Large datasets: N > 200 000 raises ValueError by default. Use max_samples=None to enable the subsample-then-assign fallback.
• Large pipeline (untested): The large tier (50K–200K) has been architecturally upgraded but has not been benchmarked or tested. Tiny, small, and medium tiers are fully validated.
• confidence: The confidence field in detailed output is a cohesion metric, not a probability.
• outlier_threshold: The auto-calibrated threshold works best when training texts are representative of the input domain. If you cluster support tickets and predict news articles, adjust the threshold manually.

Examples

python examples/01_beginner_zero_config.py
python examples/08_fit_predict_save_load.py

# Azure examples require .env with AZURE_OPENAI_* credentials:
python examples/07_advanced_azure_openai.py

License

MIT