Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

everalgo-clustering

Online incremental clustering for EverAlgo — cluster_by_geometry (sync) and acluster_by_llm (async, LLM-refined; cluster_by_llm is its sync bridge) operating on caller-owned list[Cluster] state.

Stateless: the package never embeds, never queries storage, never holds a lock. The caller owns embedding computation, persistence (list[Cluster] serialisation), and read-modify-write coordination across concurrent writers.

See the umbrella project: EverAlgo monorepo and the architecture document at docs/concepts/architecture.md.

Install

pip install everalgo-clustering

What this distribution provides

Symbol Role
Cluster Frozen Pydantic value object — one cluster snapshot. Caller-supplied id / members; algorithm supplies merged centroid, count, last_ts, preview.
cluster_by_geometry Cosine similarity + time-window filter + threshold; no LLM; sync
acluster_by_llm Top-K geometric recall → fast-path skip → LLM semantic ranking; raises on LLM failure; async
cluster_by_llm Blocking sync bridge over acluster_by_llm; only outside a running event loop

Cluster type

class Cluster(BaseModel):
    id: str | None = None           # caller-supplied business id; algo only passes through, never mints
    centroid: np.ndarray
    count: int = 1
    last_ts: int                    # Unix epoch milliseconds
    preview: list[str] = []
    members: list[str] = []         # caller-supplied entity ids; algo appends on merge, never inspects

The caller wraps each incoming item as a size-1 Cluster (count=1) before passing to either function. Both functions return Cluster | None: a merged snapshot when the item is assigned to an existing cluster, or None when no match — the caller then appends the original size-1 Cluster as a brand-new entry and mints its own id.

Quick start

import numpy as np
from everalgo.clustering import Cluster, cluster_by_geometry

existing: list[Cluster] = []  # caller loads from storage; empty on first run
vector = np.random.rand(2560).astype(np.float32)
timestamp_ms = 1_700_000_000_000

new_cluster = Cluster(centroid=vector, last_ts=timestamp_ms)
merged = cluster_by_geometry(  # sync — no await
    new_cluster,
    existing,
    threshold=0.65,        # cosine similarity floor
    time_window_days=7.0,  # ignore clusters older than this window
)

if merged is not None:
    # item assigned to an existing cluster; caller updates the matching entry
    print(f"merged into cluster id={merged.id!r}, new count={merged.count}")
else:
    # no match — caller appends new_cluster and stamps its own id
    new_cluster_with_id = new_cluster.model_copy(update={"id": "cid_001"})
    existing.append(new_cluster_with_id)
    print("created new cluster")

LLM-refined clustering

acluster_by_llm adds a semantic ranking step over the top-K geometrically-nearest candidates. It raises on LLM failure — there is no internal fallback and no retry; the caller decides whether to retry or fall back to cluster_by_geometry. The call asks for a strict {"idx": int, "reason": str} object (idx is the candidate's index in existing_clusters, -1 for a new cluster); a reply that is not exactly that raises everalgo.llm.LLMOutputError. The call sends a system message of rules and a user message holding the candidate clusters and the new item's preview. custom_instructions= adds the host's text to the rules, never replacing them: it is placed verbatim, never rendered as a template, in a <custom_instructions> block that is always the system message's last, and None or a blank string leaves it out. There is no prompt=: the bundled rules cannot be replaced per call.

Populate new_cluster.preview with the item's representative text so the LLM has something to rank against.

import asyncio
import numpy as np
from everalgo.clustering import Cluster, acluster_by_llm
from everalgo.testing import FakeLLMClient

_LLM_JSON = '{"idx": 0, "reason": "same domain"}'

async def main() -> None:
    fake = FakeLLMClient(responses=[_LLM_JSON])
    existing: list[Cluster] = []
    vector = np.random.rand(2560).astype(np.float32)

    new_cluster = Cluster(
        centroid=vector,
        last_ts=1_700_000_000_000,
        preview=["Python async retry patterns"],  # shown to the LLM
    )
    merged = await acluster_by_llm(
        new_cluster,
        existing,
        llm=fake,
        k_candidates=30,
        llm_skip_threshold=0.85,
    )
    print(f"merged: {merged}")

asyncio.run(main())

Persistence pattern

Cluster is a frozen Pydantic model — serialise with model_dump() and reconstruct with Cluster.model_validate(). The caller owns the list and the lock:

raw_list = await store.load(user_id) or []
clusters = [Cluster.model_validate(r) for r in raw_list]

async with caller.lock(f"cluster:{user_id}"):
    merged = cluster_by_geometry(new_cluster, clusters)
    if merged is not None:
        idx = next(i for i, c in enumerate(clusters) if c.id == merged.id)
        clusters[idx] = merged
    else:
        new_cluster_stamped = new_cluster.model_copy(update={"id": new_id})
        clusters.append(new_cluster_stamped)
    await store.save(user_id, [c.model_dump() for c in clusters])

API reference

def cluster_by_geometry(
    new_cluster: Cluster,
    existing_clusters: list[Cluster],
    *,
    threshold: float = 0.65,
    time_window_days: float = 7.0,
    preview_cap: int = 5,
) -> Cluster | None: ...

async def acluster_by_llm(
    new_cluster: Cluster,
    existing_clusters: list[Cluster],
    *,
    llm: LLMClient,
    k_candidates: int = 30,
    llm_skip_threshold: float = 0.85,
    custom_instructions: str | None = None,
    preview_cap: int = 5,
) -> Cluster | None: ...

cluster_by_llm = async_to_sync(acluster_by_llm)  # same signature, blocking

Both functions return the merged Cluster (existing cluster's id preserved, centroid/count/members updated) or None (no match — caller creates a new cluster entry and mints its own id).

Tested embedding model: Qwen3-Embedding-4B (2560-dim float32). Any consistent-dimension embedding works; EverAlgo does not import or manage embedding SDKs.

Metadata

Release files for everalgo-clustering 0.3.0rc1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for everalgo-clustering 0.3.0rc1
File Size Uploaded Provenance Actions
everalgo_clustering-0.3.0rc1.tar.gz 19.0 kB Without provenance Details for everalgo_clustering-0.3.0rc1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for everalgo-clustering 0.3.0rc1
File Interpreter ABI Platform Actions
everalgo_clustering-0.3.0rc1-py3-none-any.whl Python 3 none any Details for everalgo_clustering-0.3.0rc1-py3-none-any.whl

Total release size: 33.5 kB

Release files / everalgo_clustering-0.3.0rc1.tar.gz

Download URL everalgo_clustering-0.3.0rc1.tar.gz
Size 19.0 kB
Tags Source
SHA-256 checksum
How to use checksums
abe321dfefb80631f13edd8215c6d28525f5dc5339d4f2a946cc6c27355d747e
BLAKE2b-256 checksum
How to use checksums
27b3fa870a115f3e771a7a307c3d22121daa202d6f2098f3648f29de44502e8d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.12.12

Release files / everalgo_clustering-0.3.0rc1-py3-none-any.whl

Download URL everalgo_clustering-0.3.0rc1-py3-none-any.whl
Size 14.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
6806969f12724e69945c29886b25816a3221023755b83c9e9efd7f396f85e501
BLAKE2b-256 checksum
How to use checksums
2dfa301f31517552edf57998b6d940707b607ae1f40a6844dc8da27cf2f7d5f4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.12.12

Security for everalgo-clustering 0.3.0rc1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page