Intelliant: Graph-Based Algorithm for Semantic Core Extraction

Intelliant is a specialized clustering algorithm based on the Ant Colony Optimization (ACO) metaheuristic.

Unlike classical methods, the algorithm does not draw mathematical cluster boundaries in hyperspace. It transforms high-dimensional data (e.g. LLM embeddings) into a k-nearest-neighbor graph (KNN) and uses swarm intelligence to find the densest semantic centers (cores).

The algorithm is optimized with Numba and sparse matrices (CSR), which allows processing hundreds of thousands of objects in seconds on a regular CPU, avoiding the memory footprint of industrial standards like HDBSCAN.

Status and packages

Alpha. The package is under active development (research phase). The public API still changes between versions. This is a personal research tool, not a production-ready solution. See ROADMAP.md for what is done and what is ahead; long-term research ideas are in RESEARCH_NOTES.md.

• Current package: intelliant (three-class architecture).
• Old package (deprecated): intelliant-core - single-class IntelliantCoreExtractor architecture. The link is kept for those who arrived via the old name; do not install it, development moved to intelliant.

Features

• Dimension-independent. Works on a similarity graph (cosine, euclidean), not on raw coordinates. Suitable for both 2D/3D and 384D+ embeddings.
• Min-Max Ant System (MMAS). Pheromone stagnation protection prevents the graph from collapsing into a single hub "black hole".
• Elite ants. Accelerated core formation via greedy behavior of a designated agent group, with a configurable start iteration (separating exploration / exploitation phases).
• Node density heuristic. Optional: ants evaluate the local density of the target node, accelerating convergence.
• Two-stage noise absorption. Label propagation through pheromone waves, then centroid fallback for isolated points. Available both as a single call and as separate stages (for frame-by-frame visualization and intermediate state caching).
• Giant detection. Signals a suspected cluster merge/inflation by a size gap (without modifying the data).
• Transparent state. All intermediate artifacts (graph, pheromone field, raw cores, labels) are accessible via attributes for visualization, debugging, and caching between sessions.

Architecture

Clustering is split into three conceptually independent classes, each in its own module:

There is no "all-in-one" call: the three parts solve different problems (data processing, pheromone production, pheromone interpretation into clusters), and the separation is deliberate so that each stage can be tuned and inspected independently. The pheromone cutoff threshold and embedding extraction are currently prepared by the user outside the library.

Requirements

• Python: >= 3.14
• Package manager: uv
• Dependencies (installed automatically): numpy, scipy, numba, scikit-learn, pynndescent, tqdm.
• Hardware acceleration (for generating embeddings with a separate model): CUDA / MPS via PyTorch - in the embeddings dependency group.

Installation

From PyPI (alpha):

pip install intelliant
# or explicitly the alpha version:
pip install intelliant==0.1.0a2

From source (for development):

git clone https://github.com/yourdisenchantment/intelliant.git
cd intelliant
uv sync                           # library only
uv sync --all-groups --all-extras # + notebooks, embeddings, dev tools

Dependency groups: notebooks (jupyter, visualization, polars, umap), embeddings (torch, sentence-transformers, datasets), dev (pre-commit, commitizen, ruff, pyright, scipy-stubs, bandit, deptry, vulture).

To load test datasets, create a .env in the project root with a Hugging Face token:

HF_API_TOKEN=hf_your_token_string

Usage example

The pipeline consists of three steps: build the graph, run the ant colony, extract clusters. The pheromone cutoff threshold is set by the user (below - a simple example via percentile).

import numpy as np
from intelliant import GraphBuilder, PheromoneExtractor, CoreClusterer

# X - embedding matrix (N, D), prepared by the user

# 1. Similarity graph from embeddings
graph = GraphBuilder(
    n_neighbors=15,
    metric="cosine",
    mutual=True,         # mutual KNN (AND-symmetrization)
    min_connections=5,   # connectivity top-up for isolated points
    knn_method="auto",   # exact for small datasets, approx for large
    random_state=42,     # for reproducible approx search
).build(X)

# 2. Ant colony run (graph -> pheromone graph)
aco = PheromoneExtractor(
    n_ants=len(X),              # explicit ant count (critical parameter)
    n_iterations=20,
    use_elite_ants=True,
    elite_start_iteration=10,   # elite kicks in from the middle of the run
    random_state=42,            # colony seed (independent of graph seed)
)
aco.fit(graph)
pheromones = aco.pheromone_matrix_

# 3. Cutoff threshold (user computes it; here - percentile)
threshold = np.percentile(pheromones.data, 90)

# 4. Core extraction and noise absorption
clusterer = CoreClusterer(min_cluster_size=50, batch_size=200_000)
cores = clusterer.extract_cores(pheromones, threshold)   # cores 0..k-1, noise -1
labels = clusterer.absorb(pheromones, X)                 # noise fill-in

# Intermediate state is available for diagnostics and visualization:
#   clusterer.cores_             - raw cores before absorption
#   clusterer.labels_pheromone_  - after pheromone waves (stage 1)
#   clusterer.labels_            - final labels

Absorption stages can be called separately (absorb_pheromone, then absorb_centroid) - this gives three state snapshots (raw cores -> after waves -> final) and allows saving intermediate results between sessions.

Each class stores its result in an attribute with a trailing underscore (graph_, pheromone_matrix_, cores_, labels_pheromone_, labels_) per sklearn convention and also returns it from the method.

Project structure

src/intelliant/     # public API: three classes + threshold module
tests/              # pytest suite (224 tests, 20 files; 223 default + 1 slow)
notebooks/          # calibration and dataset notebooks
old_notebooks/      # first-iteration experiments (see below)
utils/              # notebook support scripts (metrics, tee)
data/, results/     # data and results (gitignored, not in repo)

old_notebooks/

First-iteration notebooks from the early experimental phase of intelliant. Two subdirectories:

• old_notebooks/1/ - initial single-class prototypes (2D/3D synthetic tests, AG News, HDBSCAN comparison).
• old_notebooks/2/ - second round of experiments before the three-class refactor (synthetic 2D/3D, AG News, benchmarks).

These are kept for historical reference. The current architecture (src/intelliant/) supersedes them.

Roadmap

• What is ahead (current and near-term work) - ROADMAP.md.
• Long-term research ideas (multilevel hierarchical clustering, etc.) - RESEARCH_NOTES.md. These are notes for the future, not commitments of the current project.

License

MIT.