splitscope

Does your training set actually cover your eval set?

splitscope clusters a train dataset, projects an eval dataset onto those clusters, and tells you how well the eval examples fit the train distribution. Use it to spot distribution gaps, decide how aggressively you can filter or subsample training data, and understand which parts of your data matter for evaluation.

• Fast CPU features: hashed word + char n-grams, TF-IDF, optional LSA. No GPU needed.
• LLM embeddings: pooled hidden states from any Hugging Face model (mean or last pooling).
• Multiple clustering backends: KMeans, MiniBatchKMeans, FAISS (CPU or GPU).
• Library and CLI.

Installation

pip install splitscope                # CPU features + sklearn KMeans / MiniBatchKMeans
pip install "splitscope[faiss]"       # + FAISS (CPU)
pip install "splitscope[faiss-gpu]"   # + FAISS (CUDA)
pip install "splitscope[llm]"         # + LLM embeddings (torch, transformers)
pip install "splitscope[all]"         # everything (CPU FAISS)

Optional dependencies are never installed automatically at import time. If a backend is missing, you get a clear error with the install command.

Quick start

Python

import splitscope as ss

train = ss.load_texts("train.jsonl")
ev = ss.load_texts("test.jsonl")

result = ss.analyze(
    train, ev,
    features="hashing",       # "hashing" | "llm"
    clusterer="minibatch",    # "kmeans" | "minibatch" | "faiss"
    n_clusters=150,
)
print(result.summary())

Command line

# CPU only
splitscope --train train.jsonl --eval test.jsonl --clusterer minibatch

# LLM embeddings + FAISS
splitscope --train train.jsonl --eval test.jsonl \
           --features llm --model Qwen/Qwen3-4B \
           --llm-max-length auto --clusterer faiss

Input format

.jsonl (one JSON object per line) or .json (a list of objects). By default, each record's text is built from these fields:

Text = instruct + input + output. If neither prompt field exists, all values in the record are concatenated. Change the field names with TextConfig or the --instruction-col, --input-col and --output-col flags:

cfg = ss.TextConfig(instruction_col="question", input_col=None, output_col="answer")
train = ss.load_texts("train.jsonl", cfg)

How it works

1. Featurize train texts (hashing or LLM embeddings).
2. Cluster the train set only.
3. Project eval texts into the same feature space and assign each to its nearest train centroid.
4. Compare each eval point's distance to its centroid against that cluster's train distance distribution:
   • 1σ band: within mean ± std of train distances.
   • Robust band: within a MAD-based robust z-score of 3.5.
5. Recommend a data-reduction strategy based on how much of eval falls inside the train core.

Output

analyze() returns an AnalysisResult:

Advanced usage

Fit once, analyze several eval sets:

from splitscope import ClusterAnalyzer

analyzer = ClusterAnalyzer(
    features="llm",
    clusterer="faiss",
    n_clusters=200,
    features_kwargs=dict(model_name="Qwen/Qwen3-4B", pooling="last",
                         max_length="auto", batch_size=8),
)
analyzer.fit(train)
res_a = analyzer.analyze(eval_a)
res_b = analyzer.analyze(eval_b)

Inspect a cluster:

cid = res_a.eval_cluster_counts.most_common(1)[0][0]
s = res_a.stats[cid]
print(s.n, s.mean, s.std)
print(train[s.representative_idx])

Choosing a backend

Notes:

• hashing features default to LSA (96 dims). Without it, centroids are dense over a very wide space, which is slow and memory-hungry.
• With features="llm", max_length="auto" uses the P95 token length of the train texts. Longer examples are truncated.
• LLM features are pooled hidden states from a base model, not retrieval-trained embeddings. A dedicated embedding model usually gives sharper clusters.

CLI reference

Development

git clone https://github.com/antony-ro/splitscope
cd splitscope
pip install -e ".[dev]"
pytest

License

MIT