splitscope

Does your training set actually cover your eval set?

splitscope clusters a train dataset, projects an eval dataset onto those clusters, and tells you how well the eval examples fit the train distribution. Use it to spot distribution gaps, decide how aggressively you can filter or subsample training data, and understand which parts of your data matter for evaluation.

• Fast CPU features: hashed word + char n-grams, TF-IDF, optional LSA. No GPU needed.
• LLM embeddings: pooled hidden states from any Hugging Face model (mean or last pooling).
• Multiple clustering backends: KMeans, MiniBatchKMeans, FAISS (CPU or GPU).
• Cluster-balanced train/eval splitting of a single dataset, with outlier handling and an optional cache.
• Library and CLI.

Installation

pip install splitscope                # CPU features + sklearn KMeans / MiniBatchKMeans
pip install "splitscope[faiss]"       # + FAISS (CPU)
pip install "splitscope[faiss-gpu]"   # + FAISS (CUDA)
pip install "splitscope[llm]"         # + LLM embeddings (torch, transformers)
pip install "splitscope[all]"         # everything (CPU FAISS)

Optional dependencies are never installed automatically at import time. If a backend is missing, you get a clear error with the install command.

Quick start

Python

import splitscope as ss

train = ss.load_texts("train.jsonl")
ev = ss.load_texts("test.jsonl")

result = ss.analyze(
    train, ev,
    features="hashing",       # "hashing" | "llm"
    clusterer="minibatch",    # "kmeans" | "minibatch" | "faiss"
    n_clusters=150,
)
print(result.summary())

Command line

# analyze: how well does train cover eval? (CPU only)
splitscope analyze --train train.jsonl --eval test.jsonl --clusterer minibatch

# analyze with LLM embeddings + FAISS
splitscope analyze --train train.jsonl --eval test.jsonl \
                   --features llm --model Qwen/Qwen3-4B \
                   --llm-max-length auto --clusterer faiss

# split: one dataset -> train.jsonl + eval.jsonl
splitscope split --data data.jsonl --eval-ratio 0.1 --out-dir splits/

Input format

.jsonl (one JSON object per line) or .json (a list of objects). By default, each record's text is built from these fields:

Text = instruct + input + output. If neither prompt field exists, all values in the record are concatenated. Change the field names with TextConfig or the --instruction-col, --input-col and --output-col flags:

cfg = ss.TextConfig(instruction_col="question", input_col=None, output_col="answer")
train = ss.load_texts("train.jsonl", cfg)

How it works

1. Featurize train texts (hashing or LLM embeddings).
2. Cluster the train set only.
3. Project eval texts into the same feature space and assign each to its nearest train centroid.
4. Compare each eval point's distance to its centroid against that cluster's train distance distribution:
   • 1σ band: within mean ± std of train distances.
   • Robust band: within a MAD-based robust z-score of 3.5.
5. Recommend a data-reduction strategy based on how much of eval falls inside the train core.

Output

analyze() returns an AnalysisResult:

Splitting a dataset

Have one dataset and no eval set yet? split_dataset clusters it and carves out an eval set whose cluster distribution matches the train set.

import splitscope as ss

cfg = ss.SplitConfig(eval_ratio=0.1, outliers="drop", clusterer="minibatch")

# SplitResult only
result = ss.split_dataset("data.jsonl", cfg)

# SplitResult + the train rows, the eval rows, or both (always in this order)
result, train = ss.split_dataset("data.jsonl", cfg, return_train=True)
result, eval_ = ss.split_dataset("data.jsonl", cfg, return_eval=True)
result, train, eval_ = ss.split_dataset("data.jsonl", cfg, return_train=True, return_eval=True)

print(result.summary())
result.save("splits/", which="both")   # "train" | "eval" | "both"

How it works:

• Duplicates stay together. Rows are grouped by a content hash first, so an exact duplicate can never land in both train and eval.
• Balanced coverage. Each cluster receives eval slots in proportion to its size. Every cluster keeps at least one row in train.
• Which rows become eval: strategy="quantile" (default) picks rows evenly spaced by distance to the centroid, so eval covers the core and the tails of each cluster. strategy="random" is stratified random sampling.
• Outliers (far from their centroid by a robust z-score, or in a cluster smaller than min_cluster_size) follow the outliers policy: drop, train, eval or keep (no outlier detection).
• Reference eval (optional). Pass example eval sets as reference_eval=[...] to shift eval slots toward the clusters those examples fall into. reference_weight (0 to 1) controls how strongly.

Rows dropped as outliers are available as result.dropped.

Caching

Results are cached under ~/.cache/splitscope (override with cache_dir= or the SPLITSCOPE_CACHE environment variable). The cache key combines an order-independent hash of the dataset rows with all split parameters, so the same data and settings return the stored assignment without re-clustering. Only row hashes and labels are stored, not your data.

ss.split_dataset("data.jsonl", cfg, cache=False)   # skip the cache entirely

Advanced usage

Fit once, analyze several eval sets:

from splitscope import ClusterAnalyzer

analyzer = ClusterAnalyzer(
    features="llm",
    clusterer="faiss",
    n_clusters=200,
    features_kwargs=dict(model_name="Qwen/Qwen3-4B", pooling="last",
                         max_length="auto", batch_size=8),
)
analyzer.fit(train)
res_a = analyzer.analyze(eval_a)
res_b = analyzer.analyze(eval_b)

Inspect a cluster:

cid = res_a.eval_cluster_counts.most_common(1)[0][0]
s = res_a.stats[cid]
print(s.n, s.mean, s.std)
print(train[s.representative_idx])

Choosing a backend

Notes:

• hashing features default to LSA (96 dims). Without it, centroids are dense over a very wide space, which is slow and memory-hungry.
• With features="llm", max_length="auto" uses the P95 token length of the train texts. Longer examples are truncated.
• LLM features are pooled hidden states from a base model, not retrieval-trained embeddings. A dedicated embedding model usually gives sharper clusters.

CLI reference

splitscope split

It also accepts the feature and clusterer flags below (--features, --clusterer, --model, ...).

splitscope analyze

License

MIT