divsel
Diverse subset selection that actually has a guarantee. A native (Rust) implementation of GIST — max-min diversification with submodular utility — with Python bindings that are zero-copy for metric="euclidean" (the default, metric="cosine", makes exactly one L2-normalised copy).
Status: 0.1.0, the first release. Changes:
CHANGELOG.md; how a release is cut:docs/RELEASE.md.
Install
pip install divsel # Python: one abi3 wheel covers CPython 3.11-3.14 per platform
cargo add divsel # Rust: the core crate
From a checkout instead: pip install . (needs a Rust toolchain, 1.83+).
Free-threaded CPython 3.14t is served by a separate
version-specific cp314t wheel, not the abi3 one. Adapter extras:
pip install "divsel[langchain]" / "divsel[llamaindex]".
Quick start
import numpy as np
import divsel
vectors = np.random.default_rng(0).standard_normal((50, 8), dtype=np.float32)
picked = divsel.gist_select(vectors, k=5, lam=1.0) # diverse-but-relevant row indices
full = divsel.gist_select_full(vectors, k=5, lam=1.0) # + objective, diversity, threshold, stage
print(picked, full["f_value"])
# The default is metric="cosine": rows are L2-normalised into one copy, and the
# (1/2 - eps) and (2/3 - eps) bounds are proven for a true metric, so raw cosine
# distance (the paper's own experimental setting) is a well-behaved heuristic.
# metric="euclidean" is the zero-copy path, and the one the proofs cover:
picked = divsel.gist_select(vectors, k=5, lam=1.0, metric="euclidean")
gist_select runs on rayon's process-global thread pool (sized from
RAYON_NUM_THREADS, built on the first call, never shut down). On Linux, forking
after a first call — multiprocessing with the default fork start method —
leaves the child with a pool whose workers do not exist; use spawn/forkserver,
or make the first call inside each child.
The problem
You retrieved 50 candidates and need the best 5 to put in a prompt, a training batch, or a recommendation slate. Take the top 5 by score and you get five near-duplicates. The standard fix is MMR (Maximal Marginal Relevance, 1998) — a greedy heuristic with no approximation guarantee and an uninterpretable λ.
divsel solves the same shape of problem with a proof attached:
maximize f(S) = g(S) + λ · min-pairwise-distance(S) subject to |S| ≤ k
where g is any monotone submodular utility (relevance, coverage, facility location). GIST achieves (1/2 − ε)·OPT for submodular g and (2/3 − ε)·OPT for linear g — the latter provably tight, since no polynomial-time (2/3 + ε) algorithm exists unless P = NP. The problem is NP-hard to approximate beyond ≈0.5584.
Why this exists
GIST — Greedy Independent Set Thresholding for Max-Min Diversification with Submodular Utility, Fahrbach, Ramalingam, Zadimoghaddam, Ahmadian, Citovsky & DeSalvo (Google Research), NeurIPS 2025, arXiv:2405.18754 — is a strong result with no production-grade implementation. As of August 2026:
- crates.io returns one result for
submodular, and it is unrelated. No maintained Rust crate does submodular maximization, facility location, or GIST. - The two Python implementations are a 1-commit release with no LICENSE file in its repository (the PyPI wheel does carry one) and a 6-commit repo that was never published to PyPI.
submodlib(131★) implements no combinedg(S) + λ·div(S)objective at all, has had no release since 0.0.3 (PyPI upload 2025-05-14), and ships no Windows wheels and no sdist — you cannot install it on Windows, or on Python 3.13/3.14, at any price.
So divsel aims at three things nobody currently offers together: a real license, wheels that install everywhere, and benchmarks you can reproduce from the repo. The measurements behind these claims — the installability matrix (all 48 cells measured: Windows locally and Linux/Windows/macOS on CI; the abi3 wheel does not cover free-threaded 3.14t), the comparison against gist-select, gist-sampling and MMR, and the incumbents' own README numbers re-run — are in docs/benchmarks/README.md, produced by bench/compare.py.
Design commitments
- The paper is the spec. Every constant transcribed from arXiv:2405.18754v3. Where the paper is silent (argmax tie-breaking),
divseldocuments its choice as a choice rather than inventing a citation. - Proven, not asserted. A brute-force oracle enumerates
OPTon small instances and asserts the approximation ratio actually holds — the test no incumbent ships. - Installs everywhere. abi3 wheels for Python 3.11 → 3.14, Linux · macOS · Windows, x86_64 and aarch64.
- Reference implementation. Exports
golden-selection.json; the ports in Aura (Python) andlimbic(TypeScript) conform to it. - Checked against an independent rewrite. A 90,000-instance differential (2026-08-26) between the Rust core and Aura's pure-Python port — written from
docs/CONFORMANCE.mdalone, never from this source — found zero algorithmic disagreements: all 1,164 differing selections reproduced divsel's answer once the port was fed divsel's own f32 distance matrix. Not a claim that the two always agree — a claim that where they differ, it is the width of the distance arithmetic. Details, parameter ranges and the two contract defects it exposed: Independent verification. - Apache-2.0.
Drop-in for MMR
Both adapters live behind optional extras; plain import divsel never imports either framework.
LangChain — pip install "divsel[langchain]"
from divsel.adapters.langchain import DivselRetriever
retriever = DivselRetriever(vectorstore=vs, k=5, fetch_k=20, lam=1.0)
docs = retriever.invoke("your query") # replaces vs.as_retriever(search_type="mmr")
Like MMR it fetches fetch_k candidates by query similarity, then returns the k of them maximizing g(S) + λ·min-distance(S) — with the guarantee instead of the greedy heuristic. Honest caveats: candidate texts are re-embedded through vectorstore.embeddings (stores do not expose their stored vectors uniformly), and when the store exposes no embeddings at all the retriever emits DivselFallbackWarning and returns plain, undiversified top-k (strict=True raises instead).
LlamaIndex — pip install "divsel[llamaindex]"
from divsel.adapters.llamaindex import DivselNodePostprocessor
engine = index.as_query_engine(
similarity_top_k=20, # this is the fetch_k — the candidate pool
node_postprocessors=[DivselNodePostprocessor(k=5, lam=1.0)],
)
There is no fetch_k parameter here: the retriever's similarity_top_k already fixes the candidate pool, and the postprocessor diversifies it down to k. Vectors come from node.embedding when every node carries one, else from an optional embed_model; with neither, it warns (DivselFallbackWarning) and returns top-k by score (strict=True raises).
Name
divsel = diverse selection. Verified free on both crates.io and PyPI, 2026-08-21.
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file divsel-0.1.0.tar.gz.
File metadata
- Download URL: divsel-0.1.0.tar.gz
- Upload date:
- Size: 186.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3574c42d442bbd1ef5414e66d7ba67e17844aead4887975628c84bd971942bc6
|
|
| MD5 |
81c23302d7c9379928a11308a43c424f
|
|
| BLAKE2b-256 |
ea1d06b1224bf5f5450c7d3f8e1299073019e5351bd826299156618cccc2cb1e
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-win_arm64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-win_arm64.whl
- Upload date:
- Size: 225.6 kB
- Tags: CPython 3.14t, Windows ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
bf41666402bfa96206f923930ece490573167174969e226d58189333b9f1f9a5
|
|
| MD5 |
d603aea7df017398ed6e6d1f877812d2
|
|
| BLAKE2b-256 |
1cd38e1736c39b675177bfa581b456c097708921b506ee1bb36b3b4732e573eb
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-win_amd64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-win_amd64.whl
- Upload date:
- Size: 237.2 kB
- Tags: CPython 3.14t, Windows x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
edc0738811db939b612a3b47af830735dda6e2621f7fae1d77905c9b377d3873
|
|
| MD5 |
6d689d4b0b860191bcb474826b7b6849
|
|
| BLAKE2b-256 |
09356e3d271a1c543711ce97c580da8454508bcd3c73c2efc8147dc75bcf4544
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 410.6 kB
- Tags: CPython 3.14t, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d155fdbb4b41af59987cf668a8f9129ea8bfb30af3b3314c72c8a16abad29bc3
|
|
| MD5 |
2de4f9aae2aacd210502dba641030e5d
|
|
| BLAKE2b-256 |
ec8cf8d00fd78d8e2b7ddbda1fcdce60639dc98947c2e2a7266537d626b30081
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 394.2 kB
- Tags: CPython 3.14t, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7a352570c11a56b4eec32bd16b1298b0aefe0025a645003dddc77b0d8ceaabe7
|
|
| MD5 |
5ba965e80965d6d22c689b2d07661a62
|
|
| BLAKE2b-256 |
62bc6ec3eb0fede56ff507fa79ce2c7d70116084b71826904774e9b69a892a95
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-macosx_11_0_arm64.whl
- Upload date:
- Size: 352.8 kB
- Tags: CPython 3.14t, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1bbc3f97edd8f9c3de87041c61e005187d851138ec874fea51d96651444f8edd
|
|
| MD5 |
d6ae86ddc87071e5078e3376a5c2f3ef
|
|
| BLAKE2b-256 |
954410be8180b946dab9ee06836c73e45c30190fcd1dbaa4f33e92453a6dbb19
|
File details
Details for the file divsel-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl.
File metadata
- Download URL: divsel-0.1.0-cp314-cp314t-macosx_10_12_x86_64.whl
- Upload date:
- Size: 361.2 kB
- Tags: CPython 3.14t, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3718b3eaf87fd7e007347b03b13f1500aee6d186e59fe0ea96f9aceeb8cab8f6
|
|
| MD5 |
68c2065f260421e3e7e7bbbcb62efaa6
|
|
| BLAKE2b-256 |
73d69e757554cadb22cf29c02f9f7f295b912d9611fd4876756622cda85f38d5
|
File details
Details for the file divsel-0.1.0-cp311-abi3-win_arm64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-win_arm64.whl
- Upload date:
- Size: 229.6 kB
- Tags: CPython 3.11+, Windows ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
12fb47ba21a8c12f96430cb426ad5f5e4f80161329be7a383d2e8a05fb61a02e
|
|
| MD5 |
06db7549573e783c91e47c7669d6bb77
|
|
| BLAKE2b-256 |
92393fa4caea0bed314117b55839cf06270a2aa057a6fe6aac280710da3d40c9
|
File details
Details for the file divsel-0.1.0-cp311-abi3-win_amd64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-win_amd64.whl
- Upload date:
- Size: 240.5 kB
- Tags: CPython 3.11+, Windows x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b5b53f440a30d9f2505be6c2382b03623ba48b5cc6b4d7ee3cb869de46ddf8a3
|
|
| MD5 |
c5c4b578bc37b4f3f7026cdee6d2f66b
|
|
| BLAKE2b-256 |
502d4c106b57313341103a5f4eb5212ff90cd7dab7aefb9cc22732cc4d6a492c
|
File details
Details for the file divsel-0.1.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 414.9 kB
- Tags: CPython 3.11+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
28699211a798b4230ae950ebac5680ce8f076ad0092c5e1c980cedfd64eedebb
|
|
| MD5 |
94b70b610d55e60dc5dda107b3f7183a
|
|
| BLAKE2b-256 |
7876858c5387d1f5415598e9306a470b0e263226863b2002821a3f8092a2210a
|
File details
Details for the file divsel-0.1.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 399.4 kB
- Tags: CPython 3.11+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b0b6783ad192b81d08181d4f34ebc5521cc0722841a28ef94741452754156e85
|
|
| MD5 |
af2722f121eed744531b5c905a8b2815
|
|
| BLAKE2b-256 |
a12a1207f0ecda68635e339c9471e409f7da552215d48c9a0eefb28759e47b09
|
File details
Details for the file divsel-0.1.0-cp311-abi3-macosx_11_0_arm64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 358.7 kB
- Tags: CPython 3.11+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
64447339b8880743a444ab39ec2ef684f94af31b442fa5ffbf1691dfae618001
|
|
| MD5 |
ec35f6d84903c26fc3dccdc3e60079c3
|
|
| BLAKE2b-256 |
2689a097fa24e4174675b8bd30795f05a506a3b8ab88fd46413168ebeebf6e22
|
File details
Details for the file divsel-0.1.0-cp311-abi3-macosx_10_12_x86_64.whl.
File metadata
- Download URL: divsel-0.1.0-cp311-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 363.2 kB
- Tags: CPython 3.11+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
uv/0.12.9 {"installer":{"name":"uv","version":"0.12.9","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d725efc5628d9aa9bb9484f42a1239bc66fad1d71795f93b993f675bc032cfef
|
|
| MD5 |
1a2627543dd2bd9f7b7bce7f67b71c45
|
|
| BLAKE2b-256 |
943884cb00a0704ef47df7af3fb2ab7e8e7d699ef12fd631cd54ad5c1cbb104b
|