roaringrange (Python)

Build static, range-fetchable search datasets from Python, then search millions of records in the browser with no backend. These bindings wrap the core Rust build module, so the files they emit are byte-identical to the Go and Rust builders and are read by the same WASM reader. Two index types: a trigram text index (Builder) and a similarity / vector index (VectorBuilder).

What it produces

Builder.build(out_dir) writes the four files the text reader serves over HTTP Range; VectorBuilder.build(path) writes one .rrvi similarity index:

file format contents
index.rrs RRSI trigram text index (popularity-split postings)
index.rrf RRSF facet sidecar (field → category → doc-ID bitmap, with counts)
records.idx / records.bin RRSR per-doc record bytes (your encoding)
*.rrvi RRVI IVFPQ similarity index (range-fetched coarse clusters + PQ codes)

Upload them to S3/CloudFront and point the WASM reader at the URLs.

Install

Prebuilt abi3 wheels (one wheel for CPython 3.8+) are published to PyPI:

pip install roaringrange

CI builds and tests the extension on CPython 3.12, 3.13, and 3.14.

From source (dev)

cd python
maturin develop --release      # builds + installs into the active venv
# or: maturin build --release   # produces a wheel in target/wheels/

Requires a Rust toolchain and pip install maturin.

Usage

import roaringrange as rr, json

b = rr.Builder(gram_size=3)
for row in rows:                              # rows from a DataFrame, DB, JSONL, …
    b.add(
        rank=row["citations"],                # higher rank = listed first (doc-ID order)
        text=f'{row["title"]} {row["abstract"]}',   # tokenized into trigram keys
        record=json.dumps({"t": row["title"], "y": row["year"]}).encode(),
        facets={"year": [str(row["year"])], "type": [row["type"]]},  # field → categories
    )

stats = b.build("out/")        # writes out/index.rrs, index.rrf, records.idx, records.bin
print(stats)                   # BuildStats(docs=..., ngrams=..., fields=...)

rr.tokenize(text, gram_size=3) returns the n-gram keys a string maps to — useful for understanding why a query does or doesn't match.

Vector / similarity search

VectorBuilder trains an IVFPQ index over your embeddings and writes a single .rrvi file that the WASM reader range-fetches like the text index. Use the same doc_id as the text index so a vector hit maps to the same record (and can hybridize with trigram search). Vectors are L2-normalized for the default "ip" (cosine) metric.

import roaringrange as rr

vb = rr.VectorBuilder(dim=256, nlist=4096, m=32, metric="ip")  # m must divide dim
for doc_id, embedding in enumerate(embeddings):     # embeddings: any float sequences
    vb.add(doc_id, embedding.tolist())              # numpy row → list of floats
# or in one call: vb.add_many([(i, e.tolist()) for i, e in enumerate(embeddings)])

stats = vb.build("out/vectors.rrvi")
print(stats)   # VectorBuildStats(vectors=..., dim=256, nlist=..., m=32, nbits=8)

Parameters: nlist coarse clusters (≈ 4·√N, clamped to the vector count), m PQ subquantizers (must divide dim), nbits (1–8) → 2^nbits codes per subspace, metric "ip"/"cosine" or "l2". Training is deterministic (seed, kmeans_iters). One .rrvi per embedding model — each model is a different vector space. See ../VECTORS.md for the byte layout.

This pure-Rust trainer suits small/medium corpora and tests; at very large scale train with FAISS and export the same RRVI layout (the reader is identical).

Scale: train with FAISS, export to RRVI

For large corpora, train OPQ,IVF,PQ with FAISS and export the trained parts — no retraining in Rust. python/scripts/faiss_to_rrvi.py does this end to end (install the extra: pip install 'roaringrange[train]' for numpy + faiss-cpu):

from faiss_to_rrvi import export_to_rrvi
stats = export_to_rrvi(vectors, doc_ids, "vectors.rrvi", nlist=4096, m=32, metric="ip")

Under the hood it calls the low-level roaringrange.write_rrvi_from_faiss(...), which takes the FAISS arrays (OPQ rotation, coarse centroids, PQ codebooks, per-vector cluster + 8-bit codes) as little-endian byte buffers — so the wheel needs no numpy dependency. The export is verified against the Rust reader (recall@10 ≈ 0.9995 vs FAISS's own search on the same index).

Embedding text (mode 2: model2vec, no backend)

python/scripts/model2vec_embed.py embeds text with a model2vec static model (minishlab/potion-retrieval-32M, 512-d, mean-pooled token vectors — no transformer, fast on CPU) and builds a .rrvi. Install the extra: pip install 'roaringrange[embed]'.

from model2vec_embed import build_rrvi_from_texts
stats, _ = build_rrvi_from_texts(titles, doc_ids, "vectors.rrvi", nlist=256, m=32)

It's "mode 2" because the same model2vec recipe can run in the browser at query time, so similarity search needs no backend at all. The query embedding must use the identical model + pooling as the corpus, or the spaces won't match.

Notes

  • Ranking is baked in. Doc IDs are assigned in descending rank, so the top-K of any query is free at read time (no query-time scoring). Pick a good rank signal (citations, holdings, popularity, …).
  • Records are opaque. record= is raw bytes; the format never dictates your schema. Decode them however you like on the client.
  • In-memory build. This builds the whole index in RAM — ideal for up to many millions of records. For corpora whose index exceeds memory, the core crate's chunked path (build::chunk) is the route; exposing it here is a follow-up.

MIT — see ../LICENSE.

Project links

Data verified by PyPI on Jul 7, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.42.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for freeeve from gravatar.com freeeve

Credits

Author: freeeve

GitHub Statistics

Data verified by PyPI on Jul 7, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.8

Provides Extra

embed gemma train

Tags

search full-text roaring static-site serverless

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

roaringrange-0.29.0-cp38-abi3-win_amd64.whl (470.6 kB view details)

Uploaded CPython 3.8+Windows x86-64

roaringrange-0.29.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (577.5 kB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ x86-64

roaringrange-0.29.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (546.8 kB view details)

Uploaded CPython 3.8+manylinux: glibc 2.17+ ARM64

roaringrange-0.29.0-cp38-abi3-macosx_11_0_arm64.whl (515.7 kB view details)

Uploaded CPython 3.8+macOS 11.0+ ARM64

roaringrange-0.29.0-cp38-abi3-macosx_10_12_x86_64.whl (530.9 kB view details)

Uploaded CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file roaringrange-0.29.0-cp38-abi3-win_amd64.whl.

File metadata

  • Download URL: roaringrange-0.29.0-cp38-abi3-win_amd64.whl
  • Upload date:
  • Size: 470.6 kB
  • Tags: CPython 3.8+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for roaringrange-0.29.0-cp38-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 26f2b6c6ef2803d46381c29c13ce3c79bfd1772aa3ace1f334c644e4f9fa490d
MD5 fd6b2e0bd89f8c07d06e42df1ac79977
BLAKE2b-256 c9e30c22d13ba583706de89f4919a2f3e1888310baf8cf6503e7bd9f2c8b3810

See more details on using hashes here.

Provenance

The following attestation bundles were made for roaringrange-0.29.0-cp38-abi3-win_amd64.whl:

Publisher: release.yml on freeeve/roaringrange

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file roaringrange-0.29.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for roaringrange-0.29.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 570a969af1795c41367468ad73c7f1a05d4f9c6614931bea65c93b862d97a582
MD5 9de99b45af0ebfafb7f3f8e3802f9da2
BLAKE2b-256 cf41b041ecfb55ce706adaaa70e45f7a1492b1dee0f8d6e40e08acef390de455

See more details on using hashes here.

Provenance

The following attestation bundles were made for roaringrange-0.29.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on freeeve/roaringrange

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file roaringrange-0.29.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for roaringrange-0.29.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 b10b19a7106c10b496e5fc8d974e7d382388b626886ceaf62e8c2e30bce74643
MD5 615a67aafa4ece19d7a7bdabd7adb3f4
BLAKE2b-256 fe1b8c9ade6e237edda8325dfaecba033cb40fedcd0730dbcf840812b0922efc

See more details on using hashes here.

Provenance

The following attestation bundles were made for roaringrange-0.29.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on freeeve/roaringrange

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file roaringrange-0.29.0-cp38-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for roaringrange-0.29.0-cp38-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 70d72386bf961c3c23a87e2a882b560545c7aaa39fe2791e4b08d6d782e1cd60
MD5 c52b9b40eca6c41d80c282e53778d8a3
BLAKE2b-256 f6adac5f729e4acb773de8ae1fbd4b42c8e972a809d08c66ebaee6ac561067f1

See more details on using hashes here.

Provenance

The following attestation bundles were made for roaringrange-0.29.0-cp38-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on freeeve/roaringrange

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file roaringrange-0.29.0-cp38-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for roaringrange-0.29.0-cp38-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 d5acffa7171f81ac9ffb7ab97ed5e736b8aab72bb25cf4b44c96049df7474b12
MD5 40f35b61ba4e482b085237098dece207
BLAKE2b-256 dfaab6c7c2fb7829bd49233df4b94955d372c5b9756d93e98c20dba20d1c2d93

See more details on using hashes here.

Provenance

The following attestation bundles were made for roaringrange-0.29.0-cp38-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on freeeve/roaringrange

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.42.0

5 files

0.41.0

5 files

0.40.0

5 files

0.39.1

5 files

0.39.0

5 files

0.38.1

5 files

0.38.0

5 files

0.37.1

5 files

0.37.0

5 files

0.36.0

5 files

0.35.0

5 files

0.34.0

5 files

0.33.0

5 files

0.32.0

5 files

0.31.0

5 files

0.30.0

5 files

This release

0.29.0 This release

5 files

0.27.0

5 files

0.26.0

5 files

0.25.0

5 files

0.24.3

5 files

0.24.2

5 files

0.1.1

5 files

0.1.0

5 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page