Skip to main content

lexindex-polars

A Polars expression plugin over lexindex: map a string column to dense ids from an index built once and stored in a file, and map the ids back to their keys.

pip install lexindex-polars
import lexindex as lx
import polars as pl
import lexindex_polars  # noqa: F401  -- the import registers the namespace

lx.DictIndex(sorted(set(catalog))).save("tracks.bdx")  # once, offline

df.with_columns(
    pl.col("track").lexindex.id("tracks.bdx").alias("track_id"),
)

examples/polars_ids.py runs end to end: a million-row string column to ids and back, with the sizes of both and of the index file.

The expressions are elementwise, so they run in the engine's own threads, inside a lazy plan and under the streaming engine, and hold no GIL. The blob is read once per process and shared by every chunk and thread.

The four expressions

expression takes gives on a key the index does not hold
.lexindex.id(path) a string column UInt64 null
.lexindex.id_unchecked(path) a string column UInt64 some id below the key count
.lexindex.contains(path) a string column Boolean false
.lexindex.key(path) an integer column String null

null in, null out, for all four. Each also takes pin=; see Reading the blob.

What each index kind answers

An index blob names its own kind, so one expression serves all seven and refuses what a kind cannot do — with a ComputeError naming the kind and the file, not a wrong answer.

kind id id_unchecked contains key
DictIndex exact — exact ✅
StringIndex exact — exact ✅
DoubleArrayIndex exact — exact —
HashedDictIndex exact at 0 bits, else 2^-bits ✅ same ✅
PerfectHashIndex exact ✅ exact ✅
CompactHashIndex 2^-bits ✅ 2^-bits —
ClosedHashIndex every key gets one ✅ — —

2^-bits is the rate at which a key the index does not hold is answered as though it did — the fingerprint width chosen when the index was built. Where the table says —, the expression raises.

Which kind to build is lexindex's own question: DictIndex when the ids must be the key order and size matters, HashedDictIndex when they must be the key order and id is the hot path, CompactHashIndex when only string → id is needed and a rare false positive is acceptable. lexindex plan keys.txt prices every one of them against your own keys.

Reading the blob

The file named by an expression is read whole on first use and cached for the process. Nothing stays mapped, because a mapping's contract is that nothing writes the file, and a plugin cannot promise that about a file any process may rebuild; this crate writes no unsafe.

An expression is pinned to the file it saw when it was built: its size, modification time and, on unix, device and inode. A chunk gets that generation from the cache without touching the file, or reads the file while it still is that generation, or raises a ComputeError saying the index was rebuilt. Ids from two generations are not comparable, so one query never sees both. After a rebuild, build the expression again. That reads the new file and drops the old generation from the cache, so a query still running on it raises rather than mixes. pin=False on any expression reads whatever the file holds when a chunk runs, so a rebuild reaches a running query's later chunks. A file absent when the expression is built is read that way too, and a plan serialized and run elsewhere carries its pin, so it needs pin=False if the copy there is another file.

A file may be anyone's: a crafted one is a ComputeError naming it. A StringIndex goes through lexindex's from_untrusted_bytes for that, one walk over its transducer each time a file is read.

Paths are expanded and made absolute when the expression is built, not when it runs, so a query keeps reading one file even if a symlink is swapped under it.

The threads of a first query wait for one read rather than each making their own. The cache holds one generation a path, keyed by the same fields, and has no limit of its own. To give it one, set LEXINDEX_POLARS_CACHE_BYTES: past that many bytes of blobs, counted by their files' sizes, the least recently used go, and the newest stays whatever its size. Eviction costs a later chunk a read, and never a wrong answer.

lexindex_polars.cache_info()  # {"entries": 1, "bytes": 2636721, "hits": 12, "loads": 1}
lexindex_polars.clear_cache()  # what no running chunk holds goes back

Versions

The plugin is compiled against polars' own FFI: the wheel names the polars it was built and tested against (polars>=1.44), and a much older polars will refuse to load it. lexindex itself is a normal dependency — it is what writes the blobs this reads.

The wheel is abi3 for CPython 3.11 and up, which is what every polars plugin ships. On a free-threaded interpreter (cp314t) pip will not take an abi3 wheel, so build it from source there — pip install 'lexindex-polars @ git+https://github.com/ilgrad/lexindex#subdirectory=polars', which needs a Rust toolchain.

Licence

MIT, the same as lexindex. Issues and discussion live in the lexindex repository.

Metadata

Release files for lexindex-polars 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for lexindex-polars 0.2.0
File Actions
lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl CPython 3.11 abi3 Windows x86-64 Details for lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl
lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 abi3 Linux glibc 2.17+ x86-64 Details for lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.11 abi3 Linux glibc 2.17+ ARM64 Details for lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl CPython 3.11 abi3 macOS 11.0+ ARM64 Details for lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl CPython 3.11 abi3 macOS 10.12+ x86-64 Details for lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl

Total release size: 19.8 MB

Release files / lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl

Download URL lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl
Size 4.2 MB
Tags CPython 3.11 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
7a965bbd5aa6e6d726164b8e8e10346b88be9112de414e0773704ff808f75259
BLAKE2b-256 checksum
How to use checksums
af5a54b591fef3025f606f4c534f2e6a769ad0a6513aba6123929ca3f60042aa
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 4.1 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
a50b719e330222b69890bc16371aa6ffaa10bab3d242e5bfa08936ad4d79549d
BLAKE2b-256 checksum
How to use checksums
979d48cbf85b5181165432d8827319105e81befb0efb94c0be3cc498e695791d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 3.9 MB
Tags CPython 3.11 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
dc491c777d9d6174afd6714aed01239701a04dbf1fddd0c3c2814a92f6ce336c
BLAKE2b-256 checksum
How to use checksums
6fcc072336569cb2f832993d735fbaacafe6e8ceaa0d6b75c3dc9120a82e7f07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl

Download URL lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
Size 3.7 MB
Tags CPython 3.11 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
baa9b61b65848e9b0f0b3aed87bad002a63b695e42471236d7687924b3b5765c
BLAKE2b-256 checksum
How to use checksums
aa0583f44a4b29ff5e13ce75935f1a8fe473edc1718e7f6acc4bf8f8dd67b5b4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Release files / lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl

Download URL lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl
Size 4.0 MB
Tags CPython 3.11 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
060ce21a8b43cf3e18092e4821397eda434c3ba814b5083272cf1bdcd4103bb8
BLAKE2b-256 checksum
How to use checksums
414dd8d9a5c6e38694cc001e73db7de51329a3135cb15780dcc7485e243f7c7b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

Security for lexindex-polars 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 5 files in this release have been uploaded with provenance metadata.

Select from 5 files

Attestation for lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page