lexindex-polars
A Polars expression plugin over lexindex: map a string column to dense ids from an index built once and stored in a file, and map the ids back to their keys.
pip install lexindex-polars
import lexindex as lx
import polars as pl
import lexindex_polars # noqa: F401 -- the import registers the namespace
lx.DictIndex(sorted(set(catalog))).save("tracks.bdx") # once, offline
df.with_columns(
pl.col("track").lexindex.id("tracks.bdx").alias("track_id"),
)
examples/polars_ids.py
runs end to end: a million-row string column to ids and back, with the sizes of both and of the
index file.
The expressions are elementwise, so they run in the engine's own threads, inside a lazy plan and under the streaming engine, and hold no GIL. The blob is read once per process and shared by every chunk and thread.
The four expressions
| expression | takes | gives | on a key the index does not hold |
|---|---|---|---|
.lexindex.id(path) |
a string column | UInt64 |
null |
.lexindex.id_unchecked(path) |
a string column | UInt64 |
some id below the key count |
.lexindex.contains(path) |
a string column | Boolean |
false |
.lexindex.key(path) |
an integer column | String |
null |
null in, null out, for all four. Each also takes pin=; see Reading the blob.
What each index kind answers
An index blob names its own kind, so one expression serves all seven and refuses what a kind cannot
do — with a ComputeError naming the kind and the file, not a wrong answer.
| kind | id |
id_unchecked |
contains |
key |
|---|---|---|---|---|
DictIndex |
exact | — | exact | ✅ |
StringIndex |
exact | — | exact | ✅ |
DoubleArrayIndex |
exact | — | exact | — |
HashedDictIndex |
exact at 0 bits, else 2^-bits |
✅ | same | ✅ |
PerfectHashIndex |
exact | ✅ | exact | ✅ |
CompactHashIndex |
2^-bits |
✅ | 2^-bits |
— |
ClosedHashIndex |
every key gets one | ✅ | — | — |
2^-bits is the rate at which a key the index does not hold is answered as though it did — the
fingerprint width chosen when the index was built. Where the table says —, the expression raises.
Which kind to build is lexindex's own
question: DictIndex when the ids must be the key
order and size matters, HashedDictIndex when they must be the key order and id is the hot path,
CompactHashIndex when only string → id is needed and a rare false positive is acceptable.
lexindex plan keys.txt prices every one of them against your own keys.
Reading the blob
The file named by an expression is read whole on first use and cached for the process. Nothing
stays mapped, because a mapping's contract is that nothing writes the file, and a plugin cannot
promise that about a file any process may rebuild; this crate writes no unsafe.
An expression is pinned to the file it saw when it was built: its size, modification time and,
on unix, device and inode. A chunk gets that generation from the cache without touching the file,
or reads the file while it still is that generation, or raises a ComputeError saying the index was
rebuilt. Ids from two generations are not comparable, so one query never sees both. After a rebuild,
build the expression again. That reads the new file and drops the old generation from the cache, so
a query still running on it raises rather than mixes. pin=False on any expression reads whatever
the file holds when a chunk runs, so a rebuild reaches a running query's later chunks. A file absent
when the expression is built is read that way too, and a plan serialized and run elsewhere carries
its pin, so it needs pin=False if the copy there is another file.
A file may be anyone's: a crafted one is a ComputeError naming it. A StringIndex goes through
lexindex's from_untrusted_bytes for that, one walk over its transducer each time a file is read.
Paths are expanded and made absolute when the expression is built, not when it runs, so a query keeps reading one file even if a symlink is swapped under it.
The threads of a first query wait for one read rather than each making their own. The cache holds
one generation a path, keyed by the same fields, and has no limit of its own. To give it one, set
LEXINDEX_POLARS_CACHE_BYTES: past that many bytes of blobs, counted by their files' sizes, the
least recently used go, and the newest stays whatever its size. Eviction costs a later chunk a read,
and never a wrong answer.
lexindex_polars.cache_info() # {"entries": 1, "bytes": 2636721, "hits": 12, "loads": 1}
lexindex_polars.clear_cache() # what no running chunk holds goes back
Versions
The plugin is compiled against polars' own FFI: the wheel names the polars it was built and tested
against (polars>=1.44), and a much older polars will refuse to load it. lexindex itself is a
normal dependency — it is what writes the blobs this reads.
The wheel is abi3 for CPython 3.11 and up, which is what every polars plugin ships. On a
free-threaded interpreter (cp314t) pip will not take an abi3 wheel, so build it from source
there — pip install 'lexindex-polars @ git+https://github.com/ilgrad/lexindex#subdirectory=polars',
which needs a Rust toolchain.
Licence
MIT, the same as lexindex. Issues and discussion live in the lexindex repository.
Metadata
Release files for lexindex-polars 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl | CPython 3.11 | abi3 | Windows x86-64 | Details for lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl |
| lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl | CPython 3.11 | abi3 | Linux glibc 2.17+ x86-64 | Details for lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
| lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl | CPython 3.11 | abi3 | Linux glibc 2.17+ ARM64 | Details for lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
| lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl | CPython 3.11 | abi3 | macOS 11.0+ ARM64 | Details for lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
| lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl | CPython 3.11 | abi3 | macOS 10.12+ x86-64 | Details for lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl |
Total release size: 19.8 MB
Release files / lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl
| Download URL | lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.11 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
7a965bbd5aa6e6d726164b8e8e10346b88be9112de414e0773704ff808f75259
|
|
BLAKE2b-256 checksum How to use checksums |
af5a54b591fef3025f606f4c534f2e6a769ad0a6513aba6123929ca3f60042aa
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logRelease files / lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | lexindex_polars-0.2.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
a50b719e330222b69890bc16371aa6ffaa10bab3d242e5bfa08936ad4d79549d
|
|
BLAKE2b-256 checksum How to use checksums |
979d48cbf85b5181165432d8827319105e81befb0efb94c0be3cc498e695791d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logRelease files / lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | lexindex_polars-0.2.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 3.9 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
dc491c777d9d6174afd6714aed01239701a04dbf1fddd0c3c2814a92f6ce336c
|
|
BLAKE2b-256 checksum How to use checksums |
6fcc072336569cb2f832993d735fbaacafe6e8ceaa0d6b75c3dc9120a82e7f07
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logRelease files / lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl
| Download URL | lexindex_polars-0.2.0-cp311-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.7 MB |
| Tags | CPython 3.11 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
baa9b61b65848e9b0f0b3aed87bad002a63b695e42471236d7687924b3b5765c
|
|
BLAKE2b-256 checksum How to use checksums |
aa0583f44a4b29ff5e13ce75935f1a8fe473edc1718e7f6acc4bf8f8dd67b5b4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logRelease files / lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl
| Download URL | lexindex_polars-0.2.0-cp311-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 4.0 MB |
| Tags | CPython 3.11 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
060ce21a8b43cf3e18092e4821397eda434c3ba814b5083272cf1bdcd4103bb8
|
|
BLAKE2b-256 checksum How to use checksums |
414dd8d9a5c6e38694cc001e73db7de51329a3135cb15780dcc7485e243f7c7b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logSecurity for lexindex-polars 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 5 files in this release have been uploaded with provenance metadata.
Attestation for lexindex_polars-0.2.0-cp311-abi3-win_amd64.whl
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 6, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.