npids
This package provides time- and space-efficient bi-directional lookups for identifiers. Contents are mmap'd, eliminating most load times and allowing for efficient caching through the file system.
Motivation
It's often helpful to map an external string identifier to an integer index and vice versa. Existing techniques for doing this in Python are either slow or require a lot of memory.
Getting Started
Install via pip:
pip install npids
Prebuilt wheels are available for Linux (x86_64, aarch64, musl x86_64), macOS (x86_64, arm64) and Windows (x86_64), for CPython 3.9 and later. Other platforms build from source, which needs a Rust toolchain.
Build a lookup:
from npids import Lookup
Lookup.build(['id1', 'id2', 'id3'], 'path/to/lookup.npids')
Perform forward lookups (index to ID)
lookup = Lookup('path/to/lookup.npids')
# individual indices (negative indices count from the end)
lookup.fwd[0] # -> 'id1'
lookup.fwd[-1] # -> 'id3'
# multiple indices
lookup.fwd[0,1] # -> ['id1', 'id2']
lookup.fwd[1:] # -> ['id2', 'id3']
# works with numpy too
lookup.fwd[np.array([0,1])] # -> array(['id1', 'id2'], dtype='<U3')
Perform inverse lookups (ID to index)
lookup = Lookup('path/to/lookup.npids')
# individual IDs
lookup.inv['id1'] # -> 0
lookup.inv['id3'] # -> 2
lookup.inv['nope'] # -> KeyError
lookup.inv.get('nope') # -> None
# multiple IDs (-1 for IDs that are not present)
lookup.inv['id1', 'id3', 'nope'] # -> [0, 2, -1]
# works with numpy too
lookup.inv[np.array(['id1', 'id3'])] # -> array([0, 2])
lookup[...] picks a forward or inverse lookup based on the type of its argument. IDs may repeat;
an inverse lookup returns the first index of an ID.
Appending
Building into an existing file appends to it. Nothing is committed until the builder closes, so a crash or an exception leaves the file as it was:
with Lookup.builder('path/to/lookup.npids') as builder:
builder.add('id4')
builder.add_many(['id5', 'id6'])
Appends only pay for the IDs they add. Over many appends, Lookup.compact(path) rewrites the file
without leftover space and merges its inverse indexes.
Readers can keep using a file while it's being appended to; they see the IDs that were present when they opened it. One writer at a time is allowed (enforced with a file lock).
Metadata
You can store a JSON object with the IDs:
Lookup.build(ids, path, meta={'dataset': 'msmarco-passage'})
Lookup(path).meta # -> {'dataset': 'msmarco-passage'}
Files from npids 0.1
npids 1.0 writes a new file format (v2). Lookup(path) still opens files written by npids 0.x (as a
read-only npids.legacy.Lookup), so existing files keep working without changes. They can't be
appended to, and lookups are faster on v2 files. Convert them with npids.upgrade:
import npids
npids.upgrade('path/to/lookup.npids') # in place
npids.upgrade('old.npids', new_path='new.npids') # or to a new file
For large batches, lookup._reader.get_many_raw(idxs) returns the IDs as an Arrow-style (offsets, data)
pair without creating a Python object per ID (e.g. for pyarrow.StringArray.from_buffers).
How it works
IDs are stored in fixed-size chunks (65,536 IDs by default). Each chunk picks a template that all of
its IDs match, such as D{dec}, {hex:8}-{hex:4}-{hex:4}-{hex:4}-{hex:12} or doc-{dec:5}, and
stores only the template's fields, compressed as constants, arithmetic sequences or bit-packed
integers. IDs that don't follow a pattern are stored as plain strings ({str}).
Inverse lookups use the same templates where they can: when IDs are in increasing order (like
0, 1, 2, ... or sorted docnos), finding an ID is a binary search over the forward data, at no extra
storage cost. Otherwise, npids builds a perfect hash (about 3 bits per ID) plus a table from hash slot
to index.
The core is written in Rust. The file format is specified in FORMAT.md.
Benchmarks
bench/compare.py builds each sample in test/samples with npids 0.1 (format v1) and 1.0 (format
v2), and times 10k random lookups with a warm cache (Apple M3).
| sample | IDs | version | build | size | fwd 10k | inv 10k |
|---|---|---|---|---|---|---|
msmarco-passage (0, 1, ...) |
8.8M | 0.1 | 3.45s | 206B | 3.1ms | 4.9ms |
| 1.0 | 1.37s | 9.8KB | 0.8ms | 1.4ms | ||
msmarco-document (D1555982, ...) |
3.2M | 0.1 | 3.65s | 40.5MB | 4.1ms | 7.8ms |
| 1.0 | 1.28s | 18.1MB | 1.0ms | 1.4ms | ||
| neuclir/1/fa (UUIDs) | 2.2M | 0.1 | 4.72s | 58.6MB | 10.5ms | 17.7ms |
| 1.0 | 1.23s | 39.2MB | 1.6ms | 4.6ms | ||
| beir/fever (Wikipedia titles) | 5.4M | 0.1 | 10.15s | 1.1GB | 4.3ms | 12.1ms |
| 1.0 | 2.58s | 137.1MB | 1.5ms | 2.2ms | ||
| beir/dbpedia-entity | 4.6M | 0.1 | 8.03s | 833.1MB | 4.9ms | 14.3ms |
| 1.0 | 2.34s | 120.6MB | 1.6ms | 3.1ms | ||
| disks45/nocr | 528k | 0.1 | 0.39s | 10.7MB | 6.2ms | 11.2ms |
| 1.0 | 0.21s | 5.3MB | 1.2ms | 1.7ms | ||
| cord19 | 193k | 0.1 | 0.08s | 3.2MB | 2.0ms | 4.0ms |
| 1.0 | 0.08s | 2.7MB | 0.8ms | 1.1ms |
Development
pip install maturin
maturin develop --release
cargo test --release
pytest test/
Metadata
Release files for npids 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| npids-1.0.0.tar.gz | 65.2 kB | With provenance | Details for npids-1.0.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| npids-1.0.0-cp39-abi3-win_amd64.whl | CPython 3.9 | abi3 | Windows x86-64 | Details for npids-1.0.0-cp39-abi3-win_amd64.whl |
| npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl | CPython 3.9 | abi3 | Linux musl 1.2+ x86-64 | Details for npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl |
| npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl | CPython 3.9 | abi3 | Linux glibc 2.17+ x86-64 | Details for npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
| npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl | CPython 3.9 | abi3 | Linux glibc 2.17+ ARM64 | Details for npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
| npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl | CPython 3.9 | abi3 | macOS 11.0+ ARM64 | Details for npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl |
| npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl | CPython 3.9 | abi3 | macOS 10.12+ x86-64 | Details for npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl |
Total release size: 2.6 MB
Release files / npids-1.0.0.tar.gz
| Download URL | npids-1.0.0.tar.gz |
|---|---|
| Size | 65.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
693284365642722f6a572c04d41cae258f936d80fa439d61690b86dbfeb75d64
|
|
BLAKE2b-256 checksum How to use checksums |
967b0625c5f7391ed9c374efe0cf14e32261955d3a369b1fb3c62dcb556dbdb5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-win_amd64.whl
| Download URL | npids-1.0.0-cp39-abi3-win_amd64.whl |
|---|---|
| Size | 302.8 kB |
| Tags | CPython 3.9 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
900e9a70c853f7de40789f2847da4d693c0c1e397a2c9ab56fa102305e6a8730
|
|
BLAKE2b-256 checksum How to use checksums |
9ee6a82ade37a09359636a50299ebe7e10ccfac7db6ee79a28fec7779b5f1c77
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl
| Download URL | npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 633.0 kB |
| Tags | CPython 3.9 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
ec14fdef7e43b385462161668f843a5dc3c63f91f4198b518f4be6aa29c824ff
|
|
BLAKE2b-256 checksum How to use checksums |
d5de149526f86154262e247e6c848a19d4aacc560a1b6b06b2d76100c29b7f15
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 420.6 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
0be2289a27da829b814e43db6b7416167c7a13ad4a54ce3cf170591fe5a216d0
|
|
BLAKE2b-256 checksum How to use checksums |
5424e4029800163736ef120133856122b6376c51bd5844c442339f6c9a462f4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 417.9 kB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
66ab811567c3ef4b7317996fbe30693df5d6615992d1b1b46b77cf938d7622e5
|
|
BLAKE2b-256 checksum How to use checksums |
b38ba95196ff017b8913c758a21169e1e43eeb661a8c11f166cc3f955791e486
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl
| Download URL | npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 377.8 kB |
| Tags | CPython 3.9 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
08c97806385b8e329ec3b539c6f69fa9864a6896bc3bf710f5b55537e067f0f9
|
|
BLAKE2b-256 checksum How to use checksums |
11f9d06ad5a7ef50e853c53409c7b9792d7d36665b3854a0f2539b5fdae412dc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logRelease files / npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl
| Download URL | npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 393.3 kB |
| Tags | CPython 3.9 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
573aacf4360abc5482d21862cf8fb51e83297ebf1cdbe36a0ca5177cd2bad6f4
|
|
BLAKE2b-256 checksum How to use checksums |
2bfb2a2bff59b988fa316cbf153436f219b2329eddf13400d25d4bf4a9f8d5f5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logSecurity for npids 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 7 files in this release have been uploaded with provenance metadata.
Attestation for npids-1.0.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.