Skip to main content

npids

This package provides time- and space-efficient bi-directional lookups for identifiers. Contents are mmap'd, eliminating most load times and allowing for efficient caching through the file system.

Motivation

It's often helpful to map an external string identifier to an integer index and vice versa. Existing techniques for doing this in Python are either slow or require a lot of memory.

Getting Started

Install via pip:

pip install npids

Prebuilt wheels are available for Linux (x86_64, aarch64, musl x86_64), macOS (x86_64, arm64) and Windows (x86_64), for CPython 3.9 and later. Other platforms build from source, which needs a Rust toolchain.

Build a lookup:

from npids import Lookup
Lookup.build(['id1', 'id2', 'id3'], 'path/to/lookup.npids')

Perform forward lookups (index to ID)

lookup = Lookup('path/to/lookup.npids')
# individual indices (negative indices count from the end)
lookup.fwd[0] # -> 'id1'
lookup.fwd[-1] # -> 'id3'
# multiple indices
lookup.fwd[0,1] # -> ['id1', 'id2']
lookup.fwd[1:] # -> ['id2', 'id3']
# works with numpy too
lookup.fwd[np.array([0,1])] # -> array(['id1', 'id2'], dtype='<U3')

Perform inverse lookups (ID to index)

lookup = Lookup('path/to/lookup.npids')
# individual IDs
lookup.inv['id1'] # -> 0
lookup.inv['id3'] # -> 2
lookup.inv['nope'] # -> KeyError
lookup.inv.get('nope') # -> None
# multiple IDs (-1 for IDs that are not present)
lookup.inv['id1', 'id3', 'nope'] # -> [0, 2, -1]
# works with numpy too
lookup.inv[np.array(['id1', 'id3'])] # -> array([0, 2])

lookup[...] picks a forward or inverse lookup based on the type of its argument. IDs may repeat; an inverse lookup returns the first index of an ID.

Appending

Building into an existing file appends to it. Nothing is committed until the builder closes, so a crash or an exception leaves the file as it was:

with Lookup.builder('path/to/lookup.npids') as builder:
    builder.add('id4')
    builder.add_many(['id5', 'id6'])

Appends only pay for the IDs they add. Over many appends, Lookup.compact(path) rewrites the file without leftover space and merges its inverse indexes.

Readers can keep using a file while it's being appended to; they see the IDs that were present when they opened it. One writer at a time is allowed (enforced with a file lock).

Metadata

You can store a JSON object with the IDs:

Lookup.build(ids, path, meta={'dataset': 'msmarco-passage'})
Lookup(path).meta # -> {'dataset': 'msmarco-passage'}

Files from npids 0.1

npids 1.0 writes a new file format (v2). Lookup(path) still opens files written by npids 0.x (as a read-only npids.legacy.Lookup), so existing files keep working without changes. They can't be appended to, and lookups are faster on v2 files. Convert them with npids.upgrade:

import npids
npids.upgrade('path/to/lookup.npids')                   # in place
npids.upgrade('old.npids', new_path='new.npids')         # or to a new file

For large batches, lookup._reader.get_many_raw(idxs) returns the IDs as an Arrow-style (offsets, data) pair without creating a Python object per ID (e.g. for pyarrow.StringArray.from_buffers).

How it works

IDs are stored in fixed-size chunks (65,536 IDs by default). Each chunk picks a template that all of its IDs match, such as D{dec}, {hex:8}-{hex:4}-{hex:4}-{hex:4}-{hex:12} or doc-{dec:5}, and stores only the template's fields, compressed as constants, arithmetic sequences or bit-packed integers. IDs that don't follow a pattern are stored as plain strings ({str}).

Inverse lookups use the same templates where they can: when IDs are in increasing order (like 0, 1, 2, ... or sorted docnos), finding an ID is a binary search over the forward data, at no extra storage cost. Otherwise, npids builds a perfect hash (about 3 bits per ID) plus a table from hash slot to index.

The core is written in Rust. The file format is specified in FORMAT.md.

Benchmarks

bench/compare.py builds each sample in test/samples with npids 0.1 (format v1) and 1.0 (format v2), and times 10k random lookups with a warm cache (Apple M3).

sample IDs version build size fwd 10k inv 10k
msmarco-passage (0, 1, ...) 8.8M 0.1 3.45s 206B 3.1ms 4.9ms
1.0 1.37s 9.8KB 0.8ms 1.4ms
msmarco-document (D1555982, ...) 3.2M 0.1 3.65s 40.5MB 4.1ms 7.8ms
1.0 1.28s 18.1MB 1.0ms 1.4ms
neuclir/1/fa (UUIDs) 2.2M 0.1 4.72s 58.6MB 10.5ms 17.7ms
1.0 1.23s 39.2MB 1.6ms 4.6ms
beir/fever (Wikipedia titles) 5.4M 0.1 10.15s 1.1GB 4.3ms 12.1ms
1.0 2.58s 137.1MB 1.5ms 2.2ms
beir/dbpedia-entity 4.6M 0.1 8.03s 833.1MB 4.9ms 14.3ms
1.0 2.34s 120.6MB 1.6ms 3.1ms
disks45/nocr 528k 0.1 0.39s 10.7MB 6.2ms 11.2ms
1.0 0.21s 5.3MB 1.2ms 1.7ms
cord19 193k 0.1 0.08s 3.2MB 2.0ms 4.0ms
1.0 0.08s 2.7MB 0.8ms 1.1ms

Development

pip install maturin
maturin develop --release
cargo test --release
pytest test/

Metadata

Release files for npids 1.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for npids 1.0.0
File Size Uploaded Provenance Actions
npids-1.0.0.tar.gz 65.2 kB With provenance Details for npids-1.0.0.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for npids 1.0.0
File Actions
npids-1.0.0-cp39-abi3-win_amd64.whl CPython 3.9 abi3 Windows x86-64 Details for npids-1.0.0-cp39-abi3-win_amd64.whl
npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl CPython 3.9 abi3 Linux musl 1.2+ x86-64 Details for npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl
npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.9 abi3 Linux glibc 2.17+ x86-64 Details for npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.9 abi3 Linux glibc 2.17+ ARM64 Details for npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl CPython 3.9 abi3 macOS 11.0+ ARM64 Details for npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl
npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl CPython 3.9 abi3 macOS 10.12+ x86-64 Details for npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl

Total release size: 2.6 MB

Release files / npids-1.0.0.tar.gz

Download URL npids-1.0.0.tar.gz
Size 65.2 kB
Tags Source
SHA-256 checksum
How to use checksums
693284365642722f6a572c04d41cae258f936d80fa439d61690b86dbfeb75d64
BLAKE2b-256 checksum
How to use checksums
967b0625c5f7391ed9c374efe0cf14e32261955d3a369b1fb3c62dcb556dbdb5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-win_amd64.whl

Download URL npids-1.0.0-cp39-abi3-win_amd64.whl
Size 302.8 kB
Tags CPython 3.9 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
900e9a70c853f7de40789f2847da4d693c0c1e397a2c9ab56fa102305e6a8730
BLAKE2b-256 checksum
How to use checksums
9ee6a82ade37a09359636a50299ebe7e10ccfac7db6ee79a28fec7779b5f1c77
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl

Download URL npids-1.0.0-cp39-abi3-musllinux_1_2_x86_64.whl
Size 633.0 kB
Tags CPython 3.9 Linux musl 1.2+ x86-64 abi3
SHA-256 checksum
How to use checksums
ec14fdef7e43b385462161668f843a5dc3c63f91f4198b518f4be6aa29c824ff
BLAKE2b-256 checksum
How to use checksums
d5de149526f86154262e247e6c848a19d4aacc560a1b6b06b2d76100c29b7f15
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL npids-1.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 420.6 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
0be2289a27da829b814e43db6b7416167c7a13ad4a54ce3cf170591fe5a216d0
BLAKE2b-256 checksum
How to use checksums
5424e4029800163736ef120133856122b6376c51bd5844c442339f6c9a462f4e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL npids-1.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 417.9 kB
Tags CPython 3.9 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
66ab811567c3ef4b7317996fbe30693df5d6615992d1b1b46b77cf938d7622e5
BLAKE2b-256 checksum
How to use checksums
b38ba95196ff017b8913c758a21169e1e43eeb661a8c11f166cc3f955791e486
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl

Download URL npids-1.0.0-cp39-abi3-macosx_11_0_arm64.whl
Size 377.8 kB
Tags CPython 3.9 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
08c97806385b8e329ec3b539c6f69fa9864a6896bc3bf710f5b55537e067f0f9
BLAKE2b-256 checksum
How to use checksums
11f9d06ad5a7ef50e853c53409c7b9792d7d36665b3854a0f2539b5fdae412dc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Release files / npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl

Download URL npids-1.0.0-cp39-abi3-macosx_10_12_x86_64.whl
Size 393.3 kB
Tags CPython 3.9 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
573aacf4360abc5482d21862cf8fb51e83297ebf1cdbe36a0ca5177cd2bad6f4
BLAKE2b-256 checksum
How to use checksums
2bfb2a2bff59b988fa316cbf153436f219b2329eddf13400d25d4bf4a9f8d5f5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

Security for npids 1.0.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 7 files in this release have been uploaded with provenance metadata.

Select from 7 files

Attestation for npids-1.0.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 17, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page