Skip to main content

v11-tokenizer — Python bindings

Pure-Rust knowledge-first tokenizer, exposed as a Python module via PyO3.

Install

pip install v11-tokenizer

The distribution is v11-tokenizer; the import is v11. Wheels are built against PyO3's stable ABI, so one wheel per platform covers every Python >= 3.9 — Linux (x86_64/aarch64), macOS (Intel/Apple silicon) and Windows x64. No Rust toolchain needed. (Use 0.1.1 or newer: 0.1.0 shipped only a macOS arm64 / cp312 wheel, so everywhere else pip fell back to building the sdist.)

For development, from source:

cd v-tokenizers/v11/python
maturin develop --release

Then from any Python script in the same venv:

import v11

tok = v11.Tokenizer.from_file("../v11/artifacts/v11.vocab.bin")
print(tok)                                         # <v11.Tokenizer vocab_size=71260>
print(tok.vocab_size)                              # 71260
print(tok.pad_id, tok.unk_id, tok.bos_id, tok.eos_id)  # 0 1 2 3

ids = tok.encode("def fibonacci(n):")
pieces = tok.encode_pieces("def fibonacci(n):")
text = tok.decode(ids)

API

method returns notes
Tokenizer.from_file(path) Tokenizer Load from a .vocab.bin file
tok.encode(text) list[int] Tokenize to ids
tok.encode_pieces(text) list[str] Tokenize to piece strings
tok.decode(ids) str Reverse to text
tok.id_to_piece(id) str | None Look up a single id
tok.piece_to_id(piece) int | None Look up a single piece
tok.vocab_size int Total pieces
tok.pad_id, unk_id, bos_id, eos_id int Special token ids
len(tok) int Same as vocab_size

Performance

Encode throughput on a single thread (release build):

~9.3M tokens/sec
~28 MB/sec

That's roughly 10-40× faster than the Python sentencepiece package on comparable inputs, because the longest-match is a single Aho-Corasick pass with no C++ FFI hops.

Build a wheel for distribution

maturin build --release
ls target/wheels/
# v11_tokenizer-0.1.2-cp39-abi3-macosx_11_0_arm64.whl

The cp39-abi3 tag is the stable-ABI build: that one file installs on any CPython >= 3.9, whatever interpreter built it. Releases produce one per platform via the matrix in .github/workflows/publish.yml.

License

Apache 2.0

Metadata

Release files for v11-tokenizer 0.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for v11-tokenizer 0.1.2
File Size Uploaded Provenance Actions
v11_tokenizer-0.1.2.tar.gz 20.4 kB Without provenance Details for v11_tokenizer-0.1.2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for v11-tokenizer 0.1.2
File Actions
v11_tokenizer-0.1.2-cp39-abi3-win_amd64.whl CPython 3.9 abi3 Windows x86-64 Details for v11_tokenizer-0.1.2-cp39-abi3-win_amd64.whl
v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl CPython 3.9 abi3 Linux glibc 2.17+ x86-64 Details for v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl CPython 3.9 abi3 Linux glibc 2.17+ ARM64 Details for v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
v11_tokenizer-0.1.2-cp39-abi3-macosx_11_0_arm64.whl CPython 3.9 abi3 macOS 11.0+ ARM64 Details for v11_tokenizer-0.1.2-cp39-abi3-macosx_11_0_arm64.whl
v11_tokenizer-0.1.2-cp39-abi3-macosx_10_12_x86_64.whl CPython 3.9 abi3 macOS 10.12+ x86-64 Details for v11_tokenizer-0.1.2-cp39-abi3-macosx_10_12_x86_64.whl

Total release size: 1.1 MB

Release files / v11_tokenizer-0.1.2.tar.gz

Download URL v11_tokenizer-0.1.2.tar.gz
Size 20.4 kB
Tags Source
SHA-256 checksum
How to use checksums
4e4eed1a93fa50cdf9610d040e34d4dcd406874fe9c70154168622f926a1518e
BLAKE2b-256 checksum
How to use checksums
a462d2aa3959ae9fe97cbd08ab299ef13859c41461370ccad4e6b3180f387a5e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Release files / v11_tokenizer-0.1.2-cp39-abi3-win_amd64.whl

Download URL v11_tokenizer-0.1.2-cp39-abi3-win_amd64.whl
Size 149.0 kB
Tags CPython 3.9 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
1bcbd864755b4ca6a5677eb300ad4aa5510b51e1085a0c9c70887338b5f6fb8e
BLAKE2b-256 checksum
How to use checksums
e192f0a0052cb290929e106922fcdbe9c8935aca9d413cdadc1ca9f569884026
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Release files / v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Download URL v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Size 248.6 kB
Tags CPython 3.9 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
24921f244179430db7a67e66cbbeb4bc4ab18ef6af044216f08583f135353f0b
BLAKE2b-256 checksum
How to use checksums
b172e748c24eff8568712ee69824372d9f16fa0114a5fb218de18a384289235d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Release files / v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl

Download URL v11_tokenizer-0.1.2-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Size 236.5 kB
Tags CPython 3.9 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
30f3e00c57b74c1181365e37098c06e6eba55f44fa694b8f7db5e086c5842a8e
BLAKE2b-256 checksum
How to use checksums
3b1f5a9983d2dcdc0698c1486cf468b33c6633922f36bfb8009b83033496514d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Release files / v11_tokenizer-0.1.2-cp39-abi3-macosx_11_0_arm64.whl

Download URL v11_tokenizer-0.1.2-cp39-abi3-macosx_11_0_arm64.whl
Size 216.7 kB
Tags CPython 3.9 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
4c3f37556a3465153a825d762f69e2a477e657c1d83fcf406d8e1e46badafbd9
BLAKE2b-256 checksum
How to use checksums
12d8b8656c40cd1b8f1f31e66b0b7665c55e3c1282f7168856e802f8e0865839
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Release files / v11_tokenizer-0.1.2-cp39-abi3-macosx_10_12_x86_64.whl

Download URL v11_tokenizer-0.1.2-cp39-abi3-macosx_10_12_x86_64.whl
Size 220.4 kB
Tags CPython 3.9 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
35c455ce0449f48e575b289c4904595d555a5daeb0f6c6682dfb57db4ffac541
BLAKE2b-256 checksum
How to use checksums
2a563406d386c1d716b3c1e09352e45c6e29838cbca7073bd49eb3be5201ecd8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via maturin/1.14.1

Security for v11-tokenizer 0.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page