toktier-fastokens

A pinned build of fastokens 0.3.1 with five patches maintained by the toktier project. It is not affiliated with or endorsed by the upstream project.

It provides the same import name as upstream:

import fastokens

so it is a drop-in for code written against fastokens 0.3.1.

Install only one of the two

Because both distributions install a top-level fastokens package, installing this one and upstream fastokens into the same environment leaves whichever was installed last in place, silently. Pick one:

pip uninstall -y fastokens
pip install toktier-fastokens

or, if you want upstream instead:

pip uninstall -y toktier-fastokens
pip install fastokens

What the patches change

Five patches close cases where the engine returned a token-id list that differs from the reference tokenizer for the same input, with no error raised. CHANGES-toktier.md lists them one by one, NOTICE states which files each one touches, and PATCHES/ carries the patches verbatim.

Applying the five patches in order to upstream commit fe854299553524f2156a22036a2cb4d1f2ef4d97 (tag v0.3.1) reproduces the exact source tree this build comes from: git tree hash aa1924284ec4abaedcc8ed5823ee17e7959c55c5.

Rebuilding

build_pinned.sh in the toktier repository rebuilds this distribution from a clean upstream checkout with a pinned Rust toolchain and a pinned maturin. It checks the patch digests and the resulting tree hash before it builds.

Licensing

Upstream fastokens is Apache-2.0; the upstream LICENSE text is included verbatim, as is the upstream NOTICES.txt (reproduced inside NOTICE, which also carries the statement of modification required by section 4(b)).

Scope of the guarantee

The readings that motivated this build were taken on the source tree named above, with tokenizers==0.22.2 as the reference. They describe that tree, not every possible build of it: a wheel built elsewhere is a different artifact with a different content digest. toktier recognises the builds it publishes by content digest and says so explicitly; see the toktier documentation for what each state reports.