Tokenizer-Changer

Python script for manipulating the existing tokenizer.

The solution was tested on Llama3-8B tokenizer.

Installation

Installation from PyPI:

pip install tokenizerchanger

Requirements

Python 3.9+
tokenizers>=0.21.0
transformers>=4.47.0
tqdm>=4.66.4

Docs

https://tokenizer-changer.readthedocs.io/en/latest/

Metadata

Release files for tokenizerchanger 1.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tokenizerchanger 1.1.1
File	Size	Uploaded	Provenance	Actions
tokenizerchanger-1.1.1.tar.gz	13.3 kB	Feb 12, 2026	Without provenance	Details for tokenizerchanger-1.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tokenizerchanger 1.1.1
File	Interpreter	ABI	Platform	Actions Reset
tokenizerchanger-1.1.1-py3-none-any.whl 13.3 kB Feb 12, 2026 Without provenance	Python 3	none	any	Details for tokenizerchanger-1.1.1-py3-none-any.whl

Total release size: 26.5 kB

Release files / tokenizerchanger-1.1.1.tar.gz

Download URL	tokenizerchanger-1.1.1.tar.gz
Size	13.3 kB
Tags	Source
SHA-256 checksum How to use checksums	`21e7bffb2d60352f862c1513f3f0bea2d34dca4ec7babda95f06668b7163b70b`
BLAKE2b-256 checksum How to use checksums	`d66fca6597c712ab308e42b905e14e10fdf469909165198dced0151d34d24a2d`
Upload date	Feb 12, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.11`

Release files / tokenizerchanger-1.1.1-py3-none-any.whl

Download URL	tokenizerchanger-1.1.1-py3-none-any.whl
Size	13.3 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`68c2d7551e563c8cb143749f2731465ab8735ca574825571f7cc996fbc8cf41a`
BLAKE2b-256 checksum How to use checksums	`11598368f3acae5036273374b623c0d1aabc1339f5d311914ec5fd0018c630f0`
Upload date	Feb 12, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/5.1.1 CPython/3.10.11`

Security for tokenizerchanger 1.1.1

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.