BPE Tokenizer Editor

A high-performance Python library for editing HuggingFace BPE tokenizer.json files with consistency guarantees. Built with Rust for maximum performance.

Features

✅ Validate merges - Check that all merge results exist in vocabulary
✅ Add tokens - Add new tokens with automatic merge chain creation
✅ Remove tokens - Remove tokens with cascade deletion of dependent merges
✅ Shrink vocab - Remove N longest tokens to reduce vocabulary size
✅ Sync single-chars - Copy all single-character tokens from source tokenizer
✅ Keep vocab size fixed - Add tokens while automatically removing others to maintain size
✅ Reindex vocab - Make vocabulary IDs sequential by removing gaps
🚀 Rust-powered - Native performance with Python convenience

Installation

pip install bpe-tokenizer-editor

Requirements

Python 3.8+
No additional dependencies required (Rust code is compiled into the wheel)

Quick Start

from bpe_tokenizer_editor import BPETokenizerEditor

# Load a tokenizer
editor = BPETokenizerEditor("tokenizer.json")

# Check stats
print(f"Vocabulary size: {editor.vocab_size}")
print(f"Number of merges: {editor.merges_count}")

# Validate merges
validation = editor.validate_merges()
print(f"Valid merges: {validation.valid_count}")
print(f"Invalid merges: {validation.invalid_count}")

# Add a new token
result = editor.add_token("merhaba")
print(f"Added: {result.added}, Method: {result.method}")

# Save the modified tokenizer
editor.save("tokenizer_modified.json")

API Reference

Loading and Saving

# Load from file
editor = BPETokenizerEditor("tokenizer.json")

# Load from JSON string
editor = BPETokenizerEditor.from_json(json_string)

# Save to file
editor.save("output.json")

# Export to JSON string
json_str = editor.to_json()

Properties

editor.vocab_size    # Current vocabulary size
editor.merges_count  # Current number of merge rules

Token Operations

# Check if token exists
editor.has_token("hello")  # Returns bool

# Get token ID
editor.get_token_id("hello")  # Returns int or None

# Get token by ID
editor.get_token_by_id(1000)  # Returns str or None

# Get all tokens as dict
vocab = editor.get_vocab()  # Returns Dict[str, int]

# Get all merges
merges = editor.get_merges()  # Returns List[Tuple[str, str]]

# Get single-character tokens
single_chars = editor.get_single_char_tokens()  # Returns List[Tuple[str, int]]

Statistics

stats = editor.get_stats()
print(f"Vocab size: {stats.vocab_size}")
print(f"Merges: {stats.merges_count}")
print(f"Single chars: {stats.single_char_count}")
print(f"Special tokens: {stats.special_token_count}")
print(f"ID range: {stats.min_token_id} - {stats.max_token_id}")
print(f"Length distribution: {stats.length_distribution}")

Validation

# Validate merges (check if merge results exist in vocab)
result = editor.validate_merges()
print(f"Valid: {result.valid_count}, Invalid: {result.invalid_count}")

for idx, token_a, token_b in result.invalid_merges:
    print(f"  Invalid merge at {idx}: '{token_a}' + '{token_b}'")

# Fix invalid merges (remove them)
removed_count = editor.remove_invalid_merges()
print(f"Removed {removed_count} invalid merges")

Adding Tokens

# Add a single token with automatic merge chain creation
result = editor.add_token("newtoken")
print(f"Added: {result.added}")
print(f"Method: {result.method}")  # 'single_char', 'longest_prefix', or 'char_chain'
print(f"Added merges: {result.added_merges}")

# Add multiple tokens
results = editor.add_tokens(["token1", "token2", "token3"])

# Add special token without merge chain
editor.add_token_atomic("<special>")

# Add tokens while keeping vocabulary size fixed
result = editor.add_tokens_keep_size(
    tokens=["new1", "new2", "new3"],
    whitelist=["protected_token"]  # Optional: tokens that should never be removed
)
print(f"Added: {result['tokens_added']}, Removed: {result['tokens_removed']}")

Removing Tokens

# Remove a token and all its dependencies (cascade removal)
result = editor.remove_token("unwanted")
print(f"Root token: {result.root_token}")
print(f"Removed tokens: {result.removed_tokens}")
print(f"Removed merges: {result.removed_merges}")

# Remove multiple tokens
results = editor.remove_tokens(["token1", "token2"])

Shrinking Vocabulary

# Preview what would be removed
tokens_to_remove = editor.find_tokens_to_shrink(count=1000, min_id=50000)
for token, id, length in tokens_to_remove[:10]:
    print(f"  {token} (ID: {id}, length: {length})")

# Actually shrink the vocabulary
result = editor.shrink(count=1000, min_id=50000)
print(f"Initial size: {result.initial_vocab_size}")
print(f"Final size: {result.final_vocab_size}")
print(f"Tokens removed: {result.total_tokens_removed}")
print(f"Merges removed: {result.total_merges_removed}")

Syncing with Another Tokenizer

# Sync single-character tokens from source tokenizer
# (useful for preserving Unicode coverage)
result = editor.sync_single_chars(
    source_path="original_tokenizer.json",
    min_id=50000  # Only remove tokens with ID >= 50000
)
print(f"Chars added: {result['chars_added_count']}")
print(f"Tokens removed: {result['total_tokens_removed']}")

Reindexing Vocabulary

After sync operations or modifications, vocabulary IDs may have gaps (e.g., IDs jump from 73000 to 74000). This creates a sparse vocabulary. Use reindex to make all IDs sequential:

# Check for gaps in vocabulary IDs
has_gaps, total_gaps, min_id, max_id = editor.check_vocab_gaps()
if has_gaps:
    print(f"Found {total_gaps} gaps in ID space (range: {min_id}-{max_id})")

# Reindex to make IDs sequential
result = editor.reindex_vocab()
print(f"Remapped {result.ids_remapped} IDs")
print(f"Removed {result.gaps_removed} gaps")
print(f"New ID range: {result.new_min_id} - {result.new_max_id}")

Note: The sync_single_chars method automatically calls reindex_vocab after completing, so you typically don't need to call it manually after syncing.


## Examples

### Training a Domain-Specific Tokenizer

```python
from bpe_tokenizer_editor import BPETokenizerEditor

# After training your custom tokenizer, validate and fix it
editor = BPETokenizerEditor("my_custom_tokenizer.json")

# Check for problems
validation = editor.validate_merges()
if validation.invalid_count > 0:
    print(f"Found {validation.invalid_count} invalid merges, fixing...")
    editor.remove_invalid_merges()

# Sync Unicode characters from original tokenizer
result = editor.sync_single_chars(
    source_path="original_tokenizer.json",
    min_id=50000
)
print(f"Added {result['chars_added_count']} missing characters")

# Add domain-specific tokens
domain_tokens = ["merhaba", "dünya", "Türkiye"]
for token in domain_tokens:
    result = editor.add_token(token)
    if result.added:
        print(f"Added '{token}' using {result.method} method")

# Save the result
editor.save("my_final_tokenizer.json")

Reducing Vocabulary Size

from bpe_tokenizer_editor import BPETokenizerEditor

editor = BPETokenizerEditor("large_tokenizer.json")
print(f"Original size: {editor.vocab_size}")

# Preview what will be removed
preview = editor.find_tokens_to_shrink(count=10000, min_id=50000)
print(f"Will remove {len(preview)} tokens")
print("First 5 tokens to be removed:")
for token, id, length in preview[:5]:
    print(f"  '{token}' (length={length}, id={id})")

# Perform the shrink
result = editor.shrink(count=10000, min_id=50000)
print(f"New size: {editor.vocab_size}")
print(f"Total tokens removed (including cascade): {result.total_tokens_removed}")

editor.save("smaller_tokenizer.json")

Adding Tokens with Fixed Vocabulary Size

from bpe_tokenizer_editor import BPETokenizerEditor

editor = BPETokenizerEditor("tokenizer.json")
original_size = editor.vocab_size

# Add new tokens while keeping the same vocabulary size
result = editor.add_tokens_keep_size(
    tokens=["custom1", "custom2", "custom3"],
    whitelist=["<pad>", "<eos>", "<bos>"]  # Never remove these
)

print(f"Tokens added: {result['tokens_added']}")
print(f"Tokens removed to maintain size: {result['tokens_removed']}")
print(f"Final vocab size: {editor.vocab_size}")  # Should equal original_size

editor.save("tokenizer_with_custom_tokens.json")

Performance

The library is written in Rust and compiled to native code, providing excellent performance:

Operation	Time (256K vocab)
Load tokenizer	~50ms
Validate merges	~100ms
Get stats	~50ms
Shrink 1000 tokens	2-5s
Sync 18K chars	30-60s

Building from Source

If you want to build the package from source:

# Install maturin
pip install maturin

# Build and install in development mode
maturin develop --features python

# Build wheel for distribution
maturin build --release --features python

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bpe_tokenizer_editor-0.1.0.tar.gz (39.9 kB view details)

Uploaded Aug 23, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

bpe_tokenizer_editor-0.1.0-cp38-abi3-win_arm64.whl (328.6 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+Windows ARM64

bpe_tokenizer_editor-0.1.0-cp38-abi3-win_amd64.whl (353.0 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+Windows x86-64

bpe_tokenizer_editor-0.1.0-cp38-abi3-win32.whl (325.8 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+Windows x86

bpe_tokenizer_editor-0.1.0-cp38-abi3-musllinux_1_2_x86_64.whl (651.0 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+musllinux: musl 1.2+ x86-64

bpe_tokenizer_editor-0.1.0-cp38-abi3-musllinux_1_2_i686.whl (681.3 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+musllinux: musl 1.2+ i686

bpe_tokenizer_editor-0.1.0-cp38-abi3-musllinux_1_2_armv7l.whl (692.5 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+musllinux: musl 1.2+ ARMv7l

bpe_tokenizer_editor-0.1.0-cp38-abi3-musllinux_1_2_aarch64.whl (589.6 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+musllinux: musl 1.2+ ARM64

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (452.7 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ x86-64

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl (478.8 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ s390x

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (549.7 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ ppc64le

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_i686.manylinux2014_i686.whl (475.1 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ i686

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (415.4 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ ARMv7l

bpe_tokenizer_editor-0.1.0-cp38-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (411.5 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+manylinux: glibc 2.17+ ARM64

bpe_tokenizer_editor-0.1.0-cp38-abi3-macosx_11_0_arm64.whl (388.5 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+macOS 11.0+ ARM64

bpe_tokenizer_editor-0.1.0-cp38-abi3-macosx_10_12_x86_64.whl (410.7 kB view details)

Uploaded Aug 23, 2026 CPython 3.8+macOS 10.12+ x86-64

File details

Details for the file bpe_tokenizer_editor-0.1.0.tar.gz.

File metadata

Download URL: bpe_tokenizer_editor-0.1.0.tar.gz
Upload date: Aug 23, 2026
Size: 39.9 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

File hashes

Hashes for bpe_tokenizer_editor-0.1.0.tar.gz
Algorithm	Hash digest
SHA256	`ce5653f90115022d9745735c0138312398429af9074fcbe629dd9e3bd09c4d34`
MD5	`eac0246141824dda9633db8bf0861db5`
BLAKE2b-256	`01e701ec9ad6ac6a29592f3ceb896a7adc35c4612c3ef282aa6326e0d957cda5`

Algorithm	Hash digest
SHA256	`d4610643cd5a97d415d1e9263274516c3d5cec20c3aa1f4811076b71559f340f`
MD5	`8ce009cdb919ebd255f262a27c11bc4d`
BLAKE2b-256	`bdaf1c6de43fc71125fb4075ca4fe826e776fd76851748a97c9283ccc66fe3a3`

Algorithm	Hash digest
SHA256	`85c0c6c1a7353488b0d72691dddee6f4b6fa99105e1c5bb244c3d50482107788`
MD5	`af7f7f1c3454d93f484ce4a0ccdb31d5`
BLAKE2b-256	`f27f738d641cd6286655fb12f32b8fbe70e2386a152f15d8a851ae8db2e8e993`

Algorithm	Hash digest
SHA256	`e5b61e0911f36a21f2d33f841ead653f2d61ae8ecff5817fcc55936f80b01e5b`
MD5	`94d6b1f5b0ebbb500e691cd4ec2c5f2b`
BLAKE2b-256	`203199d4240b212e41e926600b85d5865fd32d30fd80a4f34e279873608b7b7f`

Algorithm	Hash digest
SHA256	`5f43a0acaae800567eeac261918e39ae270d9e9d41afce77c0fca0ebafdfc25d`
MD5	`bf9a96a2136aeb6ba6ea65b757a66125`
BLAKE2b-256	`997497cf1cb4523cc173cbefe91391f6c8e44df9de2a5b72e3aa4321a2eb93a8`

Algorithm	Hash digest
SHA256	`a1085c8c22fddd55c481f86e9ecb9223b22de913e8246ac6ecc140c8735b76f3`
MD5	`8c774f85d02b11b089f5361c9ac26bed`
BLAKE2b-256	`8cae320fe57459e30318fa778435ac0e9fd1d218ddf08b268af82c6826349e7b`

Algorithm	Hash digest
SHA256	`e63b2e30108520c4b4008d78a544cd3a21e3262a4c71bfff617a7fe8d813db2b`
MD5	`e0e0f531ec7131c5f60db046b2ae4216`
BLAKE2b-256	`5667673daba1d0c7b0a7f47f4a73bf8dab7934163442c3cc6c2d1733152f794f`

Algorithm	Hash digest
SHA256	`40a85624a4af76c90c694229f1d89235ead866fac9119234b0ea8710dd52e075`
MD5	`7f9597440c78ce820f1c600a3c890670`
BLAKE2b-256	`120330d8409a28d90de3a8f86fcb6f1d0bc53b8f3d9b05b5e68db3ac577b5b9c`

Algorithm	Hash digest
SHA256	`6405ad129fc1918a164fc714998322ebf73659ca295a823b389b24c50757065f`
MD5	`35988a6569f795a94c7004f5523ae92b`
BLAKE2b-256	`4d6a04f390fdeb37199f991ef6b15794f6a06be6b10f055ec324e4978fcfe970`

Algorithm	Hash digest
SHA256	`162e259c327808a28b72ae7453a2234eef4ff4da13763e0d6d9d55f1247d0816`
MD5	`fa1c9113b652b1e4764f641cb0b487ba`
BLAKE2b-256	`3c8b1a89b6f1d2e4ab243726b93e6d1ea128f1802514956ecfc0e3aed13ec900`

Algorithm	Hash digest
SHA256	`e08c691fbc87117a9a7fe0af2562cad878ccff2dd9c6256284046fe49dde6041`
MD5	`adf2d7eaf64f5fa7593194d706498fb3`
BLAKE2b-256	`e6e49294725688c2eeda35424a8df034096a808dc670b63e89328f589f237693`

Algorithm	Hash digest
SHA256	`fc034d08cfa583c74681d7e02957a8e4e7b64a3e2afd3eaed77f0f03caedd432`
MD5	`c9bd101c54bd5eb43012464980e190c3`
BLAKE2b-256	`5db40e4661530fc75744139ba5daf0874c115981190d0b6d593e6812ef11568a`

Algorithm	Hash digest
SHA256	`20e6a78da7abd2ed589ea1110ac3779d2367b906ce1ce1c33b6c2c7d245dabd8`
MD5	`382110c1120f71a2407a15181c74366f`
BLAKE2b-256	`84b95bb5b593628658f1ea78e38b8194283b41059881e3e988639f64503c7422`

Algorithm	Hash digest
SHA256	`e91e0c16c8279e1765817fa9decfe8c62dee3eda2706955bf3ef518ce7e0dda0`
MD5	`030a678ae7ea730485bc6307073870b1`
BLAKE2b-256	`458a4ed019e6a4b423a702f13d2c33a0303d84a06185bf60123581ff4fb8fa0a`

Algorithm	Hash digest
SHA256	`5ebd09fe41a0fdcbb4a0c8c4ed038be27b124e0687b08661b383303fd1eede61`
MD5	`f2eb5c46b1e635f56e7ee2f725a7155a`
BLAKE2b-256	`ae29ee37de296d359bbfbaafcad455ab409d4790a8cb0009f487fc83e0f31a4c`

Algorithm	Hash digest
SHA256	`ce2dfb8ad3089999b4db0ec49455e4d4ffdc861b925aa886d14981e283bb8759`
MD5	`4c3915eee2fcfd1bb736a36e50a9176a`
BLAKE2b-256	`0ba6bbdf795c503316c0405b978f7db22e8d234a949ba08f60a697342ff5a1cf`

bpe-tokenizer-editor 0.1.0

BPE Tokenizer Editor

Features

Installation

Requirements

Quick Start

API Reference

Loading and Saving

Properties

Token Operations

Statistics

Validation

Adding Tokens

Removing Tokens

Shrinking Vocabulary

Syncing with Another Tokenizer

Reindexing Vocabulary

Reducing Vocabulary Size

Adding Tokens with Fixed Vocabulary Size

Performance

Building from Source

Contributing

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata

File hashes

File details

File metadata