BPE Tokenizer Editor

A high-performance Python library for editing HuggingFace BPE tokenizer.json files with consistency guarantees. Built with Rust for maximum performance.

Features

• ✅ Validate merges - Check that all merge results exist in vocabulary
• ✅ Add tokens - Add new tokens with automatic merge chain creation
• ✅ Remove tokens - Remove tokens with cascade deletion of dependent merges
• ✅ Shrink vocab - Remove N longest tokens to reduce vocabulary size
• ✅ Sync single-chars - Copy all single-character tokens from source tokenizer
• ✅ Merge tokenizers - Convert source tokens to the target byte format and prioritize their merges
• ✅ Keep vocab size fixed - Add tokens while automatically removing others to maintain size
• ✅ Reindex vocab - Make vocabulary IDs sequential by removing gaps
• 🚀 Rust-powered - Native performance with Python convenience

Installation

pip install bpe-tokenizer-editor

Requirements

• Python 3.8+
• No additional dependencies required (Rust code is compiled into the wheel)

Quick Start

from bpe_tokenizer_editor import BPETokenizerEditor

# Load a tokenizer
editor = BPETokenizerEditor("tokenizer.json")

# Check stats
print(f"Vocabulary size: {editor.vocab_size}")
print(f"Number of merges: {editor.merges_count}")

# Validate merges
validation = editor.validate_merges()
print(f"Valid merges: {validation.valid_count}")
print(f"Invalid merges: {validation.invalid_count}")

# Add a new token
result = editor.add_token("merhaba")
print(f"Added: {result.added}, Method: {result.method}")

# Save the modified tokenizer
editor.save("tokenizer_modified.json")

API Reference

Loading and Saving

# Load from file
editor = BPETokenizerEditor("tokenizer.json")

# Load from JSON string
editor = BPETokenizerEditor.from_json(json_string)

# Save to file
editor.save("output.json")

# Export to JSON string
json_str = editor.to_json()

Properties

editor.vocab_size    # Current vocabulary size
editor.merges_count  # Current number of merge rules

Token Operations

# Check if token exists
editor.has_token("hello")  # Returns bool

# Get token ID
editor.get_token_id("hello")  # Returns int or None

# Get token by ID
editor.get_token_by_id(1000)  # Returns str or None

# Get all tokens as dict
vocab = editor.get_vocab()  # Returns Dict[str, int]

# Get all merges
merges = editor.get_merges()  # Returns List[Tuple[str, str]]

# Get single-character tokens
single_chars = editor.get_single_char_tokens()  # Returns List[Tuple[str, int]]

Merging Tokenizers

# The editor is the target tokenizer whose configuration is preserved.
editor = BPETokenizerEditor("target-tokenizer.json")
result = editor.merge_from(
    "source-tokenizer.json",
    max_vocab_size=2**18,
)
editor.save("merged-tokenizer.json")

print(result.representation)       # "ByteLevel (Ġ)" or "space marker (▁)"
print(result.tokens_injected)
print(result.final_vocab_size)     # Includes added/special tokens

merge_from converts source tokens to the target tokenizer's native internal representation, adds any required UTF-8 byte bridges, and ranks source merges before original target merges. Source-specific added/special tokens are not injected.

Statistics

stats = editor.get_stats()
print(f"Vocab size: {stats.vocab_size}")
print(f"Merges: {stats.merges_count}")
print(f"Single chars: {stats.single_char_count}")
print(f"Special tokens: {stats.special_token_count}")
print(f"ID range: {stats.min_token_id} - {stats.max_token_id}")
print(f"Length distribution: {stats.length_distribution}")

Validation

# Validate merges (check if merge results exist in vocab)
result = editor.validate_merges()
print(f"Valid: {result.valid_count}, Invalid: {result.invalid_count}")

for idx, token_a, token_b in result.invalid_merges:
    print(f"  Invalid merge at {idx}: '{token_a}' + '{token_b}'")

# Fix invalid merges (remove them)
removed_count = editor.remove_invalid_merges()
print(f"Removed {removed_count} invalid merges")

Adding Tokens

# Add a single token with automatic merge chain creation
result = editor.add_token("newtoken")
print(f"Added: {result.added}")
print(f"Method: {result.method}")  # 'single_char', 'longest_prefix', or 'char_chain'
print(f"Added merges: {result.added_merges}")

# Add multiple tokens
results = editor.add_tokens(["token1", "token2", "token3"])

# Add special token without merge chain
editor.add_token_atomic("<special>")

# Add tokens while keeping vocabulary size fixed
result = editor.add_tokens_keep_size(
    tokens=["new1", "new2", "new3"],
    whitelist=["protected_token"]  # Optional: tokens that should never be removed
)
print(f"Added: {result['tokens_added']}, Removed: {result['tokens_removed']}")

Removing Tokens

# Remove a token and all its dependencies (cascade removal)
result = editor.remove_token("unwanted")
print(f"Root token: {result.root_token}")
print(f"Removed tokens: {result.removed_tokens}")
print(f"Removed merges: {result.removed_merges}")

# Remove multiple tokens
results = editor.remove_tokens(["token1", "token2"])

Shrinking Vocabulary

# Preview what would be removed
tokens_to_remove = editor.find_tokens_to_shrink(count=1000, min_id=50000)
for token, id, length in tokens_to_remove[:10]:
    print(f"  {token} (ID: {id}, length: {length})")

# Actually shrink the vocabulary
result = editor.shrink(count=1000, min_id=50000)
print(f"Initial size: {result.initial_vocab_size}")
print(f"Final size: {result.final_vocab_size}")
print(f"Tokens removed: {result.total_tokens_removed}")
print(f"Merges removed: {result.total_merges_removed}")

Syncing with Another Tokenizer

# Sync single-character tokens from source tokenizer
# (useful for preserving Unicode coverage)
result = editor.sync_single_chars(
    source_path="original_tokenizer.json",
    min_id=50000  # Only remove tokens with ID >= 50000
)
print(f"Chars added: {result['chars_added_count']}")
print(f"Tokens removed: {result['total_tokens_removed']}")

Reindexing Vocabulary

After sync operations or modifications, vocabulary IDs may have gaps (e.g., IDs jump from 73000 to 74000). This creates a sparse vocabulary. Use reindex to make all IDs sequential:

# Check for gaps in vocabulary IDs
has_gaps, total_gaps, min_id, max_id = editor.check_vocab_gaps()
if has_gaps:
    print(f"Found {total_gaps} gaps in ID space (range: {min_id}-{max_id})")

# Reindex to make IDs sequential
result = editor.reindex_vocab()
print(f"Remapped {result.ids_remapped} IDs")
print(f"Removed {result.gaps_removed} gaps")
print(f"New ID range: {result.new_min_id} - {result.new_max_id}")

Note: The sync_single_chars method automatically calls reindex_vocab after completing, so you typically don't need to call it manually after syncing.

## Examples

### Training a Domain-Specific Tokenizer

```python
from bpe_tokenizer_editor import BPETokenizerEditor

# After training your custom tokenizer, validate and fix it
editor = BPETokenizerEditor("my_custom_tokenizer.json")

# Check for problems
validation = editor.validate_merges()
if validation.invalid_count > 0:
    print(f"Found {validation.invalid_count} invalid merges, fixing...")
    editor.remove_invalid_merges()

# Sync Unicode characters from original tokenizer
result = editor.sync_single_chars(
    source_path="original_tokenizer.json",
    min_id=50000
)
print(f"Added {result['chars_added_count']} missing characters")

# Add domain-specific tokens
domain_tokens = ["merhaba", "dünya", "Türkiye"]
for token in domain_tokens:
    result = editor.add_token(token)
    if result.added:
        print(f"Added '{token}' using {result.method} method")

# Save the result
editor.save("my_final_tokenizer.json")

Reducing Vocabulary Size

from bpe_tokenizer_editor import BPETokenizerEditor

editor = BPETokenizerEditor("large_tokenizer.json")
print(f"Original size: {editor.vocab_size}")

# Preview what will be removed
preview = editor.find_tokens_to_shrink(count=10000, min_id=50000)
print(f"Will remove {len(preview)} tokens")
print("First 5 tokens to be removed:")
for token, id, length in preview[:5]:
    print(f"  '{token}' (length={length}, id={id})")

# Perform the shrink
result = editor.shrink(count=10000, min_id=50000)
print(f"New size: {editor.vocab_size}")
print(f"Total tokens removed (including cascade): {result.total_tokens_removed}")

editor.save("smaller_tokenizer.json")

Adding Tokens with Fixed Vocabulary Size

from bpe_tokenizer_editor import BPETokenizerEditor

editor = BPETokenizerEditor("tokenizer.json")
original_size = editor.vocab_size

# Add new tokens while keeping the same vocabulary size
result = editor.add_tokens_keep_size(
    tokens=["custom1", "custom2", "custom3"],
    whitelist=["<pad>", "<eos>", "<bos>"]  # Never remove these
)

print(f"Tokens added: {result['tokens_added']}")
print(f"Tokens removed to maintain size: {result['tokens_removed']}")
print(f"Final vocab size: {editor.vocab_size}")  # Should equal original_size

editor.save("tokenizer_with_custom_tokens.json")

Performance

The library is written in Rust and compiled to native code, providing excellent performance:

Building from Source

If you want to build the package from source:

# Install maturin
pip install maturin

# Build and install in development mode
maturin develop --features python

# Build wheel for distribution
maturin build --release --features python

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.