ultra-tokenizer 0.1.3

Advanced tokenizer with support for BPE, WordPiece, and Unigram algorithms

Ultra-Tokenizer

Features

• Multiple Tokenization Algorithms: Supports BPE, WordPiece, and Unigram algorithms
• High Performance: Optimized for speed and memory efficiency
• Easy Integration: Simple API for training and using tokenizers
• Production Ready: Battle-tested with comprehensive test coverage
• Type Hints: Full Python type support for better development experience

Installation

Install the latest stable version from PyPI:

pip install ultra-tokenizer

Quick Start

Basic Usage

from ultra_tokenizer import Tokenizer

# Initialize tokenizer with default settings
tokenizer = Tokenizer()

# Tokenize text
text = "Hello, world! This is Ultra-Tokenizer in action."
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['Hello', ',', 'world', '!', 'This', 'is', 'Ultra', '-', 'Token', '##izer', 'in', 'action', '.']

Training a New Tokenizer

from ultra_tokenizer import TokenizerTrainer

# Initialize trainer
trainer = TokenizerTrainer(
    vocab_size=30000,
    min_frequency=2,
    show_progress=True
)

# Train on text files
tokenizer = trainer.train_from_files(["file1.txt", "file2.txt"])

# Save tokenizer
tokenizer.save("my_tokenizer.json")

# Load tokenizer
from ultra_tokenizer import Tokenizer
tokenizer = Tokenizer.from_file("my_tokenizer.json")

Advanced Usage

Custom Tokenization Rules

from ultra_tokenizer import Tokenizer
import re

# Custom tokenization with regex pattern
custom_tokenizer = Tokenizer(
    tokenization_pattern=r"\b\w+\b|\S"  # Words or non-whitespace characters
)

Batch Processing

# Process multiple texts efficiently
texts = ["First sentence.", "Second sentence.", "Third sentence."]
all_tokens = [tokenizer.tokenize(text) for text in texts]

Customization

Special Tokens

from ultra_tokenizer import Tokenizer

# Initialize with custom special tokens
tokenizer = Tokenizer(
    special_tokens={
        "unk_token": "[UNK]",
        "pad_token": "[PAD]",
        "cls_token": "[CLS]",
        "sep_token": "[SEP]"
    }
)

Custom Preprocessing

def custom_preprocessor(text):
    # Your custom preprocessing logic here
    text = text.lower()
    text = re.sub(r'\s+', ' ', text).strip()
    return text

tokenizer = Tokenizer(preprocessing_fn=custom_preprocessor)

Fine-tuning

Updating an Existing Tokenizer

# Continue training on new data
with open("new_data.txt", "r", encoding="utf-8") as f:
    trainer = TokenizerTrainer(vocab_size=35000)  # Slightly larger vocab
    tokenizer = trainer.train_from_files(
        ["new_data.txt"],
        initial_tokenizer=tokenizer  # Start from existing tokenizer
    )

Domain-Specific Fine-tuning

# Fine-tune on domain-specific data
domain_trainer = TokenizerTrainer(
    vocab_size=32000,
    min_frequency=1,  # Include rare terms
    special_tokens={"additional_special_tokens": ["[MED]", "[DISEASE]", "[TREATMENT]"]}
)

domain_tokenizer = domain_trainer.train_from_files(
    ["medical_corpus.txt"],
    initial_tokenizer=tokenizer  # Start from base tokenizer
)

Performance Optimization

# Optimize for inference
tokenizer.enable_caching()  # Cache tokenization results
fast_tokens = tokenizer.tokenize("Optimized for speed!")

Documentation

For detailed documentation, examples, and API reference, please visit:

Ultra-Tokenizer Documentation

Contributing

Contributions are welcome! Please read our Contributing Guide to get started.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Contact

For questions or feedback, please open an issue or contact [e-mail].