ultra-tokenizer 0.1.2

Advanced tokenizer with support for BPE, WordPiece, and Unigram algorithms

Ultra-Tokenizer

Features

• Multiple Tokenization Algorithms: Supports BPE, WordPiece, and Unigram algorithms
• High Performance: Optimized for speed and memory efficiency
• Easy Integration: Simple API for training and using tokenizers
• Production Ready: Battle-tested with comprehensive test coverage
• Type Hints: Full Python type support for better development experience

Installation

Install the latest stable version from PyPI:

pip install ultra-tokenizer

Quick Start

Basic Usage

from ultra_tokenizer import Tokenizer

# Initialize tokenizer with default settings
tokenizer = Tokenizer()

# Tokenize text
text = "Hello, world! This is Ultra-Tokenizer in action."
tokens = tokenizer.tokenize(text)
print(tokens)
# Output: ['Hello', ',', 'world', '!', 'This', 'is', 'Ultra', '-', 'Token', '##izer', 'in', 'action', '.']

Training a New Tokenizer

from ultra_tokenizer import TokenizerTrainer

# Initialize trainer
trainer = TokenizerTrainer(
    vocab_size=30000,
    min_frequency=2,
    show_progress=True
)

# Train on text files
tokenizer = trainer.train_from_files(["file1.txt", "file2.txt"])

# Save tokenizer
tokenizer.save("my_tokenizer.json")

# Load tokenizer
from ultra_tokenizer import Tokenizer
tokenizer = Tokenizer.from_file("my_tokenizer.json")

Documentation

For detailed documentation, examples, and API reference, please visit:

Ultra-Tokenizer Documentation

Contributing

Contributions are welcome! Please read our Contributing Guide to get started.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Contact

For questions or feedback, please open an issue or contact [e-mail].