Sujit-Tokenizer

A custom Byte-Level BPE (Byte Pair Encoding) Tokenizer implemented from scratch in Python.

Features

• Byte-level tokenization
• Custom BPE training
• Text encoding and decoding
• Save and load tokenizer models
• UTF-8 support
• Lightweight and dependency-free

Installation

Clone Repository

git clone https://github.com/iamsspm07/Sujit-Tokenizer.git
cd Sujit-Tokenizer

Install Package

pip install -e .

Project Structure

Sujit-Tokenizer/
│
├── Sujit_Tokenizer/
│   ├── __init__.py
│   └── tokenizer.py
│
├── corpus.txt
├── train_and_test.py
│
├── README.md
├── setup.py
├── pyproject.toml
└── LICENSE

Quick Start

Import Tokenizer

from Sujit_Tokenizer import CustomByteLevelBPETokenizer

Train Tokenizer

corpus = [
    "Transformers are amazing.",
    "Machine Learning is powerful.",
    "Python is widely used in AI."
]

tokenizer = CustomByteLevelBPETokenizer(
    vocab_size=1000
)

tokenizer.train(corpus)

Save Model

tokenizer.save_model(
    "tokenizer.model"
)

Load Model

tokenizer = CustomByteLevelBPETokenizer()

tokenizer.load_model(
    "tokenizer.model"
)

Encode Text

encoded = tokenizer.encode(
    "Transformers use attention."
)

print(encoded)

Example Output:

[2, 451, 723, 812, 3]

Decode Text

decoded = tokenizer.decode(
    encoded
)

print(decoded)

Output:

Transformers use attention.

Training Workflow

Corpus
   ↓
Byte Conversion
   ↓
Pair Frequency Counting
   ↓
BPE Merging
   ↓
Vocabulary Construction
   ↓
Tokenizer Model

Special Tokens

Example

from Sujit_Tokenizer import CustomByteLevelBPETokenizer

tokenizer = CustomByteLevelBPETokenizer(
    vocab_size=1000
)

corpus = [
    "Artificial Intelligence",
    "Machine Learning",
    "Deep Learning"
]

tokenizer.train(corpus)

tokenizer.save_model(
    "tokenizer.model"
)

tokenizer.load_model(
    "tokenizer.model"
)

text = "Machine Learning"

encoded = tokenizer.encode(text)
print(encoded)

decoded = tokenizer.decode(encoded)
print(decoded)

Use Cases

• NLP experiments
• Tokenization research
• Educational projects
• Understanding BPE internals
• Building custom language models
• Learning tokenizer architecture

Future Enhancements

• Faster BPE training
• WordPiece tokenizer
• SentencePiece tokenizer
• Parallel processing
• Vocabulary statistics
• Token frequency analysis
• Hugging Face compatibility

Author

Sujit Maity, Praveen Kumar Kannan

License

MIT License