inclitoken 0.1.0

Inclitoken is implementation of Byte Pair Encoding Tokenizer from scratch.

IncliToken

A simple Byte Pair Encoding (BPE) tokenizer implementation from scratch in Python.

Installation

uv add inclitoken

Or you can use pip:

pip install inclitoken

Usage

from inclitoken.tokenizer import BPETokenizer

# Initialize tokenizer
tokenizer = BPETokenizer()

# Train on your text
text = "Hello world! This is a simple example."
tokenizer.train(text, turns=100, verbose=False)

# Encode text to token IDs
ids = tokenizer.encode("Hello world!")
print(ids)

# Decode token IDs back to text
decoded = tokenizer.decode(ids)
print(decoded)

Features

• Train custom BPE tokenizers on your text
• Encode text into token IDs
• Decode token IDs back into text
• Track merge operations and vocabulary

Requirements

• Python >= 3.14
• tqdm

Author

Built by Adarsh Dubey