Indic-BPE

A lightweight Byte Pair Encoding (BPE) tokenizer built from scratch for Hindi written in Devanagari.

Simple implementation • Efficient BPE training • Small Python API

Features

• Hindi / Devanagari BPE tokenizer
• Train BPE merges from a text corpus
• Heap-based most-frequent pair selection
• Linked-list based merge operations
• Incremental local pair updates during training
• Encode and decode support
• Vocabulary and merge serialization
• Simple Python package API
• Streamlit tokenizer playground
• Automated test suite
• Training and tokenizer benchmarks

Architecture

Installation

From PyPI

pip install indic-bpe

Development Installation

git clone https://github.com/Rktim/indic-bpe.git
cd indic-bpe
uv sync

Quick Start

from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

tokenizer.load(
    "corpus/processed/hindi_vocab.json",
    "corpus/processed/hindi_merges.json",
)

text = "यह एक हिंदी भाषा का परीक्षण है।"

tokens = tokenizer.encode(text)
print(tokens)

decoded = tokenizer.decode(tokens)
print(decoded)

Example:

Tokens:
['य', 'ह ', 'ए', 'क ', 'ह', 'ि', 'ंद', 'ी ', 'भ', 'ा', 'ष', 'ा क', 'ा ',
 'प', 'र', 'ी', 'क्ष', 'ण', ' ', 'है', '।']

Decoded:
यह एक हिंदी भाषा का परीक्षण है।

Training

A tokenizer can be trained directly from a Hindi text corpus:

from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

tokenizer.train(
    loader,
    num_merges=1000,
)

The trainer repeatedly selects the most frequent adjacent symbol pair and merges it into a new token.

The implementation maintains pair statistics incrementally rather than rebuilding all pair frequencies after every merge.

Save and Load

Save a trained tokenizer:

tokenizer.save(
    "hindi_vocab.json",
    "hindi_merges.json",
)

Load it later:

tokenizer.load(
    "hindi_vocab.json",
    "hindi_merges.json",
)

The tokenizer stores:

• Vocabulary in vocab.json
• Merge rules in merges.json

API

train()

tokenizer.train(loader, num_merges=1000)

Train the BPE tokenizer from a corpus.

encode()

tokens = tokenizer.encode(text)

Convert Hindi text into BPE tokens.

decode()

text = tokenizer.decode(tokens)

Convert tokens back into the original text.

save()

tokenizer.save(vocab_path, merges_path)

Save the vocabulary and merge rules.

load()

tokenizer.load(vocab_path, merges_path)

Load a previously trained tokenizer.

Tokenizer Evaluation

Example:

यह एक हिंदी भाषा का परीक्षण है।

Current evaluation:

Characters: 31
Tokens: 21

Example tokens:

य | ह  | ए | क  | ह | ि | ंद | ी  | भ | ा | ष | ा क | ा  | प | र | ी | क्ष | ण |   | है | ।

The evaluation verifies that:

decoded == original_text

Streamlit Demo

Indic-BPE includes a small Hindi tokenizer playground.

Run:

uv run streamlit run scripts/tokenizer_app.py

The demo provides:

• Hindi text input
• Character count
• Token count
• Colored token visualization
• Token text view
• Token ID view
• Token breakdown

Package Validation

The 0.0.1 package has been validated by:

1. Building a wheel and source distribution.
2. Installing the wheel into a clean virtual environment.
3. Importing BPETokenizer from the installed package.
4. Running a Hindi encode/decode round-trip.
5. Loading trained Hindi vocabulary and merge files.
6. Running a trained-tokenizer Hindi round-trip.

Example:

from indic_bpe import BPETokenizer

tokenizer = BPETokenizer()

text = "यह एक हिंदी भाषा का परीक्षण है।"

tokens = tokenizer.encode(text)
decoded = tokenizer.decode(tokens)

assert decoded == text

Project Structure

indic-bpe/
├── indic_bpe/
│   ├── __init__.py
│   ├── corpus.py
│   ├── decoder.py
│   ├── encoder.py
│   ├── merges.py
│   ├── serialization.py
│   ├── tokenizer.py
│   ├── trainer.py
│   ├── utils.py
│   ├── version.py
│   └── vocabulary.py
│
├── corpus/
│   ├── raw/
│   └── processed/
│
├── scripts/
│   ├── benchmark_tokenizer.py
│   ├── evaluate_tokenizer.py
│   └── tokenizer_app.py
│
├── tests/
├── assets/
├── README.md
├── LICENSE
├── pyproject.toml
└── .gitignore

Current Status

Version: 0.0.1

Indic-BPE 0.0.1 is an early release focused on Hindi / Devanagari BPE tokenization.

Completed:

• BPE trainer
• Optimized pair tracking
• Encoder
• Decoder
• Vocabulary handling
• Serialization
• Tokenizer API
• Test suite
• Benchmark scripts
• Streamlit demo
• Python package build
• Clean wheel installation validation

License

MIT License. See LICENSE.