🔤 LexiByte Engine ("Lexi")

LexiByte is a production-grade, algorithmically optimized Byte-Pair Encoding (BPE) tokenizer, inspired by the architecture used in modern Large Language Models (LLMs) like GPT-2, GPT-4, and Llama.

Built entirely in Python, LexiByte bridges the gap between raw text and Neural Networks by safely compressing UTF-8 byte streams into integer token sequences.

✨ Production-Grade Features

• Sennrich Frequency Optimization: Escapes the $O(N^2)$ training bottleneck by compressing the corpus into a frequency dictionary. It trains on millions of words in seconds by mathematically weighting unique pairs.
• Inference Memoization Cache: Bypasses BPE loops during inference by caching tokenized sequences, delivering $O(1)$ constant time lookups for previously seen words.
• Regex Pre-Splitting Guardrails: Implements GPT-2/GPT-4 style regex boundaries to prevent unnatural merges (e.g., merging punctuation with words, or trailing spaces with letters).
• UTF-8 Byte Level Base: Starts with a base vocabulary of 256 standard UTF-8 bytes, meaning it can theoretically encode any string (including emojis and non-English scripts) without out-of-vocabulary (OOV) errors.
• Special Token Support: Safely handles control tokens (e.g., <|endoftext|>).

📚 Documentation

Detailed documentation has been separated into the following guides:

• Architecture & Internal Mechanics - Understand how BPE works, why regex guardrails matter, and view visual pipeline diagrams.
• Training & Inference Deep Dive - A concrete, step-by-step look at how the merges and vocab maps are built and how production optimization works.
• Usage Guide - Step-by-step instructions on training, encoding, decoding, and saving your tokenizer.
• Developer & Publishing Guide - Instructions for maintainers on how to build and publish the package to PyPI.

🚀 Quick Start

Installation (For Users)

You can install LexiByte directly from PyPI using pip.

pip install lexibyte

(If you are developing locally from the source repository, run pip install -r requirements.min.txt instead).

Basic Example

from lexibyte import LexiByteTokenizer

# Initialize tokenizer
tokenizer = LexiByteTokenizer()

# Sample corpus
text = "hello world! 👋 This is the LexiByte engine."

# Train the tokenizer to reach a vocabulary of 276 (256 base bytes + 20 learned merges)
tokenizer.train(text, vocab_size=276, verbose=True)

# Encode text to token IDs
encoded = tokenizer.encode("hello world!")
print("Encoded:", encoded)

# Decode token IDs back to text
decoded = tokenizer.decode(encoded)
print("Decoded:", decoded)

🧠 Context

This project was developed as an advanced Systems Engineering project to bridge the gap between educational tokenizers and scalable production logic.