🔤 LexiByte Engine ("Lexi")

LexiByte is a production-grade, from-scratch implementation of a Byte-Pair Encoding (BPE) tokenizer, inspired by the architecture used in modern Large Language Models (LLMs) like GPT-2, GPT-4, and Llama.

Built entirely in Python, LexiByte bridges the gap between raw text and Neural Networks by safely compressing UTF-8 byte streams into integer token sequences.

✨ Key Features

• Pure BPE Algorithm: Iteratively merges the most frequent adjacent byte/token pairs to build a compressed vocabulary.
• Regex Pre-Splitting Guardrails: Implements GPT-2/GPT-4 style regex boundaries to prevent unnatural merges (e.g., merging punctuation with words, or trailing spaces with letters).
• UTF-8 Byte Level Base: Starts with a base vocabulary of 256 standard UTF-8 bytes, meaning it can theoretically encode any string (including emojis and non-English scripts) without out-of-vocabulary (OOV) errors.
• Save/Load Functionality: Export learned merges and special tokens to a JSON file, ready to be plugged into custom transformer models.
• Special Token Support: Safely handles control tokens (e.g., <|endoftext|>).

📚 Documentation

Detailed documentation has been separated into the following guides:

• Architecture & Internal Mechanics - Understand how BPE works, why regex guardrails matter, and view visual pipeline diagrams.
• Training & Inference Deep Dive - A concrete, step-by-step look at how the merges and vocab maps are built and utilized.
• Usage Guide - Step-by-step instructions on training, encoding, decoding, and saving your tokenizer.
• Developer & Publishing Guide - Instructions for maintainers on how to build and publish the package to PyPI.

🚀 Quick Start

Installation (For Users)

You can install LexiByte directly from PyPI using pip.

pip install lexibyte

(If you are developing locally from the source repository, run pip install -r requirements.min.txt instead).

Basic Example

from lexibyte import LexiByteTokenizer

# Initialize tokenizer
tokenizer = LexiByteTokenizer()

# Sample corpus
text = "hello world! 👋 This is the LexiByte engine."

# Train the tokenizer to reach a vocabulary of 276 (256 base bytes + 20 learned merges)
tokenizer.train(text, vocab_size=276, verbose=True)

# Encode text to token IDs
encoded = tokenizer.encode("hello world!")
print("Encoded:", encoded)

# Decode token IDs back to text
decoded = tokenizer.decode(encoded)
print("Decoded:", decoded)

Running the Test Script (Local Dev)

If you cloned the repo and want to verify the local build:

python test.py

🧠 Inspiration & Context

This project was developed as part of a High-Impact AI/ML Portfolio, heavily inspired by Andrej Karpathy's "Neural Networks: Zero to Hero" series, specifically the lecture on building the GPT Tokenizer.