shredword-trainer 0.0.2

BPE & Unigram trainers for Shredword tokenizer

ShredWord

ShredWord is a byte-pair encoding (BPE) based tokenizer-trainer designed for fast, efficient, and flexible text processing & vocab training. It offers training, and text normalization functionalities and is backed by a C/C++ core with a Python interface for easy integration into machine learning workflows.

Unigram code doesn't work, I lack intelligence capabilites for fixing it.

Features

1. Efficient Tokenization: Utilizes BPE for compressing text data and reducing the vocabulary size, making it well-suited for NLP tasks.
2. Customizable Vocabulary: Allows users to define the target vocabulary size during training.
3. Save and Load Models: Supports saving and loading trained tokenizers for reuse.
4. Python Integration: Provides a Python interface for seamless integration and usability.

How It Works

Byte-Pair Encoding (BPE)

BPE is a subword tokenization algorithm that compresses a dataset by merging the most frequent pairs of characters or subwords into new tokens. This process continues until a predefined vocabulary size is reached.

Key steps:

1. Initialize the vocabulary with all unique characters in the dataset.
2. Count the frequency of character pairs.
3. Merge the most frequent pair into a new token.
4. Repeat until the target vocabulary size is achieved.

ShredWord implements this process efficiently in C/C++, exposing training, encoding, and decoding methods through Python.

Installation

Prerequisites

• Python 3.11+
• GCC or a compatible compiler (for compiling the C/C++ code)

Steps

1. Install the Python package from PyPI.org:

   pip install shredword-trainer
   

Usage

Below is a simple example demonstrating how to use ShredWord for training, encoding, and decoding text.

Example

from shredword.trainer import BPETrainer

trainer = BPETrainer(target_vocab_size=500, min_pair_freq=1000)
trainer.load_corpus("test data/final.txt")
trainer.train()
trainer.save("model/merges_1k.model", "model/vocab_1k.vocab")

API Overview

Core Methods

• train(text, vocab_size): Train a tokenizer on the input text to a specified vocabulary size.
• save(file_path): Save the trained tokenizer to a file.

Properties

• merges: View or set the merge rules for tokenization.
• vocab: Access the vocabulary as a dictionary of token IDs to strings.
• pattern: View or set the regular expression pattern used for token splitting.
• special_tokens: View or set special tokens used by the tokenizer.

Advanced Features

Saving and Loading

Trained tokenizers can be saved to a file and reloaded for use in future tasks. The saved model includes merge rules and any special tokens or patterns defined during training.

# Save the trained model
tokenizer.save("vocab/trained_vocab.model")

# Load the model
tokenizer.load("vocab/trained_vocab.model")

Customization

Users can define special tokens or modify the merge rules and pattern directly using the provided properties.

# Set special tokens
special_tokens = [("<PAD>", 0), ("<UNK>", 1)]
tokenizer.special_tokens = special_tokens

# Update merge rules
merges = [(101, 32, 256), (32, 116, 257)]
tokenizer.merges = merges

a project by Shivendra