LangToken 0.1.1

Word Tokenizer to create each word into token and decode back

LangToken

Overview

LangToken is a Python library designed for text preprocessing and tokenization. It converts text into token IDs and provides functionality to encode and decode text, ensuring consistent mapping between words and their token representations.

This project includes a robust implementation, detailed documentation, and unit tests to validate its functionality.

Features

• Vocabulary Creation: Automatically generates a token-to-ID mapping and its inverse from text.
• Encoding: Converts text into a list of token IDs.
• Decoding: Converts a list of token IDs back into readable text.
• File Integration: Reads text files and processes them for tokenization.
• Handling Unknown Tokens: Maps unknown words to a special <|unk|> token.
• Special Tokens: Includes <|unk|> for unknown words and <|endoftext|> for file-ending markers.

Class Diagram

Use

from word_token.tokenizer import Tokenizer

# Initialize the tokenizer
tokenizer = Tokenizer()

# Pass a file to the tokenizer
tokenizer.pass_file("example.txt", "utf-8")

# Generate the vocabulary
tokenizer.fit()

# Encode a text string
encoded = tokenizer.encode("Hello, world!")
print("Encoded:", encoded)

# Decode the encoded IDs
decoded = tokenizer.decode(encoded)
print("Decoded:", decoded)

Output

Encoded: [0, 1, 2, 3]
Decoded: Hello, world!