vrox-lex 0.2.6

A fast, customizable lexer for tokenizing text using regex rules.

📜 VroxLexer

VroxLexer is a simple yet powerful Python-based lexer designed to tokenize input text based on user-defined regular expression rules. It efficiently processes large files by reading in chunks and writing tokens directly to an output file.

✨ Features

• Customizable Tokenization – Users can define custom tokenization rules using regular expressions.
• Efficient File Processing – Processes large files efficiently by reading in chunks instead of loading the entire file into memory.
• Regex-based Matching – Uses precompiled regular expressions for fast token detection.
• Automatic Output Writing – Tokens are written directly to an output file in real time.
• Callback Support – Allows users to specify a callback function that runs after tokenization is complete.
• Extensible API – Easily add custom token rules for any syntax.

📦 Installation

pip install vrox-lex

🔗 Optional Dependency

For higher-level regex abstraction, you can install VroxRegex:

pip install vrox-regex

📝 Note: VroxLexer works without VroxRegex, but using it simplifies complex regex definitions.

🚀 Usage

1️⃣ Create a Lexer Instance

To start using VroxLexer, create an instance of it:

    from vrox_lexer import VroxLexer
    lexer = VroxLexer()

2️⃣ Add Tokenization Rules

Use the add_rule(name, regex) method to define token rules.

Method 1: One-by-One Rule Addition

lexer.add_rule("KEYWORD", r"\b(if|else|while|for|return)\b")
lexer.add_rule("NUMBER", r"\b\d+(\.\d+)?\b")
lexer.add_rule("STRING", r"\".*?\"|'.*?'")
lexer.add_rule("IDENTIFIER", r"\b[a-zA-Z_][a-zA-Z0-9_]*\b")
lexer.add_rule("OPERATOR", r"[+\-*/=]")
lexer.add_rule("PUNCTUATION", r"[\(\)\{\},;]")

Method 2: Chainable API for Better Readability (Recommended)

lexer = VroxLexer()\
        .add_rule("KEYWORD", r"\b(if|else|while|for|return)\b")\
        .add_rule("NUMBER", r"\b\d+(\.\d+)?\b")\
        .add_rule("STRING", r"\".*?\"|'.*?'")\
        .add_rule("IDENTIFIER", r"\b[a-zA-Z_][a-zA-Z0-9_]*\b")\
        .add_rule("OPERATOR", r"[+\-*/=]")\
        .add_rule("PUNCTUATION", r"[\(\)\{\},;]")

3️⃣ Tokenize Text from a File

def on_tokenization_complete(output_path):
    print(f"✔ Tokenization complete! Tokens saved in {output_path}")

lexer.tokenize("input.txt", "output.txt", on_tokenization_complete)

📢 Example Output (output.txt)

KEYWORD: if
IDENTIFIER: count
OPERATOR: =
NUMBER: 10
PUNCTUATION: ;

🔗 Final Notes

• VroxLexer is lightweight, fast, and designed for extensibility.
• Using precompiled regex improves performance significantly.
• The chainable API makes rule definition more intuitive.
• The callback function enables post-processing after tokenization.