PyLGEN

From prototype to production: a Python-native compiler framework that brings the "Dragon Book" to life in Python, with clarity throughout.

PyLGEN gives you complete control over every stage of language processing. Build interpreters and compilers from scratch without leaving the Python ecosystem. Prototype rapidly in pure Python, then compile to native speed with Cython for production workloads.

Why PyLGEN?

• Full pipeline control: You own every step: lexer, parser, AST, semantic analysis, and evaluation.

• Dual‑nature design: Pure Python for development and debugging; Cython for near‑C performance.

• Handles large inputs: 2‑million‑line files parsed in ~61.62 s, about 1.89x faster than Lark + lark_cython.

• Python ecosystem integration: Leverage NumPy, SciPy, or any library from within your language.

📊 Benchmark at a Glance

more details in the benchmark section of documentation →

PyLGEN is a framework for building interpreters, covering the full compilation pipeline from lexical analysis to execution.

🚀 Minimal Example

from pylgen.lexer import Lexer
from pylgen.grammar import AttributedGrammar
from pylgen.parser import ParserBuilder, ParserType
from pylgen.common.enums import TokenType
from pylgen.common.types import Symbol

class TokenTypeEnum(TokenType):
    NUMBER = 'NUMBER'
    PLUS = 'PLUS'

    # ...

def mapping_function(t:TokenTypeEnum,tx:str) -> Symbol:
    # ....

# 1. Define tokens & lexer
lexer = Lexer(mapping_function, r'\s+')
lexer[0, TokenTypeEnum.NUMBER] = r'\d+'
lexer[1, TokenTypeEnum.PLUS]   = r'\+'

# 2. Define grammar with AST builders
# Assumes the following are already defined:
#   E, T, plus, binary_reductor, single_reductor
G = AttributedGrammar(Symbol('E'))
G[E] += (E, plus, T), binary_reductor
G[E] += (T,),        single_reductor

# 3. Build the parser
parser = ParserBuilder.build_parser_from_attributed(G, ParserType.LALR1)

# 4. Parse, analyse, execute
text = '1 + 2' # your text to parse
lexer.load_text(text)
ast = parser.parse(lexer.tokens)
# ... your semantic visitors & evaluator ...

🧩 Architecture

flowchart LR
    Source["Source Code"] --> Lexer["Lexer"]
    Lexer --> Parser["Parser LALR(1)"]
    Parser --> AST["AST"]
    AST --> Visitors["Visitors"]
    Visitors --> Result["Result / Errors"] 

All modules are fully usable from Python and Cython, prototype in Python, ship in C.

⚡ Quick Start

pip install pylgen-core

Then build your interpreter step by step, following the complete tutorial in the documentation.

📖 Learn More

• Step‑by‑step tutorial: Build a full arithmetic REPL from scratch.

• VecLang case study: Production‑grade language with vectors, functions, and slicing.

• Deep‑dive API tour: Understand every module inside out.

• Under the hood: Build a lexer and parser from scratch, using the same building blocks that the framework uses internally.

• Examples: See practical examples of using PyLGEN to build interpreters and DSLs.

Read the full documentation →

PyLGEN: Where compiler theory meets Python pragmatism.

Contributing

Contributions are welcome. Please open an issue or submit a pull request. See Contributing.md for details.

License

This project is licensed under the BSD 3-Clause License. See LICENSE for details.

Changelog

All notable changes to this project are documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.7.2] - 2026-10-09

Added

• Support for extended Latin characters (accented vowels, diaeresis, and eñe: á é í ó ú ü ñ and their uppercase forms) as literals in regular expressions.
• Support for Spanish punctuation marks (¿ ¡ « ») as literals.

Changed

• Breaking: the lexer and the constants (\D, \S, \W, ., and the default negated class) now operate over the extended alphabet instead of string.printable. Inputs containing non-ASCII characters that previously produced a tokenization error are now recognized as CHAR.
• Breaking: the complement of a character class ([^...]) is computed against the extended alphabet, not string.printable.

Fixed

• Regular expressions containing Spanish letters (niño, camión, pingüino, …) failed in the lexer because those characters were not in string.printable.
• The ¡ and ¿ signs are no longer discarded as invalid characters.

[0.7.1] - 2026-09-28

Fixed

• BaseLexer wrong behavior with idented languages.

[0.7.0] - 2026-09-27

Changed

• AST now exposes start_position and end_position properties instead of separate line and column. The constructor now receives start_line, start_column, end_line, and end_column instead of just line and column. Breaking change for any subclass of AST or any reductor that instantiates AST nodes.

• Moved ErrorType from pylgen.analysis.error_type to pylgen.common.enums. Update your imports accordingly. Breaking change.

• Moved Error, LexicalError, SyntaxError, SemanticError and RuntimeError from pylgen.analysis.error to pylgen.common.types. Update your imports accordingly. Breaking change.

[0.6.2] - 2026-08-29

Fixed

• Fixed type mismatches in BottomUpParser with the enum BottomUpParserAction.