dbrownell-parserlib 0.12.0

Project:

Package:

Development:

Contents

• Overview
• Installation
• Development
• Additional Information
• License

Overview

dbrownell_ParserLib is a Python library that provides utility functionality for creating parsers, with a primary focus on ANTLR (Another Tool for Language Recognition) integration. The library simplifies the process of building parsers that produce strongly-typed abstract syntax trees (ASTs) with comprehensive source location tracking.

Key Features

• ANTLR Integration: Streamlined wrapper around ANTLR for Python, handling grammar compilation and parser generation
• AST Representation: Element-based AST nodes with automatic parent-child relationship management
• Source Location Tracking: Every AST node tracks its source file location (filename, line, column ranges) for detailed error reporting
• Visitor Pattern: Flexible visitor pattern implementation with fine-grained traversal control
• Error Handling: Rich error objects that associate messages with source locations and extract Python exception tracebacks
• Multi-threaded Parsing: Built-in support for parsing multiple files concurrently
• Workspace Support: Parse translation units organized across multiple workspace roots
• Whitespace Handling: A single visitor mixin supports both insignificant and significant whitespace grammars (e.g., Python-like indentation); grammars that define INDENT/DEDENT tokens are detected automatically
• Multiline String Extraction: Extract multiline string content that must be vertically aligned with its opening token, with detailed errors on misalignment

How to use dbrownell_ParserLib

The typical workflow for creating a parser with this library involves the following steps. For detailed examples, see the test files in the repository, particularly the end-to-end tests that demonstrate complete parsing workflows.

1. Define an ANTLR Grammar

Create a .g4 grammar file defining your language syntax:

grammar Calculator;

expr : expr ('*'|'/') expr   # BinaryOp
     | expr ('+'|'-') expr   # BinaryOp
     | INT                   # Number
     ;

INT : [0-9]+ ;
WS : [ \t\n\r]+ -> skip ;

2. Build the Grammar

Use BuildAntlrGrammar to generate the lexer and parser:

from pathlib import Path

from dbrownell_ParserLib.antlr.build_antlr_grammar import BuildAntlrGrammar


BuildAntlrGrammar(
    dm,  # DoneManager instance
    Path("Calculator.g4"),
    Path("output_dir"),
)

3. Create Custom Visitor

Implement a visitor that converts ANTLR parse trees to Element objects:

from dataclasses import dataclass

from dbrownell_ParserLib.antlr.antlr_visitor_mixin import AntlrVisitorMixin
from dbrownell_ParserLib.element import Element
from dbrownell_ParserLib.terminal_element import TerminalElement

from CalculatorVisitor import CalculatorVisitor as GeneratedVisitor


@dataclass(eq=False)
class BinaryExpression(Element):
    left: Element
    operator: TerminalElement[str]
    right: Element

    def _GenerateAcceptDetails(self):
        yield "left", self.left
        yield "operator", self.operator
        yield "right", self.right


class CalculatorVisitor(AntlrVisitorMixin, GeneratedVisitor):
    def visitBinaryOp(self, ctx):
        operator = TerminalElement[str](self.CreateRegion(ctx.children[1]), ctx.children[1].getText())

        children = self.GetChildren(ctx)
        assert len(children) == 2, children

        self._stack.append(
            BinaryExpression(
                self.CreateRegion(ctx),
                children[0],
                operator,
                children[1],
            )
        )

    def visitNumber(self, ctx):
        self._stack.append(
            TerminalElement[int](self.CreateRegion(ctx), int(ctx.getText()))
        )

4. Create Parser Instance

Use CreateAntlrParser to build a callable parser. Grammars that define INDENT and DEDENT tokens (significant whitespace) are detected automatically; no additional configuration is required.

from dbrownell_ParserLib.antlr.antlr_parser import CreateAntlrParser


parser = CreateAntlrParser(
    CalculatorLexer,
    CalculatorParser,
    CalculatorVisitor,
    lambda p: p.expr(),  # Entry point rule
)

5. Parse Files

Parse single files, multiple files, or entire workspaces:

from dbrownell_ParserLib.errors import Error


# Parse a single file
results = parser(dm, Path("input.txt"), None)

# Parse multiple files
results = parser(dm, [Path("file1.txt"), Path("file2.txt")], None)

# Check results
for filepath, result in results.items():
    if isinstance(result, Error):
        print(f"Parse error in {filepath}: {result}")
    else:
        # result is the visitor containing the parsed AST
        ast = result._stack[0]
        # Process the AST

6. Traverse the AST

Use the visitor pattern to process your AST:

from contextlib import contextmanager

from dbrownell_ParserLib.visitors import ElementVisitorHelper, VisitResult


class EvaluatorVisitor(ElementVisitorHelper):
    @contextmanager
    def OnBinaryExpression(self, element):
        # Process binary expressions
        yield VisitResult.Continue


ast.Accept(EvaluatorVisitor())

Installation

Verifying Signed Artifacts

Artifacts are signed and verified using py-minisign and the public key in the file ./minisign_key.pub.

To verify that an artifact is valid, visit the latest release and download the .minisign signature file that corresponds to the artifact, then run the following command, replacing <filename> with the name of the artifact to be verified:

uv run --with py-minisign python -c "import minisign; minisign.PublicKey.from_file('minisign_key.pub').verify_file('<filename>'); print('The file has been verified.')"

Development

Please visit Contributing and Development for information on contributing to this project.

Additional Information

Additional information can be found at these locations.

License

dbrownell_ParserLib is licensed under the MIT license.